Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Abstract

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

1. Introduction

Information theory is a powerful tool widely used in many disciplines, including, for example, neuroscience, machine learning, and communication technology [1,2,3,4,5,6,7]. As it is often notoriously difficult to effectively calculate Shannon mutual information in many practical applications [8], various approximation methods have been proposed to estimate the mutual information, such as those based on asymptotic expansion [9,10,11,12,13], k-nearest neighbor [14], and minimal spanning trees [15]. Recently, Safaai et al. proposed a copula method for estimation of mutual information, which can be nonparametric and potentially robust [16]. Another approach for estimating the mutual information is to simplify the calculations by approximations based on information-theoretic bounds, such as the Cramér–Rao lower bound [17] and the van Trees’ Bayesian Cramér–Rao bound [18].

In this paper, we focus on mutual information estimation based on asymptotic approximations [19,20,21,22,23,24]. For encoding of continuous variables, asymptotic relations between mutual information and Fisher information have been presented by several researchers [19,20,21,22]. Recently, Huang and Zhang [24] proposed an improved approximation formula, which remains accurate for high-dimensional variables. A significant advantage of this approach is that asymptotic approximations are sometimes very useful in analytical studies. For instance, asymptotic approximations allow us to prove that the optimal neural population distribution that maximizes the mutual information between stimulus and response can be solved by convex optimization [24]. Unfortunately this approach does not generalize to discrete variables since the calculation of Fisher information requires partial derivatives of the likelihood function with respect to the encoded variables. For encoding of discrete variables, Kang and Sompolinsky [23] represented an asymptotic relationship between mutual information and Chernoff information for statistically independent neurons in a large population. However, Chernoff information is still hard to calculate in many practical applications.

Discrete stimuli or variables occur naturally in sensory coding. While some stimuli are continuous (e.g., the direction of movement, and the pitch of a tone), others are discrete (e.g., the identities of faces, and the words in human speech). For definiteness, in this paper, we frame our questions in the context of neural population coding; that is, we assume that the stimuli or the input variables are encoded by the pattern of responses elicited from a large population of neurons. The concrete examples used in our numerical simulations were based on Poisson spike model, where the response of each neuron is taken as the spike count within a given time window. While this simple Poisson model allowed us to consider a large neural population, it only captured the spike rate but not any temporal structure of the spike trains [25,26,27,28]. Nonetheless, our mathematical results are quite general and should be applicable to other input–output systems under suitable conditions to be discussed later.

In the following, we first derive several upper and lower bounds on Shannon mutual information using Kullback-Leibler divergence and Rényi divergence. Next, we derive several new approximation formulas for Shannon mutual information in the limit of large population size. These formulas are more convenient to calculate than the mutual information in our examples. Finally, we confirm the validity of our approximation formulas using the true mutual information as evaluated by Monte Carlo simulations.

2. Theory and Methods

2.1. Notations and Definitions

Suppose the input $\mathbf{x}$ is a K-dimensional vector, $\mathbf{x}={(x_1,\cdots ,x_K)}^T$, which could be interpreted as the parameters that specifies a stimulus for a sensory system, and the outputs is an N-dimensional vector, $\mathbf{r}={(r_1,\cdots ,r_N)}^T$, which could be interpreted as the responses of N neurons. We assume N is large, generally $N\gg K$. We denote random variables by upper case letters, e.g., random variables X and R, in contrast to their vector values $\mathbf{x}$ and $\mathbf{r}$. The mutual information between X and R is defined by

$$I=I(X;R)={〈ln\frac{p\left(\mathbf{r}\right|\mathbf{x})}{p\left(\mathbf{r}\right)}〉}_{\mathbf{r},\mathbf{x}},$$

(1)

where $\mathbf{x}\in \mathcal{X}\subseteq {\mathbb{R}}^K$, $\mathbf{r}\in \mathcal{R}\subseteq {\mathbb{R}}^N$, and ${〈·〉}_{\mathbf{r},\mathbf{x}}$ denotes the expectation with respect to the probability density function $p(\mathbf{r},\mathbf{x})$. Similarly, in the following, we use ${〈·〉}_{\mathbf{r}|\mathbf{x}}$ and ${〈·〉}_{\mathbf{x}}$ to denote expectations with respect to $p\left(\mathbf{r}\right|\mathbf{x})$ and $p\left(\mathbf{x}\right)$, respectively.

If $p\left(\mathbf{x}\right)$ and $p\left(\mathbf{r}\right|\mathbf{x})$ are twice continuously differentiable for almost every $\mathbf{x}\in \mathcal{X}$, then for large N we can use an asymptotic formula to approximate the true value of I with high accuracy [24]:

$$I\simeq I_G=\frac{1}{2}{〈ln\left(det\left(\frac{\mathbf{G}\left(\mathbf{x}\right)}{2\pi e}\right)\right)〉}_{\mathbf{x}}+H\left(X\right),$$

(2)

which is sometimes reduced to

$$I\simeq I_F=\frac{1}{2}{〈ln\left(det\left(\frac{\mathbf{J}\left(\mathbf{x}\right)}{2\pi e}\right)\right)〉}_{\mathbf{x}}+H\left(X\right),$$

(3)

where $det\left(·\right)$ denotes the matrix determinant, $H\left(X\right)=-{〈lnp\left(\mathbf{x}\right)〉}_{\mathbf{x}}$ is the stimulus entropy,

$$\begin{array}{c}\mathbf{G}\left(\mathbf{x}\right)=\mathbf{J}\left(\mathbf{x}\right)+\mathbf{P}\left(\mathbf{x}\right),\hfill \end{array}$$

(4)

$$\begin{array}{c}\mathbf{P}\left(\mathbf{x}\right)=-{\displaystyle \frac{{\partial }^{2}lnp\left(\mathbf{x}\right)}{\partial \mathbf{x}\partial {\mathbf{x}}^T}},\hfill \end{array}$$

(5)

and

$$\mathbf{J}\left(\mathbf{x}\right)=-{〈\frac{{\partial }^{2}lnp\left(\mathbf{r}\right|\mathbf{x})}{\partial \mathbf{x}\partial {\mathbf{x}}^T}〉}_{\mathbf{r}|\mathbf{x}}={〈\frac{\partial lnp\left(\mathbf{r}\right|\mathbf{x})}{\partial \mathbf{x}}\frac{\partial lnp\left(\mathbf{r}\right|\mathbf{x})}{\partial {\mathbf{x}}^T}〉}_{\mathbf{r}|\mathbf{x}}$$

(6)

is the Fisher information matrix.

We denote the Kullback-Leibler divergence as

$$D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)={〈ln\frac{p\left(\mathbf{r}|\mathbf{x}\right)}{p\left(\mathbf{r}|\widehat{\mathbf{x}}\right)}〉}_{\mathbf{r}|\mathbf{x}},$$

(7)

and denote Rényi divergence [29] of order $\beta +1$ as

$$D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)=\frac{1}{\beta }ln{〈{\left(\frac{p\left(\mathbf{r}\right|\mathbf{x})}{p\left(\mathbf{r}|\widehat{\mathbf{x}}\right)}\right)}^{\beta }〉}_{\mathbf{r}|\mathbf{x}}.$$

(8)

Here, $\beta D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$ is equivalent to Chernoff divergence of order $\beta +1$ [30]. It is well known that $D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\to D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$ in the limit $\beta \to 0$.

We define

$$\begin{array}{cc}\hfill I_u& =-{〈ln{〈exp\left(-D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)〉}_{\widehat{\mathbf{x}}}〉}_{\mathbf{x}},\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill I_e& =-{〈ln{〈exp\left(-e^{-1}D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)〉}_{\widehat{\mathbf{x}}}〉}_{\mathbf{x}},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill I_{\beta ,\alpha }& =-{〈ln{〈exp\left(-\beta D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)+\left(1-\alpha \right)ln\frac{p\left(\mathbf{x}\right)}{p\left(\widehat{\mathbf{x}}\right)}\right)〉}_{\widehat{\mathbf{x}}}〉}_{\mathbf{x}},\hfill \end{array}$$

(11)

where in $I_{\beta ,\alpha }$ we have $\beta \in \left(0,1\right)$ and $\alpha \in \left(0,\infty \right)$ and assume $p\left(\mathbf{x}\right)>0$ for all $\mathbf{x}\in \mathcal{X}$.

In the following, we suppose $\mathbf{x}$ takes M discrete values, ${\mathbf{x}}_m$, $m\in \mathcal{M}=\left\{1,2,\cdots ,M\right\}$, and $p\left({\mathbf{x}}_m\right)>0$ for all m. Now, the definitions in Equations (9)–(11) become

$$\begin{array}{cc}\hfill I_u& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(\sum _{\widehat{m}=1}^M\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill I_e& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(\sum _{\widehat{m}=1}^M\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill I_{\beta ,\alpha }& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(\sum _{\widehat{m}=1}^M{\left(\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}\right)}^{\alpha }exp\left(-\beta D_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right).\hfill \end{array}$$

(14)

Furthermore, we define

$$\begin{array}{cc}\hfill I_d& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(1+\sum _{\widehat{m}\in {\mathcal{M}}_m^u}\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill I_u^d& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(1+\sum _{\widehat{m}\in {\mathcal{M}}_m^u}\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill I_{\beta ,\alpha }^d& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(1+\sum _{\widehat{m}\in {\mathcal{M}}_m^{\beta }}{\left(\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}\right)}^{\alpha }exp\left(-\beta D_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill I_D& =-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(1+\sum _{\widehat{m}\in {\mathcal{M}}_m^u}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right),\hfill \end{array}$$

(18)

where

$$\begin{array}{c}{\check{\mathcal{M}}}_m^{\beta }=\left\{\widehat{m}:\widehat{m}=\underset{\check{m}\in \mathcal{M}-{\widehat{\mathcal{M}}}_m^{\beta }}{\arg \min }{D}_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\check{m}}\right)\right\},\hfill \end{array}$$

(19)

$$\begin{array}{c}{\check{\mathcal{M}}}_m^u=\left\{\widehat{m}:\widehat{m}=\underset{\check{m}\in \mathcal{M}-{\widehat{\mathcal{M}}}_m^u}{\arg \min }D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\check{m}}\right)\right\},\hfill \end{array}$$

(20)

$$\begin{array}{c}{\widehat{\mathcal{M}}}_m^{\beta }=\left\{\widehat{m}:D_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)=0\right\},\hfill \end{array}$$

(21)

$$\begin{array}{c}{\widehat{\mathcal{M}}}_m^u=\left\{\widehat{m}:D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)=0\right\},\hfill \end{array}$$

(22)

$$\begin{array}{c}{\mathcal{M}}_m^{\beta }={\check{\mathcal{M}}}_m^{\beta }\cup {\widehat{\mathcal{M}}}_m^{\beta }-\left\{m\right\},\hfill \end{array}$$

(23)

$$\begin{array}{c}{\mathcal{M}}_m^u={\check{\mathcal{M}}}_m^u\cup {\widehat{\mathcal{M}}}_m^u-\left\{m\right\}.\hfill \end{array}$$

(24)

Here, notice that, if $\mathbf{x}$ is uniformly distributed, then by definition $I_d$ and $I_D$ become identical. The elements in set ${\check{\mathcal{M}}}_m^{\beta }$ are those that make $D_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\check{m}}\right)$ take the minimum value, excluding any element that satisfies the condition $D_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)=0$. Similarly, the elements in set ${\check{\mathcal{M}}}_m^u$ are those that minimize $D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\check{m}}\right)$ excluding the ones that satisfy the condition $D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)=0$.

2.2. Theorems

In the following, we state several conclusions as theorems and prove them in Appendix A.

Theorem 1.

The mutual information I is bounded as follows:

$$I_{\beta ,\alpha }\le I\le I_u.$$

(25)

Theorem 2.

The following inequalities are satisfied,

$$I_{{\beta }_1,1}\le I_e\le I_u$$

(26)

where $I_{{\beta }_1,1}$ is a special case of $I_{\beta ,\alpha }$ in Equation (11) with ${\beta }_1=e^{-1}$ so that

$$I_{{\beta }_1,1}=-{〈ln{〈exp\left(-{\beta }_1{D}_{{\beta }_1}\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)〉}_{\widehat{\mathbf{x}}}〉}_{\mathbf{x}}.$$

(27)

Theorem 3.

If there exist ${\gamma }_1>0$ and ${\gamma }_2>0$ such that

$$\begin{array}{cc}\hfill \beta D_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{m_1}\right)& \ge {\gamma }_{1}lnN,\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{m_2}\right)& \ge {\gamma }_{2}lnN,\hfill \end{array}$$

(29)

for discrete stimuli ${\mathbf{x}}_m$, where $m\in \mathcal{M}$, $m_1\in \mathcal{M}-{\mathcal{M}}_m^{\beta }$ and $m_2\in \mathcal{M}-{\mathcal{M}}_m^u$, then we have the following asymptotic relationships:

$$I_{\beta ,\alpha }=I_{\beta ,\alpha }^d+O\left(N^{-{\gamma }_1}\right)\le I\le I_u=I_u^d+O\left(N^{-{\gamma }_2}\right)$$

(30)

and

$$I_e=I_d+O\left(N^{-{\gamma }_2/e}\right).$$

(31)

Theorem 4.

Suppose $p\left(\mathbf{x}\right)$ and $p\left(\mathbf{r}\right|\mathbf{x})$ are twice continuously differentiable for $\mathbf{x}\in \mathcal{X}$, $∥q{}^{\prime }\left(\mathbf{x}\right)∥<\infty$, $∥q^{″}\left(\mathbf{x}\right)∥<\infty$, where $q\left(\mathbf{x}\right)=lnp\left(\mathbf{x}\right)$ and ${}^{\prime }$ and ${}^{″}$ denote partial derivatives $\partial /\partial \mathbf{x}$ and ${\partial }^2/\partial \mathbf{x}\partial {\mathbf{x}}^T$, and ${\mathbf{G}}_{\gamma }\left(\mathbf{x}\right)$ is positive definite with $∥N{\mathbf{G}}_{\gamma }^{-1}\left(\mathbf{x}\right)∥=O\left(1\right)$, where $∥·∥$ denotes matrix Frobenius norm,

$${\mathbf{G}}_{\gamma }\left(\mathbf{x}\right)=\gamma \left(\mathbf{J}\left(\mathbf{x}\right)+\mathbf{P}\left(\mathbf{x}\right)\right),$$

(32)

$\gamma =\beta \left(1-\beta \right)$ and $\beta \in \left(0,1\right)$. If there exist an $\omega =\omega \left(\mathbf{x}\right)>0$ such that

$$\begin{array}{c}det{\left(\mathbf{G}\left(\mathbf{x}\right)\right)}^{1/2}{\int }_{{\overline{\mathcal{X}}}_{\epsilon }\left(\mathbf{x}\right)}p\left(\widehat{\mathbf{x}}\right)exp\left(-D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}=O\left(N^{-1}\right),\hfill \end{array}$$

(33)

$$\begin{array}{c}det{\left({\mathbf{G}}_{\gamma }\left(\mathbf{x}\right)\right)}^{1/2}{\int }_{{\overline{\mathcal{X}}}_{\epsilon }\left(\mathbf{x}\right)}p\left(\widehat{\mathbf{x}}\right)exp\left(-\beta D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}=O\left(N^{-1}\right),\hfill \end{array}$$

(34)

for all $\mathbf{x}\in \mathcal{X}$ and $\epsilon \in \left(0,\omega \right)$, where ${\overline{\mathcal{X}}}_{\omega }\left(\mathbf{x}\right)=\mathcal{X}-{\mathcal{X}}_{\omega }\left(\mathbf{x}\right)$ is the complementary set of ${\mathcal{X}}_{\omega }\left(\mathbf{x}\right)=\left\{\breve{\mathbf{x}}\in {\mathbb{R}}^K:{\left(\breve{\mathbf{x}}-\mathbf{x}\right)}^T\mathbf{G}\left(\mathbf{x}\right)\left(\breve{\mathbf{x}}-\mathbf{x}\right)<N{\omega }^2\right\}$, then we have the following asymptotic relationships:

$$I_{\beta ,\alpha }\le I_{{\gamma }_0}+O\left(N^{-1}\right)\le I\le I_u=I_G+K/2+O\left(N^{-1}\right),$$

(35)

$$I_e=I_G+O\left(N^{-1}\right),$$

(36)

$$I_{\beta ,\alpha }=I_{\gamma }+O\left(N^{-1}\right),$$

(37)

where

$$I_{\gamma }=\frac{1}{2}{\int }_{\mathcal{X}}p\left(\mathbf{x}\right)ln\left(det\left(\frac{{\mathbf{G}}_{\gamma }\left(\mathbf{x}\right)}{2\pi }\right)\right)d\mathbf{x}+H\left(X\right)$$

(38)

and ${\gamma }_0={\beta }_0\left(1-{\beta }_0\right)=1/4$ with ${\beta }_0=1/2$.

Remark 1.

We see from Theorems 1 and 2 that the true mutual information I and the approximation $I_e$ both lie between $I_{{\beta }_1,1}$ and $I_u$, which implies that their values may be close to each other. For discrete variable $\mathbf{x}$, Theorem 3 tells us that $I{}_e$ and $I_d$ are asymptotically equivalent (i.e., their difference vanishes) in the limit of large N. For continuous variable $\mathbf{x}$, Theorem 4 tells us that $I{}_e$ and $I_G$ are asymptotically equivalent in the limit of large N, which means that $I_e$ and I are also asymptotically equivalent because $I_G$ and I are known to be asymptotically equivalent [24].

Remark 2.

To see how the condition in Equation (33) could be satisfied, consider the case where $D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$ has only one extreme point at $\widehat{\mathbf{x}}=\mathbf{x}$ for $\widehat{\mathbf{x}}\in {\mathcal{X}}_{\omega }\left(\mathbf{x}\right)$ and there exists an $\eta >0$ such that $N^{-1}D\left(\mathbf{x}|\widehat{\mathbf{x}}\right)\ge \eta$ for $\widehat{\mathbf{x}}\in {\overline{\mathcal{X}}}_{\omega }\left(\mathbf{x}\right)$. Now, the condition in Equation (33) is satisfied because

$$\begin{array}{c}det{\left(\mathbf{G}\left(\mathbf{x}\right)\right)}^{1/2}{\int }_{{\overline{\mathcal{X}}}_{\epsilon }\left(\mathbf{x}\right)}p\left(\widehat{\mathbf{x}}\right)exp\left(-D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\hfill \\ \le det{\left(\mathbf{G}\left(\mathbf{x}\right)\right)}^{1/2}{\int }_{{\overline{\mathcal{X}}}_{\epsilon }\left(\mathbf{x}\right)}p\left(\widehat{\mathbf{x}}\right)exp\left(-\widehat{\eta }\left(\epsilon \right)N\right)d\widehat{\mathbf{x}}\hfill \\ =O\left(N^{K/2}{e}^{-\widehat{\eta }\left(\epsilon \right)N}\right),\hfill \end{array}$$

(39)

where by assumption we can find an $\widehat{\eta }\left(\epsilon \right)>0$ for any given $\epsilon \in \left(0,\omega \right)$. The condition in Equation (34) can be satisfied in a similar way. When ${\beta }_0=1/2$, ${\beta }_0{D}_{{\beta }_0}\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$ is the Bhattacharyya distance [31]:

$${\beta }_0{D}_{{\beta }_0}\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)=-ln\left({\int }_{\mathcal{R}}\sqrt{p\left(\mathbf{r}\right|\mathbf{x})p\left(\mathbf{r}\right|\widehat{\mathbf{x}})}d\mathbf{r}\right),$$

(40)

and we have

$$\mathbf{J}\left(\mathbf{x}\right)={\left.\frac{{\partial }^2\left(D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)}{\partial \widehat{\mathbf{x}}\partial {\widehat{\mathbf{x}}}^T}\right|}_{\widehat{\mathbf{x}}=\mathbf{x}}={\left.\frac{{\partial }^2\left(4{\beta }_0{D}_{{\beta }_0}\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)}{\partial \widehat{\mathbf{x}}\partial {\widehat{\mathbf{x}}}^T}\right|}_{\widehat{\mathbf{x}}=\mathbf{x}}={\left.\frac{{\partial }^2\left(8H_l^2\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)}{\partial \widehat{\mathbf{x}}\partial {\widehat{\mathbf{x}}}^T}\right|}_{\widehat{\mathbf{x}}=\mathbf{x}},$$

(41)

where $H_l\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$ is the Hellinger distance [32] between $p\left(\mathbf{r}\right|\mathbf{x})$ and $p\left(\mathbf{r}\right|\widehat{\mathbf{x}})$:

$$H_l^2\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)=\frac{1}{2}{\int }_{\mathcal{R}}{\left(\sqrt{p\left(\mathbf{r}\right|\mathbf{x})}-\sqrt{p\left(\mathbf{r}\right|\widehat{\mathbf{x}})}\right)}^{2}d\mathbf{r}.$$

(42)

By Jensen’s inequality, for $\beta \in \left(0,1\right)$ we get

$$0\le D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\le D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right).$$

(43)

Denoting the Chernoff information [8] as

$$C\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)=\underset{\beta \in \left(0,1\right)}{max}\left(\beta D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)\right)={\beta }_m{D}_{{\beta }_m}\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right),$$

(44)

where $\beta D_{\beta }\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$ achieves its maximum at ${\beta }_m$, we have

$$\begin{array}{c}{I}_{\beta ,\alpha }-H\left(X\right)\hfill \\ \le h_c=-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(\sum _{\widehat{m}=1}^M\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-C\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)\hfill \end{array}$$

(45)

$$\begin{array}{c}\le h_d=-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(\sum _{\widehat{m}=1}^M\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-{\beta }_m{D}_{\beta }\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right).\hfill \end{array}$$

(46)

By Theorem 4,

$$\begin{array}{c}\underset{\beta \in \left(0,1\right)}{max}{I}_{\beta ,\alpha }=I_{{\gamma }_0}+O\left(N^{-1}\right),\hfill \end{array}$$

(47)

$$\begin{array}{c}{I}_{{\gamma }_0}=I_G-\frac{K}{2}ln\frac{4}{e}.\hfill \end{array}$$

(48)

If ${\beta }_m=1/2$, then, by Equations (46)–(48) and (50), we have

$$\begin{array}{cc}\hfill \underset{\beta \in \left(0,1\right)}{max}{I}_{\beta }+\frac{K}{2}ln\frac{4}{e}+O\left(N^{-1}\right)& \le I_e=I+O\left(N^{-1}\right)\hfill \\ & \le h_d+H\left(X\right)\le I_u.\hfill \end{array}$$

(49)

Therefore, from Equations (45), (46) and (49), we can see that $I_e$ and I are close to $h_c+H\left(X\right)$.

2.3. Approximations for Mutual Information

In this section, we use the relationships described above to find effective approximations to true mutual information I in the case of large but finite N. First, Theorems 1 and 2 tell us that the true mutual information I and its approximation $I_e$ lie between lower and upper bounds given by: $I_{\beta ,\alpha }\le I\le I_u$ and $I_{{\beta }_1,1}\le I_e\le I_u$. As a special case, I is also bounded by $I_{{\beta }_1,1}\le I\le I_u$. Furthermore, from Equations (2) and (36) we can obtain the following asymptotic equality under suitable conditions:

$$I=I_e+O\left(N^{-1}\right).$$

(50)

Hence, for continuous stimuli, we have the following approximate relationship for large N:

$$I\simeq I_e\simeq I_G.$$

(51)

For discrete stimuli, by Equation (31) for large but finite N, we have

$$I\simeq I_e\simeq I_d=-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(1+\sum _{\widehat{m}\in {\mathcal{M}}_m^u}\frac{p\left({\mathbf{x}}_{\widehat{m}}\right)}{p\left({\mathbf{x}}_m\right)}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right).$$

(52)

Consider the special case $p\left({\mathbf{x}}_{\widehat{m}}\right)\simeq p\left({\mathbf{x}}_m\right)$ for $\widehat{m}\in {\mathcal{M}}_m^u$. With the help of Equation (18), substitution of $p\left({\mathbf{x}}_{\widehat{m}}\right)\simeq p\left({\mathbf{x}}_m\right)$ into Equation (52) yields

$$\begin{array}{cc}\hfill I& \simeq I_D=-\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)ln\left(1+\sum _{\widehat{m}\in {\mathcal{M}}_m^u}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)\right)+H\left(X\right)\hfill \\ & \simeq -\sum _{m=1}^{M}p\left({\mathbf{x}}_m\right)\sum _{\widehat{m}\in {\mathcal{M}}_m^u}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)+H\left(X\right)\hfill \\ & =I_D^0\hfill \end{array}$$

(53)

where $I_D^0\le I_D$ and the second approximation follows from the first-order Taylor expansion assuming that the term ${\displaystyle \sum _{\widehat{m}\in {\mathcal{M}}_m^u}}exp\left(-e^{-1}D\left({\mathbf{x}}_m\left|\right|{\mathbf{x}}_{\widehat{m}}\right)\right)$ is sufficiently small.

The theoretical discussion above suggests that $I_e$ and $I_d$ are effective approximations to true mutual information I in the limit of large N. Moreover, we find that they are often good approximations of mutual information I even for relatively small N, as illustrated in the following section.

3. Results of Numerical Simulations

Consider Poisson model neuron whose responses (i.e., numbers of spikes within a given time window) follow a Poisson distribution [24]. The mean response of neuron n, with $n\in \left\{1,2,\cdots ,N\right\}$, is described by the tuning function $f\left(x;{\theta }_n\right)$, which takes the form of a Heaviside step function:

$$f\left(x;{\theta }_n\right)=\left\{\begin{array}{cc}\hfill A,& \mathrm{if} x\ge {\theta }_n,\hfill \\ \hfill 0,& \mathrm{if} x<{\theta }_n,\hfill \end{array}\right.$$

(54)

where the stimulus $x\in \left[-T,T\right]$ with $T=10$, $A=10$, and the centers ${\theta }_1$, ${\theta }_2$, ⋯, ${\theta }_N$ of the N neurons are uniformly spaced in interval $\left[-T,T\right]$, namely, ${\theta }_n=\left(n-1\right)d-T$ with $d=2T/(N-1)$ for $N\ge 2$, and ${\theta }_n=0$ for $N=1$. We suppose that the discrete stimulus x has $M=21$ possible values that are evenly spaced from $-T$ to T, namely, $x\in \mathcal{X}=\left\{x_m:x_m=2\left(m-1\right)T/(M-1)-T,m=1,2,\cdots ,M\right\}$. Now, the Kullback-Leibler divergence can be written as

$$D\left(x_m\left|\right|x_{\widehat{m}}\right)=f\left(x_m;{\theta }_n\right)log\left(\frac{f\left(x_m;{\theta }_n\right)}{f\left(x_{\widehat{m}};{\theta }_n\right)}\right)+f\left(x_{\widehat{m}};{\theta }_n\right)-f\left(x_m;{\theta }_n\right).$$

(55)

Thus, we have $exp\left(-e^{-1}D\left(x_m\left|\right|x_{\widehat{m}}\right)\right)=1$ when $f\left(x_m;{\theta }_n\right)=f\left(x_{\widehat{m}};{\theta }_n\right)$, $exp\left(-e^{-1}D\left(x_m\left|\right|x_{\widehat{m}}\right)\right)=exp\left(-e^{-1}A\right)$ when $f\left(x_m;{\theta }_n\right)=0$ and $f\left(x_{\widehat{m}};{\theta }_n\right)=A$, and $exp\left(-e^{-1}D\left(x_m\left|\right|x_{\widehat{m}}\right)\right)=0$ when $f\left(x_m;{\theta }_n\right)=A$ and $f\left(x_{\widehat{m}};{\theta }_n\right)=0$. Therefore, in this case, we have

$$I_e=I_d.$$

(56)

More generally, this equality holds true whenever the tuning function has binary values.

In the first example, as illustrated in Figure 1, we suppose the stimulus has a uniform distribution, so that the probability is given by $p\left(x_m\right)=1/M$. Figure 1a shows graphs of the input distribution $p\left(x\right)$ and a representative tuning function $f\left(x;\theta \right)$ with the center $\theta =0$.

To assess the accuracy of the approximation formulas, we employed Monte Carlo (MC) simulation to evaluate the mutual information I [24]. In our MC simulation, we first sampled an input $x_j\in \mathcal{X}$ from the uniform distribution $p\left(x_j\right)=1/M$, then generated the neural responses ${\mathbf{r}}_j$ by the conditional distribution $p\left({\mathbf{r}}_j\right|x_j)$ based on the Poisson model, where $j=1$, 2, ⋯, $j_{max}$. The value of mutual information by MC simulation was calculated by

$$I_{MC}^{\ast }=\frac{1}{j_{max}}\sum _{j=1}^{j_{max}}ln\left(\frac{p\left({\mathbf{r}}_j\right|x_j)}{p\left({\mathbf{r}}_j\right)}\right),$$

(57)

where

$$p\left({\mathbf{r}}_j\right)=\sum _{m=1}^{M}p\left({\mathbf{r}}_j\right|x_m)p\left(x_m\right).$$

(58)

To assess the precision of our MC simulation, we computed the standard deviation of repeated trials by bootstrapping:

$$I_{std}=\sqrt{\frac{1}{i_{max}}\sum _{i=1}^{i_{max}}{\left(I_{MC}^i-I_{MC}\right)}^2},$$

(59)

where

$$\begin{array}{cc}\hfill I_{MC}^i& =\frac{1}{j_{max}}\sum _{j=1}^{j_{max}}ln\left(\frac{p\left({\mathbf{r}}_{{\mathsf{\Gamma }}_{j,i}}\right|x_{{\mathsf{\Gamma }}_{j,i}})}{p\left({\mathbf{r}}_{{\mathsf{\Gamma }}_{j,i}}\right)}\right),\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill I_{MC}& =\frac{1}{i_{max}}\sum _{i=1}^{i_{max}}{I}_{MC}^i,\hfill \end{array}$$

(61)

and ${\mathsf{\Gamma }}_{j,i}\in \left\{1,2,\cdots ,j_{max}\right\}$ is the $\left(j,i\right)$-th entry of the matrix $\mathsf{\Gamma }\in {\mathbb{N}}^{j_{max}\times i_{max}}$ with samples taken randomly from the integer set $\{1$, 2, ⋯, $j_{max}\}$ by a uniform distribution. Here, we set $j_{max}=5\times {10}^5$, $i_{max}=100$ and $M={10}^3$.

For different $N\in \{1,2,3,4,6,10,14,20,30,50,100,200,400,700,1000\}$, we compared $I_{MC}$ with $I_e$, $I_d$ and $I_D$, as illustrated in Figure 1b–d. Here, we define the relative error of approximation, e.g., for $I_e$, as

$$D{I}_e=\frac{I_e-I_{MC}}{I_{MC}},$$

(62)

and the relative standard deviation

$$D{I}_{std}=\frac{I_{std}}{I_{MC}}.$$

(63)

For the second example, we only changed the probability distribution of stimulus $p\left(x_m\right)$ while keeping all other conditions unchanged. Now, $p\left(x_m\right)$ is a discrete sample from a Gaussian function:

$$p\left(x_m\right)=Z^{-1}\exp \left(-\frac{x_m^2}{2{\sigma }^2}\right),m=1,2,\cdots ,M,$$

(64)

where $Z$ is the normalization constant and $\sigma =T/2$. The results are illustrated in Figure 2.

Next, we changed each tuning function $f\left(x;{\theta }_n\right)$ to a rectified linear function:

$$f\left(x;{\theta }_n\right)=max\left(0,x-{\theta }_n\right),$$

(65)

Figure 3 and Figure 4 show the results under the same conditions of Figure 1 and Figure 2 except for the shape of the tuning functions.

Finally, we let the tuning function $f\left(x;{\theta }_n\right)$ have a random form:

$$f\left(x;{\theta }_n\right)=\left\{\begin{array}{cc}\hfill B,& \mathrm{if}x\in {\theta }_n=\left\{{\theta }_n^1,{\theta }_n^2,\cdots ,{\theta }_n^K\right\},\hfill \\ \hfill 0,& \mathrm{otherwise},\hfill \end{array}\right.$$

(66)

where the stimulus $x\in \mathcal{X}=\{1,2,\cdots 999,1000\}$, $B=10$, the values of $\left\{{\theta }_n^1,{\theta }_n^2,\cdots ,{\theta }_n^K\right\}$ are distinct and randomly selected from the set $\mathcal{X}$ with $K=10$. In this example, we may regard $\mathcal{X}$ as a list of natural objects (stimuli), and there are a total of N sensory neurons, each of which responds only to K randomly selected objects. Figure 5 shows the results under the condition that $p\left(x\right)$ is a uniform distribution. In Figure 6, we assume that $p\left(x\right)$ is not flat but a half Gaussian given by Equation (64) with $\sigma =500$.

In all these examples, we found that the three formulas, namely, $I_e$, $I_d$ and $I_D$, provided excellent approximations to the true values of mutual information as evaluated by Monte Carlo method. For example, in the examples in Figure 1 and Figure 5, all three approximations were practically indistinguishable. In general, all these approximations were extremely accurate when $N>100$.

In all our simulations, the mutual information tended to increase with the population size N, eventually reaching a plateau for large enough N. The saturation of information for large N is due to the fact that it requires at most ${log}_{2}M$ bits of information to completely distinguish all M stimuli. It is impossible to gain more information than this maximum amount regardless of how many neurons are used in the population. In Figure 1, for instance, this maximum is ${log}_{2}21=4.39$ bits, and in Figure 5, this maximum is ${log}_{2}1000=9.97$ bits.

For relatively small values of N, we found that $I_D$ tended to be less accurate than $I_e$ or $I_d$ (see Figure 5 and Figure 6). Our simulations also confirmed two analytical results. The first one is that $I_d=I_D$ when the stimulus distribution is uniform; this result follows directly from the definitions of $I_d$ and $I_D$ and is confirmed by the simulations in Figure 1, Figure 3, and Figure 5. The second result is that $I_d=I_e$ (Equation (56)) when the tuning function is binary, as confirmed by the simulations in Figure 1, Figure 2, Figure 5, and Figure 6. When the tuning function allows many different values, $I_e$ can be much more accurate than $I_d$ and $I_D$, as shown by the simulations in Figure 3 and Figure 4. To summarize, our best approximation formula is $I_e$ because it is more accurate than $I_d$ and $I_D$, and, unlike $I_d$ and $I_D$, it applies to both discrete and continuous stimuli (Equations (10) and (13)).

4. Discussion

We have derived several asymptotic bounds and effective approximations of mutual information for discrete variables and established several relationships among different approximations. Our final approximation formulas involve only Kullback-Leibler divergence, which is often easier to evaluate than Shannon mutual information in practical applications. Although in this paper our theory is developed in the framework of neural population coding with concrete examples, our mathematical results are generic and should hold true in many related situations beyond the original context.

We propose to approximate the mutual information with several asymptotic formulas, including $I_e$ in Equation (10) or Equation (13), $I_d$ in Equation (15) and $I_D$ in Equation (18). Our numerical experimental results show that the three approximations $I_e$, $I_d$ and $I_D$ were very accurate for large population size N, and sometimes even for relatively small N. Among the three approximations, $I_D$ tended to be the least accurate, although, as a special case of $I_d$, it is slightly easier to evaluate than $I_d$. For a comparison of $I_e$ and $I_d$, we note that $I_e$ is the universal formula, whereas $I_d$ is restricted only to discrete variables. The two formulas $I_e$ and $I_d$ become identical when the responses or the tuning functions have only two values. For more general tuning functions, the performance of $I_e$ was better than $I_d$ in our simulations.

As mentioned before, an advantage of of $I_e$ is that it works not only for discrete stimuli but also for continuous stimuli. Theoretically speaking, the formula for $I_e$ is well justified, and we have proven that it approaches the true mutual information I in the limit of large population. In our numerical simulations, the performance of $I_e$ was excellent and better than that of $I_d$ and $I_D$. Overall, $I_e$ is our most accurate and versatile approximation formula, although, in some cases, $I_d$ and $I_D$ are slightly more convenient to calculate.

The numerical examples considered in this paper were based on an independent population of neurons whose responses have Poisson statistics. Although such models are widely used, they are appropriate only if the neural responses can be well characterized by the spike counts within a fixed time window. To study the temporal patterns of spike trains, one has to consider more complicated models. Estimation of mutual information from neural spike trains is a difficult computational problem [25,26,27,28]. In future work, it would be interesting to apply the asymptotic formulas such as $I_e$ to spike trains with small time bins each containing either one spike or nothing. A potential advantage of the asymptotic formula is that it might help reduce the bias caused by small samples in the calculation of the response marginal distribution $p\left(\mathbf{r}\right)={\sum }_{\mathbf{x}}p\left(\mathbf{r}|\mathbf{x}\right)p\left(\mathbf{x}\right)$ or the response entropy $H\left(R\right)$ because here one only needs to calculate the Kullback-Leibler divergence $D\left(\mathbf{x}\left|\right|\widehat{\mathbf{x}}\right)$, which may have a smaller estimation error.

Finding effective approximation methods for computing mutual information is a key step for many practical applications of the information theory. Generally speaking, Kullback-Leibler divergence (Equation (7)) is often easier to evaluate and approximate than either Chernoff information (Equation (44)) or Shannon mutual information (Equation (1)). In situations where this is indeed the case, our approximation formulas are potentially useful. Besides applications in numerical simulations, the availability of a set of approximation formulas may also provide helpful theoretical tools in future analytical studies of information coding and representations.

As mentioned in the Introduction, various methods have been proposed to approximate the mutual information [9,10,11,12,13,14,15,16]. In future work, it would be useful to compare different methods rigorously under identical conditions in order to asses their relative merits. The approximation formulas developed in this paper are relatively easy to compute for practical problems. They are especially suitable for analytical purposes; for example, they could be used explicitly as objective functions for optimization or learning algorithms. Although the examples used in our simulations in this paper are parametric, it should be possible to extend the formulas to nonparametric problem, possibly with help of the copula method to take advantage of its robustness in nonparametric estimations [16].