Asymptotic Normality for Plug-In Estimators of Generalized Shannon’s Entropy

Abstract

Shannon’s entropy is one of the building blocks of information theory and an essential aspect of Machine Learning (ML) methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon’s entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon’s entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon’s entropy. The asymptotic distribution for Shannon’s entropy’s plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon’s entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow for interval estimation and statistical tests with generalized Shannon’s entropy.

1. Introduction

1.1. Introduction and Related Work

Shannon’s entropy, introduced by [1], is one of the building blocks of Information Theory and a key aspect of Machine Learning (ML) methods (e.g., Random Forests). It is one of the most popular quantities on countable alphabet (An countable alphabet is a space that could be either finite, or countably infinite; the elements in an alphabet can be either ordinal (e.g., numbers) or non-ordinal (e.g., letters)), particularly on non-ordinal space with categorical data. For example, in [2], all reviewed feature selection methods on non-ordinal space boiled down to a function of Shannon’s entropy. In addition, Shannon’s entropy is one of the most important foundations for all tree-based ML algorithms, sometimes substitutable with the Gini impurity index [3,4,5].

As one of the essential information-theoretical quantities, Shannon’s entropy and its estimation are widely studied in the past decades [6,7,8,9,10,11,12]. In particular, [9] proved that an unbiased estimator of Shannon’s entropy does not exist. Current state-of-art Shannon’s entropy point estimator was provided in [10] with the fastest bias decaying rate (exponentially-decaying).

Nevertheless, Shannon’s entropy is only finitely defined for distributions with fast decaying tails [13].

It is never known if the real distribution yields a finite Shannon’s entropy in practice. Furthermore, all existing results on Shannon’s entropy require it to be finitely defined, which results in a usage restriction when adopting the entropy-based methods. This is, in fact, a void in the foundation of all Shannon’s entropy-related results.

Example 1

(Unbounded Shannon’s Entropy). Let a distribution $\mathcal{P}$ be ${\mathcal{P}}_k=c/(k{ln}^{2}k)$ for $k\ge 2$, where c is the constant that makes $\mathcal{P}$ a valid probability distribution. Such c uniquely exists because ${\sum }_{k=2}^{\infty }[1/(k{ln}^{2}k)]$ converges. Then Shannon’s entropy of $\mathcal{P}$, $H(\mathcal{P})$ is unbounded because

$$\begin{array}{cc}\hfill H(\mathcal{P})=& -\sum _{k=2}^{\infty }\left[{\mathcal{P}}_{k}ln{\mathcal{P}}_k\right]\hfill \\ \hfill =& -\sum _{k=2}^{\infty }\left[\frac{c}{k{ln}^{2}k}ln\frac{c}{k{ln}^{2}k}\right]\hfill \\ \hfill =& -\sum _{k=2}^{\infty }\left[\frac{c}{k{ln}^{2}k}lnc\right]+\sum _{k=2}^{\infty }\left[\frac{c}{k{ln}^{2}k}lnk\right]+\sum _{k=2}^{\infty }\left[\frac{c}{k{ln}^{2}k}ln\left({ln}^{2}k\right)\right]\hfill \\ \hfill =& -\sum _{k=2}^{\infty }\left[\frac{c}{k{ln}^{2}k}lnc\right]+\sum _{k=2}^{\infty }\left[\frac{c}{klnk}\right]+\sum _{k=2}^{\infty }\left[\frac{2clnlnk}{k{ln}^{2}k}\right]\hfill \\ \hfill =& A\mathit{Finite}\mathit{Value}+\infty +A\mathit{Finite}\mathit{Value}=\infty .\hfill \end{array}$$

The effort to generalize Shannon’s entropy has been long and extensive in the existing literature. As summarized in [14], the main perspective in the generalization in the existing literature is based on axiomatic characterization of Shannon’s entropy [15,16]. For example, Refs. [17,18] are efforts with respect to the functional form, $H={\sum }_{k\ge 1}h\left(p_k\right)$, under certain desirable axioms, $h(p)=-plogp$ is uniquely determined up to a multiplicative constant; if the strong additivity axiom is relaxed to be one of the weaker versions, say $\alpha$-additivity or composability, then $h(p)$ may be of other forms, which give rise to Rényi’s entropy [19], and the Tsallis entropy [20]. However, all such generalization effort does not seem to lead to an information measure on a joint alphabet that would possess all the desirable properties of mutual information, which is supported by an argument via the Kullback–Leibler divergence [21]. Interested readers may refer to [14] for details.

To further address the deficiency of Shannon’s entropy [14] proposed generalized Shannon’s entropy (GSE) and showed that GSE enjoys all properties of a finite Shannon’s entropy. In addition, GSE is finitely defined on all distributions. Due to the advantages of GSE and the deficiency of Shannon’s entropy, the use of Shannon’s entropy should eventually be transited to GSE.

1.2. Summary and Contribution

To aid the transition, the estimation of GSE needs to be studied. In practice, the plug-in estimator is one of the most popular estimation approaches. For plug-in estimation of GSE, asymptotic properties are required for statistical tests and confidence intervals. This article studies the asymptotic properties for plug-in estimators of GSE.

As a summary of the article’s results, Theorem 1 and Corollary 1 provide asymptotic normality properties for the plug-in estimators of GSE for all orders (An explanation of the order is given in Definition 2) on countably infinite alphabet. Corollary 2 provides the asymptotic normality properties for the plug-in estimators of GSE for all orders on finite alphabet, except the underlying distribution being uniform (Under a uniform distribution, the estimation of GSE is reduced to an estimation of population size. Interested readers can read [22]). The presented asymptotic normality properties immediately allow interval estimation and hypothesis testing with plug-in estimators of GSE. The numerical results in Section 3 show that the developed asymptotic properties converge fast, especially when the order is 2.

The presented properties and performance of GSE plug-in estimators suggest that GSE’s use is full of promising potential. One may be concerned the construction of CDOTC (Defined in Definition 1) would bring additional estimation challenges to the already-difficult estimation of Shannon’s entropy. Yet, the convergence speed for GSE plug-in estimators is fast. To further unlock the potentials of GSE, additional estimation methods of GSE and asymptotic properties of functions of GSE (e.g., Generalized Mutual Information, also originated in [14]) shall be visited. This article’s results and proofs’ approaches provide a solid direction toward that end.

The rest of this paper is organized as follows. Section 2 formally states the problem and gives our main results. In Section 3, we provide a small-scale simulation study. In Section 4, we discuss the potential of GSE. Proofs are postponed to Section 5.

2. Main Results

Let Z be a random element on a countable alphabet $\mathcal{Z}=\left\{z_k;k\ge 1\right\}$ with an associated distribution $\mathbf{p}=\left\{p_k;k\ge 1\right\}$. Let the cardinality or support on $\mathcal{Z}$ be denoted $K={\sum }_{k\ge 1}1\left[p_k>0\right]$, where $1[·]$ is the indicator function. K is possibly finite or infinite. Let $\mathcal{P}$ denote the family of all distributions on $\mathcal{Z}$. Shannon’s entropy, H, is defined as

$$H=H(Z)=-\sum _{k\ge 1}{p}_{k}ln{p}_k.$$

To state our main result, we need to state Definitions 1 and 2 given by [14], and Definition 3.

Definition 1

(Conditional Distribution of Total Collision (CDOTC)). Given $\mathcal{Z}=\left\{z_k;k\ge 1\right\}$ and $\mathbf{p}=\left\{p_k\right\}$, consider the experiment of drawing an identically and independently distributed (iid) sample of size m ($m\ge 2)$. Let $C_m$ denote the event that all observations of the sample take on a same letter in $\mathcal{Z}$, and let $C_m$ be referred to as the event of total collision. The conditional probability, given $C_m$, that the total collision occurs at letter $z_k$ is

$$p_{m,k}=\frac{p_k^m}{{\sum }_{i\ge 1}{p}_i^m},$$

where $m\ge 2$. ${\mathbf{p}}_m=\left\{p_{m,k}\right\}$ is defined as the m-th order CDOTC.

Definition 2

(Generalized Shannon’s Entropy (GSE)). Given $\mathcal{Z}=\left\{z_k;k\ge 1\right\}$, $\mathbf{p}=\left\{p_k\right\}$, and ${\mathbf{p}}_m=\left\{p_{m,k}\right\}$, generalized Shannon’s entropy (GSE) is defined as

$$H_m(Z)=-\sum _{k\ge 1}{p}_{m,k}ln{p}_{m,k},$$

where $p_{m,k}$ is defined in Definition 1, and $m=2,3,\dots$ is the order of GSE. GSE with order m is referred to as the m-th order GSE.

It is clear that ${\mathbf{p}}_m$ is a probability distribution induced from $\mathbf{p}=\left\{p_k\right\}$. Furthermore, for each m, $m\ge 2$, $\mathbf{p}$ and ${\mathbf{p}}_m$ uniquely determined each other (Lemma 1 in [14]). To help understand Definitions 1 and 2, Examples 2 and 3 are provided as follows.

Example 2

(The 2nd order CDOTC). Given $\mathcal{Z}=\left\{z_k;k\ge 1\right\}$ and $\mathbf{p}=\left\{p_k\right\}=\{6k^{-2}/{\pi }^2;k=1,2,3,\dots \}$, the 2nd order CDOTC is then defined as

$${\mathbf{p}}_2=\left\{p_{2,k}\right\},$$

where

$$p_{2,k}=\frac{p_k^2}{{\sum }_{i\ge 1}{p}_i^2}=\frac{36k^{-4}/{\pi }^4}{{\sum }_{i\ge 1}\left[36i^{-4}/{\pi }^4\right]}=\frac{k^{-4}}{{\sum }_{i\ge 1}{i}^{-4}}$$

for $k=1,2,3,\dots$.

Example 3

(The 2nd order GSE). Given $\mathcal{Z}=\left\{z_k;k\ge 1\right\}$, $\mathbf{p}=\left\{p_k\right\}=\{6k^{-2}/{\pi }^2;k=1,2,3,\dots \}$, and ${\mathbf{p}}_2=\left\{p_{2,k}\right\}=\{\frac{k^{-4}}{{\sum }_{i\ge 1}{i}^{-4}};k=1,2,\dots \}$, the 2nd order GSE, $H_2(Z)$, is then defined as

$$H_2(Z)=-\sum _{k\ge 1}{p}_{2,k}ln{p}_{2,k},$$

where $p_{2,k}$ is given in Example 2.

The definition of the plug-in estimator of GSE is stated in Definition 3.

Definition 3

(Plug-in estimator of GSE). Let $Z_1,Z_2,\dots ,Z_n$ be independent and identically distributed (iid) random variables taking values in $\mathcal{Z}=\left\{z_k;k\ge 1\right\}$ with distribution $\mathbf{p}$. For each $k=1,2,\dots$, let $Y_k={\sum }_{i=1}^{n}1[Z_i=z_k]$ be the sample count of observations in category $z_k$, and let ${\widehat{p}}_k=Y_k/n$ be the sample proportion. The plug-in estimator for the m-th order GSE, ${\widehat{H}}_m(Z)$, is defined as

$$\begin{array}{cc}\hfill {\widehat{H}}_m(Z)=& -\sum _{k\ge 1}\left[{\widehat{p}}_{m,k}ln{\widehat{p}}_{m,k}\right]\hfill \\ \hfill =& -\sum _{k\ge 1}\left[\frac{{\widehat{p}}_k^m}{{\sum }_{i\ge 1}{\widehat{p}}_i^m}ln\frac{{\widehat{p}}_k^m}{{\sum }_{i\ge 1}{\widehat{p}}_i^m}\right].\hfill \end{array}$$

Our main results are stated in Theorem 1, Corollary 1 and 2.

Theorem 1.

Let $\mathbf{p}=\left\{p_k\right\}$ be a probability distribution supported by a countably infinite alphabet, without any further conditions,

$$\sqrt{n}\left({\widehat{H}}_m(Z)-H_m(Z)\right)\stackrel{d}{\to }N(0,{\sigma }_m^2),$$

where

$${\sigma }_m^2=\sum _{k=1}^{\infty }{\left[\frac{m^2}{p_k}\left(p_{m,k}ln{p}_{m,k}+p_{m,k}{H}_m(Z)\right)\right]}^2.$$

Corollary 1.

Let $\mathbf{p}=\left\{p_k\right\}$ be a probability distribution supported by a countably infinite alphabet, without any further conditions,

$$\sqrt{n}\left(\frac{{\widehat{H}}_m(Z)-H_m(Z)}{{\widehat{\sigma }}_m}\right)\stackrel{d}{\to }N(0,1),$$

where

$${\widehat{\sigma }}_m^2=\sum _{k=1}^{\infty }{\left[\frac{m^2}{{\widehat{p}}_k}\left({\widehat{p}}_{m,k}ln{\widehat{p}}_{m,k}+{\widehat{p}}_{m,k}{\widehat{H}}_m(Z)\right)\right]}^2.$$

(1)

Corollary 2.

Let $\mathbf{p}=\{p_k;k=1,2,\dots ,K\}$ be a non-uniform probability distribution on a finite alphabet, without any further conditions,

$$\sqrt{n}\left(\frac{{\widehat{H}}_m(Z)-H_m(Z)}{{\widehat{\sigma }}_m}\right)\stackrel{d}{\to }N(0,1),$$

where

$${\widehat{\sigma }}_m^2=\sum _{k=1}^K{\left[\frac{m^2}{{\widehat{p}}_k}\left({\widehat{p}}_{m,k}ln{\widehat{p}}_{m,k}+{\widehat{p}}_{m,k}{\widehat{H}}_m(Z)\right)\right]}^2.$$

Corollary 2 is a special case of Theorem 1. All proofs are provided in Section 5.

3. Simulations

One of the main applications of our results is the ability to construct confidence intervals, and hence testing hypothesis. Specifically, Corollary 1 implies that an asymptotic $(1-\alpha )100\%$ confidence interval for $H_m$ is given by

$$\left({\widehat{H}}_m-z_{\alpha /2}\frac{{\widehat{\sigma }}_m}{\sqrt{n}},{\widehat{H}}_m+z_{\alpha /2}\frac{{\widehat{\sigma }}_m}{\sqrt{n}}\right),$$

(2)

where ${\widehat{\sigma }}_m$ is given by (1) and $z_{\alpha /2}$ is a number such that $P\left(Z>z_{\alpha /2}\right)=\alpha /2$ and $Z\sim$ $N(0,1)$. In this section, we give a small scale simulation study to check the finite sample performance of this confidence interval.

We consider Zeta distribution

$$P(x=k)=\frac{1}{\zeta (s)}{k}^{-s},k\in \{1,2,\dots \}$$

with $s=1.5$ and 2.5, where $\zeta (s)$ is the Riemann zeta function given by

$$\zeta (s)=\sum _{n=1}^{\infty }\frac{1}{n^s}.$$

The simulations were performed as follows. For the given distribution, we obtained a random sample of size n and used it to evaluate a $95\%$ confidence interval for a given index using (2). We then checked to see if the true value of the $H_m$ was in the interval or not. This was repeated 5000 times, and the proportion of times when the true value was in the interval was calculated. When the asymptotics works well, this proportion should be close to $0.95$. We repeated this for sample sizes ranging from 10 to 1000 in increments of 10. The results for $s=1.5$, order $m=2$ and $m=3$ are given in Figure 1 and Figure 2; the results for $s=2.5$, order $m=2$ and $m=3$ are given in Figure 3 and Figure 4.

The results suggest that convergence is fast, particularly when the order is $m=2$. We conjecture that this may be caused by the fact that, when m is larger, the probabilities in the corresponding CDOTC are smaller and hence require a larger sample size for convergence. For the same reason, the results with $s=1.5$ converge faster than that of $s=2.5$, because $s=2.5$ yields a thinner tail distribution which requires a larger sample size for convergence. Although GSE with order $m\ge 3$ may shed some light on specific information, GSE with order $m=2$ is enough to well exist with asymptotic properties for any valid underlying probability distribution $\mathbf{p}$.

4. Discussion

The proposed asymptotic properties in Corollary 1 and 2 make it possible for interval estimation and statistical tests. Based on the simulation results, the convergence is quite fast, particularly under order $m=2$. Note that a GSE with order $m=2$ already enjoys all asymptotic properties without any assumption on original distribution $\mathbf{p}$.

We recommend using GSE with order $m=2$ in place of Shannon’s entropy in all entropy-based methods when applicable. By replacing Shannon’s entropy with GSE, one still enjoys all the benefits of Shannon’s entropy with a fast convergence speed. Moreover, using GSE is risk-free compared to Shannon’s entropy because Shannon’s entropy (1) does not exist on some thick-tailed distributions and (2) requires thinner tail distribution for some asymptotic properties [11]. Additional research is required to aid the transition. The proposed asymptotic results allow interval estimation and statistical tests on the modified entropy-based methods that replaced Shannon’s entropy with GSE. Future research should aim to provide additional estimation methods of GSE and statistical properties of functions of GSE, such as GMI. The proposed asymptotic properties in this article directly provide asymptotic normality for the plug-in estimator of GMI when the real underlying GMI is not 0. The asymptotic behavior for the plug-in estimator of GMI when the real underlying GMI is 0 remains an open question, which we will address in future work.

5. Proofs

The proofs require several lemmas. The first lemma is state below.

Lemma 1

([11,23]). Assume that ${\sum }_{k=1}^{\infty }{p}_k{\left|log{p}_k\right|}^2<\infty$ and that there is a deterministic sequence $K(n)$ with $K(n)\to \infty$ such that ${lim}_{n\to \infty }K(n)/\sqrt{n}\to 0$ and

$$\underset{n\to \infty }{lim}\sqrt{n}\sum _{k=K(n)}^{\infty }{p}_{k}log{p}_k=0.$$

In this case

$$\sqrt{n}\left({\widehat{H}}_n-H\right)\stackrel{d}{\to }N\left(0,{\sigma }^2\right),$$

where

$${\sigma }^2=\sum _{k=1}^{\infty }{p}_k{\left(log{p}_k\right)}^2-{\left(\sum _{k=1}^{\infty }{p}_{k}log{p}_k\right)}^2.$$

Furthermore, if $\sigma >0$,

$$\sqrt{n}\left(\frac{{\widehat{H}}_n-H}{{\widehat{\sigma }}_n}\right)\stackrel{d}{\to }N(0,1)$$

where

$${\widehat{\sigma }}_n^2=\sum _{k=1}^{\infty }{\widehat{p}}_k{\left(log{\widehat{p}}_k\right)}^2-{\left(\sum _{k=1}^{\infty }{\widehat{p}}_{k}log{\widehat{p}}_k\right)}^2.$$

Different proofs of Lemma 1 are provided in [11,23].

The spirit for proof of Theorem 1 is to regard CDOTC as an original distribution and utilize the result from Lemma 1. Toward that end, several lemmas are needed and stated below.

Lemma 2

(Equivalent conditions in Lemma 1). For any valid distribution $\mathbf{p}$, let the corresponding CDOTC with order m be denoted as ${\mathbf{p}}_m$, then

$$\sum _{k=1}^{\infty }{p}_{m,k}{\left|log{p}_{m,k}\right|}^2<\infty$$

and that there is a deterministic sequence $K(n)$ with $K(n)\to \infty$ such that ${lim}_{n\to \infty }K(n)/\sqrt{n}\to 0$ and

$$\underset{n\to \infty }{lim}\sqrt{n}\sum _{k=K(n)}^{\infty }{p}_{m,k}log{p}_{m,k}=0.$$

Lemma 3

(${\sigma }_m^2$ in Theorem 1). In Theorem 1,

$${\sigma }_m^2=\sum _{k=1}^{\infty }{\left[\frac{m^2}{p_k}\left(p_{m,k}ln{p}_{m,k}+p_{m,k}{H}_m(Z)\right)\right]}^2.$$

Lemma 4

(${\widehat{\sigma }}_m^2$ in Corollary 1). In Corollary 1,

$${\widehat{\sigma }}_m^2=\sum _{k=1}^{\infty }{\left[\frac{m^2}{{\widehat{p}}_k}\left({\widehat{p}}_{m,k}ln{\widehat{p}}_{m,k}+{\widehat{p}}_{m,k}{\widehat{H}}_m(Z)\right)\right]}^2.$$

Proof of Lemma 2.

Note that for any $\mathbf{p}$ to be a valid distribution, the tail of $\mathbf{p}$ must be thicker than $1/(klnk)$ because ${\sum }_{k\ge 2}1/(klnk)$ diverges. Hence ${\mathbf{p}}_m$ is thicker than $1/(k^2{ln}^{2}k)$ for any $m\ge 2$ by definition. It is shown in Example 3 of [11] that such tail satisfies the mentioned conditions. □

Proof of Lemma 3.

Because of Lemma 2, ${\sigma }^2$ can be obtained under finite K and then let $K\to \infty$. For a finite K, it can be verified that for $i=1,\dots ,K-1$,

$$\frac{\partial H_m}{\partial p_i}=\left(ln{p}_{m,K}-ln{p}_{m,i}\right)\frac{m{p}_{m,i}}{p_i}-m\left(\frac{p_{m,i}}{p_i}-\frac{p_{m,K}}{p_K}\right)\left(H_m+ln{p}_{m,K}\right).$$

Let

$$\begin{array}{cc}& v={\left(p_1,\dots ,p_{K-1}\right)}^{\tau },\hfill \\ & \widehat{v}={\left({\widehat{p}}_1,\dots ,{\widehat{p}}_{K-1}\right)}^{\tau }.\hfill \end{array}$$

We have $\sqrt{n}(\widehat{v}-v)\stackrel{L}{\to }MVN(0,\Sigma (v))$, where $\Sigma (v)$ is the $(K-1)\times (K-1)$ covariance matrix given by

$$\Sigma (v)=\left(\begin{array}{cccc}{p}_1\left(1-p_1\right)& -p_1{p}_2& \dots & -p_1{p}_{K-1}\\ -p_2{p}_1& p_2\left(1-p_2\right)& \dots & -p_2{p}_{K-1}\\ \dots & \dots & \dots & \dots \\ -p_{K-1}{p}_1& -p_{K-1}{p}_2& \dots & p_{K-1}\left(1-p_{K-1}\right)\end{array}\right)$$

According to the first-order Delta method,

$${\sigma }_K^2=\nabla H_m^T\Sigma \nabla H_m=\sum _{k=1}^K{\left[\frac{m^2}{p_k}\left(p_{m,k}ln{p}_{m,k}+p_{m,k}{H}_m(Z)\right)\right]}^2.$$

Given Lemma 2, let $K\to \infty$,

$${\sigma }^2=\sum _{k=1}^{\infty }{\left[\frac{m^2}{p_k}\left(p_{m,k}ln{p}_{m,k}+p_{m,k}{H}_m(Z)\right)\right]}^2.$$

 □

Proof of Lemma 4.

Lemma 4 is because of ${\widehat{\sigma }}_m^2\stackrel{p}{\to }{\sigma }_m^2$. □

Proof of Theorem 1 and Corollary 1.

With Lemmas 1–4, and Slutsky’s theorem, Theorem 1 and Corollary 1 are proved. □

Proof of Corollary 2.

Corollary 2 is a directly result of Theorem 1, except under uniform distribution when $\nabla H_m=0$ for all $m\ge 2$. □