Several Basic Elements of Entropic Statistics

Abstract

Inspired by the development in modern data science, a shift is increasingly visible in the foundation of statistical inference, away from a real space, where random variables reside, toward a nonmetrized and nonordinal alphabet, where more general random elements reside. While statistical inferences based on random variables are theoretically well supported in the rich literature of probability and statistics, inferences on alphabets, mostly by way of various entropies and their estimation, are less systematically supported in theory. Without the familiar notions of neighborhood, real or complex moments, tails, et cetera, associated with random variables, probability and statistics based on random elements on alphabets need more attention to foster a sound framework for rigorous development of entropy-based statistical exercises. In this article, several basic elements of entropic statistics are introduced and discussed, including notions of general entropies, entropic sample spaces, entropic distributions, entropic statistics, entropic multinomial distributions, entropic moments, and entropic basis, among other entropic objects. In particular, an entropic-moment-generating function is defined and it is shown to uniquely characterize the underlying distribution in entropic perspective, and, hence, all entropies. An entropic version of the Glivenko–Cantelli convergence theorem is also established.

1. Introduction and Summary

Let $\mathcal{X}=\{{\ell }_k;k\ge 1\}$ be a countable alphabet and let $\mathbf{p}=\{p_k;k\ge 1\}$ be a probability distribution on $\mathcal{X}$. Let $\mathcal{P}$ be the collection of all probability distributions on $\mathcal{X}$. Let ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$ be the nonincreasingly rearranged $\mathbf{p}$, that is, $p_{\left(k\right)}\ge p_{(k+1)}$ for every $k\ge 1$. Let ${\mathcal{P}}_{\downarrow }$ be the collection of all possible ${\mathbf{p}}_{\downarrow }$. It follows that ${\mathcal{P}}_{\downarrow }\subset \mathcal{P}$ is an aggregated version of $\mathcal{P}$ in the sense that $\mathcal{P}$ is partitioned and represented by ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$.

Across a wide spectrum of scientific investigation, a random system is often described as a probability distribution on a countable alphabet, $\{\mathcal{X},\mathbf{p}\}$; however many complex system properties of interest, such as those studied in information theory and statistical mechanics, are often described by functions of ${\mathbf{p}}_{\downarrow }$, for example, the Shannon entropy

$$H=-\sum _{k\ge 1}{p}_{k}ln{p}_k$$

as in [1], the members of the Rényi entropy family

$$R_{\alpha }=ln\sum _{k\ge 1}{p}_k^{\alpha }/(1-\alpha )$$

where $\alpha \in (0,1)\cup (1.\infty )$, as in [2], and the members of the Tsallis entropy family

$$T_{\alpha }=(1-\sum _{k\ge 1}{p}_k^{\alpha })/(\alpha -1)$$

where $\alpha \in (-\infty ,1)\cup (1,\infty )$, as in [3]. Other similar functions come under the names of diversity indices, for example, the Gini–Simpson index

$$\zeta =1-\sum _{k\ge 1}{p}_k^2$$

as in [4], the generalized Simpson’s indices

$${\zeta }_{u,v}=\sum _{k\ge 1}{p}_k^u{(1-p_k)}^v$$

where $u\ge 1$ and $v\ge 0$ are integers, as described in [5], Hill’s diversity numbers

$$H_{\alpha }={(\sum _{k\ge 1}{p}_k^{\alpha })}^{1/(1-\alpha )}$$

where $\alpha \in (0,1)\cup (1,\infty )$, as in [6], Emlen’s index

$$D=\sum _{k\ge 1}{p}_k{e}^{-p_k},$$

as in [7], and the richness index

$$K=\sum _{k\ge 1}1[p_k>0],$$

where $1[·]$ is the indicator function. While all the abovementioned functions each have their unique significance in their respective fields of study, they share one characteristic in common: they are all functions of ${\mathbf{p}}_{\downarrow }$.

The word entropy has ancient Greek roots, en and tropē, that is, inward and change respectively, in English, or internal change collectively. As such, it is a label-independent concept. For generality and conciseness of the presentation in this article, let the following definition be adopted.

Definition 1.

Let $f\left(\mathbf{p}\right)$ be a function defined for every $\mathbf{p}\in \mathcal{P}$. The function $f\left(\mathbf{p}\right)$ is referred to as an entropy if $f\left(\mathbf{p}\right)$ depends on $\mathbf{p}$ only through ${\mathbf{p}}_{\downarrow }$, that is, $f\left(\mathbf{p}\right)=f\left({\mathbf{p}}_{\downarrow }\right)$.

By Definition 1, all entropies and diversity indices mentioned about are indeed entropies. In addition, $p_{\left(1\right)}$, or more generally $p_{\left(k\right)}$ for any positive integer k, is an entropy, and therefore ${\mathbf{p}}_{\downarrow }$ is an array of entropies. One important property to be noted about entropies is that ${\mathbf{p}}_{\downarrow }$ is independent of labels of the alphabet, $\{{\ell }_k;k\ge 1\}$. Another fact to be noted is that all entropies are uniquely determined by ${\mathbf{p}}_{\downarrow }$. For clarity of terminologies throughout this article, let it be noted that any properties of the underlying random system that are described by one or more entropies are referred to as entropic properties. Furthermore, $\mathbf{p}$ is referred to as the underlying probability distribution, or simply the distribution, of a random system, and ${\mathbf{p}}_{\downarrow }$ is referred to as the entropic distribution associated with $\mathbf{p}$. It is also to be noted that ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$ is not a probability distribution in the usual sense since it is not associated with any specific probability experiment. It is merely an array of nonincreasingly ordered positive parameters that sum up to one.

Let $\{X_1,\cdots ,X_n\}$, drawn from $\mathcal{X}$ according to $\mathbf{p}$, be a random sample of size n. The sample may be summarized into $\mathbf{Y}=\{Y_k;k\ge 1\}$, where $Y_k$ is the observed frequency of letter ${\ell }_k$, or into $\widehat{\mathbf{p}}=\{{\widehat{p}}_k=Y_k/n;k\ge 1\}$. Let ${\mathbf{Y}}_{\downarrow }=\{Y_{\left(k\right)};k\ge \}$ and ${\widehat{\mathbf{p}}}_{\downarrow }=\{{\widehat{p}}_{\left(k\right)}=Y_{\left(k\right)}/n;k\ge 1\}$ be the nonincreasingly rearranged $\mathbf{Y}$ and $\widehat{\mathbf{p}}$, respectively, where $Y_{\left(k\right)}\ge Y_{(k+1)}$ and ${\widehat{p}}_{\left(k\right)}\ge {\widehat{p}}_{(k+1)}$ for every k. Under the assumption that the study interest of the underlying random system only lies with the properties described by indices that are functions of the form $f\left({\mathbf{p}}_{\downarrow }\right)$, that is, entropies by Definition 1, there are two conceptual perspectives to the associated with statistical inference. The first is a framework of estimating $f\left(\mathbf{p}\right)$ based on $\widehat{\mathbf{p}}$, and the second is one of estimating $f\left(\mathbf{p}\right)=f\left({\mathbf{p}}_{\downarrow }\right)$ based on ${\widehat{\mathbf{p}}}_{\downarrow }$. For lack of better terms, let the first framework be referred to as the classical statistics and the second framework as the entropic statistics. These two frameworks are not equivalent and, in particular, the entropic framework has its special and useful implications.

The literature of statistical estimation of entropies, mostly in the specific form of the Shannon entropy, begins with the early works, as in [8,9,10], and expands in width and depth in works by, for example, [11,12,13]. Many other worthy references on entropy estimation may be found in the literature review in [14]. The general entropies of Definition 1, however, allow a discussion on the foundational elements of the statistics in entropic perspective, or entropic statistics, in a broader sense. This article focuses on three basic basic issues.

First, a notion of entropic sample space is introduced in Section 2 below. An entropic sample space is an aggregated sample space to register; not a single data point, but an ensemble of data points. It is a sample space of the entropic statistics, ${\mathbf{Y}}_{\downarrow }$ or ${\widehat{\mathbf{p}}}_{\downarrow }$, and hence is label-independent. The said label-independence in turn allows an entropic sample space to accommodate statistical sampling into a population that is not necessarily prescribed, that is, the labels of alphabet $\mathcal{X}$ need not be completely specified a priori. This property of an entropic sample space gives new meaning to statistical learning and lends foundational support for statistical exploration into an unknown, or partially known, universe.

Second, an entropic characteristic function, $\varphi \left(t\right)={\sum }_{k\ge 1}{p}_{\left(k\right)}^t$ for $t\ge 1$, is introduced. It is obvious that $\varphi \left(t\right)$ is an entropy by Definition 1 and that it always exists. It is established in Section 3 that $\varphi \left(t\right)$ in an arbitrarily small neighborhood of any interior point of $[1,\infty )$ uniquely determines the ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$ and vice versa. Therefore, it is immediately implied that any and all entropic properties of a random system, including statistical inferences, may be approached by way of $\varphi \left(t\right)$.

Third, it is established in Section 4 that the entropic statistics converges almost surely and uniformly to the underlying entropic distribution, that is, ${\widehat{\mathbf{p}}}_{\downarrow }\stackrel{a.s.}{⟶}{\mathbf{p}}_{\downarrow }$ uniformly, for any ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$. In light of the entropic sampling space and an entropic characterization of the associated entropic sampling distribution, the Glivenko–Cantelli-like convergence theorem provides a fundamental support in theory for exercises in entropic statistics.

The article ends with an appendix where a lengthy proof is found.

2. Things Entropic

2.1. Sample Spaces in Different Resolutions

Consider the experiment of randomly drawing a marble from urn 1, which contains marbles of $K=3$ known colors, red, white, and blue. In anticipating the outcome of the experiment, one may introduce an index k, $k=1,2,3$, to label the possible outcomes by ${\ell }_1=\mathit{red}$, ${\ell }_2=\mathit{white}$, and ${\ell }_3=\mathit{blue}$, and denote the corresponding proportions by $p_1$, $p_2$, and $p_3$. In this case, the sample space is

$${\Omega }_1=\{{\ell }_1,{\ell }_2,{\ell }_3\},$$

(1)

the event space is $\mathcal{B}=\{\varnothing ,\left\{{\ell }_1\right\},\left\{{\ell }_2\right\},\left\{{\ell }_3\right\},\{{\ell }_1,{\ell }_2\},\{{\ell }_1,{\ell }_3\},\{{\ell }_2,{\ell }_3\},\{{\ell }_1,{\ell }_2,{\ell }_3\}\}$, and the point mass probability measure $\mu (·)$ assigns $p_1$ to ${\ell }_1$, $p_2$ to ${\ell }_2$, and $p_3$ to ${\ell }_3$. Let X denote the random outcome of the experiment. The following model of probability distribution,

$$\begin{array}{|cccc|}\hline X& {\ell }_1& {\ell }_2& {\ell }_3\\ P\left(x\right)& p_1& p_2& p_3\\ \hline\end{array}$$

(2)

or in a different form $\mathbf{p}=\{p_1,p_2,p_3\}$ on $\mathcal{X}={\Omega }_1=\{{\ell }_1,{\ell }_2,{\ell }_3\}$, is well defined with three parameters, $p_1$, $p_2$, and $p_3$, subject to the constraints, $0\le p_k\le 1$ for each k and ${\sum }_{k=1}^3{p}_k=1$. The result of drawing $n=1$ marble from the urn may also be represented by a triplet of random variables $\mathbf{Y}=\{1[X={\ell }_1],1[X={\ell }_2],1[X={\ell }_3]\}$. If $\mathbf{Y}$ is used to represent the outcome of the experiment, the sample space may be denoted as ${\Omega }_1=\{\{1,0,0\},\{0,1,0\},\{0,0,1\}\}$ with corresponding probability distribution $P(\mathbf{Y}=\{1,0,0\})=p_1$, $P(\mathbf{Y}=\{0,1,0\})=p_2$ and $P(\mathbf{Y}=\{0,0,1\})=p_3$. For clarity in terminology, X is referred to as a random element but $\mathbf{Y}$ is a set of random variables. In general, random results of an experiment that are represented by numerical values are referred to as random variables, and those by non-numerical symbols are random elements.

For a given experiment, the sample space may be chosen at different levels of resolution depending on the experimenter’s interest in the study. Suppose the experimenter is to randomly draw $n=3$ marbles from urn 1 with replacement in sequence, resulting in $\mathbf{X}=\{X_1,X_2,X_3\}$ where $X_i$, $i=1,2,3$, is the color of the ith marble drawn in the sequence. The sample space associated with $\mathbf{X}$ may be represented by

$${\Omega }_s=\left\{\begin{array}{ccc}\{{\ell }_1,{\ell }_1,{\ell }_1\},\hfill & \{{\ell }_2,{\ell }_2,{\ell }_2\},\hfill & \{{\ell }_3,{\ell }_3,{\ell }_3\},\hfill \\ \{{\ell }_1,{\ell }_1,{\ell }_2\},\hfill & \{{\ell }_2,{\ell }_1,{\ell }_1\},\hfill & \{{\ell }_1,{\ell }_2,{\ell }_1\},\hfill \\ \{{\ell }_1,{\ell }_1,{\ell }_3\},\hfill & \{{\ell }_3,{\ell }_1,{\ell }_1\},\hfill & \{{\ell }_1,{\ell }_3,{\ell }_1\},\hfill \\ \{{\ell }_2,{\ell }_2,{\ell }_1\},\hfill & \{{\ell }_1,{\ell }_2,{\ell }_2\},\hfill & \{{\ell }_2,{\ell }_1,{\ell }_2\},\hfill \\ \{{\ell }_2,{\ell }_2,{\ell }_3\},\hfill & \{{\ell }_3,{\ell }_2,{\ell }_2\},\hfill & \{{\ell }_2,{\ell }_3,{\ell }_2\},\hfill \\ \{{\ell }_3,{\ell }_3,{\ell }_1\},\hfill & \{{\ell }_1,{\ell }_3,{\ell }_3\},\hfill & \{{\ell }_3,{\ell }_1,{\ell }_3\},\hfill \\ \{{\ell }_3,{\ell }_3,{\ell }_2\},\hfill & \{{\ell }_2,{\ell }_3,{\ell }_3\},\hfill & \{{\ell }_3,{\ell }_2,{\ell }_3\},\hfill \\ \{{\ell }_1,{\ell }_2,{\ell }_3\},\hfill & \{{\ell }_1,{\ell }_3,{\ell }_2\},\hfill & \{{\ell }_2,{\ell }_1,{\ell }_3\},\hfill \\ \{{\ell }_2,{\ell }_3,{\ell }_1\},\hfill & \{{\ell }_3,{\ell }_1,{\ell }_2\},\hfill & \{{\ell }_3,{\ell }_2,{\ell }_1\}\hfill \end{array}\right\},$$

(3)

where the subscript “s” stands for sequential. There are 27 distinct elements in (3). In this case, the sample space may also be expressed as ${\Omega }_s={\{{\ell }_1,{\ell }_2,{\ell }_3\}}^3$. This sample space may be adopted if the order of the $n=3$ observations is observable and is of interest.

Suppose in the above experiment the order of the observations is not observable or not of interest. Then the relevant information in $\mathbf{X}=\{X_1,X_2,X_3\}$ may be represented in the form of $\mathbf{Y}=\{Y_1,Y_2,Y_3\}$, where $Y_k$, $k=1,2$, and 3 is the number of ${\ell }_k$s observed in the sample. The sample space associated with $\mathbf{Y}$ is

$${\Omega }_m=\left\{\begin{array}{ccccc}\{3,0,0\},\hfill & \{0,3,0\},\hfill & \{0,0,3\},\hfill & \{2,1,0\},\hfill & \{2,0,1\},\hfill \\ \{0,2,1\},\hfill & \{1,2,0\},\hfill & \{0,1,2\},\hfill & \{1,0,2\},\hfill & \{1,1,1\}\hfill \end{array}\right\},$$

(4)

where the subscript “m” stands for multinomial. There are 10 distinct elements in (4). In fact, $\mathbf{Y}=\{Y_1,Y_2,Y_3\}$ is the usual multinomial random vector with $K=3$ categories and category probabilities $p_1$, $p_2$, and $p_3$.

The two sample spaces, ${\Omega }_s$ and ${\Omega }_m$, serve different statistical interests in various situations. ${\Omega }_s$ is well defined if $\mathbf{X}=\{X_1,X_2,X_3\}$ is observable. ${\Omega }_m$ is well defined if $\mathbf{X}=\{X_1,X_2,X_3\}$ is observable or only $\mathbf{Y}=\{Y_1,Y_2,Y_3\}$ is observable. Noting that $\mathbf{X}=\{X_1,X_2,X_3\}$ implies $\mathbf{Y}=\{Y_1,Y_2,Y_3\}$, a lower-resolution sample space may always be adopted if a higher-resolution sample space may, but not vice versa. For example, if the order of the draws is not observable, then only ${\Omega }_m$ is appropriate since $\mathbf{Y}=\{Y_1,Y_2,Y_3\}$ is not linked uniquely to the elements of ${\Omega }_s$.

${\Omega }_m$ is an aggregated form of ${\Omega }_s$ and is hence of lower resolution; however, ${\Omega }_m$ may be further reduced in resolution. Let

$${\mathbf{Y}}_{\downarrow }=\{Y_{\left(1\right)},Y_{\left(2\right)},Y_{\left(3\right)}\},$$

(5)

where $Y_{\left(1\right)},Y_{\left(2\right)},Y_{\left(3\right)}$ are nonincreasingly ordered observed frequencies of the three colors. The sample space associated with ${\mathbf{Y}}_{\downarrow }$ is

$${\Omega }_e=\left\{\begin{array}{ccc}\{3,0,0\},\hfill & \{2,1,0\},\hfill & \{1,1,1\}\hfill \end{array}\right\},$$

(6)

where the subscript “e” stands for entropic. ${\Omega }_e$ is yet an aggregated form of ${\Omega }_m$ and hence of lower resolution still than that of ${\Omega }_m$. Noting that ${\Omega }_e$ is label-independent, it is an example of entropic sample space.

It is easily verified that the probability distribution of ${\mathbf{Y}}_{\downarrow }$ may be expressed in terms of ${\mathbf{p}}_{\downarrow }$ as follows.

$$\begin{array}{|cccc|}\hline {\mathbf{Y}}_{\downarrow }& \{3,0,0\}& \{2,1,0\}& \{1,1,1\}\\ P\left({\mathbf{y}}_{\downarrow }\right)& \sum _{k\ge 1}{p}_{\left(k\right)}^3& 3\sum _{k\ge 1}{p}_{\left(k\right)}^2\left(1-p_{\left(k\right)}\right)& 6p_{\left(1\right)}{p}_{\left(2\right)}{p}_{\left(3\right)}\\ \hline\end{array}$$

(7)

Let it be noted that all the probabilities in (7) are label-independent, and therefore they are entropies by Definition 1.

In the case of sampling $n=3$ marbles from urn 1 in sequence, a subscription to the entropic sample space, ${\Omega }_e$, is by choice since both ${\Omega }_m$ and ${\Omega }_e$ are available. There are situations when the subscription to an entropic sample space may be by necessity.

Consider the experiment of randomly drawing $n=3$ marbles in sequence from urn 2, which contains marbles of $K=3$ unknown but distinguishable colors. In this case, the sample spaces, ${\Omega }_1$ of (1) and ${\Omega }_m$ of (3), are not well defined due to the lack of knowledge of the color labels. However, the entropic sample space, ${\Omega }_e$, is available for subscription regardless of what the colors are, known or unknown, as long as they are distinguishable.

In general, consider drawing a random sample of size n from $\mathcal{X}=\{{\ell }_k;k\ge 1\}$ under $\mathbf{p}=\{p_k;k\ge 1\}$ in sequence. The sequential sample space is of the form ${\Omega }_1={\mathcal{X}}^n$. The aggregated sample space,

$${\Omega }_m=\{\{y_k;k\ge 1\}:y_k\ge 0 \mathrm{for} \mathrm{every} k\ge 1 \mathrm{and} {\sum }_{k\ge 1}{y}_k=n\},$$

(8)

is that of the mutinomial array, $\mathbf{Y}=\{Y_k;k\ge 1\}$, with probability mass function

$$P\left(\{y_k;k\ge 1\}\right)=\frac{n!}{{\prod }_{k\ge 1}{y}_k!}\prod _{k\ge 1}{p}_k^{y_k}$$

(9)

where $0\le y_k\le n$ for every $k\ge 0$ and ${\sum }_{k\ge 1}{y}_k=n$. Moreover, ${\Omega }_m$ may be further aggregated into a sample space, ${\Omega }_e$, for ${\mathbf{Y}}_{\downarrow }=\{Y_{\left(k\right)};k\ge 1\}$, that is,

$${\Omega }_e=\{\{y_{\left(k\right)};k\ge 1\}:y_{\left(k\right)}\ge 0 \mathrm{and} y_{\left(k\right)}\ge y_{(k+1)} \mathrm{for} \mathrm{every} k\ge 1, \mathrm{and} {\sum }_{k\ge 1}{y}_{\left(k\right)}=n\}.$$

(10)

Let ${\Omega }_e$ of (10) be referred to as the entropic sample space. The associated probability distribution is

$$P\left(\{y_{\left(k\right)};k\ge 1\}\right)=\sum _{*}P\left(\{y_k;k\ge 1\}\right)$$

(11)

where ${\sum }_{*}$ is summation of (9) over all $\{y_k;k\ge 1\}$s in ${\Omega }_m$ sharing the same given $\{y_{\left(k\right)};k\ge 1\}$.

Given a ${\mathbf{y}}_{\downarrow }=\{y_{\left(k\right)};k\ge 1\}$, (11) is an entropy. This may be seen in two steps. First, let $\widehat{K}={\sum }_{k\ge 1}1[y_{\left(k\right)}\ge 1]$ be the number of distinct letters of $\mathcal{X}$ represented in a sample of size n, and let $\mathbf{z}=\{z_1,\cdots ,z_{\widehat{K}}\}$ be the set of $\widehat{K}$ positive integer values of ${\mathbf{y}}_{\downarrow }$. $\widehat{K}$ is a positive finite integer. Let the cardinality of $\mathcal{X}$ be denoted as $K={\sum }_{k\ge 1}1[p_k>0]$. $K\ge 1$ may be finite or countably infinite. Consider an array $\mathbf{a}\left({\mathbf{y}}_{\downarrow }\right)=\{a_k\left({\mathbf{y}}_{\downarrow }\right);k\ge 1\}$ of length K whose entries are a particular allocation of the $\widehat{K}$ values of $z_j$, $j=1,\dots ,\widehat{K}$, with the other $K-\widehat{K}$ values of $\mathbf{a}\left({\mathbf{y}}_{\downarrow }\right)$ being zeros. Let $\mathbf{A}\left({\mathbf{y}}_{\downarrow }\right)$ be the complete collection of all such distinct $\mathbf{a}\left({\mathbf{y}}_{\downarrow }\right)$s. Then it is clear that ${\mathbf{y}}_{\downarrow }$ uniquely implies $\mathbf{A}\left({\mathbf{y}}_{\downarrow }\right)$.

Second, the probability in (11) may be re-expressed as

$$P\left(\{y_{\left(k\right)};k\ge 1\}\right)=\frac{n!}{{\prod }_{k\ge 1}{y}_{\left(k\right)}!}\sum _{**}\left(\prod _{k\ge 1}{p}_{\left(k\right)}^{a_k\left({\mathbf{y}}_{\downarrow }\right)}\right)$$

(12)

where ${\sum }_{**}$ is summation over all $\mathbf{a}\left({\mathbf{y}}_{\downarrow }\right)\in \mathbf{A}\left({\mathbf{y}}_{\downarrow }\right)$, given a ${\mathbf{y}}_{\downarrow }=\{y_{\left(k\right)};k\ge 1\}$. Equation (12) implies that $P\left(\{y_{\left(k\right)};k\ge 1\}\right)$ is a function of ${\mathbf{p}}_{\downarrow }$ and hence an entropy. Let $P\left(\{y_{\left(k\right)};k\ge 1\}\right)$ of (11) or (12) be referred to as the entropic distribution associated with the entropic sample space, ${\Omega }_e$.

2.2. Entropic Objects

Let the adjective “entropic” be used to describe objects that are label-independent. Several such objects are defined or summarized below.

• A function, $f\left(\mathbf{p}\right)=f\left({\mathbf{p}}_{\downarrow }\right)$ for all ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$, is an entropy.
• The elements of ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$ are the entropic parameters, as compared to the elements of $\mathbf{p}=\{p_k;k\ge 1\}$, which are multinomial parameters.
• The elements of ${\mathbf{Y}}_{\downarrow }=\{Y_{\left(k\right)};k\ge 1\}$ or equivalently of ${\widehat{\mathbf{p}}}_{\downarrow }=\{{\widehat{p}}_{\left(k\right)};k\ge 1\}$ are entropic statistics, as compared to the elements of $\mathbf{Y}=\{Y_k;k\ge 1\}$ or equivalently $\widehat{\mathbf{p}}=\{{\widehat{p}}_k;k\ge 1\}$, which are multinomial statistics.
• ${\Omega }_e$ of (10) is the entropic (multinomial) sample space, as compared to ${\Omega }_m$ of (8), which is the multinomial sample space.
• The distribution $P\left(\{y_{\left(k\right)};k\ge 1\}\right)$ of (11) or (12), is the entropic probability distribution, while $P\left(\{y_k;k\ge 1\}\right)$ of (9) is the multinomial probability distribution.
• Entropic statistics is the collection of statistical methodologies that help to make inference on the characteristics of a random system exclusively via entropies.

In addition, there are several useful other entropic objects. First, letting ${\zeta }_v={\sum }_{k\ge 1}{p}_k{(1-p_k)}^v$ for all non-negative integers $v\ge 0$, $\zeta =\{{\zeta }_v;v\ge 0\}$ is referred to as the entropic basis. The name comes from the fact that, for any well-behaved function, $h\left(p\right)$ for $p\in [0,1]$, an entropy of the form $H={\sum }_{k\ge 1}{p}_{k}h\left(p_k\right)$ may be expressed as a linear combination $H={\sum }_{v\ge 1}w\left(v\right){\zeta }_v$. For example, the Shannon entropy, provided that it is finite, may be written as

$$H=-\sum _{k\ge 1}{p}_{k}ln{p}_k=\sum _{v\ge 1}(1/v){\zeta }_{v-1}.$$

The entropic basis is useful because it unfolds many entropies into simple and linearly additive forms.

Second, letting ${\eta }_u={\sum }_{k\ge 1}{p}_k^u$ for all positive integers $u\ge 1$, $\eta =\{{\eta }_u;u\ge 1\}$ is often referred to as the entropic moment. The elements of both $\zeta$ and $\eta$ have good estimators. A detailed discussion may be found in [14].

Definition 2.

Let X be an random element on a countable alphabet $\mathcal{X}=\{{\ell }_k;k\ge 1\}$ with a corresponding probability distribution $\mathbf{p}\in \mathcal{P}$ and its associated entropic distribution ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$. The function,

$$\varphi \left(t\right)=\sum _{k\ge 1}{p}_k^t,for t\ge 0,$$

(13)

is referred to as the entropic-moment-generating function of X, of $\mathbf{p}$, or of ${\mathbf{p}}_{\downarrow }$. The two complementary parts of its domain, $[1,\infty )$ and $[0,1)$, are, respectively, referred to as the primary domain and the secondary domain of the entropic-moment-generating function.

Depending on context, $\varphi \left(t\right)$ may be denoted as ${\varphi }_X\left(t\right)$, ${\varphi }_{\mathbf{p}}\left(t\right)$, or ${\varphi }_{{\mathbf{p}}_{\downarrow }}\left(t\right)$ whenever appropriate. Obviously, $\varphi \left(t\right)$ is uniformly bounded above by one for all $\mathbf{p}\in \mathcal{P}$ in the primary domain but is not necessarily finitely defined in the secondary domain. However, in the case of a finite alphabet, that is, $K={\sum }_{k\ge 1}1[p_k>0]<\infty$, $\varphi \left(t\right)$ is finitely defined for each and every $t\in \mathbb{R}$, in particular for $t\ge 0$. The characteristic utility of $\varphi \left(t\right)$ is further explored in Section 3 below.

2.3. Examples of Entropic Statistics

Example 1.

Consider the Bernoulli experiment of tossing a coin, where $P\left(h\right)=p$ and $P\left(t\right)=1-p$. The question of whether the coin is fair may be formulated in the usual classical sense, that is, whether $p=0.5$. The question may be approached by estimating p based on a sample proportion, $\widehat{p}$, if it is observable which trials lead to “h” and which lead to “t”. The question may alternatively be formulated by an equivalent entropic statement, for example, whether $H=p(1-p)=0.25$. More generally, if $K={\sum }_{k\ge 1}1\left[p_k\right]$ is finite and known, then the uniformity of $\mathbf{p}$ on $\mathcal{X}$ may be formulated entropically by, for example, $H={\sum }_{k\ge 1}{p}_k^2=1/K$, $H={\sum }_{k\ge 1}{p}_k(1-p_k)=(K-1)/K$, or $H=-{\sum }_{k\ge 1}{p}_{k}ln{p}_k=lnK$. The validity of these entropic statements may then be gauged statistically.

Example 2.

Consider a two-stage sampling scheme: a random sample of size n, $\{X_1,\cdots ,X_n\}$, and then a single extra observation $X_{n+1}$ are taken. The sample of size n may be summarized into letter frequencies, $\mathbf{Y}=\{Y_k;k\ge 1\}$. Let ${\pi }_0={\sum }_{k\ge 1}{p}_{k}1[Y_k=0]$. Clearly, ${\pi }_0$ is label-independent and therefore an entropic random variable. Given the sample of size n, ${\pi }_0$ may be thought of as the probability of that $X_{n+1}$ assumes a letter in $\mathcal{X}$ that is not represented in the sample of size n. In some context, ${\pi }_0$ may be thought of as the probability of new discovery. Let $N_1={\sum }_{k\ge 1}1[Y_k=1]$ and $T_n=N_1/n$. $T_n$ is commonly known as Turing’s formula, introduced in [15], but credited largely to Alan Turing. It is to be noted that $N_1$ is label-independent and, therefore, so is $T_n$. $T_n$ is an good estimator of ${\pi }_0$ and a discussion on many of its statistical properties may be found in [14].

Example 3.

In developing a decision tree classifier, the data space is partitioned into an ensemble of small subspaces, in each of which a local classification rule is sought. The central spirit of every local classification may be described by a two-step scheme.

1. 

First, a random sample of size n, $\{X_1,\cdots ,X_n\}$, is taken from $\mathcal{X}=\{{\ell }_k;k\ge 1\}$, under an unknown $\mathbf{p}=\{p_k;k\ge 1\}$, which is summarized into $\mathbf{Y}=\{Y_k;k\ge 1\}$.

2. 

The data-based local classification rule is as follows: the next observation, $X_{n+1}$, is predicted to be the letter which is observed most frequently in the sample of size n. For simplicity, let it be assumed that $p_{\left(1\right)}>p_{\left(2\right)}$, and a letter with the sample maximum frequency is unique (if not, some randomization may be employed).

Obviously, the designated letter based on a sample is not necessarily the letter associated with the letter corresponding to the maximum of $p_k$s. In such a setup, the performance of the tree classifier may be gauged by evaluating (calculating or estimating) the probability of the event that “the designated letter is the same letter of $\mathcal{X}$ with probability $p_{\left(1\right)}$”, that is,

$$P\left(\underset{{\ell }_k;k\ge 1}{arg\; max}\{p\left({\ell }_k\right);k\ge 1\}=\underset{{\ell }_k;k\ge 1}{arg\; max}\{\widehat{p}\left({\ell }_k\right);k\ge 1\}\right).$$

(14)

Note that the event in (14) is label-independent and hence the probability is an entropy, which may be estimated. The probability in (14) may reasonably called the confidence level of the simple classifier.

For illustration purpose, consider the special case of a binary X, with $n=2m+1$ for some positive integer m. For simplicity, n is chosen to be odd here so that $Y_{\left(1\right)}>Y_{\left(2\right)}$ always holds true. Suppose that $p_1=p_{\left(1\right)}>p_{\left(2\right)}=1-p_{\left(1\right)}$. The event that a classifier based on the sample of n correctly identifies the letter of maximum probability may be equivalently expressed as $Y_1\ge m+1$. The probability of such an event, (14), is

$$\begin{array}{cc}\hfill P\left({\ell }_1=\underset{{\ell }_1,{\ell }_2}{arg\; max}\{{\widehat{p}}_1,1-{\widehat{p}}_1\}\right)& =P(Y_1\ge m+1)\hfill \\ \hfill & =\sum _{y\ge m+1}\frac{n!}{y!(n-y)!}{p}_{\left(1\right)}^y{(1-p_{\left(1\right)})}^{n-y},\hfill \end{array}$$

(15)

which is independent of the assumption that $p_1>p_2=1-p_1$ and, therefore, is an entropy. More specifically, (15) is computed for several combinations of n and ${\mathbf{p}}_{\downarrow }$ and the resulting values are tabulated in

Table 1. Confidence Levels of Simple Binary Classifier.

${\mathbf{p}}_{\downarrow }$	$\mathit{n}=3$	$\mathit{n}=5$	$\mathit{n}=7$
$(1/2,1/2)$	0.5000	0.5000	0.5000
$(2/3,1/3)$	0.7407	0.7901	0.8267
$(3/4,1/4)$	0.8438	0.8965	0.9294
$(4/5,1/5)$	0.8960	0.9421	0.9667
$(5/6,1/6)$	0.9259	0.9645	0.9824

. Table 1, and its likes, may be used in two different ways. First, given a fixed ${\mathbf{p}}_{\downarrow }$, it indicates how large a sample is needed to assure a reliability level of the classifier. On the other hand, at a given level of n and a particular ${\mathbf{p}}_{\downarrow }$, the classifier may be evaluated by the probabilities in the table. In practice, ${\mathbf{p}}_{\downarrow }$ is unknown but may be estimated.

3. Entropic Characterization

Entropic statistics focuses on making inference via entropies; it is therefore of interest to find a function which may characterize ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$. Since the function $\varphi \left(t\right)$ in its primary domain and $\mathit{\eta }=\{{\eta }_u;u\ge 1\}$, where ${\eta }_u={\sum }_{k\ge 1}{p}_k^u$ and u is an integer, imply each other (see Lemma 1 below), it follows immediately that (13) uniquely determines all entropies. However, the following theorem claims that the characteristic property of the entropic-moment-generating function, $\varphi \left(t\right)$, remains intact in any arbitrarily neighborhood of any $t\in (1,\infty )$.

Theorem 1.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ and $\mathbf{q}=\{q_k;k\ge 1\}$ be two probability distributions on a same countable alphabet, $\mathcal{X}=\{{\ell }_k;k\ge 1\}$. Let ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$ and ${\mathbf{q}}_{\downarrow }=\{q_{\left(k\right)};k\ge 1\}$ be the respective corresponding entropic distributions of $\mathbf{p}$ and $\mathbf{q}$. Then ${\mathbf{p}}_{\downarrow }={\mathbf{q}}_{\downarrow }$ if and only if ${\varphi }_{\mathbf{p}}\left(t\right)={\varphi }_{\mathbf{q}}\left(t\right)$ for all $t\in (a,b)$ where $(a,b)$ is an arbitrary interval such that $1\le a<b<\infty$.

Lemma 1.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ and $\mathbf{q}=\{q_k;k\ge 1\}$ be two probability distributions in $\mathcal{P}$ with two corresponding associated entropic distributions ${\mathbf{p}}_{\downarrow }$ and ${\mathbf{q}}_{\downarrow }$ in ${\mathcal{P}}_{\downarrow }$. Then ${\mathbf{p}}_{\downarrow }={\mathbf{q}}_{\downarrow }$ if and only if ${\sum }_{k\ge 1}{p}_k^n={\sum }_{k\ge 1}{q}_k^n$ for all positive integers $n\ge 1$.

A proof of Lemma 1 may be found on pages 50 and 51 in [14]. To prove Theorem 1, it suffices to show that $\varphi \left(t\right)$ in an arbitrarily small neighborhood of any interior point of $[1,\infty )$ determines the function globally.

Proof of Theorem 1.

If ${\mathbf{p}}_{\downarrow }={\mathbf{q}}_{\downarrow }$, then it immediately follows that ${\varphi }_{\mathbf{p}}\left(t\right)={\varphi }_{\mathbf{q}}\left(t\right)$ for all $t\in [1,\infty )$ and, therefore, for $t\in (a,b)$ specifically. To prove the theorem, it suffices to show the converse.

Consider the series

$$f\left(z\right)=\sum _{k=1}^{\infty }{p}_k^z$$

where $z\in \mathbb{C}$ is a complex variable. Denote the real and the imaginary parts of a complex value z by $Re\left(z\right)$ and $Im\left(z\right)$, respectively.

Let $\mathbb{D}=\{z:Re(z)>1\}$ be the subset of $\mathbb{C}$ such that the real part of z is greater than 1. For every $z\in \mathbb{D}$, since $p_k^{Re(z-1)}\le 1$ and $\left|p_k^{iIm\left(z\right)}\right|=1$, for every k, where $\left|z\right|$ is the modulus of z, it follows that

$$f\left(z\right)=\sum _{k=1}^{\infty }{p}_k{p}_k^{z-1}=\sum _{k=1}^{\infty }{p}_k{p}_k^{Re(z-1)}{p}_k^{iIm\left(z\right)}$$

and

$$\left|f\left(z\right)\right|\le \sum _{k=1}^{\infty }{p}_k.$$

(16)

Letting ${\alpha }_k=ln(1/p_k)$,

$$f\left(z\right)=\sum _{k=1}^{\infty }{e}^{-{\alpha }_{k}z},$$

(17)

and the functions $e^{-{\alpha }_{k}z}$, $k\ge 1$, are analytic on $\mathbb{C}$.

Since the series in (17), for $z\in \mathbb{D}$, is dominated by the convergent series ${\sum }_{k\ge 1}{p}_k$ as in (16), by the Weierstrass uniform convergence theorem, $f\left(z\right)$ is analytic on $\mathbb{D}$. By a similar argument, $g\left(z\right)={\sum }_{k=1}^{\infty }{q}_k^z$ is also analytic on $\mathbb{D}$.

Assuming that ${\varphi }_{\mathbf{p}}\left(t\right)={\varphi }_{\mathbf{q}}\left(t\right)$ for $t\in (a,b)$ where $1\le a<b<\infty$, there exists a convergent sequence, $\{z_n;n\ge 1\}$ in $(a,b)$ such that ${lim}_{n\to \infty }{z}_n=z_0\in (a,b)$. Noting that $(a,b)\subset \mathbb{D}$ and $f\left(z_n\right)={\varphi }_{\mathbf{p}}\left(z_n\right)={\varphi }_{\mathbf{q}}\left(z_n\right)=g\left(z_n\right)$ for $n\ge 0$, by the identity theorem for analytic functions, $f\left(z\right)=g\left(z\right)$ for all $z\in \mathbb{D}$. It follows that ${\varphi }_{\mathbf{p}}\left(t\right)=f\left(t\right)=g\left(t\right)={\varphi }_{\mathbf{q}}\left(t\right)$ for all $t\in [1,\infty )$, specifically, ${\sum }_{k\ge 1}{p}_k^n={\sum }_{k\ge 1}{q}_k^n$ for all $n\ge 1$. Finally, by Lemma 1, ${\mathbf{p}}_{\downarrow }={\mathbf{q}}_{\downarrow }$. □

Theorem 1 immediately implies that a subfamily of the Rényi entropy $R_{\alpha }$ with $\alpha \in (a,b)\subset (1,\infty )$, a subfamily of the Tsallis entropy $T_{\alpha }$ with $\alpha \in (a,b)\subset (1,\infty )$, and a subfamily of Hill’s diversity numbers $H_{\alpha }$ with $\alpha \in (a,b)\subset (1,\infty )$, respectively, characterizes ${\mathbf{p}}_{\downarrow }$ and, hence, characterizes all entropies.

The characterization of ${\mathbf{p}}_{\downarrow }$ in Theorem 1 may be equivalently stated only on any infinitely countable subset of $(a,b)$.

Corollary 1.

Let $\mathbf{p}$ and $\mathbf{q}$ be two probability distributions on a same countable alphabet, $\mathcal{X}$. Let ${\mathbf{p}}_{\downarrow }$ and ${\mathbf{q}}_{\downarrow }$ be the corresponding entropic distributions of $\mathbf{p}$ and $\mathbf{q}$, respectively. Then ${\mathbf{p}}_{\downarrow }={\mathbf{q}}_{\downarrow }$ if and only if ${\varphi }_{\mathbf{p}}\left(t\right)={\varphi }_{\mathbf{q}}\left(t\right)$ on any infinite sequence of distinct values, $\{t_n;n\ge 1\}$, such that ${lim}_{n\to \infty }{t}_n=c\in (1,\infty )$.

Proof. 

Both ${\varphi }_{\mathbf{p}}\left(t\right)$ and ${\varphi }_{\mathbf{q}}\left(t\right)$ are analytic at $t=c$, and therefore $h\left(t\right)={\varphi }_{\mathbf{p}}\left(t\right)-{\varphi }_{\mathbf{q}}\left(t\right)$ is analytic at $t=c$. Let it be first shown, by induction, that all derivatives of $h\left(t\right)$ at $t=c$ are zero, that is, $h^{\left(m\right)}\left(c\right)=0$ for $m\ge 0$. Note first that $h\left(c\right)=h^{\left(0\right)}=0$ by the fact that both ${\varphi }_{\mathbf{p}}\left(t\right)$ and ${\varphi }_{\mathbf{q}}\left(t\right)$ are continuous and ${lim}_{n\to \infty }{\varphi }_{\mathbf{p}}\left(t_n\right)={\varphi }_{\mathbf{p}}\left(c\right)={\varphi }_{\mathbf{q}}\left(c\right)={lim}_{n\to \infty }{\varphi }_{\mathbf{q}}\left(t_n\right)$. Suppose that $h^{\left(0\right)}\left(c\right)=h^{\left(1\right)}\left(c\right)=h^{\left(2\right)}\left(c\right)=\cdots =h^{\left(m\right)}\left(c\right)=0$ but $h^{(m+1)}\left(c\right)\ne 0$. Then there exists an interval $(c-\epsilon ,c+\epsilon )$ such that $h\left(t\right)\ne 0$ for $t\in (c-\epsilon ,c+\epsilon )$. However, there is at least one $t_n\in (c-\epsilon ,c+\epsilon )$ such that $h\left(t_n\right)=0$ by assumption. This is a contradiction and therefore $h^{\left(m\right)}\left(c\right)=0$ for all $m\ge 1$. □

Corollary 2.

Let $\mathbf{p}$ and $\mathbf{q}$ be two probability distributions on a same countable alphabet, $\mathcal{X}$. Let ${\mathbf{p}}_{\downarrow }$ and ${\mathbf{q}}_{\downarrow }$ be the corresponding entropic distributions of $\mathbf{p}$ and $\mathbf{q}$, respectively. Then ${\mathbf{p}}_{\downarrow }={\mathbf{q}}_{\downarrow }$ if and only if ${\varphi }_{\mathbf{p}}\left(t\right)={\varphi }_{\mathbf{q}}\left(t\right)$ on any infinite sequence of distinct values, $\{t_n;n\ge 1\}\in (a,b)$ where $1\le a<b<\infty$.

Proof. 

Noting that the infinitely many $t_n$s are in an bounded interval, there exists an infinite subset of $\{t_n;n\ge 1\}$ that converges to a constant $c\in [a,b]$. The corollary follows Corollary 1. □

Consider a pair of random elements, $(X,Y)$, on a countable joint alphabet, $\mathcal{X}\times \mathcal{Y}=\{(l_i,m_j);i\ge 1,j\ge 1\}$, with a corresponding joint probability distribution, ${\mathbf{p}}_{X,Y}=\{p_{i,j};i\ge 1,j\ge 1\}$. Let ${\mathbf{p}}_X=\{p_{i,·};i\ge 1\}$ and ${\mathbf{p}}_Y=\{p_{·,j};j\ge 1\}$, where $p_{i,·}={\sum }_{j\ge 1}{p}_{i,j}$ and $p_{·,j}={\sum }_{i\ge 1}{p}_{i,j}$, be the two marginal probability distributions of X and Y, respectively.

Corollary 3.

X and Y are independent if and only if

$${\varphi }_{X,Y}\left(t\right)={\varphi }_X\left(t\right)\times {\varphi }_Y\left(t\right)$$

(18)

for all $t\in (a,b)$, where a and b are two arbitrary real numbers such that $1\le a<b<\infty$.

Proof. 

If X and Y are independent, then (18) follows immediately. Conversely, suppose that (18) holds. Consider another pair of independent random elements, $(U,V)$, on the same countable joint alphabet $\mathcal{X}\times \mathcal{Y}$ and with identical marginal distributions to those of $(X,Y)$, that is, ${\mathbf{p}}_X$ and ${\mathbf{p}}_Y$. It then follows, by (18) and Theorem 2, that ${\mathbf{p}}_{U,V}={\mathbf{p}}_{X,Y}$, which in turn implies that X and Y are independent. □

Corollary 3 provides a characterization of independence on a general countable joint alphabet, and its utility may be explored further.

4. A Basic Convergence Theorem

From an entropic perspective, the convergence of ${\widehat{\mathbf{p}}}_{\downarrow }$ to ${\mathbf{p}}_{\downarrow }$, to be distinguished from that of $\widehat{\mathbf{p}}$ to $\mathbf{p}$, is of fundamental interest.

For clarity of presentation in this section, let it be noted that, whenever necessary, the subindex n may be added to $\mathbf{Y}$, $Y_k$, $\widehat{\mathbf{p}}$, ${\widehat{p}}_k$, ${\widehat{\mathbf{p}}}_{\downarrow }$, and ${\widehat{p}}_{\left(k\right)}$ to highlight the dynamic nature of these previously defined quantities as n changes, that is, $\mathbf{Y}={\mathbf{Y}}_n$, $Y_k=Y_{k,n}$, $\widehat{\mathbf{p}}={\widehat{\mathbf{p}}}_n$, ${\widehat{p}}_k={\widehat{p}}_{k,n}$, ${\widehat{\mathbf{p}}}_{\downarrow ,n}={\widehat{\mathbf{p}}}_{\downarrow }$ and ${\widehat{p}}_{\left(k\right)}={\widehat{p}}_{\left(k\right),n}$, respectively.

The main result established in this section is the uniform almost-sure convergence of ${\widehat{\mathbf{p}}}_{\downarrow }$ to ${\mathbf{p}}_{\downarrow }$, which is made more precise in Theorem 2 below.

Consider the experiment of repeatedly and independently drawing a letter from $\mathcal{X}$ under $\mathbf{p}$, resulting in a sequence of randomly selected letters, $\omega =\{x_1,x_2,\cdots \}$. Let the collection of all possible such sequences or paths be denoted $\Omega$. A sample of size n is a partial sequence of the first n randomly selected letters in an $\omega$, $\{x_1,\cdots ,x_n\}$.

Let ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$ and ${\widehat{\mathbf{p}}}_{\downarrow }=\{{\widehat{p}}_{\left(k\right)};k\ge 1\}$ be defined as above. It is to be specifically noted that the rearrangement of the observed relative frequencies, ${\widehat{\mathbf{p}}}_{\downarrow }$, is performed independently based on the observed values of ${\widehat{p}}_k$ for all $k\ge 1$, with no regard to the arrangement of the probabilities, ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$. Consequently, the letter of which the relative frequency ${\widehat{p}}_{\left(k\right)}$ is observed is not necessarily the same letter with which the probability $p_{\left(k\right)}$ is associated. This is, in fact, the essence of entropic perspective.

Theorem 2.

For any $\mathbf{p}\in \mathcal{P}$, let ${\mathbf{p}}_{\downarrow }$, $\widehat{\mathbf{p}}$ and ${\widehat{\mathbf{p}}}_{\downarrow }$ be defined as above. Then

$$\underset{k\ge 1}{max}\left|{\widehat{p}}_{\left(k\right)}-p_{\left(k\right)}\right|\stackrel{a.s.}{⟶}0.$$

(19)

A proof of Theorem 2 requires Lemmas 2 and 3 below.

Lemma 2.

For any $\mathbf{p}\in \mathcal{P}$, let $\widehat{\mathbf{p}}$ be as defined above. Then

$$\underset{k\ge 1}{max}\left|{\widehat{p}}_k-p_k\right|\stackrel{a.s.}{⟶}0.$$

(20)

Proof. 

For each k, by the strong law of large numbers, ${\widehat{p}}_k\stackrel{a.s.}{⟶}{p}_k$ or equivalently $|{\widehat{p}}_k-p_k|\stackrel{a.s}{⟶}0$. Let the collection of paths $\omega =\{x_1,x_2,\cdots \}$ in $\Omega$ that satisfies ${lim}_{n\to \infty }|{\widehat{p}}_k-p_k|=0$ be denoted as ${\Omega }_k\subseteq \Omega$. It follows that $P\left({\Omega }_k\right)=1$, that the complement of ${\Omega }_k$, ${\Omega }_k^{\prime }$, is of probability zero, that ${\cup }_{k\ge 1}{\Omega }_k^{\prime }$ is of probability zero, and that, letting ${\Omega }^{*}={\cap }_{k\ge 1}{\Omega }_k$, $P\left({\Omega }^{*}\right)=1-P\left({\cup }_{k\ge 1}{\Omega }_k^{\prime }\right)=1$.

For each and every path $\omega \in {\Omega }^{*}$ and every k, ${lim}_{n\to \infty }|{\widehat{p}}_k-p_k|=0$. Note the fact that $|{\widehat{p}}_k-p_k|\le {\widehat{p}}_k+p_k$ and, therefore, ${\sum }_{k\ge 1}|{\widehat{p}}_k-p_k|\le {\sum }_{k\ge 1}({\widehat{p}}_k+p_k)=2$, by the bounded convergence theorem,

$$\underset{n\to \infty }{lim}\sum _{k\ge 1}|{\widehat{p}}_k-p_k|=\sum _{k\ge 1}\underset{n\to \infty }{lim}|{\widehat{p}}_k-p_k|=0,$$

(21)

that is, ${\sum }_{k\ge 1}|{\widehat{p}}_k-p_k|\stackrel{a.s.}{⟶}0$. By (21), the lemma follows from the fact that

$$\underset{k\ge 1}{max}|{\widehat{p}}_k-p_k|\le \sum _{k\ge 1}|{\widehat{p}}_k-p_k|\stackrel{a.s.}{⟶}0.$$

 □

Lemma 2 may be viewed as a version of the Glivenko–Cantelli theorem on countable alphabets with respect to observed data from a classical multinomial experiment. The uniformity of the convergence in (20) is of essential importance in the proof of Theorem 2, which is given below by way of Lemma 3.

Lemma 3.

For each $k\ge 1$,

$${\widehat{p}}_{\left(k\right)}-p_{\left(k\right)}\stackrel{a.s.}{⟶}0.$$

(22)

A proof of Lemma 3 is given in Appendix A. Let it be noted that $\Omega$ is the sample space of a perpetual multinomial iid sampling scheme on $\mathcal{X}$ under a probability distribution $\mathbf{p}\in \mathcal{P}$. Each path in $\Omega$ may be represented by $\{{\widehat{\mathbf{p}}}_n;n\ge 1\}$ where ${\widehat{\mathbf{p}}}_n=\{{\widehat{p}}_{k,n};k\ge 1\}$. For each such path $\{{\widehat{\mathbf{p}}}_n;n\ge 1\}\in \Omega$, there exists a corresponding path $\{{\widehat{\mathbf{p}}}_{\downarrow ,n};n\ge 1\}$, which is the rearranged $\{{\widehat{\mathbf{p}}}_n;n\ge 1\}$ over all k for every n. Let the total collection of all rearranged paths of $\Omega$ be denoted as ${\Omega }_{\downarrow }$, and let the collection of all rearranged paths of ${\Omega }^{*}$ be denoted as ${\Omega }_{\downarrow }^{*}$. It follows that $P\left({\Omega }_{\downarrow }^{*}\right)=P\left({\Omega }^{*}\right)=1$. Lemma 3 states that, in each path of ${\Omega }_{\downarrow }^{*}$, the k th component of ${\widehat{\mathbf{p}}}_{\downarrow ,n}$ converges to the k th component of ${\mathbf{p}}_{\downarrow }$, namely, $p_{\left(k\right)}$, for each k.

Proof of Theorem 2.

For any $\omega \in {\Omega }_{\downarrow }^{*}$, note that ${\sum }_{k\ge 1}|{\widehat{p}}_{\left(k\right)}-p_{\left(k\right)}|\le 2$, by the bounded convergence theorem and Lemma 3, ${lim}_{n\to \infty }{max}_{k\ge 1}|{\widehat{p}}_{\left(k\right)}-p_{\left(k\right)}|\le {lim}_{n\to \infty }{\sum }_{k\ge 1}|{\widehat{p}}_{\left(k\right)}-p_{\left(k\right)}|={\sum }_{k\ge 1}{lim}_{n\to \infty }|{\widehat{p}}_{\left(k\right)}-p_{\left(k\right)}|=0.$ The theorem follows the fact that $P\left({\Omega }_{\downarrow }^{*}\right)=1$. □

Theorem 2 may be viewed as a version of the Glivenko–Cantelli theorem on countable alphabets with respect to observed data from an entropic multinomial experiment. Theorem 2 immediately implies almost sure convergence for estimators of several key quantities in classification procedures.

Example 4.

${\widehat{p}}_{\left(1\right)}\stackrel{a.s.}{\to }{p}_{\left(1\right)}$.

Example 5.

Suppose that $p_{\left(1\right)}>p_{\left(2\right)}$, that is, there exists a unique letter in $\mathcal{X}$, denoted ${\ell }_0$, associated with probability $p_{\left(1\right)}$. Then the probability of a correct classification, that is, ${\ell }_0={arg\; max}_{\mathcal{X}}\{{\widehat{p}}_k;k\ge 1\}$, converges almost surely to one. This is so because, for any path in ${\Omega }_{\downarrow }^{*}$ and any $\epsilon <(p_{\left(1\right)}-p_{\left(2\right)})/2$, there exists an N such that, for any $n>N$, $|{\widehat{p}}_{\left(1\right)}-p_{\left(1\right)}|<\epsilon$ and $|{\widehat{p}}_{\left(1\right)}-p_{\left(k\right)}|>\epsilon$ for all $k\ge 2$.

The results of Examples 4 and 5 lend fundamental support for classification algorithms based on maximum observed frequency, used widely in exercises of modern data science, for example, decision trees, as mentioned in Example 3.

Many entropies of interest across a wide spectrum of studies are of the additive form, $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)h\left(p_{\left(k\right)}\right)$, where $g\left(p\right)\ge 0$ and $h\left(p\right)\ge 0$ are functions of $p\in [0,1]$. The almost-sure convergence of Theorem 2 may be passed on to the plug-in estimators of some such entropies by way of a rather trivial statement in the proposition below.

Proposition 1.

Let $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)h\left(p_{\left(k\right)}\right)$ where $g\left(p\right)\ge 0$ and $0\le h\left(p\right)\le M$ for some $M>0$ are continuous functions of $p\in I=[0,1]$. Suppose that $\mathbf{p}\in \mathcal{P}$ such that

1. 

${\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)<\infty$, and

2. 

${\sum }_{k\ge 1}g\left({\widehat{p}}_{\left(k\right),n}\right)\stackrel{a.s.}{\to }{\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)$.

Then $H\left({\widehat{\mathbf{p}}}_{\downarrow }\right)\stackrel{a.s.}{\to }H\left({\mathbf{p}}_{\downarrow }\right)$.

Proof. 

Noting that $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)h\left(p_{\left(k\right)}\right)\le M{\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)<\infty$, it follows by Conditions 1 and 2 that

$$|H\left({\widehat{\mathbf{p}}}_{\downarrow }\right)-H\left({\mathbf{p}}_{\downarrow }\right)|\le M\sum _{k\ge 1}g\left({\widehat{p}}_{\left(k\right)}\right)+M\sum _{k\ge 1}g\left(p_{\left(k\right)}\right)<\infty .$$

(23)

Let ${\Omega }_{\downarrow }^{**}\subseteq {\Omega }_{\downarrow }$ be the total collection of paths such that Condition 2 holds. For each path, $\{{\widehat{\mathbf{p}}}_{n,\downarrow };n\ge 1\}\in {\Omega }_{\downarrow }^{**}$, by (23), the proposition follows by the bounded convergence theorem and the fact that $P\left({\Omega }_{\downarrow }^{**}\right)=1$. □

Example 6.

Let $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}{p}_{\left(k\right)}^s{(1-p_{\left(k\right)})}^t$ where $s\ge 1$ and $t\ge 0$ are two real constants. In the setup of Proposition 1, $h\left(p\right)={(1-p)}^t\le 1$ on $I=[0,1]$, and $g\left(p\right)=p^s$ satisfying ${\sum }_{k\ge 1}{p}_{\left(k\right)}^s\le 1$ for all ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$ without qualification, and therefore also for ${\widehat{\mathbf{p}}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$, that is, ${\sum }_{k\ge 1}{\widehat{p}}_{\left(k\right)}^s\le 1$, which implies, by the bounded convergence theorem, ${\sum }_{k\ge 1}{\widehat{p}}_{\left(k\right)}^s\to {\sum }_{k\ge 1}{p}_{\left(k\right)}^s$ along each and every path in ${\Omega }^{*}$. By Proposition 1, ${\sum }_{k\ge 1}{\widehat{p}}_k^s{(1-{\widehat{p}}_{\left(k\right)})}^t\stackrel{a.s.}{\to }{\sum }_{k\ge 1}{p}_k^s{(1-p_{\left(k\right)})}^t$. More specifically, when s and t take integers $u\ge 1$ and $v\ge 0$, the plug-in estimator of the generalized Simpson’s diversity index $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}{p}_{\left(k\right)}^u{(1-p_{\left(k\right)})}^v$ (see [5,16]) converges almost surely.

Example 6 implies that the plug-in estimator of $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}{p}_{\left(k\right)}^s$ where $s\ge 1$ converges almost surely, which in turn implies that the plug-in estimators of members of the Rényi entropy family and the Tsallis entropy family converge almost surely for all ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$ without qualification when $\alpha \ge 1$. However, it is not known whether the plug-in estimators of the members of the families with $\alpha \in (0,1)$ converge almost surely when ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$ without other qualification (also, see [17]).

Example 7.

The plug-in estimator of the Shannon entropy, $H\left({\mathbf{p}}_{\downarrow }\right)=-{\sum }_{k\ge 1}{p}_{\left(k\right)}ln{p}_{\left(k\right)}$, converges almost surely when ${\mathbf{p}}_{\downarrow }$ is such that $K={\sum }_{k\ge 1}{1}_{[p_{\left(k\right)}>0]}<\infty$. In this case, even though $-lnp$ is not bounded above on $I=[0,1]$, $h\left(p\right)=-p^{\alpha }lnp\le 1/\left(\alpha e\right)$ is for any $\alpha \in (0,1)$. Writing $H\left({\mathbf{p}}_{\downarrow }\right)={\sum }_{k\ge 1}g\left(p_{\left(k\right)}\right)h\left(p_{\left(k\right)}\right)$ where $g\left(p\right)=p^{1-\alpha }$ and $h\left(p\right)=-p^{\alpha }lnp$, it suffices to show ${\sum }_{k\ge 1}{\widehat{p}}_{\left(k\right)}^{1-\alpha }$ converges almost surely. However, this is the case since, by Theorem 2, for every path $\{{\widehat{\mathbf{p}}}_{n,\downarrow };n\ge 1\}\in {\Omega }_{\downarrow }^{*}$, ${\sum }_{k\ge 1}{\widehat{p}}_{\left(k\right)}^{1-\alpha }\to {\sum }_{k\ge 1}{p}_{\left(k\right)}^{1-\alpha }$, due to the fact that $K<\infty$ and $P\left({\Omega }_{\downarrow }^{*}\right)=1$.

It is not known whether the plug-in estimator of the Shannon entropy converges almost surely when ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$ without further qualification.

The Shannon entropy has utilities across a wide spectrum of scientific investigations (see [18]). However, it is not finitely defined for all distributions in $\mathcal{P}$. A family of the generalized Shannon entropies, for any ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$, is proposed as follows:

$$H_m\left({\mathbf{p}}_{\downarrow }\right)=-\sum _{k\ge 1}\left(\frac{p_{\left(k\right)}^m}{{\sum }_{j\ge 1}{p}_{\left(j\right)}^m}\right)ln\left(\frac{p_{\left(k\right)}^m}{{\sum }_{j\ge 1}{p}_{\left(j\right)}^m}\right)$$

(24)

in [19], where $m\ge 1$ is an integer. The Shannon entropy is a special family member corresponding to $m=1$. It may be verified that each member of the family, except the Shannon entropy, is finitely defined for all $\mathbf{p}\in \mathcal{P}$ and offers all important utilities that the Shannon entropy offers, including the fact that the mutual information derived based on each member with $m\ge 2$ is zero if and only if the two underlying random elements are independent.

Example 8.

The plug-in estimator of (24) converges almost surely for any ${\mathbf{p}}_{\downarrow }\in {\mathcal{P}}_{\downarrow }$ whenever $m\ge 2$. To see this, let it be first noted that the plug-in estimator of $-{\sum }_{k\ge 1}{p}_k^{m}ln{p}_k$ converges almost surely. This fact follows from Proposition 1 with $g\left(p\right)=p$, $h\left(p\right)=-p^{m-1}lnp$ which is uniformly bounded above on $I=[0,1]$. The claimed almost-sure convergence then follows the fact that, in the re-expression of (24) below,

$$H_m\left({\mathbf{p}}_{\downarrow }\right)=\left(\frac{1}{{\sum }_{j\ge 1}{p}_{\left(j\right)}^m}\right)\left[\left(-m\sum _{k\ge 1}{p}_{\left(k\right)}^{m}ln{p}_{\left(k\right)}\right)+\left(\sum _{k\ge 1}{p}_{\left(k\right)}^m\right)\left(ln\sum _{j\ge 1}{p}_{\left(j\right)}^m\right)\right],$$

(25)

and the fact that the plug-in estimator of each of the four series converges almost surely.

5. Conclusions and Discussion

This article introduces a perspective termed entropic statistics. One of the motivations of the perspective is to accommodate probability experiments on sample spaces which may include outcomes that are known to exist (and therefore are prescribed) and those whose existence is not known (and therefore not prescribable). Such a framework allows statistical exploration into a general population with possibly infinitely many previously unobserved and unknown outcomes, or new discoveries. The key concept to foster such a framework is the label-independence, that is, all parameters and statistics do not depend of the labels of an alphabet as long as they are distinguishable. Consequently, in this article an array of label-independent objects are defined and termed entropic objects. In particular, a general entropy, entropic parameters, entropic statistics, entropic sample spaces, entropic probability distributions, and an entropic-moment-generating function are defined.

Based on the defined entropic objects, two basic theorems are established. Theorem 1 provides a characterization of the entropic probability distribution on the alphabet via the entropic-moment-generating function, and Theorem 2 establishes the almost-sure convergence of the entropic statistics to the entropic parameters and, hence, provides a foundational support to the entropic framework.

On the other hand, this article merely provides a few basic results in entropic statistics. On a broader spectrum, many other issues may be fruitfully considered on at least three fronts, namely, fundamental, probabilistic, and statistical. To begin with, the fundamental question of what constitutes entropy may be explored in many directions. One of the most cited sets of axioms is that discussed by Khinchin [20], under which the Shannon entropy is proved to be unique. However under slightly less restrictive axioms, many other entropies exist and enjoy almost all the desirable utilities of the Shannon entropy; for example, see [19]. The existing literature on generalization of entropy is extensive in physics and information theory; for example, see [21,22]. The collective effort to better understand what entropy is and how it may help to describe an underlying random system is ongoing. Further research in understanding generalized entropies and their implications could greatly enrich the framework of entropic statistics.

Entropy in general is often thought of as summary of a profile state, however measured numerically, of inner energy or chaos within a random system. As such, it is independent of any labeling systems, regardless of whether the state is observable or not. A key conceptual shift introduced in this article is from statistical inference on $\mathbf{p}$ (or a function of $\mathbf{p}$) based on the multinomial frequencies $\mathbf{Y}$ to that on ${\mathbf{p}}_{\downarrow }$ (or a function of ${\mathbf{p}}_{\downarrow }$) based on the entropic frequencies ${\mathbf{Y}}_{\downarrow }$. Such a framework shift, by necessity or by choice, triggers a long array of basic probability and statistics questions, under different degrees of model restriction, ranging from parametric forms of $p_k=p(k,\mathit{\theta })$ for some parameter $\mathit{\theta }$ to the nonparametric form, $\{p_{\left(k\right)};k\ge 1\}$. It may be interesting to note that even for the nonparametric form, there are several qualitatively different forms, that of a known $K={\sum }_{k\ge 1}{1}_{[p_{\left(k\right)}>0]}<\infty$, that of an unknown $K={\sum }_{k\ge 1}{1}_{[p_{\left(k\right)}>0]}<\infty$, and that of $K={\sum }_{k\ge 1}{1}_{[p_{\left(k\right)}>0]}=\infty$. Each of these model classes could imply a very different stochastic behavior of ${\mathbf{Y}}_{\downarrow }$ as the sample size n increases. Even long before the notion of information entropy was coined by Shannon in [1], the behavior of ${\mathbf{Y}}_{\downarrow }$ had been discussed in the literature by, for example, Auerbach [23] and Zipf [24]. More recently, several articles [25,26] discussed domains of attraction in the total collection of all distributions on a countable alphabet by a tail index, ${\tau }_n=n{\sum }_{k\ge 1}{p}_{\left(k\right)}{(1-p_{\left(k\right)})}^n$. Each domain characterizes the decay rate of the tail of the underlying entropic distribution and, in turn, dictates the rates of convergence of various statistical estimators of various entropies. Further advances on that front would enhance the understanding of probabilistic behavior of the entropic statistics and, in turn, the estimated entropies of interest.

In terms of statistical estimation, a large proportion of the existing literature mainly focuses on the Shannon entropy and variations of the plug-in estimators under various conditions, most of which are described and referenced in [14]. There are also non-plug-in estimators of different types, for example, the Bayes estimators [27,28,29], the hierarchical Bayes estimators [30], the James–Stein estimators [31], the coverage-adjusted estimators [32,33,34], and an unbiased estimator based on sequential data proposed by Montgomery-Smith and Schürmann. In general, the asymptotic distributions of the plug-in estimators and their variants seem to have been studied and described to some extent; for example, see [12,35,36,37,38]. However, it is fair to say that many, if not most, of the proposed estimators of various types have not yet been assigned asymptotic distributions. Any advances in that direction could much benefit applications of these estimators.

In short, the landscape of entropic statistics is quite porous in comparison to that of richly supported classical statistics. Many basic and important questions are yet to be answered, from the axiomatic foundation, to the definitions of basic elements, to the theoretical supporting architecture, and to the relevance in applications. However, the same said porosity also offers opportunities for interesting contemplation.