Khinchin’s Fourth Axiom of Entropy Revisited

Abstract

The Boltzmann–Gibbs–Shannon (BGS) entropy is the only entropy form satisfying four conditions known as Khinchin’s axioms. The uniqueness theorem of the BGS entropy, plus the fact that Shannon’s mutual information completely characterizes independence between the two underlying random elements, puts the BGS entropy in a special place in many fields of study. In this article, the fourth axiom is replaced by a slightly weakened condition: an entropy whose associated mutual information is zero if and only if the two underlying random elements are independent. Under the weaker fourth axiom, other forms of entropy are sought by way of escort transformations. Two main results are reported in this article. First, there are many entropies other than the BGS entropy satisfying the weaker condition, yet retaining all the desirable utilities of the BGS entropy. Second, by way of escort transformations, the newly identified entropies are the only ones satisfying the weaker axioms.

1. Introduction and Summary

Let X be a random element in a countable alphabet $\mathcal{X}=\{x_k;k\ge 1\}$, where $x_k$, $k\ge 1$ are distinct letters or labels with a probability distribution $\mathbf{p}=\{p_k;k\ge 1\}\in \mathcal{P}$, where $\mathcal{P}$ is the collection of all possible probability distributions on $\mathcal{X}$. Many random system properties of interest may be described by entropic quantities or entropies, that is, functions of $\mathbf{p}$ that are label-independent. For generality of discussion, letting ${\mathbf{p}}_{\downarrow }=\{p_{\left(k\right)};k\ge 1\}$ be a non-increasingly ordered $\mathbf{p}$, a function of $\mathbf{p}$ that satisfies $H=H\left(\mathbf{p}\right)=H\left({\mathbf{p}}_{\downarrow }\right)$ for all $\mathbf{p}\in \mathcal{P}$ is referred to as an entropy. An entropy may also be denoted as $H\left(X\right)$ for notation simplicity. For example, the Boltzmann–Gibbs–Shannon entropy $H_{\mathrm{BGS}}=-{\sum }_{k\ge 1}{p}_{k}ln{p}_k$, the Rényi entropy $H_{\mathrm{R}}={(1-\alpha )}^{-1}ln\left({\sum }_{k\ge 1}{p}_k^{\alpha }\right)$ for $\alpha >0$ and $\alpha \ne 1$ is a constant, and the Tsallis entropy $H_{\mathrm{T}}={(\alpha -1)}^{-1}(1-{\sum }_{k\ge 1}{p}_k^{\alpha })$ for any $\alpha \ne 0$ are of high utility across many fields of study, such as statistical mechanics and information theory.

Ever since the information-theoretic utility of $H_{\mathrm{BGS}}$ was unlocked in [1], a large volume of research has been amassed in relation to $H_{\mathrm{BGS}}$. A considerable amount of the research effort has been placed in framing $H_{\mathrm{BGS}}$ in general forms. Many articles have been published under the theme of generalized entropy, but from different perspectives. One particular perspective is the axiomatic system studied by Khinchin in [2].

Let $(X,Y)$ be a pair of random elements on a joint countable alphabet, $\mathcal{X}\times \mathcal{Y}=\{(x_i,y_j);i\ge 1,j\ge 1\}$, where $(x_i,y_j)$ for $i\ge 1$ and $j\ge 1$ are distinct labels with a joint probability distribution, ${\mathbf{p}}_{\mathrm{X},\mathrm{Y}}=\{p_{i,j};i\ge 1,j\ge 1\}$, and the two marginal distributions are ${\mathbf{p}}_{\mathrm{X}}=\{p_{i,·};i\ge 1\}$ and ${\mathbf{p}}_{\mathrm{Y}}=\{p_{·,j};j\ge 1\}$, where $p_{i,·}={\sum }_{j\ge 1}{p}_{i,j}$ and $p_{·,j}={\sum }_{i\ge 1}{p}_{i,j}$ for X and Y, respectively. Let $H\left(\mathbf{p}\right)$ be an entropy. The four axioms of Khinchin are given below.

$K_1$:

(Continuity) $H\left(\mathbf{p}\right)$ is continuous with respect to all elements of $\mathbf{p}$.

$K_2$:

(Maximality) Given $K={\sum }_{k\ge 1}{1}_{[p_k>0]}<\infty$, $H\left(\mathbf{p}\right)$ is maximized in $\mathcal{P}$ at $p_k=1/K$ for $k=1,\cdots ,K$.

$K_3$:

(Expansibility) Letting ${\mathbf{p}}_0=\{0,p_1,p_2,\cdots \}$, $H\left(\mathbf{p}\right)=H\left({\mathbf{p}}_{\mathbf{0}}\right)$.

$K_4$:

(Separability) For any pair of random elements $(X,Y)$ on $\mathcal{X}\times \mathcal{Y}$ with a joint probability distribution ${\mathbf{p}}_{\mathrm{X},\mathrm{Y}}$,

$$H(X,Y)=H\left(X\right)+H\left(Y\right|X)$$

(1)

where $H\left(Y\right|X)={\sum }_{i\ge 1}{p}_{i,·}H\left(Y\right|X=x_i)$ and $H\left(Y\right|X=x_i)$ is the entropy of the conditional distribution of Y given $X=x_i$.

$K_4$ is sometimes also known as Strong Additivity.

Khinchin famously proved the following fact in [2].

Fact 1

(The uniqueness theorem of entropy). For any $\mathbf{p}\in$$\mathcal{P}$, if an entropy $H\left(\mathbf{p}\right)$, such that $H\left(\mathbf{p}\right)<\infty$, satisfies all four axioms, $K_1$–$K_4$, then $H\left(\mathbf{p}\right)$ must be uniquely of the form $H_{BGS}=-{\sum }_{k\ge 1}{p}_{k}ln{p}_k$ up to a multiplicative constant.

Let

$$MI=MI(X,Y)=H_{\mathrm{BGS}}\left(X\right)+H_{\mathrm{BGS}}\left(Y\right)-H_{\mathrm{BGS}}(X,Y)$$

(2)

which is often referred to as Shannon’s mutual information. The following fact is due to Shannon.

Fact 2.

Let ${\mathbf{p}}_{X,Y}$ be a joint probability distribution of $(X,Y)$ satisfying $H\left(X\right)<\infty$ and $H\left(Y\right)<\infty$. Then, X and Y are independent if and only if $MI(X,Y)=0$.

The fact that the independence between X and Y may be characterized by a single-valued index $MI$ under a general joint distribution on $\mathcal{X}\times \mathcal{Y}$ puts $MI$ in a very important place in information theory. Furthermore the uniqueness theorem of entropy adds a special aura to $H_{\mathrm{BGS}}$.

However, $H_{\mathrm{BGS}}$, which satisfies

$$H_{\mathrm{BGS}}(X,Y)=H_{\mathrm{BGS}}\left(X\right)+H_{\mathrm{BGS}}\left(Y\right)$$

(3)

under independence of X and Y, is considered by many physicists to be overly rigid. In search of more general forms of entropy, Khinchin’s axiom $K_4$ is weakened in various ways, and a large number of research articles has been published under the weaker conditions. Many of these articles may be found in two excellent review articles, see [3,4].

The fourth axiom, $K_4$, may be weakened to different degrees across a spectrum. At one end of it, $K_4$ is ignored and generalized entropy forms are sought only under $\{K_1,K_2,K_3\}$. Other versions of the weakened $K_4$ are mostly given in more general forms of (3), under independence of X and Y. For example, an entropy $H\left(\mathbf{p}\right)$ may be required to satisfy

$$H(X,Y)=\Phi \left(H\right(X),H(Y\left)\right)$$

(4)

for some symmetric function of two variables, $\Phi$, of which a special case is

$$H(X,Y)=H\left(X\right)+H\left(Y\right)+(1-\alpha )H\left(X\right)H\left(Y\right)$$

(5)

where $\alpha \ne 1$ is a real number. The condition in (5) is a centerpiece of non-extensive statistical mechanics. The Tsallis entropy satisfies (4) in general and (5) in specific. It is to be particularly noted that conditions in (4) and (5) are necessary conditions of X and Y being independent, but are not sufficient conditions.

Non-extensive physics aside, the rigidity of $H_{\mathrm{BGS}}$ has its remarkable utility, namely Fact 2: a single-valued index characterizes the stochastic association between two random elements on a non-meterized joint alphabet under general distributions. (Also see standardized mutual information in Chapter 5 of [5]).

It may be interesting to ask whether there exist other entropies, $H\left(\mathbf{p}\right)$, such that, in addition to satisfying $\{K_1,K_2,K_3\}$, it also satisfies

$$K_4^{♭}:H\left(X\right)+H\left(Y\right)-H(X,Y)=0\mathrm{if} \mathrm{and} \mathrm{only} \mathrm{if} X \mathrm{and} Y \mathrm{are} \mathrm{independent}.$$

(6)

Let it be noted that $K_4^{♭}$ is weaker than $K_4$ in the sense that $\{K_1,K_2,K_3,K_4\}$ implies $\{K_1,K_2,K_3,K_4^{♭}\}$. In the same sense, $K_4^{♭}$ is a stronger condition than

$$K_4^{♭♭}:H\left(X\right)+H\left(Y\right)-H(X,Y)=0\mathrm{if} X \mathrm{and} Y \mathrm{are} \mathrm{independent},$$

(7)

since $K_4^{♭♭}$ is a necessary condition of X and Y being independent, but not a sufficient condition. The condition in (7) is a special case of (4).

Example 1.

The Rényi entropy, $H_T$, satisfies $K_4^{♭♭}$ but not $K_4^{♭}$.

Example 2.

The no-name entropy, $H_N=-{\sum }_{k\ge 1}{p}_k^{\alpha }ln{p}_k^{\alpha }$ for any $\alpha >1$, satisfies $K_4^{♭♭}$ but not $K_4^{♭}$. By the way, it may be interesting to note that ${lim}_{\alpha \to 1}{H}_N=H_{BGS}$.

As it turns out, there are many entropies satisfying $\{K_1,K_2,K_3,K_4^{♭}\}$. Consider the following family of entropies,

$$H_{\alpha }\left(\mathbf{p}\right)=-\sum _{k\ge 1}\left(\frac{p_k^{\alpha }}{{\sum }_{i\ge 1}{p}_i^{\alpha }}\right)ln\left(\frac{p_k^{\alpha }}{{\sum }_{i\ge 1}{p}_i^{\alpha }}\right),$$

(8)

for $\alpha >1$, and its implied mutual information,

$$M{I}_{\alpha }(X,Y)=H_{\alpha }\left(X\right)+H_{\alpha }\left(Y\right)-H_{\alpha }(X,Y).$$

(9)

Obviously the family of entropies in (8) also satisfies ${lim}_{\alpha \to 1}{H}_{\alpha }\left(\mathbf{p}\right)=H_{\mathrm{BGS}}$, which, however, is only finitely defined for some of the distributions in $\mathcal{P}$. A significant advantage of (8) is that every member is finitely defined for each and every $\mathbf{p}\in \mathcal{P}$. The first main result established in this article is the fact that X and Y are independent if and only if $M{I}_{\alpha }(X,Y)=0$ for any $\alpha \in (1,\infty )$, which immediately implies that there are many entropies satisfying $\{K_1,K_2,K_3,K_4^{♭}\}$.

A second question of interest is whether there exist other forms of entropy satisfying $\{K_1,K_2,K_3,K_4^{♭}\}$ beyond those in (8). The answer to this question is unknown in general. However, under certain restrictions on the functional forms of $H\left(\mathbf{p}\right)$, uniqueness of (8) may be established. This, in fact, is the second main result of this article.

In Section 2, the statements of the two main results are made more precise and are established in two separate subsections. The article ends with Section 3, where several related minor results are summarized.

2. Main Results

The path leading to both main results of this article goes through a mapping, $\mathbf{\Phi }$: $\mathcal{P}\Rightarrow {\mathcal{P}}^{*}\subseteq \mathcal{P}$, denoted ${\mathbf{p}}^{*}=\mathbf{\Phi }\left(\mathbf{p}\right)$ and referred to as the escort distribution of $\mathbf{p}$ on the same alphabet $\mathcal{X}$. ${\mathbf{p}}^{*}$ is constructed in a special way, according to the concept of escort distributions introduced in [6]. For a given function $\varphi \left(p\right)\ge 0$ on $[0,1]$ and a distribution $\mathbf{p}\in \mathcal{P}$, provided that $0<{\sum }_{i\ge 1}\varphi \left(p_i\right)=C(\mathbf{p},\varphi (·))<\infty$, ${\mathbf{p}}^{*}=\{p_k^{*};k\ge 1\}$, where

$$p_k^{*}=\frac{\varphi \left(p_k\right)}{{\sum }_{i\ge 1}\varphi \left(p_i\right)},$$

(10)

is referred to as the escort distribution of $\mathbf{p}$ associated with the deformation function (also known as the the escort function), $\varphi \left(p\right)$. The utility of escort distributions is discussed by many researchers, most notably [7] in the context of statistical mechanics, and [8] regarding information geometry. Escort distributions, originally as natural scanners of a single underpinning probability distribution, $\mathbf{p}$, in a multifractal structure, have been shown to be useful in a great variety of places and ways, ranging from information theory and coding theory to multifractal neural networks. For example, many interesting results and applications may be found in [9,10], and the references there within.

Consider a special family of power escort functions,

$$\{\varphi \left(p\right)=\lambda p^{\alpha }:\alpha >1,\lambda >0\}.$$

(11)

When $\varphi \left(p\right)$ is a member of (11), (10) becomes

$$p_k^{*}=\frac{p_k^{\alpha }}{{\sum }_{i\ge 1}{p}_i^{\alpha }}.$$

(12)

The Boltzmann–Gibbs–Shannon entropy of the escort distribution, ${\mathbf{p}}^{*}=\{p_k^{*};k\ge 1\}$, becomes $H_{\alpha }\left(\mathbf{p}\right)$ of (8), and $M{I}_{\alpha }$ of (9) becomes its implied Shannon’s mutual information.

2.1. Characterization of Independence

Theorem 1.

X and Y are independent if and only if $M{I}_{\alpha }(X,Y)=0$, for any fixed $\alpha >1$.

Let $(X,Y)$ be a pair of random elements on $\mathcal{X}\times \mathcal{Y}$, with a joint probability distribution, ${\mathbf{p}}_{\mathrm{X},\mathrm{Y}}=\{p_{i,j};i\ge 1,j\ge 1\}$, and two marginal distributions, ${\mathbf{p}}_{\mathrm{X}}$ and ${\mathbf{p}}_{\mathrm{Y}}$. For a fixed $\alpha >1$, consider another pair of random elements $(X^{*},Y^{*})$ in the same alphabet $\mathcal{X}\times \mathcal{Y}$, but with an induced joint probability distribution, ${\mathbf{p}}_{\mathrm{X},\mathrm{Y}}^{*}=\{p_{i,j}^{*};i\ge 1,j\ge 1\}$, and two marginal distributions, ${\mathbf{p}}_{\mathrm{X}}^{*}=\{p_{i,·}^{*};i\ge 1\}$ and ${\mathbf{p}}_{\mathrm{Y}}^{*}=\{p_{·,j}^{*};j\ge 1\}$, where, for some $\alpha >1$,

$$p_{i,j}^{*}=\frac{p_{i,j}^{\alpha }}{{\sum }_{s\ge 1,t\ge 1}{p}_{s,t}^{\alpha }},p_{i,·}^{*}=\frac{{\sum }_{j\ge 1}{p}_{i,j}^{\alpha }}{{\sum }_{s\ge 1,t\ge 1}{p}_{s,t}^{\alpha }},\mathrm{and}{p}_{·,j}^{*}=\frac{{\sum }_{i\ge 1}{p}_{i,j}^{\alpha }}{{\sum }_{s\ge 1,t\ge 1}{p}_{s,t}^{\alpha }}.$$

(13)

Since (8) is the Boltzmann–Gibbs–Shannon entropy of the escort distribution, ${\mathbf{p}}^{*}=\{p_k^{*};k\ge 1\}$, that is, $H_{\alpha }\left(X\right)=H_{\mathrm{BGS}}\left(X^{*}\right)$, and $M{I}_{\alpha }(X,Y)=MI(X^{*},Y^{*})$, by Fact 2, Theorem 1 is an immediate consequence of the following lemma.

Lemma 1.

X and Y are independent if and only if $X^{*}$ and $Y^{*}$ are independent.

Proof. 

For each $\alpha >1$, if $p_{i,j}=p_{i,·}\times p_{·,j}$ for all pairs $(i,j)$, $i\ge 1$ and $j\ge 1$, then

$$p_{i,j}^{*}=\frac{p_{i,j}^{\alpha }}{{\sum }_{s\ge 1,t\ge 1}{p}_{s,t}^{\alpha }}=\left(\frac{p_{i,·}^{\alpha }}{{\sum }_{s\ge 1}{p}_{s,·}^{\alpha }}\right)\times \left(\frac{p_{·,j}^{\alpha }}{{\sum }_{t\ge 1}{p}_{·,t}^{\alpha }}\right)=p_{i,·}^{*}\times p_{·,j}^{*}.$$

(14)

Therefore, the fact that $H_{\alpha }(X,Y)$ is the Boltzmann–Gibbs–Shannon entropy of $(X^{*},Y^{*})$ implies $M{I}_{\alpha }(X,Y)=0$.

Conversely, if $M{I}_{\alpha }(X,Y)=0$, then (14) holds, which implies for each pair $(i,j)$, $i\ge 1$ and $j\ge 1$,

$$\begin{array}{cc}\hfill p_{i,j}& ={\left(\frac{p_{i,·}^{\alpha }}{{\sum }_{s\ge 1}{p}_{s,·}^{\alpha }}\right)}^{1/\alpha }\times {\left(\frac{p_{·,j}^{\alpha }}{{\sum }_{t\ge 1}{p}_{·,t}^{\alpha }}\right)}^{1/\alpha }\times {\left(\sum _{s\ge 1,t\ge 1}{p}_{s,t}^{\alpha }\right)}^{1/\alpha }\hfill \\ \hfill & =p_{i,·}\times p_{·,j}\times {\left(\frac{{\sum }_{s\ge 1,t\ge 1}{p}_{s,t}^{\alpha }}{{\sum }_{s\ge 1}{p}_{s,·}^{\alpha }\times {\sum }_{t\ge 1}{p}_{·,t}^{\alpha }}\right)}^{1/\alpha }.\hfill \end{array}$$

Noting that the third factor in the expression above does not depend on i or j, the lemma immediately follows the factorization theorem.    □

Remark 1.

It is acknowledged that the proof of Lemma 1 above is inspired by a similar proof in [11], where a similar result with a more restrictive family of escort functions is established.

2.2. A Uniqueness Theorem

In Theorem 1, it is established that X and Y are independent if and only if $X^{*}$ and $Y^{*}$ are independent, where $(X,Y)$ and $(X^{*},Y^{*})$ are linked by a power escort transformation, $\varphi \left(p\right)$, through the mapping, $\mathbf{\Phi }$, between their respective joint distributions. Such an escort function may be reasonably referred to as an independence-dependence preserving deformation function. In concept, however, there may exist other such functions outside of the power family in (11). However, Theorem 2 below says otherwise.

Definition 1.

A deformation function $\varphi \left(p\right)$ on $[0,1]$ is said to be independence-dependence preserving with respect to a subclass ${\mathcal{P}}_{X,Y}^{\prime }\subseteq {\mathcal{P}}_{X,Y}$, if for each and every ${\mathbf{p}}_{X,Y}\in {\mathcal{P}}_{X,Y}^{\prime }$ and its associated escort distribution ${\mathbf{p}}_{X,Y}^{*}$, X and Y are independent if and only if $X^{*}$ and $Y^{*}$ are independent. More specifically, a deformation function $\varphi \left(p\right)$ on $[0,1]$ is said to be independence-dependence preserving if it is independence-dependence preserving with respect to ${\mathcal{P}}_{X,Y}$.

Theorem 2.

A measurable and integrable deformation function, $\varphi \left(p\right)$ for $p\in (0,1)$, is independence preserving if and only if it is a member in the family of power functions in (11).

Lemma 2.

Suppose $f\left(x\right)>0$ is a Lebesgue measurable function on $\mathbb{R}$ such that $f(x+y)=f\left(x\right)f\left(y\right)$ for all $x,y\in \mathbb{R}$. Then, $f\left(x\right)=e^{\alpha x}$ for all $x\in \mathbb{R}$ and some constant $\alpha \in \mathbb{R}$.

Lemma 2 above is due to [12]. In fact, it is also established in [12] that if the condition of Lebesgue measurability is not imposed, a nowhere-continuous $f\left(x\right)$, satisfying $f(x+y)=f\left(x\right)f\left(y\right)$ for all $x,y\in \mathbb{R}$, exists and is therefore not of the form $f\left(x\right)=e^{\alpha x}$.

Lemma 3.

Suppose $g\left(x\right)>0$ is a Lebesgue measurable function on $(0,\infty )$ such that $g\left(xy\right)=g\left(x\right)g\left(y\right)$ for all $x,y\in (0,\infty )$. Then, $g\left(x\right)=x^{\alpha }$ for all $x>0$ and some constant $\alpha \in \mathbb{R}$.

Proof. 

For any $x,y\in (0,\infty )$, consider the variable transformation $s=lnx$ and $t=lny$, and hence $x=e^s$ and $y=e^t$. Let $f\left(s\right)=g\left(e^s\right)$. It follows that, for all $x,y\in (0,\infty )$,

$$f(s+t)=g\left(e^{s+t}\right)=g\left(e^s{e}^t\right)=g\left(e^s\right)g\left(e^t\right)=f\left(s\right)f\left(t\right).$$

Since $g\left(x\right)$ is measurable, $f\left(s\right)$ is measurable, and therefore, by Lemma 2, $f\left(s\right)=e^{\alpha s}$ for some constant $\alpha >0$, or equivalently $g\left(x\right)=x^{\alpha }$.    □

Lemma 4.

If $\varphi \left(p\right)$ is a member of (11), then $\varphi \left(p\right)$ is independence-dependence preserving.

Proof. 

Let $\{p_{i,j};i\ge 1,j\ge 1\}$ be any joint distribution on $\mathcal{X}\times \mathcal{Y}$ and denote its marginal distributions as $\{p_{i,·};i\ge 1\}$ and $\{p_{·,j};j\ge 1\}$. Suppose $p_{i,j}=p_{i,·}{p}_{·,j}$ for every pair $(i,j)$. It follows that

$$\frac{\varphi \left(p_{i,j}\right)}{{\sum }_{s\ge 1,t\ge 1}\varphi \left(p_{s,t}\right)}=\frac{{\lambda }^2{p}_{i,·}^{\alpha }{p}_{·,j}^{\alpha }}{{\lambda }^2{\sum }_{s\ge 1,t\ge 1}{p}_{s,·}^{\alpha }{p}_{·,t}^{\alpha }}=\frac{\varphi \left(p_{i,·}\right)}{{\sum }_{s\ge 1}\varphi \left(p_{s,·}\right)}\times \frac{\varphi \left(p_{·,j}\right)}{{\sum }_{t\ge 1}\varphi \left(p_{·,t}\right)}.$$

(15)

Conversely, suppose

$$\frac{\varphi \left(p_{i,j}\right)}{{\sum }_{s\ge 1,t\ge 1}\varphi \left(p_{s,t}\right)}=\frac{\varphi \left(p_{i,·}\right)}{{\sum }_{s\ge 1}\varphi \left(p_{s,·}\right)}\times \frac{\varphi \left(p_{·,j}\right)}{{\sum }_{t\ge 1}\varphi \left(p_{·,t}\right)}$$

holds for every pair $(i,j)$. Noting $\lambda {\sum }_{s\ge 1,t\ge 1}\varphi \left(p_{s,t}\right)={\sum }_{s\ge 1}\varphi \left(p_{s,·}\right)\times {\sum }_{t\ge 1}\varphi \left(p_{·,t}\right)$, it follows that $\lambda \varphi \left(p_{i,j}\right)=\varphi \left(p_{i,·}\right)\varphi \left(p_{·,j}\right)$, ${\lambda }^2{p}_{i,j}^{\alpha }=\lambda p_{i,·}^{\alpha }\lambda p_{·,j}^{\alpha }$, and $p_{i,j}=p_{i,·}{p}_{·,j}$ for every pair $(i,j)$.    □

Lemma 5.

If $\varphi \left(p\right)$ is Lebesgue measurable and integrable on $(0,1)$ and is independence-dependence preserving on ${\mathcal{P}}_{X,Y}$, then $\varphi \left(p\right)$ is a member of (11).

Proof. 

Suppose X and Y are independent, that is $p_{i,j}=p_{i,·}\times p_{·,j}$ for all $\{i,j\}$. For $\varphi (·)$ to preserve independence, it is to satisfy, for all $\{i,j\}$,

$$p_{i,j}^{*}=p_{i,·}^{*}\times p_{·,j}^{*},$$

(16)

or to satisfy, for any two pairs of indices $(k,l)$ and $(s,t)$,

$$\frac{\varphi \left(p_{k,l}\right)}{\varphi \left(p_{s,t}\right)}=\frac{{\sum }_j\varphi \left(p_{k,j}\right)\times {\sum }_i\varphi \left(p_{i,l}\right)}{{\sum }_j\varphi \left(p_{s,j}\right)\times {\sum }_i\varphi \left(p_{i,t}\right)}.$$

(17)

More specifically let $k=s$, then Equation (17) is reduced to

$$\frac{\varphi \left(p_{k,l}\right)}{\varphi \left(p_{k,t}\right)}=\frac{{\sum }_i\varphi \left(p_{i,l}\right)}{{\sum }_i\varphi \left(p_{i,t}\right)}.$$

(18)

Noting $p_{i,j}=p_{i,·}\times p_{·,j}$ for all pairs of $(i,j)$, it follows from (18) that

$$\frac{\varphi (p_{k,·}\times p_{·,l})}{\varphi (p_{k,·}\times p_{·,t})}=\frac{{\sum }_i\varphi (p_{i,·}\times p_{·,l})}{{\sum }_i\varphi (p_{i,·}\times p_{·,t})}.$$

(19)

The right-hand side of (19) is independent of k, and so is the left-hand side of (19). It follows that

$$\frac{\varphi (p_{k,·}\times q_{·,l})}{\varphi (p_{k,·}\times q_{·,t})}=\frac{\varphi (p_{s,·}\times q_{·,l})}{\varphi (p_{s,·}\times q_{·,t})}$$

(20)

regardless if $k=s$ or $k\ne s$. It is to be noted that (20) implies

$$\frac{\varphi (p·q)}{\varphi (p·q^{\prime })}=\frac{\varphi (p^{\prime }·q)}{\varphi (p^{\prime }·q^{\prime })}$$

(21)

for any $p\in (0,1)$, $p^{\prime }\in (0,1)$, $q\in (0,1)$, and $q^{\prime }\in (0,1)$, but subject to the constraints $p+p^{\prime }\le 1$ and $q+q^{\prime }\le 1$.

Now, it is desired to show that, without the constraints $p+p^{\prime }\le 1$ and $q+q^{\prime }\le 1$, (21) still holds. Toward that end, the proof is given in two steps, (1) $q+q^{\prime }\le 1$ and (2) $q+q^{\prime }>1$.

Step 1: Suppose $q+q^{\prime }\le 1$. Let

$$\begin{array}{cc}\hfill P_1=& \left\{p,\frac{1}{2}(1-max(p,p^{\prime })),\cdots \right\},\hfill \\ \hfill P_2=& \left\{p^{\prime },\frac{1}{2}(1-max(p,p^{\prime })),\cdots \right\},\hfill \\ \hfill Q=& \left\{q,q^{\prime },\cdots \right\}.\hfill \end{array}$$

Applying $P_1$ and Q to Equation (20), it follows that

$$\frac{\varphi (p·q)}{\varphi (p·q^{\prime })}=\frac{\varphi (\frac{1}{2}(1-max(p,p^{\prime }))·q)}{\varphi (\frac{1}{2}(1-max(p,p^{\prime }))·q^{\prime })}$$

(22)

Applying $P_2$ and Q to Equation (20), it follows that

$$\frac{\varphi (p^{\prime }·q)}{\varphi (p^{\prime }·q^{\prime })}=\frac{\varphi (\frac{1}{2}(1-max(p,p^{\prime }))·q)}{\varphi (\frac{1}{2}(1-max(p,p^{\prime }))·q^{\prime })}$$

(23)

and that

$$\frac{\varphi (p·q)}{\varphi (p·q^{\prime })}=\frac{\varphi (p^{\prime }·q)}{\varphi (p^{\prime }·q^{\prime })}.$$

(24)

Before moving on to Step 2, let it be noted that the constraint $p+p^{\prime }\le 1$ is not used in the above proof of Step 1. That is to say that (24) holds under the condition $q+q^{\prime }\le 1$ regardless of whether $p+p^{\prime }\le 1$ or $p+p^{\prime }>1$.

Step 2: Suppose $q+q^{\prime }>1$. Let $q^{″}=(1-max(q,q^{\prime })).$ Noting $q+q^{″}\le 1$, evaluating (24) with $q^{″}$ in place of $q^{\prime }$ gives

$$\frac{\varphi (p·q)}{\varphi (p·\frac{1}{2}(1-max(q,q^{\prime })))}=\frac{\varphi (p^{\prime }·q)}{\varphi (p^{\prime }·\frac{1}{2}(1-max(q,q^{\prime })))}.$$

(25)

Noting $q^{\prime }+q^{″}\le 1$, evaluating (24) with $q^{\prime }$ in place of q gives

$$\frac{\varphi (p·q^{\prime })}{\varphi (p·\frac{1}{2}(1-max(q,q^{\prime })))}=\frac{\varphi (p^{\prime }·q^{\prime })}{\varphi (p^{\prime }·\frac{1}{2}(1-max(q,q^{\prime })))}.$$

(26)

Combining (25) and (26) again gives (24), but for all $p,p^{\prime },q,q^{\prime }\in (0,1)$ without any constraints.

Moreover, (24) may be further simplified. Let $ϵ$ be such that $0<ϵ<min\left(1/q,1/q^{\prime }\right)$. Then, for any $p\in (0,1)$, $q\in (0,1)$, $q^{\prime }\in (0,1)$,

$$\frac{\varphi (p·q)}{\varphi (p·q^{\prime })}=\frac{\varphi \left(q\right)}{\varphi \left(q^{\prime }\right)}.$$

(27)

This is so because

$$\frac{\varphi (p·q)}{\varphi (p·q^{\prime })}=\frac{\varphi \left(\frac{p}{1+ϵ}·(1+ϵ)q\right)}{\varphi \left(\frac{p}{1+ϵ}·(1+ϵ)q^{\prime }\right)}=\frac{\varphi \left(\frac{1}{1+ϵ}·(1+ϵ)q\right)}{\varphi \left(\frac{1}{1+ϵ}·(1+ϵ)q^{\prime }\right)}=\frac{\varphi \left(q\right)}{\varphi \left(q^{\prime }\right)}.$$

Next, it is to establish the fact that $\varphi \left(p\right)$ is continuous on $(0,1)$. Since $\varphi (·)$ is measurable and integrable on $(0,1)$ by assumption, substituting $u=xv$, $\Phi \left(x\right)={\int }_0^x\varphi \left(u\right)du=x{\int }_0^1\varphi \left(xv\right)dv.$ By (27), for a fixed $s\in (0,1)$,

$$\frac{\varphi \left(xv\right)}{\varphi \left(sv\right)}=\frac{\varphi \left(x\right)}{\varphi \left(s\right)}\mathrm{or}\varphi \left(xv\right)=\frac{\varphi \left(x\right)\varphi \left(sv\right)}{\varphi \left(s\right)}.$$

It follows that

$$\Phi \left(x\right)=x\varphi \left(x\right){\int }_0^1\frac{\varphi \left(sv\right)}{\varphi \left(s\right)}dv.$$

Since $\Phi$ is continuous in x, and $x\int \frac{\varphi \left(sv\right)}{\varphi \left(s\right)}dv$ is a linear function of x, it follows that $\varphi \left(x\right)$ is continuous in x on $(0,1)$. Denoting $\varphi \left(1\right)={lim}_{x\to 1}\varphi \left(x\right)$, by (27),

$$\varphi (p·q)=\frac{\varphi \left(p\right)·\varphi \left(q\right)}{\varphi \left(1\right)},$$

(28)

which implies that $\varphi \left(p\right)$ is a member of (11).    □

Lemmas 4 and 5 immediately give Theorem 2.

3. Other Results

An independence-dependence preserving deformation function, $\varphi \left(p\right)$, with respect to a subspace, ${\mathcal{P}}_{\mathrm{X},\mathrm{Y}}^{\prime }\subset {\mathcal{P}}_{\mathrm{X},\mathrm{Y}}$, needs not necessarily to be of the power form in (11). Let ${\mathcal{P}}_{K\times K}$ be the collection of all distributions of $(X,Y)$ on a $K\times K$ joint alphabet, where $K\ge 2$ is a finite integer.

Proposition 1.

A Lebesgue measurable and integrable deformation function $\varphi \left(p\right)$ is independence-dependence preserving on a subspace ${\mathcal{P}}_{X,Y}^{\prime }$ such that ${\mathcal{P}}_{3\times 3}\subseteq {\mathcal{P}}_{X,Y}^{\prime }\subseteq {\mathcal{P}}_{X,Y}$, if and only if $\varphi \left(p\right)$ is a member of power functions in (11).

The proof of Proposition 1 is trivial because the the proof of Lemma 5 directly applies in this case, noting that the existence of the constructed distributions, $P_1,P_2,Q$, in the proof of Lemma 5 only requires the joint alphabet to be $3\times 3$ or larger.

It may be interesting to note that Proposition 1 is a stronger statement than Theorem 2 in the sense that Theorem 2 may be considered a corollary of Proposition 1. This is due to the fact that, for two sub-classes of distributions ${\mathcal{P}}_{X,Y}^{\prime }\subseteq {\mathcal{P}}_{X,Y}^{″}$, an independence-dependence preserving $\varphi \left(p\right)$ with respect to ${\mathcal{P}}_{X,Y}^{″}$ is independence-dependence preserving with respect to ${\mathcal{P}}_{X,Y}^{\prime }$.

Example 3 below illustrates the fact that on some restricted class of probability distributions an independence-dependence preserving deformation function, $\varphi \left(p\right)$, needs not to be of the power form of (11).

Example 3.

Consider the collection of all distributions on a $2\times 2$ joint alphabet, denoted ${\mathcal{P}}_{2\times 2}$. The function,

$$\varphi \left(x\right)=exp(-(x-1/4)+2{(x-1/4)}^2)),$$

(29)

is independence-dependence preserving but is not in the form of a power function. To see this, let it be first noted what being independence-dependence preserving entails on a $2\times 2$ alphabet. Write the joint distribution as $\{p_{i,j};i=1,2,j=1,2\}$ and the two marginal distributions as $\{p_{1,·},p_{2,·}\}=\{p,1-p\}$ and $\{p_{·,1},p_{·,2}\}=\{q,1-q\}$. When two underlying random elements X and Y on the $2\times 2$ alphabet are independent, a qualified deformation function $\varphi \left(x\right)$ must satisfy

$$\frac{\varphi \left(p_{1,1}\right)}{{\sum }_{s=1,2;t=1,2}\varphi \left(p_{s,t}\right)}=\frac{\varphi \left(p_{1,1}\right)+\varphi \left(p_{1,2}\right)}{{\sum }_{s=1,2;t=1,2}\varphi \left(p_{s,t}\right)}\times \frac{\varphi \left(p_{1,1}\right)+\varphi \left(p_{2,1}\right)}{{\sum }_{s=1,2;t=1,2}\varphi \left(p_{s,t}\right)},$$

or, letting $\sum =\varphi \left(pq\right)+\varphi \left(p\right(1-q\left)\right)+\varphi \left(q\right(1-p\left)\right)+\varphi \left(\right(1-p\left)\right(1-q\left)\right)$,

$$\frac{\varphi \left(pq\right)}{\sum }=\frac{\varphi \left(pq\right)+\varphi \left(p\right(1-q\left)\right)}{\sum }\times \frac{\varphi \left(pq\right)+\varphi \left(q\right(1-p\left)\right)}{\sum },$$

which, after a few algebraic steps, is reduced to

$$\varphi \left(pq\right)\times \varphi \left(\right(1-p\left)\right(1-q\left)\right)=\varphi \left(p\right(1-q\left)\right)\varphi \left(q\right(1-p\left)\right).$$

(30)

It is easily verified that (29) satisfies (30).

The independence-dependence preserving property of a chosen $\varphi \left(p\right)$ may be important when dependence between two random elements is of importance in a study. However, when only a one-to-one escort deformation mapping $\mathbf{\Phi }$ is desired, the deformation function $\varphi \left(p\right)$ needs not to be a member of (11). The following proposition provides a sufficient condition.

Proposition 2.

If $\varphi \left(p\right)$ is strictly increasing for $p\in [0,1]$, then the ϕ-induced mapping Φ:$\mathcal{P}\to {\mathcal{P}}^{*}$is injective.

Proof. 

For every strictly increasing $\varphi \left(p\right)$ on $p\in [0,1]$ for which ${\mathbf{p}}^{*}$ is well-defined for each and every $\mathbf{p}\in \mathcal{P}$, it suffices to show that, given any ${\mathbf{p}}^{*}\in {\mathcal{P}}^{*}$, ${\mathbf{\Phi }}^{-1}\left({\mathbf{p}}^{*}\right)$ is unique. Toward that end, suppose there are two distinct distributions, ${\mathbf{p}}_1=\{p_{1,k};k\ge 1\}\in \mathcal{P}$ and ${\mathbf{p}}_2=\{p_{2,k};k\ge 1\}\in \mathcal{P}$, such that ${\mathbf{p}}_1^{*}={\mathbf{p}}_2^{*}={\mathbf{p}}^{*}$. It follows that there exists an index $k^{\prime }$ such that $p_{1,k^{\prime }}\ne p_{2,k^{\prime }}$. Without loss of generality, let it be supposed that

$$p_{1,k^{\prime }}<p_{2,k^{\prime }}.$$

(31)

Then, it follows further that

$$\frac{\varphi \left(p_{1,k^{\prime }}\right)}{{\sum }_{i\ge 1}\varphi \left(p_{1,i}\right)}=\frac{\varphi \left(p_{2,k^{\prime }}\right)}{{\sum }_{i\ge 1}\varphi \left(p_{2,i}\right)}=p_{k^{\prime }}^{*},\mathrm{and} \mathrm{hence}\varphi \left(p_{1,k^{\prime }}\right)=\varphi \left(p_{2,k^{\prime }}\right)\frac{{\sum }_{i\ge 1}\varphi \left(p_{1,i}\right)}{{\sum }_{i\ge 1}\varphi \left(p_{2,i}\right)}.$$

By (31) and the condition that $\varphi \left(p\right)$ is strictly increasing on $[0,1]$, it follows that $\varphi \left(p_{1,k^{\prime }}\right)<\varphi \left(p_{2,k^{\prime }}\right)$, and hence

$$r=\frac{{\sum }_{i\ge 1}\varphi \left(p_{1,i}\right)}{{\sum }_{i\ge 1}\varphi \left(p_{2,i}\right)}<1.$$

(32)

However, since $\varphi \left(p_{1,k}\right)=\varphi \left(p_{2,k}\right)r$ for every $k\ge 1$, it follows that $\varphi \left(p_{1,k}\right)<\varphi \left(p_{2,k}\right)$ for every $k\ge 1$. Since $\varphi \left(p\right)$ is strictly increasing, it follows that $p_{1,k}<p_{2,k}$ for every $k\ge 1$, which contradicts the supposition that both ${\mathbf{p}}_1$ and ${\mathbf{p}}_2$ are probability distributions, that is, ${\sum }_{i\ge 1}{p}_{1,i}={\sum }_{i\ge 1}{p}_{2,i}=1$. The said contradiction implies that ${\mathbf{p}}_1={\mathbf{p}}_2$, and the proposition follows.    □

Example 4.

Consider a special family of $\varphi \left(p\right)$, $\varphi \left(p\right)={(1+\lambda p)}^{1/\lambda }$ for $p\in [0,1]$, where $\lambda >0$ is a parameter. The escort distribution based on this deformation function is one of several well-studied families of escort distributions known as the Tsallis distributions. Every such $\varphi \left(p\right)$ is strictly increasing on $[0,1]$ and therefore, by Proposition 2, every member of the family induces an injective mapping. However, by Theorem 2, the induced mapping is not independence-dependence preserving on a countable joint alphabet $\mathcal{X}\times \mathcal{Y}$.

4. Concluding Remarks

Two main results are established in this article. First, there exist many entropies other than the BGS entropy satisfying the weaker axiomatic conditions, more specifically $\{K_1,K_2,K_3,K_4^{♭}\}$ rather than $\{K_1,K_2,K_3,K_4\}$, yet retaining the key utility that the associated mutual information preserves independence-dependence on a countable joint alphabet as the BGS entropy does. Second, by way of escort transformations, the newly identified entropies are the only ones satisfying the weaker axioms on a general countable joint alphabet.

The significance of the established results come into better focus in a broader perspective. Inspired by the development in modern data science, a shift is increasingly visible in the foundation of statistical inference, away from a real space, where random variables reside, toward a non-meterized and non-ordinal alphabet, where more general random elements reside. While statistical inferences based on random variables are theoretically well supported in the rich literature of probability and statistics, inferences on alphabets, mostly by way of various entropies and their estimation, are less systematically supported in theory. Without the familiar notions of neighborhood, real or complex moments, tails, etc., associated with random variables, probability and statistics based on random elements on alphabets need more attention to foster a clearer framework for the rigorous development of entropy-based statistical exercises, which may be more concisely termed entropic statistics. While a considerable volume of published work has been accumulated over several decades on entropy estimation, it is fair to say that the current research activities in the existing literature are sporadic in nature and the implied theoretical framework is porous. The said porosity permeates across the board, from basic axioms to a general definition of entropy, and from model interpretability to statistical inference, although some recent effort is observed to alleviate it. See, for example, [13] and [14], where a general definition of entropy and several fundamental results are given.

The exploration of this article is on the axiomatic foundation of entropy. If the primary study interest lies with the independence-dependence between two sets of random elements on a joint countable alphabet, as is usually the case in practice, then the main results of this article suggest that there is flexibility in choosing an entropy from (8) to serve the interest. In addition, the uniqueness of (8) by way of escort distributions is of interest not only in its own right theoretically but also as it provides support in practice. For example, in artificial neural networks with multi-layer fractal structures, the links between nodes in two layers are often modeled by power escort distributions propagating through the entire network. In such a context, it is of fundamental importance to know that the power escort distributions are the only type that would preserve dependence and independence. If the dependence-independence preserving property is desired, then the power escort is appropriate. If not, then some other escort is needed.