A Brief Review of Generalized Entropies

Abstract

Entropy appears in many contexts (thermodynamics, statistical mechanics, information theory, measure-preserving dynamical systems, topological dynamics, etc.) as a measure of different properties (energy that cannot produce work, disorder, uncertainty, randomness, complexity, etc.). In this review, we focus on the so-called generalized entropies, which from a mathematical point of view are nonnegative functions defined on probability distributions that satisfy the first three Shannon–Khinchin axioms: continuity, maximality and expansibility. While these three axioms are expected to be satisfied by all macroscopic physical systems, the fourth axiom (separability or strong additivity) is in general violated by non-ergodic systems with long range forces, this having been the main reason for exploring weaker axiomatic settings. Currently, non-additive generalized entropies are being used also to study new phenomena in complex dynamics (multifractality), quantum systems (entanglement), soft sciences, and more. Besides going through the axiomatic framework, we review the characterization of generalized entropies via two scaling exponents introduced by Hanel and Thurner. In turn, the first of these exponents is related to the diffusion scaling exponent of diffusion processes, as we also discuss. Applications are addressed as the description of the main generalized entropies advances.

1. Introduction

The concept of entropy was introduced by Clausius [1] in thermodynamics to measure the amount of energy in a system that cannot produce work, and given an atomic interpretation in the foundational works of statistical mechanics and gas dynamics by Boltzmann [2,3], Gibbs [4], and others. Since then, entropy has played a central role in many-particle physics, notoriously in the description of non-equilibrium processes through the second principle of thermodynamics and the principle of maximum entropy production [5,6]. Moreover, Shannon made of entropy the cornerstone on which he built his theory of information and communication [7]. Entropy and the associated entropic forces are also the main character in recent innovative approaches to artificial intelligence and collective behavior [8,9]. Our formalism is information-theoretic (i.e., entropic forms are functions of probability distributions) owing to the mathematical properties that we discuss along the way, but can be translated to a physical context through the concept of microstate.

The prototype of entropy that we are going to consider below is the Boltzmann–Gibbs–Shannon (BGS) entropy,

$$S_{BGS}(p_1,\dots ,p_W)=k\sum _{i=1}^W{p}_{i}ln\frac{1}{p_i}=-k\sum _{i=1}^W{p}_{i}ln{p}_i.$$

(1)

In its physical interpretation, $k=1.3807\times {10}^{-23}$ J/K is the Boltzmann constant, W is the number of microstates consistent with the macroscopic constraints of a given thermodynamical system, and $p_i$ is the probability (i.e., the asymptotic fraction of time) that the system is in the microstate i. In information theory, k is set equal to 1 for mathematical convenience, as we do hereafter, and $S_{BGS}$ measures the average information conveyed by the outcomes of a random variable with probability distribution $\{p_1,\dots ,p_W\}$. We use natural logarithms unless otherwise stated, although logarithms to base 2 is the natural choice in binary communications (the difference being the units, nats or bits, respectively). Remarkably enough, Shannon proved in Appendix B of his seminal paper [7] that Equation (1) follows necessarily from three properties or axioms (actually, four are needed; more on this below).

BGS entropy was later on generalized by other “entropy-like” quantities in dynamical systems (Kolmogorov–Sinai entropy [10], etc.), information theory (Rényi entropy [11], etc.), and statistical physics (Tsallis entropy [12], etc.), to mention the most familiar ones (see, e.g., [13] for an account of some entropy-like quantities and their applications, especially in time series analysis). Similar to with $S_{BGS}$, the essence of these new entropic forms was distilled into a small number of properties that allow sorting them out in a more systematic way [13,14]. Currently, the uniqueness of $S_{BGS}$ is derived from the four Khinchin–Shannon axioms (Section 2). However, the fourth axiom, called the separability or strong additivity axiom (which implies additivity, i.e., $S(A_1+A_2)=S(A_1)+S(A_2)$, where $A_1+A_2$ stands for a system composed of any two probabilistically independent subsystems $A_1$ and $A_2$), is violated by physical systems with long-range interactions [15,16]. This poses the question of what mathematical properties have the “generalized entropies” satisfying only the other three axioms. These are the primary candidates for extensive entropic forms, i.e., functions S such that $S(B_1\cup B_2)=S(B_1)+S(B_2)$, the shorthand $B_1\cup B_2$ standing for the physical system composed of the subsystems $B_1$ and $B_2$. Note that $B_1\cup B_2\ne B_1+B_2$ in non-ergodic interacting systems just because the number of states in $B_1\cup B_2$ is different from the number of states in $B_1+B_2$. A related though different question is how to weaken the separability axiom to identify the extensive generalized entropies; we come back briefly to this point in Section 2 when speaking of the composability property.

Along with $S_{BGS}$, typical examples of generalized entropies are the Tsallis entropy [12],

$$T_q(p_1,\dots ,p_W)=\frac{1}{1-q}\left(\sum _{i=1}^W{p}_i^q-1\right)$$

(2)

($q\in \mathbb{R}$, $q\ne 1$, with the proviso that for $q<0$ terms with $p_i=0$ are omitted), and the Rényi entropy [11],

$$R_q(p_1,\dots ,p_W)=\frac{1}{1-q}ln\left(\sum _{i=1}^W{p}_i^q\right)$$

(3)

($q\ge 0$, $q\ne 1$). The Tsallis and Rényi entropies are related to the BGS entropy through the limits

$$\underset{q\to 1}{lim}{T}_q(p_1,\dots ,p_W)=\underset{q\to 1}{lim}{R}_q(p_1,\dots ,p_W)=S_{BGS}(p_1,\dots ,p_W),$$

this being one of the reasons they are considered generalizations of the BGS entropy. Both $T_q$ and $R_q$ have found interesting applications [15,17]; in particular, the parametric weighting of the probabilities in their definitions endows data analysis with additional flexibility. Other generalized entropies that we consider in this paper are related to ongoing work on graphs [18]. Further instances of generalized entropies are also referred to below.

Let us remark at this point that $S_{BGS}$, $T_q$, $R_q$ and other generalized entropies considered in this review can be viewed as special cases of the $(h,\varphi )$-entropies introduced in [19] for the study of asymptotic probability distributions. In turn, $(h,\varphi )$-entropies were generalized to quantum information theory in [20]. Quantum $(h,\varphi )$-entropies, which include von Neumann’s entropy [21] as well as the quantum versions of Tsallis’ and Rényi’s entropies, have been applied, for example, to the detection of quantum entanglement (see [20] and references therein). In this review, we do not consider quantum entropies, which would require advanced mathematical concepts, but only entropies defined on classical, discrete and finite probability distributions. If necessary, the transition to continuous distributions is done by formally replacing probability mass functions by densities and sums by integrals. For other approaches to the concept of entropy in more general settings, see [22,23,24,25].

Generalized entropies can be characterized by two scaling exponents in the limit $W\to \infty$, which we call Hanel–Thurner exponents [16]. For the simplest generalized entropies, which include $T_q$ but not $R_q$ (see Section 2), these exponents allow establishing a relationship between the abstract concept of generalized entropy and the physical properties of the system they describe through their asymptotic scaling behavior in the thermodynamic limit. That is, the two exponents label equivalence classes of systems which are universal in that the corresponding entropies have the same thermodynamic limit. In this regard, it is interesting to mention that, for any pair of Hanel–Thurner exponents (at least within certain ranges), there is a generalized entropy with those exponents, i.e., systems with the sought asymptotic behavior. Furthermore, the first Hanel–Thurner exponent allows also establishing a second relation with physical properties, namely, with the diffusion scaling exponents of diffusion processes, under some additional assumptions.

The rest of this review is organized as follows. The concept of generalized entropy along with some formal preliminaries and its basic properties are discussed in Section 1. As way of illustration, we discuss in Section 3 the Tsallis and Renyi entropies, as well as more recent entropic forms. The choice of the former ones is justified by their uniqueness properties under quite natural axiomatic formulations. The Hanel–Thurner exponents are introduced in Section 4, where their computation is also exemplified. Their aforementioned relation to diffusion scaling exponents is explained in Section 5. The main messages are recapped in Section 6. There is no section devoted to the applications but, rather, these are progressively addressed as the different generalized entropies are presented. The main text has been supplemented with three appendices at the end of the paper.

2. Generalized Entropies

Let $\mathcal{P}$ be the set of probability mass distributions $\{p_1,\dots ,p_W\}$ for all $W\ge 2$. For any function $H:\mathcal{P}\to {\mathbb{R}}^{+}$ (${\mathbb{R}}^{+}$ being the nonnegative real numbers), the Shannon–Khinchin axioms for an entropic form H are the following.

SK1 

Continuity. $H(p_1,\dots ,p_W)$ depends continuously on all variables for each W.

SK2 

Maximality. For all W,

$$H(p_1,\dots ,p_W)\le H(\frac{1}{W},\dots ,\frac{1}{W}).$$

SK3 

Expansibility: For all W and $1\le i\le W$,

$$H(0,p_1,\dots ,p_W)=H(p_1,\dots ,p_i,0,p_{i+1},\dots ,p_W)=H(p_1,\dots ,p_i,p_{i+1},\dots ,p_W).$$

SK4 

Separability (or strong additivity): For all $W,U$,

$$\begin{array}{c}H(p_{11},\dots ,p_{1U},p_{21},\dots p_{2U},\dots ,p_{W1},\dots ,p_{WU})\hfill \\ =H(p_{1·},p_{2·},\dots ,p_{W·})+\sum _{i=1}^W{p}_{i·}H\left(\frac{p_{i1}}{p_{i·}},\frac{p_{i2}}{p_{i·}},\dots ,\frac{p_{iU}}{p_{i·}}\right),\hfill \end{array}$$

(4)

where $p_{i·}={\sum }_{j=1}^U{p}_{ij}$.

Let $\{p_{11},\dots ,p_{1U},p_{21},\dots p_{2U},\dots ,p_{W1},\dots ,p_{WU}\}$ be the joint probability distribution of the random variables X and Y, with marginal distributions $\{p_{i·}:1\le i\le W\}$ and $\{p_{·j}={\sum }_{i=1}^W{p}_{ij}:1\le j\le U\}$, respectively. Then, axiom SK4 can be written as

$$H(X,Y)=H(X)+H(Y\left|X\right),$$

where $H(Y\left|X\right)$ is the entropy of Y conditional on X. In particular, if X and Y are independent (i.e., $p_{ij}=p_{i·}{p}_{·j}$), then $H(Y\left|X\right)=H(Y)$ and

$$H(X,Y)=H(X)+H(Y).$$

(5)

A function H such that Equation (5) holds (for independent random variables X and Y) is called additive. Physicists prefer writing $X+Y$ for composed systems with microstate probabilities $p_{ij}=p_{i·}{p}_{·j}$; this condition holds approximately only for weakly interacting systems X and Y.

With regard to Equation (5), let us remind that, for two general random variables X and Y, the difference $I(X;Y)=H(X)+H(Y)-H(X,Y)\ge 0$ is the mutual information of X and Y. It holds $I(X;Y)=0$ if and only if X and Y are independent [26].

More generally, a function H such that

$$\begin{array}{c}H(p_1{q}_1,\dots ,p_1{q}_U,p_2{q}_1,\dots ,p_2{q}_U,\dots ,p_W{q}_1,\dots ,p_W{q}_U)\hfill \\ =H(p_1,\dots ,p_W)+H(q_1,\dots ,q_U)+(1-\alpha )H(p_1,\dots ,p_W)H(q_1,\dots ,q_U),\hfill \end{array}$$

(6)

($\alpha >0$) is called $\alpha$-additive. With the same notation as above, we can write this property as

$$H(X,Y)=H(X)+H(Y)+(1-\alpha )H(X)H(Y),$$

(7)

where, again, X and Y are independent random variables. In a statistical mechanical context, X and Y may stand also for two probabilistically independent (or weakly interacting) physical systems. If $\alpha =1$, we recover additivity (Equation (5)).

In turn, additivity and $\alpha$-additivity are special cases of composability [15,27]:

$$H(X,Y)=\Phi (H(X),H(Y)),$$

(8)

with the same caveats for X and Y. Here, $\Phi$ is a symmetric function of two variables. Composability was proposed in [15] to replace axiom SK4. Interestingly, it has been proved in [27] that, under some technical assumptions, the only composable generalized entropy of the form in Equation (10) is $T_q$, up to a multiplicative constant.

As mentioned in Section 1, a function $F:\mathcal{P}\to {\mathbb{R}}^{+}$ satisfying axioms SK1–SK4 is necessarily of the form $F(p_1,\dots ,p_W)=k{S}_{BGS}(p_1,\dots ,p_W)$ for every W, where k is a positive constant ([28], Theorem 1). The same conclusion can be derived using other equivalent axioms [14,29]. For instance, Shannon used continuity, the property that $H(1/n,\dots ,1/n)$ increases with n, and a property called grouping [29] or decomposibility [30], which he defined graphically in Figure 6 of [7]:

$$\begin{array}{ccc}\hfill H(p_1,\dots ,p_W)& =& H((p_1+\dots +p_r),(p_{r+1}+\dots +p_W))\hfill \\ & & +(p_1+\dots +p_r)H\left(\frac{p_1}{{\sum }_{i=1}^r{p}_i},\dots ,\frac{p_r}{{\sum }_{i=1}^r{p}_i}\right)\hfill \\ & & +(p_{r+1}+\dots +p_W)H\left(\frac{p_{r+1}}{{\sum }_{i=r+1}^W{p}_i},\dots ,\frac{p_W}{{\sum }_{i=r+1}^W{p}_i}\right)\hfill \end{array}$$

(9)

($1\le r\le W-1$). This property allows reducing the computation of $H(p_1,\dots ,p_W)$ to the computation of the entropy of dichotomic random variables. According to ([15], Section 2.1.2.7), Shannon missed in his uniqueness theorem to formulate the condition in Equation (5), X and Y being independent random variables.

Nonnegative functions defined on $\mathcal{P}$ that satisfy axioms SK1–SK3 are called generalized entropies [16]. In the simplest situation, a generalized entropy has the sum property [14], i.e., the algebraic form

$$F_g(p_1,\dots ,p_W)=\sum _{i=1}^{W}g(p_i),$$

(10)

with $g:[0,1]\to {\mathbb{R}}^{+}$.

The following propositions are immediate.

(i)

Symmetry: $F_g(p_1,\dots ,p_W)$ is invariant under permutation of $p_1,\dots ,p_W$.

(ii)

$F_g$ satisfies axiom SK1 if and only if g is continuous.

(iii)

If $F_g$ satisfies axiom SK2, then

$$\sum _{i=1}^{W}g(p_i)\le Wg(\frac{1}{W})$$

for all $W\ge 2$ and $p_1,\dots ,p_W$ with $p_1+\dots +p_W$$=1$.

(iv)

If g is concave (i.e., ∩-convex), then $F_g$ satisfies axiom SK2.

(v)

$F_g$ satisfies axiom SK3 if and only if $g(0)=0$.

Note that Proposition (iv) follows from the symmetry and concavity of $F_g$ (since the unique maximum of $F_g$ must occur at equal probabilities).

We conclude from Propositions (ii), (iv) and (v) that, for $F_g$ to be a generalized entropy, the following three condition suffice:

(C1)

g is continuous.

(C2)

g is concave.

(C3)

$g(0)=0$.

As in [16], we say that a macroscopic statistical system is admissible if it is described by a generalized entropy $F_g$ of the form in Equation (10) such that g verifies Conditions (C1)–(C3). By extension, we say also that the generalized entropy $F_g$ is admissible. Admissible systems and generalized entropies are the central subject of this review. Clearly, $S_{BGS}$ is admissible because

$$g(x)=-xlogx,$$

(11)

$0\le x\le 1$. On the other hand, $T_q$ corresponds to

$$g(x)=\frac{1}{1-q}(x^q-x).$$

(12)

For $T_q$ to be admissible, Condition (C1) requires $q\ge 0$ and Condition (C3) requires $q>0$.

An example of a function $F:\mathcal{P}\to {\mathbb{R}}^{+}$ with the sum property that does not qualify for admissible generalized entropy is

$$F(p_1,\dots ,p_W)=\sum _{i=1}^W{\left(p_i-\frac{1}{W}\right)}^2=\sum _{i=1}^W{p}_i^2-\frac{1}{W}.$$

(13)

Indeed, $g(x)={(x-\frac{1}{W})}^2$ is not ∩-convex but ∪-convex and $g(0)=\frac{1}{W^2}\ne 0$. This probability functional was used in [31] to classify sleep stages.

Other generalized entropies that are considered below have the form

$$F_{G,g}(p_1,\dots ,p_W)=G\left(\sum _{i=1}^{W}g(p_i)\right),$$

(14)

where G is a continuous monotonic function, and g is continuous with $g(0)=0$. By definition, $F_{G,g}$ is also symmetric, and Proposition (iii) holds with the obvious changes. However, the concavity of g is not a sufficient condition any more for $F_{G,g}$ to be a generalized entropy. Such is the case of the Rényi entropy $R_q$ (Equation (3)); here

$$G(u)=\frac{1}{1-q}lnu\mathrm{and}g(x)=x^q,$$

(15)

but $g(x)$ (and, hence, ${\sum }_{i=1}^{W}g(p_i)$) is not ∩-convex for $q>1$. Furthermore, note that axiom SK3 requires $q>0$ for $R_q$ to be a generalized entropy.

Since Equation (10) is a special case of Equation (14) (set G to be the identity map $id(u)=u$), we can refer to both cases just by using the notation $F_{G,g}$, as we do hereafter.

We say that two probability distributions $\left\{p_i\right\}$ and $\left\{p_i^{\prime }\right\}$, $1\le i\le W$, are close if

$$∥\left\{p_i\right\}-\left\{p_i^{\prime }\right\}∥={\sum }_{i=1}^W\left|p_i-p_i^{\prime }\right|\le \delta ,$$

where $0<\delta \ll 1$; other norms, such as the two-norm and the max-norm, will do as well since they are all equivalent in the metric sense. A function $F:\mathcal{P}\to {\mathbb{R}}^{+}$ is said to be Lesche-stable if for all W and $ϵ>0$ there exists $\delta >0$ such that

$$∥\left\{p_i\right\}-\left\{p_i^{\prime }\right\}∥\le \delta \Rightarrow \left|\frac{F(\left\{p_i\right\})-F(\left\{p_i^{\prime }\right\})}{F_{max}}\right|<ϵ,$$

(16)

where $F_{max}$$={max}_{\left\{p_i\right\}\in \mathcal{P}}F(\left\{p_i\right\})$. It follows that

$$\underset{\delta \to 0}{lim}\underset{W\to \infty }{lim}\left|\frac{F(\left\{p_i\right\})-F(\left\{p_i^{\prime }\right\})}{F_{max}}\right|=0.$$

Lesche stability is called experimental robustness in [15] because it guarantees that similar experiments performed on similar physical systems provide similar results for the function F. According to [16], all admissible systems are Lesche stable.

3. Examples of Generalized Entropies

As way of illustration, we put the focus in this section on two classical generalized entropies as well as on some newer ones. The classical examples are the Tsallis entropy and the Rényi entropy because they have extensively been studied in the literature from an axiomatic point of view too. As it turns out, they are unique under some natural assumptions, such as additivity, $\alpha$-additivity or composability (see below for details). The newer entropies are related to potential applications of the concept of entropy to graph theory [18]. Other examples of generalized entropies are listed in Appendix A for further references.

3.1. Tsallis Entropy

A simple way to introduce Tsallis’ entropy as a generalization of the BGS entropy is the following [15]. Given $q\in \mathbb{R}$, define the q-logarithm of a real number $x>0$ as

$${ln}_{q}x=\left\{\begin{array}{cc}lnx& \mathrm{if}q=1,\hfill \\ \frac{x^{1-q}-1}{1-q}& \mathrm{otherwise}.\hfill \end{array}\right.$$

Note that ${ln}_{1}x$ is defined by continuity since ${lim}_{q\to 1}{ln}_{q}x=lnx$. If the logarithm in the definition of $S_{BGS}$, Equation (1), is replaced by ${ln}_q$, then we obtain the Tsallis entropy:

$$T_q(p_1,\dots ,p_W)=\sum _{i=1}^W{p}_i{ln}_q(1/p_i)=\frac{1}{1-q}\left(\sum _{i=1}^W{p}_i^q-1\right).$$

(17)

As noted before, $q>0$ for $T_q$ to be an admissible generalized entropy.

Alternatively, the definition

$$S_{BGS}(p_1,\dots ,p_W)=-\frac{d}{dx}{\left.\sum _{i=1}^W{p}_i^x\right|}_{x=1}$$

can also be generalized to provide the Tsallis entropy via the q-derivative,

$$T_q(p_1,\dots ,p_W)=-D_q{\left.\sum _{i=1}^W{p}_i^x\right|}_{x=1},$$

where

$$D_{q}f(x):=\frac{f(qx)-f(x)}{qx-x}.$$

Set $qx=x+h$, i.e., $h=(q-1)x$, and let $h\to 0$ to check that $D_{1}f(x)\equiv {lim}_{q\to 1}{D}_{q}f(x)=df(x)/dx$.

Although Tsallis proposed his entropy (Equation (17)) in 1988 to go beyond the standard statistical mechanics [12], basically the same formula had already been proposed in 1967 by Havrda and Charvát (with a different multiplying factor) in the realm of cybernetics and control theory [32].

Some basic properties of $T_q$ follow.

(T1)

$T_1=S_{BGS}$ because ${ln}_1{p}_i=ln{p}_i$ (or $D_{1}f(x)=df(x)/dx$).

(T2)

$T_q$ is (strictly) ∩-convex for $q>0$. Figure 1 plots $T_q(p,1-p)$ for $q=0.5$, 1, 2 and 5. Let us mention in passing that $T_q$ is ∪-convex for $q<0$.

Figure 1. Tsallis entropy $T_q(p,1-p)$ for $q=0.5,1,2$ and 5.

(T3)

$T_q$ is Lesche-stable for all $q>0$ [33,34]. Actually, we stated at the end of Section 2 that all admissible systems are Lesche stable.

(T4)

$T_q$ is not additive but q-additive (see Equation (6) or (7) with $\alpha$ replaced by q). This property follows from [15]

$${ln}_{q}xy={ln}_{q}x+{ln}_{q}y+(1-q)({ln}_{q}x)({ln}_{q}y).$$

(T5)

Similar to what happens with the BGS entropy, Tsallis entropy can be uniquely determined (except for a multiplicative positive constant) by a small number of axioms. Thus, Abe [35] characterized the Tsallis entropy by: (i) continuity; (ii) the increasing monotonicity of $T_q(1/W,\dots ,1/W)$ with respect to W; (iii) expansivity; and (iv) a property involving conditional entropies. Dos Santos [36], on the other hand, used the previous Axioms (i) and (ii), q-additivity, and a generalization of the grouping axiom (Equation (9)). Suyari [37] derived $T_q$ from the first three Shannon–Khinchin axioms and a generalization of the fourth one. The perhaps most economical characterization of $T_q$ was given by Furuichi [38]; it consists of continuity, symmetry under the permutation of $p_1,\dots ,p_W$, and a property called q-recursivity. As mentioned in Section 2, Tsallis entropy was recently shown [27] to be the only composable generalized entropy of the form in Equation (10) under some technical assumptions. Further axiomatic characterizations of the Tsallis entropy can be found in [39].

An observable of a thermodynamical (i.e., many-particle) system, say its energy or entropy, is said to be extensive if (among other characterizations), for a large number N of particles, that observable is (asymptotically) proportional to N. For example, for a system whose particles are weakly interacting (think of a dilute gas), the additive $S_{BGS}$ is extensive, whereas the non-additive $T_q$ ($q\ne 1$) is non-extensive. The same happens with ergodic systems [40]. However, according to [15], for a non-ergodic system with strong correlations, $S_{BGS}$ can be non-extensive while $T_q$ can be extensive for a particular value of q; such is the case of a microcanonical spin system on a network with growing constant connectancy [40]. This is why $T_q$ represents a physically relevant generalization of the traditional $S_{BGS}$. Axioms SK1–SK3 are expected to hold true also in strongly interacting systems.

Further applications of the Tsallis entropy include astrophysics [41], fractal random walks [42], anomalous diffusion [43,44], time series analysis [45], classification [46,47], and artificial neural networks [48].

3.2. Rényi Entropy

A simple way to introduce Rényi’s entropy as a generalization of $S_{BGS}$ is the following [17]. By definition, the BGS entropy of the probability distribution $\{p_1,\dots ,p_W\}$ (or of a random variable X with that probability distribution) is the linear average of the information function

$$I(p_i)=ln\frac{1}{p_i},1\le i\le W,$$

or, equivalently, the expected value of the random variable $ln\frac{1}{p(X)}$:

$$S_{BGS}(p_1,\dots ,p_W)={\mathbb{E}}_p[ln\frac{1}{p(X)}]=\sum _{i=1}^W{p}_{i}I(p_i).$$

In the general theory of expected values, for any invertible function $\varphi$ and realizations $x_1,\dots ,x_W$ of X in the definition domain of $\varphi$, an expected value can be defined as

$${\mathbb{E}}_{p,\varphi }\left[X\right]={\varphi }^{-1}\left(\sum _{i=1}^W{p}_i\varphi (x_i)\right).$$

Applying this definition to $ln\frac{1}{p(X)}$, we obtain

$${\mathbb{E}}_{p,\varphi }[ln\frac{1}{p(X)}]={\varphi }^{-1}\left(\sum _{i=1}^W{p}_i\varphi (I(p_i))\right).$$

If this generalized average has to be additive for independent events, i.e., it has to satisfy Equation (6) with $\alpha =1$, then

$$\varphi (x)=c_{1}x\mathrm{or}\varphi (x)=c_2^{(1-q)x}$$

must hold, where $c_1$, $c_2$ are positive constants, and $q>0,q\ne 1$. The first case leads to $S_{BGS}$, Equation (1), after choosing $c_1=e$. The second case leads to the Rényi entropy (actually, a one-parameter family of entropies) $R_q$, Equation (3), after choosing $c_2=e$ as well.

Next, we summarize some important properties of the Rényi entropy.

(R1)

$R_q$ is additive by construction.

(R2)

$R_1\equiv {lim}_{q\to 1}{R}_q=S_{BGS}$. Indeed, use L’Hôpital’s Rule to derive

$$\begin{array}{ccc}\hfill \underset{q\to 1}{lim}\frac{1}{1-q}ln\left(\sum _{i=1}^W{p}_i^q\right)& =& -\underset{q\to 1}{lim}\frac{d}{dq}ln\left(\sum _{i=1}^W{p}_i^q\right)\hfill \\ & =& -\underset{q\to 1}{lim}\frac{1}{{\sum }_{i=1}^W{p}_i^q}\sum _{i=1}^W{p}_i^{q}ln{p}_i\hfill \\ & =& -\sum _{i=1}^W{p}_{i}ln{p}_i.\hfill \end{array}$$

(R3)

$R_q$ is ∩-convex for $0<q\le 1$ and it is neither ∩-convex nor ∪-convex for $q>1$. Figure 2 plots $R_q(p,1-p)$ for $q=0.5$, 1, 2 and 5.

Figure 2. Rényi entropy $R_q(p,1-p)$ for $q=0.5,1,2$ and 5.

(R4)

$R_q$ is Lesche-unstable for all $q>0$, $q\ne 1$ [49].

(R5)

The entropies $R_q$ are monotonically decreasing with respect to the parameter q for any distribution of probabilities, i.e.,

$$q<q^{\prime }⟹R_q\ge R_{q^{\prime }}.$$

This property follows from the formula

$$-\frac{d{R}_q}{dq}=\frac{1}{{(1-q)}^2}\sum _{i=1}^W{p}_i^{\prime }ln\frac{p_i^{\prime }}{p_i}=\frac{1}{{(1-q)}^2}D(\left\{p_i^{\prime }\right\}\left.∥\left\{p_i\right\}\right),$$

where $p_i^{\prime }=p_i^q/{\sum }_{k=1}^W{p}_k^q$, and $D(\left\{p_i^{\prime }\right\}\left.∥\left\{p_i\right\}\right)\ge 0$ is the Kullback–Leibler divergence of the probability distributions $\{p_1^{\prime },\dots ,p_W^{\prime }\}$ and $\{p_1,\dots ,p_W\}$. $D(\left\{p_i^{\prime }\right\}\left.∥\left\{p_i\right\}\right)$ vanishes only in the event that both probability distributions coincide, otherwise is positive [26].

(R6)

A straightforward relation between Rényi’s and Tsallis’ entropies is the following [50]:

$$T_q=\frac{1}{1-q}\left(e^{(1-q)R_q}-1\right)\mathrm{or}{R}_q=\frac{1}{1-q}ln\left(1+(1-q)T_q\right).$$

However, the axiomatic characterizations of the Rényi entropy are not as simple as those for the Tsallis entropy. See [27,51,52] for some contributions in this regard.

For some values of q, $R_q$ has particular names. Thus, $R_0=lnW$ is called Hartley or max-entropy, which coincides numerically with $S_{BGS}$ for an even probability distribution. We saw in (R2) that $R_q$ converges to the BGS entropy in the limit $q\to 1$. $R_2=-{\sum }_{i=1}^W{p}_i^2$ is called collision entropy. In the limit $q\to \infty$, $R_q$ converges to the min-entropy

$$R_{\infty }(p_1,\dots ,p_W)=\underset{1\le i\le W}{min}(-ln{p}_i)=-\underset{1\le i\le W}{max}ln{p}_i=-ln\underset{1\le i\le W}{max}{p}_i.$$

The name of $R_{\infty }$ is due to property (R5).

Rényi entropy has found interesting applications in random search [53], information theory (especially in source coding [54,55]), cryptography [56], time series analysis [57], and classification [46,58], as well as in statistical signal processing and machine learning [17].

3.3. Graph Related Entropies

As part of ongoing work on graph entropy [18], the following generalized entropies are defined:

$$H_1(p_1,\dots ,p_W)=\sum _{i=1}^W\left(1-{(p_i)}^{p_i}\right),$$

(18)

$$H_2(p_1,\dots ,p_W)=\prod _{i=1}^W\left(2-{(p_i)}^{p_i}\right)=exp\left(\sum _{i=1}^{W}ln\left(2-{(p_i)}^{p_i}\right)\right),$$

(19)

and

$$H_3(p_1,\dots ,p_W)=1+ln{H}_2(p_1,\dots ,p_W)=\sum _{i=1}^W\left(p_i+ln\left(2-{(p_i)}^{p_i}\right)\right).$$

(20)

Note that $H_1(\dots ,0,1,0,etc.)=0$, while $H_2(\dots ,0,1,0,\dots )=H_3(\dots ,0,1,0,\dots )=1$. Other oddities of the above entropies include the terms ${(p_i)}^{p_i}$ in their definitions, as well as the presence of products instead of sums in the definition of $H_2$.

First, $H_1$ is of the type in Equation (10) with

$$g_1(x)=\left\{\begin{array}{cc}0& \mathrm{if}x=0,\hfill \\ 1-x^x& \mathrm{if}0<x\le 1.\hfill \end{array}\right.$$

(21)

By definition, $g(x)$ is continuous (even smooth), concave on the interval $[0,1]$, and $g_1(0)=0$. Therefore (see Conditions (C1)–(C3) in Section 2), H1 satisfies the axioms SK1–SK3, hence it is a generalized entropy.

As for $H_2$, this probability functional is of the type in Equation (14) with

$$g_2(x)=\left\{\begin{array}{cc}0& \mathrm{if}x=0,\hfill \\ ln\left(2-x^x\right)& \mathrm{if}0<x\le 1,\hfill \end{array}\right.$$

(22)

and $G(u)=e^u$. To prove that $H_2$ is a generalized entropy, note that

$$ln{H}_2(p_1,\dots ,p_W)=\sum _{i=1}^{W}ln\left(2-{(p_i)}^{p_i}\right)$$

satisfies axioms SK1–SK3 for the same reasons as $H_1$ does. Therefore, the same happens with $H_2$ on account of the exponential function being continuous (SK1), increasingly monotonic (SK2), and univalued (SK3).

Finally, $H_3$ is of the type in Equation (10) with

$$g_3(x)=\left\{\begin{array}{cc}0& \mathrm{if}x=0,\hfill \\ x+g_2(x)& \mathrm{if}0<x\le 1.\hfill \end{array}\right.$$

(23)

Since $H_3=1+ln{H}_2$, it is a generalized entropy because, as shown above, $ln{H}_2$ satisfies axioms SK1–SK3.

Figure 3 depicts $H_1(p,1-p)$, $H_2(p,1-p)$, $H_3(p,1-p)$, along with $S_{BGS}(p,1-p)$ and $H_2-S_{BGS}-1$ for comparison. As a curiosity, let us point out that the scaled versions

$${\tilde{H}}_i(p,1-p)=\frac{H_i(p,1-p)-H_i(0,1)}{H_i(\frac{1}{2},\frac{1}{2})-H_i(0,1)},$$

(24)

($i=1,2,3$), see Figure 4, approximate $S_{BGS}(p,1-p)$ measured in bits very well. In particular, the relative error in the approximation of $S_{BGS}(p,1-p)$ by ${\tilde{H}}_2(p,1-p)$ is less than $2.9\times {10}^{-4}$, so their graphs overlap when plotted.

Figure 3. Entropies $H_i(p,1-p)$, $i=1,2,3$, along with $S_{BGS}(p,1-p)$ and $H_2-S_{BGS}-1$ for comparison.

Figure 4. Scaled entropies ${\tilde{H}}_i(p,1-p)$, $i=1,2,3,$ see Equation (24).

A further description of the entropies in Equations (18)–(20) is beyond the scope of this section. Let us only mention in this regard that these entropies can be extended into the realm of acyclic directed graphs.

4. Hanel–Thurner Exponents

All generalized entropies $F_{G,g}$ group in classes labeled by two exponents $(c,d)$ introduced by Hanel and Thurner [16], which are determined by the limits

$$\underset{W\to \infty }{lim}\frac{F_{G,g}(p_1,\dots ,p_{\lambda W})}{F_{G,g}(p_1,\dots ,p_W)}={\lambda }^{1-c}$$

(25)

(W being as before the cardinality of the probability distribution or the total number of microstates in the system, $\lambda >1$) and

$$\underset{W\to \infty }{lim}\frac{F_{G,g}(p_1,\dots ,p_{W^{1+a}})}{F_{G,g}(p_1,\dots ,p_W)}{W}^{a(c-1)}={(1+a)}^d$$

(26)

($a>0$). Note that the limit in Equation (26) does not depend actually on c. The limits in Equations (25) and (26) can be computed via the asymptotic equipartition property [26]. Thus,

$$F_{G,g}(p_1,\dots ,p_{\lambda W})\approx G\left(\lambda Wg\left(\frac{1}{\lambda W}\right)\right)$$

and

$$F_{G,g}(p_1,\dots ,p_{W^{1+a}})\approx G\left(W^{1+a}g\left(\frac{1}{W^{1+a}}\right)\right)$$

asymptotically with ever larger W (thermodynamic limit). Set now $x=1/W$ to derive

$$\underset{x\to 0+}{lim}\frac{G\left(\frac{\lambda }{x}g\left(\frac{x}{\lambda }\right)\right)}{G\left(\frac{1}{x}g\left(x\right)\right)}={\lambda }^{1-c}$$

(27)

and

$$\underset{x\to 0+}{lim}\frac{G\left(\frac{1}{x^{1+a}}g\left(x^{1+a}\right)\right)}{x^{a(c-1)}G\left(\frac{1}{x}g\left(x\right)\right)}={(1+a)}^d.$$

(28)

Clearly, the scaling exponents c, d of a generalized entropy $F_{G,g}$ depend on the behavior of g in an infinitesimal neighborhood $(0,\epsilon ]$ of 0 (i.e., $g(\epsilon )$ with $0<\epsilon \ll 1$), as well as on the properties of G if $G\ne id$. We call $(c,d)$ the Hanel–Thurner (HT) exponents of the generalized entropy $F_{G,g}$.

When $G=id$, Equations (27) and (28) abridge to

$$\underset{x\to 0+}{lim}\frac{g(zx)}{g(x)}=z^c$$

(29)

(after replacing ${\lambda }^{-1}$ by z), and

$$\underset{x\to 0+}{lim}\frac{g(x^{1+a})}{x^{ac}g(x)}={(1+a)}^d,$$

(30)

respectively. In this case, $0<c\le 1$, while d can be any real number. If $c=1$, the concavity of g implies $d\ge 0$ [16]. The physical properties of admissible systems are uniquely characterized by their HT exponents, i.e., by their asymptotic properties in the limit $W\to \infty$ [16]. In this sense, we can also speak of the universality class $(c,d)$.

As way of illustration, we are going to derive the HT exponents of $S_{BGS}$, $T_q$ and $R_q$.

(E1)

For the BGS entropy, $g(x)=-xlnx$ (see Equation (11)), so

$$\frac{g\left(zx\right)}{g\left(x\right)}=\frac{zxln(zx)}{xlnx}=\frac{zlnz+zlnx}{lnx}\to z$$

as $x\to 0+$. Therefore, $c=1$. Furthermore,

$$\frac{g\left(x^{1+a}\right)}{x^{ac}g\left(x\right)}=\frac{x^{1+a}ln{x}^{1+a}}{x^{a+1}lnx}=\frac{(1+a)lnx}{lnx}=1+a$$

for all $x>0$, so $d=1$.

(E2)

For the Tsallis entropy, see Equation (12),

$$g(x)=\left\{\begin{array}{cc}\frac{1}{1-q}{x}^q+\mathcal{O}(x)& \mathrm{if}0<q<1,\hfill \\ -\frac{1}{1-q}x+\mathcal{O}(x)& \mathrm{if}q>1.\hfill \end{array}\right.$$

It follows readily that $(c,d)=(q,0)$ if $0<q<1$, and $(c,d)=(1,0)$ if $q>1$. Hence, although ${lim}_{q\to 1}{T}_q=S_{BGS}$, there is no parallel convergence concerning the HT exponents.

(E3)

For the Rényi entropy, $g(x)=x^q$ and $G(u)=\frac{1}{1-q}{e}^u$ (see Equation (15)), so

$$\frac{G(\frac{\lambda }{x}g\left(\frac{x}{\lambda }\right))}{G(\frac{1}{x}g\left(x\right))}=\frac{ln\left(\frac{\lambda }{x}{(\frac{x}{\lambda })}^q\right)}{ln\left(\frac{1}{x}{x}^q\right)}=\frac{ln{x}^{q-1}-ln{\lambda }^{q-1}}{ln{x}^{q-1}}\to 1$$

as $x\to 0+$ (both for $0\le q\le 1$ and $q\ge 1$). Therefore, $c=1$. Furthermore,

$$\frac{G\left(\frac{1}{x^{1+a}}g\left(x^{1+a}\right)\right)}{G\left(\frac{1}{x}g\left(x\right)\right)}=\frac{ln\left(\frac{1}{x^{1+a}}{x}^{q(1+a)}\right)}{ln\left(\frac{1}{x}{x}^q\right)}=\frac{ln{x}^{(q-1)(1+a)}}{ln{x}^{q-1}}=1+a$$

for all $x>0$, so that $d=1$. In sum, $(c,d)=(1,1)$ for all q.

As for the generalized entropies $H_1,H_2$, and $H_3$ considered in Section 3.3, we show in Appendix B that their HT exponents are $(1,1)$, $(0,0)$, and $(1,1)$, respectively. Thus, $H_1$ and $H_3$ belong to the same universality class as $S_{BGS}$, while the HT exponents of $H_2$ and $R_q$ (both of the same the type in Equation (14)) are different. Moreover, the interested reader will find in Table 1 of [16] the HT exponents of the generalized entropies listed in Appendix A.

An interesting issue that arises at this point is the inverse question: Given $c\in (0,1]$ and $d\in \mathbb{R}$, is there an admissible system such that its HT exponents are precisely $(c,d)$? The answer is yes, at least under some restrictions on the values of c and d. Following [16], we show in Appendix C that, if

$$\begin{array}{cc}d>-1& \mathrm{for}0<c\le \frac{1}{2},\\ d\ge 1-\frac{1}{c}& \mathrm{for}\frac{1}{2}<c\le 1,\end{array}$$

(31)

then the “generalized $(c,d)$-entropy”

$$S_{c,d}(p_1,\dots ,p_W)=eA\sum _{i=1}^W\Gamma (d+1,1-cln{p}_i),$$

(32)

has HT exponents $(c,d)$. Here, $A>0$ and $\Gamma$ is the incomplete Gamma function (Section 6.5 of [59]), that is,

$$\Gamma (r,s)={\int }_s^{\infty }{t}^{r-1}{e}^{-t}dt(r>0).$$

(33)

Several application cases where generalized $(c,d)$-entropies are relevant have been discussed by Hanel and Thurner in [40] (super-diffusion, spin systems, binary processes, and self-organized critical systems) and [60] (aging random walks, i.e., random walks whose transition rates between states are path- and time-dependent).

5. Asymptotic Relation between the HT Exponent c and the Diffusion Scaling Exponent

In contrast to “non-interacting” systems, where both the additivity and extensivity of the BGS entropy $S_{BGS}$ hold, in the case of general interacting statistical systems these properties can no longer be simultaneously satisfied, requiring a more general concept of entropy [16,40]. Following [16] (Section 4), a possible generalization of $S_{BGS}$ for admissible systems is defined via the two asymptotic scaling relations in Equations (29) and (30), i.e., the HT exponents c and d, respectively. These asymptotic exponents can be interpreted as a measure of deviation from the “non-interacting” case regarding the stationary behavior.

5.1. The Non-Stationary Regime

In this section, we describe a relation between the exponent c and a similar macroscopic measure that characterizes the system in the non-stationary regime, thus providing a meaningful interpretation of the exponent. The non-stationary behavior of a system can possibly be described by the Fokker–Planck (FP) equation governing the time evolution of a probability density function $p=p(x,t)$. In this continuous limit, the generalized entropy $F_g$ is assumed to be written as $F_g\left[p(s)\right]=\int g(p(s))\mathrm{d}s$, where g is asymptotically characterized by Equation (29) and $s=s(x)$ is a time-independent scalar function of the space coordinate x (for example, a potential) [61,62].

Going beyond the scope of the simplest FP equation, we consider systems for which the correlation among their (sub-)units can be taken into account by replacing the diffusive term ${\partial }_x^{2}p$ with an effective term ${\partial }_x^2\Phi \left[p\right]$, where $\Phi \left[p\right]$ is a pre-defined functional of the probability density. $\Phi \left[p\right]$ can be either derived directly from the microscopical transition rules or it may be defined based on macroscopic assumptions. The resulting FP equation can be written as

$${\partial }_{t}p(x,t)=D\beta {\partial }_x\left(p(x,t){\partial }_{x}u(x)\right)+D{\partial }_x^2\Phi \left[p(x,t)\right],$$

(34)

where $D,\beta$ are constants and $u(x)$ is a time-independent external potential.

For simplicity, hereafter we exclusively focus on one dimensional FP equations. In the special case of $\Phi \left[p\right]=p$ and no external forces, Equation (34) reduces to the well-known linear diffusion equation

$${\partial }_{t}p(x,t)=D{\partial }_x^{2}p(x,t).$$

(35)

The above equation is invariant under the space-time scaling transformation

$$p(x,t)={\tau }^{-\gamma }p\left(\frac{x}{{\tau }^{\gamma }},\frac{t}{\tau }\right)$$

(36)

with $\gamma =\frac{1}{2}$ [63,64]. This scaling property opens up the possibility of a phenomenological and macroscopic characterization of anomalous diffusion processes [15,44] as well, which correspond to more complicated non-stationary processes described by FP equations in the form of Equation (34) with a non-trivial value of $\gamma$. With the help of the transformation in Equation (36), we can also classify correlated statistical systems according to the rate of the spread of their probability density functions over time in the asymptotic limit and, thus, quantitatively describe their behavior in the non-stationary regime.

5.2. Relation between the Stationary and Non-Stationary Regime

To reasonably and consistently relate the generalized entropies to the formalism of FP equations—corresponding to the stationary and non-stationary regime, respectively—the functional $\Phi \left[p\right]$ has to be chosen such that the stationary solution of the general FP equation becomes equivalent to the Maximum Entropy (MaxEnt) probability distribution calculated with the generalized entropies. These MaxEnt distributions can be obtained analogously to the results by Hanel and Thurner in [16,40], where they used standard constrained optimization to find the most general form of MaxEnt distributions, which turned out to be $p(ϵ)={\mathcal{E}}_{c,d,r}(-ϵ)$ with

$${\mathcal{E}}_{c,d,r}(x)\propto exp\left[-\frac{d}{1-c}{W}_k\left(B{(1-\frac{x}{r})}^{1/d}\right)\right].$$

(37)

Here, $B,r$ are constants depending only on the $c,d$ parameters and $W_k$ is the kth branch of the Lambert-W function (specifically, branch $k=0$ for $d\ge 0$ and branch $k=1$ for $d<0$). The consistency criterion imposed above accords with the fact that many physical systems tend to converge towards maximum entropy configuration over time, however, it specifies the limits of our assumptions.

Consider systems described by Equation (34) in the absence of external force, i.e.,

$${\partial }_{t}p(x,t)=D{\partial }_x^2\Phi \left[p(x,t)\right].$$

(38)

By assuming that the corresponding stationary solutions can be identified with the MaxEnt distributions in Equation (37), it can be shown that the functional form of the effective density $\Phi \left[p\right]$ must be expressed as

$$\Phi \left[p\right]\propto {\int }_0^{p}q{\partial }_q^{2}g\left(q\right)\mathrm{d}q,$$

(39)

where we neglected additive and multiplicative constant factors for the sake of simplicity. Similar implicit equations have already been investigated in [61,62,65]. Once the asymptotic phase space volume scaling relation in Equation (29) holds, it can also be shown that the generalized FP in Equation (38) (with $\Phi$ as in Equation (39)) obeys the diffusion scaling property in Equation (36) with a non-trivial value of $\gamma$ in the $p\to 0$ asymptotic limit [66] (assuming additionally the existence of the solution of Equation (38), at least from an appropriate initial condition). A simple algebraic relation between the diffusion scaling exponent $\gamma$ and the phase space volume scaling exponent c can be established [66], which can be written as

$$\gamma =\frac{1}{1+c}.$$

(40)

Therefore, this relation between c and $\gamma$ defines families of FP equations which show asymptotic invariance under the scaling relation in Equation (36).

6. Conclusions

This review concentrates on the concept of generalized entropy (Section 2), which is relevant in the study of real thermodynamical systems and, more generally, in the theory of complex systems. Possibly the first example of a generalized entropy was introduced by Rényi (Section 3.2), who was interested in the most general information measure which is additive in the sense of Equation (5), with the random variables X and Y being independent. Another very popular generalized entropy was introduced by Tsallis as a generalization of the Boltzmann–Gibbs entropy (Section 3.1) to describe the properties of physical systems with long range forces and complex dynamics in equilibrium. Some more exotic generalized entropies are considered in Section 3.3, while other examples that have been published in the last two decades are gathered in Appendix A. Our approach was to a great extent formal, with special emphasis in Section 2 and Section 3 on axiomatic formulations and mathematical properties. For expository reasons, applications are mentioned and the original references given as our description of the main generalized entropies progressed, rather than addressing them jointly in a separate section.

An alternative approach to generalized entropies other than the axiomatic one (Section 2) consists in characterizing their asymptotic behavior in the thermodynamic limit $W\to \infty$. Hanel and Thurner showed that two scaling exponents $(c,d)$ suffice for admissible generalized entropies, i.e., those entropies of the form in Equation (10) with g continuous, concave and $g(0)=0$ (Section 4); it holds $c\in (0,1]$ and $d\in \mathbb{R}$. As a result, the admissible systems fall in equivalence classes labeled by the exponents $(c,d)$ of the corresponding entropies. Conversely, to each $(c,d)$, there is a generalized entropy with those Hanel–Thurner exponents (see Equation (32)), at least for the most interesting value ranges.

It is also remarkable that, at asymptotically large times and volumes, there is a 1-to-1 relation between the equivalence class of generalized entropies with a given $c\in \left(0,1\right]$ and the equivalence class of Fokker–Planck equations in which the invariance in Equation (36) holds with $\gamma =\frac{1}{1+c}\in \left[\frac{1}{2},1\right)$ (Section 5). This means that the equivalence classes of admissible systems can generally be mapped into anomalous diffusion processes and vice versa, thus conveying the same information about the system in the asymptotic limit (i.e., when $p(x,t)\to 0$) [66]. A schematic visualization of this relation is provided in Figure 5. Moreover, the above result can actually be understood as a possible generalization of the Tsallis–Bukman relation [44].

Figure 5. Visual summary of the main result presented in Section 5 schematically depicting the relation between the exponents $\gamma$ and c. Source: [66].