Based on a diffusion-like master equation we propose a formula using the Bregman divergence for measuring entropic distance in terms of different non-extensive entropy expressions. We obtain the non-extensivity parameter range for a universal approach to the stationary distribution by simple diffusive dynamics for the Tsallis and the Kaniadakis entropies, for the Hanel–Thurner generalization, and finally for a recently suggested log-log type entropy formula which belongs to diverging variance in the inverse temperature superstatistics.
generalized q-entropy; entropic distance
Over the last decades, there have been several suggestions for generalizations of the Boltzmann–Gibbs–Shannon (BGS) entropy formula [1,2,3,4,5,6]. Most formulas can be grouped into categories either by their mathematical form (trace form or a function of the trace form) , or by the scaling properties for large systems, usually providing large entropies, S, even if not necessarily proportional to the logarithm of the number of states, [8,9,10,11]. Such entropy formulas are often presented based on axioms, like the Khinchin or Shannon axioms for the most known logarithmic entropy formula due to Boltzmann and Gibbs. Non-extensive entropy then abandons the additivity axiom and replaces it with a more general assumption. For the group entropy, the assumption of the associativity of the entropy composition rule is used . Another way to obtain a generalized entropy formula starts with physical properties, like the universality of the thermal equilibrium between two systems as described by the zeroth law of thermodynamics [12,13], and by this it also includes the associativity assumption . It is also noteworthy that starting with a general, non-associative composition prescription one arrives asymptotically at an associative one—only by repeating it in small steps and reconstructing the effective composition formula in the continuous scaling limit .
Deformations of the original Boltzmann–Gibbs–Shannon entropy formula can also be achieved by tracing it back from the canonical distribution, exponential in the subsystem energy, when parameters of this distribution are treated as stochastic ones, or otherwise randomized, and hence integrated over another—the superstatistical—distribution [16,17,18]. Instead of treating the temperature or its inverse as a fluctuating quantity, one may also consider the number of degrees of freedom, and hence the dimensionality of phase space, as a fluctuating quantity. In [19,20,21,22,23] it has been shown that simple fluctuation patterns of the particle number, N, influencing the phase space volume Ω occupied by an ideal gas, can lead both to an exponential distribution of by a Poisson N distribution, or to cut power-law like Tsallis–Pareto distribution by a negative binomial N distribution. In general, a trace form entropy,
emerges with satisfying a second order differential equation, the additivity restoration condition (ARC) . Connecting to a more traditional notation, the K function is related to the deformed logarithm function as
This form includes the one parameter deformed logarithms as well as the generalizations with more parameters [12,13,25,26,27].
In the present paper we investigate whether such entropy formulas define an entropic distance between two probability distributions, which has the following useful properties:
it is positive for any two different distributions;
it is zero for comparing any distribution with itself;
it is symmetric.
A symmetric distance measure is most naturally formulated as a sum of relative divergences, . For deformed logarithms two basic forms of suggestions occur in the literature, one based on the ratio and one based on the difference in their logarithm, cf. refs. [28,29,30,31]. Furthermore we are interested in a definition which describes a distance that is always shrinking between a dynamically evolving distribution and the one belonging to maximal entropy under a spontaneous dynamics.
In the following we show this behavior for the traditional logarithmic entropy formula and a generally state-dependent nearest neighbour master equation (defined below), and then propose a generalization of the symmetrized entropic distance measure based on the deformed logarithm function. We conclude that some proposals encertain a shrinking of the entropic distance during an approach to the stationary distribution only for a restricted range of the non-extensivity parameter(s) used in the entropy formula.
2. Probability Distributions
The Poisson distribution is ubiquitous in statistical phenomena, where the discreteness of the basic variable, n, and its positivity are natural constraints. This distribution,
is parameterized by a single parameter, coinciding with the mean value of the variable n : . The logarithm of its characteristic function,
generates all central moments of the distribution. Notably, due to the Poisson distribution narrows for growing mean value of n.
This distribution plays a central role in different areas of physics: it occurs as the statistics of spontaneous radioactive decays during given time intervals, as the number of photons in a Glauber coherent state, or as the number distribution of randomly produced hadrons in high energy experiments when uniformly filling the available phase space. In the latter case, a given total energy, E, is distributed among n newly made particles in experiment. The statistical weight to find a single particle with individual energy ω among the products is roughly proportional to the relative phase space described by a ratio of corresponding n-dimensional spheres. Its average over the Poisson distribution is given by the following remarkable formula
Several authors use the power instead of n in this formula. The difference traces back to different concepts on the hyperspheric phase space volume vs. surface. Since for large n it does not matter, we do not go into details here.
This result allows for a Gibbsean interpretation of this statistical factor with the kinetic temperature .
Beyond the Poissonian also sub- and super-Poissonian bosonic states occur with the corresponding Bernoulli or negative binomial multiplicity distributions in experiments. The above ideal phase space ratio averages in those cases to a statistical factor generalizing the exponential to a cut power-law form, also known as a canonical distribution to the Tsallis entropy, e.g.,
for the negative binomial distribution
The above Tsallis–Pareto distribution, Equation (6) can also be obtained as a canonical distribution to the Tsallis entropy formula [3,4,5],
with and .
These and similar experiences suggest that different entropy formulas can be used as a basis for defining the entropic distance. Earlier attempts mostly defined a divergence formula based on the deformed logarithm, , of the ratio of the two respective probabilities [31,32]. An alternative approach—which we propose to follow—uses the difference of the deformed logarithms as a definition, it is based on a Bregman type divergence. While for the traditional logarithm function these alternative definitions coincide (due to ), they do differ appreciably for the deformed logarithm functions in general use.
3. Master Equation
Let us denote the probability of having exactly n quanta in the observed system (part of phase space) at the time t generally by . This quantity may depend on further parameters, this is suppressed in the notation by now. However, in any blind choice case it is proportional to the quantity , to the ratio of one to the number of possible ways to realize the state with n quanta.
A wide range of phenomena can be described by a simplified dynamics, assuming that only one quantum can be exchanged in a time instant. In this case, the evolution of and depends only on the state probabilities of having one more or one less quantum. We consider here only the linearized version of this dynamics, the so called master equation:
Here, is the transition (decay) rate from a state with n quanta to the state , is the corresponding reverse (growth) rate from n to . In this case the occurrence of state n in a huge parallel ensemble (Gibbs ensemble) of systems is fed by both the and processes and it is diminished by the reverse processes. It is of special interest to investigate processes when λ and μ are related by symmetry principles, like time reversal invariance or subsystem—reservoir homogeneity.
We shall quote the general detailed balance solution of Equation (9), , as the equilibrium distribution. Since the equation is homogeneous and linear in the -s the overall normalization is not fixed by it. Being the equilibrium distribution, all -s should be zero. That condition is fulfilled only if
from which it follows that also
holds, annullating all evolution. Based on this observation, the detailed balance distribution satisfies
where can be obtained from the normalization condition
This is kept during the evolution, since upon Equation (9).
4. Entropic Distance
Based on the Boltzmann-Gibbs entropy,
a distance measure from a reference distribution, is based on the quantity
By this definition , and it gives a positive Kullback–Leibler divergence [33,34,35] between the normalized distributions and . This can be easily proved by using the Jensen inequality, applied to . The symmetric combination, based on a sum of Bregman type divergences [28,29,30,31],
is non-negative then term by term. For a fixed, time-independent the master Equation (9) causes to decrease:
The sum can be re-arranged by summation index re-definition to
This expression is non-positive for arbitrary values only if
Namely, one arrives solely in this case at the type expression under the sum. But this means that the reference distribution, to which any initial distribution approaches if evolving according to the master equation, is the detailed balance distribution! It can be shown that by the virtue of the master equation also the distance is decreasing term by term. The symmetrically summed Kullback–Leibler distance to the detailed balance distribution is also reduced: , due to
In principle different distance measures (based on deformed entropy formulas) can also be used to investigate this property of a linear master equation. If one considers an entropy defined by a deformed logarithm function,
then the Kullback–Leibler divergence and the definition of the distance between probability distributions can be modified accordingly as
This measure is zero only if for all n; otherwise it is positive if is a strictly increasing function, i.e., on the whole interval .
The condition of the convergence to the detailed balance solution then reads as
On the other hand, from the master equation one obtains
with and . Substituting Equation (25) into Equation (23) and re-arranging the summation indices finally we arrive at the requirement
This is satisfied if is strictly increasing, which means that for all possible and values; being realized only if (the last equality is due to the normalization while ). The condition for approaching the detailed balance distribution then finally reads as
It is clear that this is a more detailed restriction than just the concavity of the entropy formula. The Q-independent part of the expression in the bracket equals , whose positivity is the well-known concavity requirement for the entropy formula. In cases when also , this concaivity suffices for the uniform approach to the stationary distribution. In the opposite case we arrive at the nontrivial constraint at .
Let us now test whether some familiar suggestions for deformed logarithms satisfy this constraint. The Tsallis logarithm is defined as
It results in and due to Equation (27) the final constraint is
This is fulfilled for arbitrary probability distribution pairs if ; otherwise it might be violated. The classical logarithm is reconstructed when .
The kappa-exponential, promoted by Kaniadakis , belongs to the deformed logarithm
In this case one obtains
as the condition for the correct evolution towards the detailed balance solution by the linear master equation. This is fulfilled universally (i.e., for arbitrary and in the physical interval of ) as long as . One obtains this result as follows. Both square bracket expressions in Equation (31) are non-negative. Their sum is also non-negative if . Otherwise there are values near to zero for which one of the summands, either due to or due to , diverge to negative infinity, spoiling the inequality in this way. The classical logarithm is reconstructed for and in order to have the same power-law tail in canonical distributions . The condition for universal evolution translates to .
By now there is a much bigger variety of suggested entropy formulas, many of them having the trace form using a deformed logarithm function [37,38,39]. Here we analyse one of them, the entropy suggested by Hanel and Thurner , since it includes a number of different entropy formulas suggested earlier as particular cases. It is given as
being the incomplete gamma function. The coefficients A and B can be determined by considering the equiprobability case, , resulting in , and applying the conditions and . By this procedure one obtains and , with and .
The corresponding deformed logarithm of the probability, based on the form Equation (21), is
Here, for instance, the , choice gives the Tsallis logarithm, Equation (28) with and .
The first and second derivative of the deformed logarithm are given as
Here we used the abbreviations Γ, and without explicitely writing out the arguments and .
All this leads to the following criterion for the term by term approach to the stationary distribution by a general via the diffusion like master equation:
Suppose now that . This requires . (In the above example that translates to .) Furthermore, considering the extreme case (equivalent to the concavity condition) we deal with
This is fulfilled for and . Once this is so, observing that the lhs expression of the inequality Equation (39) is linear in , the condition has to be checked only at the other endpoint of the interval. At we have:
Although this criterion looks involved, we note, that at it simplifies to , i.e., as supposed above. Similar analysis can be done for the opposite, case.
Finally, the recently suggested doubly logarithmic entropy formula, designed for extreme large fluctuations in a reservoir by Biro et al. [23,24], considers
as the deformed logarithm, leading to
for the evolution condition . This is fulfilled for any pair of distributions, since is always true.
By now the use of non-extensive entropy formulas is clearly numerous. A review of all areas of physics where a power-law tailed canonical distribution occurs can be found in . Furthermore, applications to non-Gaussian velocity distributions in dusty plasmas can be found in [40,41]. In this paper we have concentrated on a mathematical problem in the background testing generalized entropy formulas with respect to whether they serve as a basis for a well-behaving distance measure which describes an approach to the stationary distribution of state-dependent diffusion-like master equations. We have found that, in some parameter range of the suggested modern entropy formulas, the distance measure, based on the difference of deformed logarithms of the respective distributions, behaves well as shrinking uniformly, term by term during the time evolution towards the stationary distribution of general, state-dependent diffusion-like master equations. In particular, while the Tsallis entropy and other generalized entropy formulas (involving further deformed logarithm functions) guarantee the above term by term approach only in a restricted range of parameters, the extreme case belonging to a diverging variance in the temperature superstatistics, with a log-log form entropy formula, behaves universally well in this respect.
This work was supported by the Hungarian National Research Fund (Grant K 104260) and by the Helmholtz International Center for FAIR (Facility for Antiproton and Ion Research) within the framework of the LOEWE program (Landes Offensive zur Entwicklung Wissenschaftlich-ökonomischer Exzellenz) launched by the State of Hesse.
Tamás Sándor Biró and Zsolt Schram performed and controled the calculations and prepared the manuscript alternatingly. Both authors have read and approved the final manuscript.
Conflicts of Interest
The authors declare no conflict of interest.
Rényi, A. On Measures of Entropy and Information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA, 20 June–30 July 1960; University of California Press: Berkeley, CA, USA, 1961; Volume 1, pp. 547–561. [Google Scholar]
Rényi, A. Probability Theory; North Holland: Amsterdam, The Netherlands, 1970. [Google Scholar]
Tsallis, C. Possible generalization of Boltzmann–Gibbs statistics. J. Stat. Phys.1988, 52, 479–487. [Google Scholar] [CrossRef]
Tsallis, C. Nonextensive statistics: Theoretical, experimental and computational evidences and connections. Braz. J. Phys.1999, 29. [Google Scholar] [CrossRef]
Tsallis, C. Introduction to Nonextensive Statistical Mechanics; Springer: New York, NY, USA, 2009. [Google Scholar]
Almeida, M.P. Generalized entropies from first principles. Physica A2001, 300, 424–432. [Google Scholar] [CrossRef]
Hanel, R.; Thurner, S. A comprehensive classification of complex statistical systems and axiomatic derivation of their entropy and distribution functions. Europhys. Lett.2011, 93, 20006. [Google Scholar] [CrossRef]
Hanel, R.; Thurner, S. When do generalised entropies apply? How phase space volume determines entropy. Europhys. Lett.2011, 96, 50003. [Google Scholar] [CrossRef]
Hanel, R.; Thurner, S.; Gell-Mann, M. How multiplicity determines entropy and derivation of the maximum entropy principle for complex systems. Proc. Natl. Acad. Sci. USA2014, 111, 6905–6910. [Google Scholar] [CrossRef] [PubMed][Green Version]
Hanel, R.; Thurner, S. Generalized (c,d) entropies and aging random walks. Entropy2013, 15, 5324–5337. [Google Scholar] [CrossRef]
Kaniadakis, G. Maximum entropy principle and power-law tailed distributions. Eur. Phys. J. B2009, 70, 3–13. [Google Scholar] [CrossRef]
Kaniadakis, G. Theoretical Foundations and Mathematical Formalism of the Power-Law Tailed Statistical Distributions. Entropy2013, 15, 3983–4010. [Google Scholar] [CrossRef]
Biró, T.S.; Ván, P. Zeroth law compatibility of non-additive thermodynamics. Phys. Rev. E2011, 83, 061187. [Google Scholar] [CrossRef] [PubMed]
Biró, T.S. Abstract composition rule for relativistic kinetic theory in the thermodynamical limit. Europhys. Lett.2008, 84, 56003. [Google Scholar] [CrossRef]
Beck, C. Dynamical foundations of nonextensive statistical mechanics. Phys. Rev. Lett.2001, 87, 180601. [Google Scholar] [CrossRef]
Biró, T.S.; Barnaföldi, G.G.; Ván, P. New Entropy Formula with Fluctuating Reservoir. Physica A2015, 417, 215–220. [Google Scholar] [CrossRef]
Biró, T.S.; Ván, P.; Barnaföldi, G.G.; Ürmössy, K. Statistical Power Law due to Reservoir Fluctuations and the Universal Thermostat Independence Principle. Entropy2014, 16, 6497–6514. [Google Scholar] [CrossRef]
Abe, S. A note on the q-deformation-theoretic aspect of the generalized entropies in nonextensive physics. Phys. Lett. A1997, 224, 326–330. [Google Scholar] [CrossRef]
Kaniadakis, G. Relativistic entropy and related Boltzmann kinetics. Eur. Phys. J. A2009, 40, 275–287. [Google Scholar] [CrossRef]
Ubriaco, M.R. Entropies based on factional calculus. Phys. Lett. A2009, 373, 2516–2519. [Google Scholar] [CrossRef]
Naudts, J. Deformed exponentials and logarithms in generalized tehrmostatistics. Physica A2002, 316, 323–334. [Google Scholar] [CrossRef]
Naudts, J. Continuity of a class of entropies and relative entropies. Rev. Math. Phys.2004, 16, 809–822. [Google Scholar] [CrossRef]
Naudts, J. Generalized thermostatistics based on deformed exponential and logarithmic functions. Physica A2004, 340, 32–40. [Google Scholar] [CrossRef]
Naudts, J. Generalised Exponential Families and Associated Entropy Functions. Entropy2008, 10, 131–149. [Google Scholar] [CrossRef]
Tsallis, C. Generalized entropy-based criterion for consistent testing. Phys. Rev. E1998, 58. [Google Scholar] [CrossRef]
Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat.1951, 22, 79–86. [Google Scholar] [CrossRef]
Kullback, S. Letter to the Editor: The Kullback–Leibler distance. Am. Stat.1987, 41, 340–341. [Google Scholar]
Van Erven, T.; Harremoës, P. Rényi Divergence and Kullback–Leibler Divergence. IEEE Trans. Inf. Theory2014, 60, 3797–3820. [Google Scholar] [CrossRef]