A Non-Negative Measure of Information for Continuous Probability Distributions

Abstract

In this work, we investigate the possibility of using varentropy, an information measure previously proposed for discrete probability distribution, as a measure of the probabilistic uncertainty of continuous probability distribution. We show that varentropy allows avoiding negative values and some undesirable features of informational entropy encountered while using the Boltzmann–Shannon formula (and others) for continuous probability distributions.

1. Introduction

Entropy and information are among the most fundamental concepts in thermodynamics, statistical mechanics and information theory [1,2,3,4,5,6,7]. According to the common view, entropy and information (entropy for short in what follows) are two names for the same thing. They are a measure of disorder or statistical uncertainty associated with probability. The relationship between entropy and probability has been subject to rigorous mathematical study since the work of Shannon [2] and Khinchin [7]. Nevertheless, some confusion persists concerning the expression of entropy for continuous probability distributions [4,5,8].

The best known and most employed statistical expression of informational entropy is in the form of a logarithmic functional of probability distribution (Boltzmann–Shannon entropy), a form proposed for the first time by Boltzmann in his H-theorem [1] with continuous distribution of particles, then used by Gibbs in his work on statistical mechanics [3], and by Shannon in his information theory [2]. An axiomatic derivation (or proof of the uniqueness) of the formula was given in [2] by using discrete probability distributions. For a system having W discrete microstates, each having probability $p_i$ ($i=1,2,\dots ,W)$, the Boltzmann–Shannon (BS) entropy $S$ is given by

$$S=-{\sum }_{i=1}^W{p}_i\mathrm{l}\mathrm{n}{p}_i$$

(1)

(We suppose here that the Boltzmann constant $k_B=1$). This information measure is always positive because ${1\ge p}_i\ge 0$.

When the variable $x$ of the states becomes continuous, entropy is sometimes called differential or continuous entropy given by [2,3,4,8]:

$$S=-\int \rho \left(x\right)\mathrm{l}\mathrm{n}\rho \left(x\right)dx,$$

(2)

where $\rho \left(x\right)$ is the probability density distribution giving the probability $dp\left(x\right)=\rho \left(x\right)dx$ of finding the system in the state interval between $x$ and $x+dx$, across the range of all possible states. The first use of this integral form dates back to Boltzmann [1]. Gibbs mentioned this formula in his book [3]. Shannon [2] as well intuitively took Equation (2) for granted as an analogue of Equation (1) without giving a mathematical derivation or proof of its uniqueness. As far as we know, an axiomatic derivation of Equation (2), as was done for Equation (1) by Shannon [2] and Khinchin [7], is still missing to date.

In this work, we focus on one specific mathematical property of Equation (2) concerning its sign. As mentioned above, in the case of Equation (1) with a discrete probability distribution, since $p_i$ is positive and smaller than 1, $\mathrm{l}\mathrm{n}{p}_i\le 0$, which guarantees $S\ge 0$ [2]. However, in the case of a continuous probability distribution as in Equation (2), $\rho \left(x\right)$ can be larger than 1, leading to the possibility that $\mathrm{l}\mathrm{n}\rho \left(x\right)>0$ and, therefore, to negative informational entropy. The reader can see a list of continuous entropies calculated from Equation (2) for many probability density distributions, in which most entropies can be negative. The list includes some common distributions such as uniform, normal and exponential distributions [8]. This negative entropy problem can occur to many other entropy measures proposed in different contexts as generalization of BS formula (see, for example, [9]). As all these formulas contain adjustable parameters and recover the BS formula when the parameters take specific values, they risk negative values with continuous probability distribution in the same way as Equation (2). For example, the continuous Tsallis entropy $S_q={\displaystyle \frac{1-\int {\rho }^q\left(x\right)dx}{q-1}}$ [10] and the Renyi entropy $R_q={\displaystyle \frac{\mathrm{l}\mathrm{n}\int {\rho }^q\left(x\right)dx}{1-q}}$ [11] are both negative when $\int {\rho }^q\left(x\right)dx>1$, which can happen for $\rho >1$ with $q>1$ and for $\rho <1$ with $q<1$. Most of the generalized continuous entropies containing the integral of $\int {\rho }^q\left(x\right)dx$ (counterpart of ${\sum }_{\mathrm{i}}{p}_i^q$) [12] show negative values.

As is well known, thermodynamic entropy cannot be negative due to the third law of thermodynamics [6]. It may contain a constant in addition to the function term of probability, as proposed by Boltzmann, Gibbs and Shannon [1,2,3]. It is this function, depending only on probability, that assumes the role of entropy as a measure of disorder or probabilistic uncertainty. Logically, this function is expected to be zero in a non-probabilistic situation, as is the case of all the different forms of entropy, generalized or not, for discrete probability distribution; when only one state has $p_i=1$, all other states have $p_{j\ne i}=0$. Hence, the functions of probability of both thermodynamic entropy and informational entropy are expected to have positive lower bounds. But it is not the case with those entropy functions using continuous distribution as mentioned above. Many of them enter into contradiction with the definition of entropy as an uncertainty measure and even lose the finite lower bound. For example, the continuous entropy of the uniform distribution $\rho \left(x\right)={\displaystyle \frac{1}{b-a}}$ in the interval $a\le x\le b$ is $S=\mathrm{l}\mathrm{o}\mathrm{g}(b-a)$, as given by Equation (2) [8,13]. We straightforwardly have $S=0$ if $\mathrm{b}-a=1$, while the probabilistic uncertainty of such a distribution should not be zero. On the other hand, when $b\to a$, $\rho \left(x\right)\to \infty$, which can be considered equivalent to the Dirac delta function (see Section 3.1 below) with the normalization ${\int }_{-\infty }^{\infty }\rho \left(x\right)dx=1$. The Dirac delta function is a continuous counterpart of the discrete probability distribution: $p_a=1$ and all other $p_{x\ne a}=0$, a situation where the probabilistic uncertainty should be zero from Equation (1). However, Equation (2) yields $S=\log \left(b-a\right)\to -\infty$. This loss of lower bound implies that it is impossible to add some constant to S to shift it to a positive domain as proposed by Jaynes (see Section 4 below) [4]. A similar paradox happens with other continuous entropies, for instance $S=1-\log \lambda$ [8] of the exponential distribution $\rho \left(x\right)~e^{-\lambda x}$ ($0\le x\le \infty$).

These undesirable features of continuous entropy raise the question whether Equation (2) is still suitable for measuring probabilistic uncertainty and whether it deserves the name entropy. Despite this uncertain situation, continuous entropies have been considered acceptable [8,13] and widely used in many applications (see [11,14,15], for example) sometimes with their negative values [14]. There are also other solutions to this negativity issue in using relative entropy or Kulback–Leibler divergence instead of Equation (2) [13,14], although this is not a solution to the issue of Equation (2) itself.

As is well known, the use of Equation (2) can be traced back to Boltzmann [1]. It has been proposed rather by intuition than by mathematical proof as has been done for Equation (1) by Shannon [2]. From a mathematical point of view, Equation (1) cannot be simply replaced by Equation (2) because, when $p_i$ is replaced by its continuous counterpart $dp(x)=\rho dx$, a divergent term $-\mathrm{l}\mathrm{n}dx\propto \mathrm{l}\mathrm{n}W$ appears [4,5], which implies that the replacement of $p_i$ by $\rho dx$ in Equation (1) is questionable. Jaynes has tried to avoid negative entropy using a continuous version of BS entropy $S=-{\int }_1^{\mathrm{\infty }}\rho \left(x\right)\mathrm{l}\mathrm{n}{\displaystyle \frac{\rho \left(x\right)}{m\left(x\right)}}dx+\mathrm{l}\mathrm{n}W$ where $m(x)$ is called the invariant measure of the density of discrete values of x [4,5]. The term $\mathrm{l}\mathrm{n}W$, divergent when $W\to \mathrm{\infty }$, was simply removed, giving a continuous entropy $S^c=S+{\int }_1^{\mathrm{\infty }}\rho \left(x\right)\mathrm{l}\mathrm{n}m\left(x\right)dx$. According to Jaynes [4], it is possible to choose the invariant measure $m\left(x\right)$ in an appropriate (albeit ad hoc) way for $S^c$ to be positive.

In what follows, we present an alternative informational entropy called varentropy, which possibly helps to avoid negative value and other undesirable properties of continuous entropy. Varentropy is an extension of the fundamental equation of equilibrium thermodynamics $\delta U=T\delta S+\delta W$, where $\delta U$ is a variation of internal energy $U$ in a reversible process, $\delta S$ a variation of thermodynamic entropy, $T$ the temperature, and $\delta W$ the work done during the quasi-equilibrium equilibrium process [6]. We show that varentropy, when extended to continuous probability distributions, allows us to avoid negative informational entropy in many cases where BS formula yields negative values.

2. Definition of Varentropy

Varentropy was defined on the basis of the variational form of the entropy of the second law of thermodynamics [16,17]. This is a definition from scratch without any prerequisite or postulate about the property of entropy. The motivation was to look for a probabilistic uncertainty measure that has a sound physical background and is maximizable to generate probability distribution. This objective implies that this measure cannot be defined with a given formula because the maximization of a given functional can only yield one or two distributions. One just cannot maximize a given functional for any distribution. For example, the Shannon entropy in Equation (1), which has been widely used as a universal uncertainty measure for any probability distribution, can only be maximized for exponential or uniform distribution, depending on the constraints [4,5] and the Tsallis or Renyi entropy can only be maximized for q-exponential distributions [10,11]. Hence, this measure we looked for should be something more general and fundamental than the formulas of the different entropies [9,12]; more fundamental in that it should have some physical background relative to thermodynamic entropy; more general in that it can reproduce the different entropies for different distributions, more or less in similar ways as a differential equation of physical law generates different trajectories with different interactions, as has been shown in [16,17,18].

The idea of [16] was to start from the first law of equilibrium thermodynamics given by $\delta U=T\delta S+\delta W$. As is well known, in classical statistical mechanics, the internal energy is the average $\overline{E}$ of the energy $E_i$ of all microstates i, i.e., $U=\overline{E}={\displaystyle {\sum }_{i=1}^W}{p}_i{E}_i$, and the work done in an infinitesimal reversible process is given by the average of the energy change ${\delta E}_i$ of each state: $\delta W=\overline{\delta E}={\displaystyle {\sum }_{i=1}^W}{p}_i{\delta E}_i$, where $p_i=f(E_i)$ is the probability of finding the system at the state $i=\mathrm{1,2},\dots ,W$ having energy $E_i$. The statistical expression of the fundamental equation $\delta U=T\delta S+\delta W$ becomes then $\delta {\displaystyle {\sum }_{i=1}^W}{p}_i{E}_i=T\delta S+{\displaystyle {\sum }_{i=1}^W}{p}_i{\delta E}_i$, giving a statistical expression of the variation of entropy during the process $\delta S={\displaystyle \frac{1}{T}}\left(\delta {\displaystyle {\sum }_{i=1}^W}{p}_i{E}_i-{\displaystyle {\sum }_{i=1}^W}{p}_i{\delta E}_i\right)={\displaystyle \frac{1}{T}}\left(\delta \overline{E}-\overline{\delta E}\right)$, giving $\delta S={\displaystyle \frac{1}{T}}{\displaystyle {\sum }_{i=1}^W}\delta p_i{E}_i$. From the above calculation, we see that this variational expression of thermodynamic entropy as a function of $\delta p_i$ multiplied by the random variable energy $E_i$ is a statistical form of the first law. For us, this expression reveals a general kinship between a measure of probabilistic uncertainty (S) and the related random variable ($E_i$) with its probability distribution $p_i$. This kinship was deeply hidden in the first and second law of thermodynamics and can only be seen when these laws are expressed in statistical form as shown above. In a previous work [10], we have extended this relationship to any single random variable $x_i$ with its probability distribution $p_i=f(x_i)$ and defined varentropy $S_V$ in the variational form: $\delta S_V=A{\displaystyle {\sum }_{i=1}^W}{x}_i\delta p_i$. This extension is purely mathematical; $x_i$ can be any random variable (frequency of words, price, population, position, etc.). $S_V$ measures the uncertainty in $x_i$, and its functional form and property (extensivity, additivity, concavity, etc.) are entirely determined by $x_i$ and $p_i$. $S_V$ is the thermodynamic entropy only when $x_i$ is energy $E_i$.

It turns out that this variational definition of varentropy could really generate many known entropies with discrete distributions. For example, $S_V$ has the form of the Boltzmann–Shannon entropy Equation (1) when $p_i$ is exponential distribution, it is the nonadditive Tsallis or additive Renyi entropy if $p_i$ is q-exponential distribution, and can take different forms for other distributions (Power law, stretched exponential, Cauchy, Gauss, etc.), and $S_V=-{\displaystyle \frac{1-{\sum }_i{p}_i^{1-\frac{1}{\alpha }}}{1-\frac{1}{\alpha }}}$ (nonadditive) for power law for example [16,17,18,19]. A remarkable property of varentropy is that each functional form of $S_V$ can be maximized using the calculus of variation $\delta \left(S_V-A\overline{x}\right)=0$ to generate its original distribution. For instance, maximizing Equation (1) generates exponential distribution, maximizing Tsallis entropy generates q-exponential, and maximizing the power law varentropy $S_V=-{\displaystyle \frac{1-{\sum }_i{p}_i^{1-\frac{1}{\alpha }}}{1-\frac{1}{\alpha }}}$ generates the power law $p_i\propto x_i^{-\alpha }$ ($\alpha$ positive) [16]. This property of varentropy is the meaning of the statement “maximizable measure” and is an intrinsic nature of varentropy, because its origin is in the variational definition $\delta S={\displaystyle \frac{1}{T}}{\displaystyle {\sum }_{i=1}^W}\delta p_i{E}_i$, which is equivalent to writing $\delta S_V=A\left(\delta \overline{x}-\overline{\delta x}\right)$ or $\delta \left(S_V-A\overline{x}\right)=-A\overline{\delta x}$. This means that the maximization of $S_V$ subject to the constraint of the constant $\overline{x}$, i.e., $\delta \left(S_V-A\overline{x}\right)=0$, implies $\overline{\delta x}=0$. This expression can be better understood if we consider the thermodynamic case where $x_i$ is energy $E_i$ and $\overline{\delta x}=\overline{\delta E}={\displaystyle {\sum }_{i=1}^W}{p}_i{\delta E}_i=\delta W$, which is the work in the variational process considered as the virtual work. In this way, $\delta W=0$ can be understood as prescribed by the principle of virtual work and yields the maximum entropy calculus (Maxent principle) $\delta \left(S_V-A\overline{x}\right)=0$. This kinship between two fundamental principles was studied in [20] with the conclusion that, in the case of thermodynamic entropy, maxent with the calculus $\delta \left(S-A\overline{E}\right)=0$ can be considered as a law of physics derived from the fundamental principle of virtual work.

In what follows, we extend this varentropy to continuous probability replacing $p_i=f(x_i)$ by $dp(x)=\rho (x)dx$. Considering $\overline{x}=\int x\rho \left(x\right)dx$ and $\overline{\delta x}=\int \delta x\rho \left(x\right)dx$, we arrive at

$$\delta S_V=A\left(\delta \overline{x}-\overline{\delta x}\right)=A\int x\delta \rho \left(x\right)dx,$$

(3)

where $A\in \mathbb{R}$ is a constant to be chosen according to the nature of $S_V$. For example, in a reversible thermodynamic process where $x=E$, $S_V$ is the thermodynamic entropy with $A={\displaystyle \frac{1}{T}}$, as shown above; for the exponential decay distribution, we can choose $A=1$ or $A=-1$, depending on the domain of $x$ in the considered distribution (see below). It should be noticed in the above discussion of maximization of varentropy that A also plays the role of Lagrange multiplier in the maximum varentropy to generate probability distribution. It is then natural to see different A for different varentropies and distributions due to the different physics or processes. It is worth stressing that an important role of $A$ is to guarantee $S_V$ be positive, as required both for entropy and for any measure of statistical uncertainty, which is the main aim of this work.

Suppose $f$ is bijective in $\left[a,b\right]$, so we can write $x=f^{-1}(\rho )$, Equation (3) becomes $\delta S_V=A{\int }_a^b{f}^{-1}\left[\rho \left(x\right)\right]\delta \rho \left(x\right)dx$. Then we define the function of the upper bound as follows: $\forall x\in \left[a,b\right],F[\rho (x)]={\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt$. According to the second fundamental theorem of calculus, $F$ is differentiable with respect $\rho (x)$ $\forall x\in \left[a,b\right]:{\displaystyle \frac{d}{d\rho \left(x\right)}}F[\rho (x)]=f^{-1}[\rho (x)]$, so that we can write $\delta S_V=A{\int }_a^b\left[{\displaystyle \frac{d}{d\rho (x)}}{\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt\right]\delta \rho \left(x\right)dx=A{\int }_a^b\left[\delta {\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt\right]dx=\delta \left\{A{\int }_a^b\left[{\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt\right]dx+C\right\}$, which implies

$$S_V=A{\int }_a^b\left[{\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt\right]dx+C,$$

(4)

where $C$ is a constant.

If $f$ is not bijective in $\left[a,b\right]$, we can cut the domain $\left[a,b\right]$ into w sub-domains $\left[a_k,a_{k+1}\right]$ with $k=\mathrm{0,1},2\dots w-1$ ($a_0=a,a_w=b$) in such a way that, in each sub-domain k, $f(x)$ is bijective and a local varentropy $S_V^{(k)}$ can be calculated using $S_V^{(k)}=A_k{\int }_{a_k}^{a_{k+1}}\left[{\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt\right]dx$. The total varentropy $S_V$ is then the sum $S_V=\sum _k{S}_V^{(k)}$ [19] or:

$$S_V={\sum }_{k=0}^{w-1}{A}_k{\int }_{a_k}^{a_{k+1}}\left[{\int }_0^{\rho (x)}{f}^{-1}\left(t\right)dt\right]dx+C.$$

(5)

If $f(x)$ is not bijective in $\left[a,b\right]$ but is differentiable, one can use the relationships $\delta \rho \left(x\right)=f^{\prime }\left(x\right)\delta x$ and $x{f}^{\prime }\left(x\right)\delta x=\delta {\int }_0^{x}t{f}^{\prime }\left(t\right)dt$ to change the definition of varentropy in the following way: $\delta S_V=A\int x\delta \rho \left(x\right)dx=A{\int }_a^{b}x{f}^{\prime }(x)\delta xdx={\int }_a^b\left[\delta {\int }_0^{x}t{f}^{\prime }\left(t\right)dt\right]dx$, which implies

$$S_V=A{\int }_a^b\left[{\int }_0^{x}t{f}^{\prime }\left(t\right)dt\right]dx+C,$$

(6)

which can be used for any differentiable distribution.

3. Examples of Varentropy

3.1. Uniform Distribution

The uniform distribution is given by $\mathsf{\rho }\left(\mathrm{x}\right)=\mathrm{f}(\mathrm{x})=\left\{\begin{array}{c}{\displaystyle \frac{1}{\mathrm{b}-\mathrm{a}}},\mathrm{a}\le \mathrm{x}\le \mathrm{b}\\ 0,\mathrm{x}<\mathrm{a}\mathrm{or}\mathrm{x}>\mathrm{b}\end{array}\right.$, which has the BS entropy given by $\mathrm{S}=\mathrm{l}\mathrm{n}(\mathrm{b}-\mathrm{a})$. This entropy is zero when $\mathrm{b}-\mathrm{a}=1$, becomes negative in the interval $\mathrm{b}-\mathrm{a}<1$ and tends to negative infinity when $\mathrm{b}\to \mathrm{a}$ instead of zero as expected for a deterministic case [13]. $\mathrm{S}=\mathrm{l}\mathrm{n}(\mathrm{b}-\mathrm{a})$ has another issue related to the scale of $\mathrm{x}$. Suppose $\mathrm{x}$ is a coordinate distance (idem in following discussion), logarithm of a dimensioned quantity is improper, leading to different values of entropy over the same distance but measured in different units (for example, S = 0 over 1 m, but $S=\ln 100\approx 4.61$ over 100 cm and $S<0$ over 0.001 km). The origin of this paradox is the improper use of logarithm $\mathrm{l}\mathrm{n}\rho \left(x\right)$ in Equation (2) since $\rho \left(x\right)$ has a dimension $m^{-1}$.

For calculating varentropy of this distribution, let’s consider a linear distribution between a and b: $\mathsf{\rho }\left(\mathrm{x}\right)=\mathrm{f}(\mathrm{x})=\left\{\begin{array}{c}{\displaystyle \frac{1}{\mathrm{Z}}}\left(\mathrm{s}(\mathrm{x}-\mathrm{a})+{\displaystyle \frac{1}{\mathrm{b}-\mathrm{a}}}\right),\mathrm{a}\le \mathrm{x}\le \mathrm{b}\\ 0,\mathrm{x}<\mathrm{a}\mathrm{or}\mathrm{x}>\mathrm{b}\end{array}\right.$ with a positive slope s, where normalization constant $\mathrm{Z}={\int }_{\mathrm{a}}^{\mathrm{b}}\left(\mathrm{s}(\mathrm{x}-\mathrm{a})+{\displaystyle \frac{1}{\mathrm{b}-\mathrm{a}}}\right)\mathrm{d}\mathrm{x}={\left[{\displaystyle \frac{\mathrm{s}}{2}}{(\mathrm{x}-\mathrm{a})}^2+{\displaystyle \frac{\mathrm{x}}{\mathrm{b}-\mathrm{a}}}\right]}_{\mathrm{a}}^{\mathrm{b}}={\displaystyle \frac{\mathrm{s}}{2}}{(\mathrm{b}-\mathrm{a})}^2+1$. For simplicity, we use Equation (6) where ${\mathrm{f}}^{\prime }\left(\mathrm{x}\right)={\displaystyle \frac{s}{Z}},$ leading to

$${\mathrm{S}}_{\mathrm{V}}=\mathrm{A}{\int }_{\mathrm{a}}^{\mathrm{b}}\left[{\int }_0^{\mathrm{x}}\mathrm{t}{\mathrm{f}}^{\prime }\left(\mathrm{t}\right)\mathrm{d}\mathrm{t}\right]\mathrm{d}\mathrm{x}+\mathrm{C}=\mathrm{A}{\int }_{\mathrm{a}}^{\mathrm{b}}\left[{\displaystyle \frac{\mathrm{s}}{\mathrm{Z}}}{\int }_0^{\mathrm{x}}\mathrm{t}\mathrm{d}\mathrm{t}\right]\mathrm{d}\mathrm{x}={\displaystyle \frac{\mathrm{s}\mathrm{A}}{6\mathrm{Z}}}\left({\mathrm{b}}^3-{\mathrm{a}}^3\right)+\mathrm{C}.$$

This expression of ${\mathrm{S}}_{\mathrm{V}}$ does not have the dimensional problem since slope s has the dimension $m^{-2}$ and A has the dimension $m^{-1}$ by definition in Equation (3), yielding dimensionless ${\mathrm{S}}_{\mathrm{V}}$. As A is an arbitrary constant, let A = 1/s and C = 0, and we obtain ${\mathrm{S}}_{\mathrm{V}}={\displaystyle \frac{1}{6\mathrm{Z}}}\left({\mathrm{b}}^3-{\mathrm{a}}^3\right)$, which becomes the varentropy of the uniform distribution when $\mathrm{s}\to 0$ ($\mathrm{Z}\to 1)$:

$${\mathrm{S}}_{\mathrm{V}}={\displaystyle \frac{1}{6}}\left({\mathrm{b}}^3-{\mathrm{a}}^3\right),$$

which is always positive. ${\mathrm{S}}_{\mathrm{V}}\to 0$ when $\mathrm{b}\to \mathrm{a}$, which is natural for this non-probabilistic case (no uncertainty) where $\mathsf{\rho }\left(\mathrm{x}\right)$ is a Dirac delta function at $\mathrm{x}=\mathrm{a}$, i.e., $\mathsf{\rho }(\mathrm{x}-\mathrm{a})=\left\{\begin{array}{c}\infty \mathrm{if}\mathrm{x}=\mathrm{a}\\ 0\mathrm{if}\mathrm{x}\ne \mathrm{a}\end{array}\right.$ with the normalization ${\int }_{\mathrm{a}}^{\mathrm{b}}\mathsf{\rho }(\mathrm{x}-\mathrm{a})\mathrm{d}\mathrm{x}=1$. This expression of ${\mathrm{S}}_{\mathrm{V}}$ can also be derived, with a bit longer computation, from Equation (4), which we will use for the following bijective distribution functions.

3.2. Exponential Distributions

Exponential distribution is given by $\mathsf{\rho }\left(\mathrm{x}\right)={\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-\mathsf{\alpha }\mathrm{x}}$ for $0<\mathrm{x}<\mathrm{\infty }$ and $\mathsf{\alpha }>0$, $\mathrm{Z}$ being the normalization constant. Its BS entropy has been calculated and reads $\mathrm{S}=1-\mathrm{l}\mathrm{n}\mathsf{\alpha }$ [6], which is inevitably negative when $\mathsf{\alpha }$ is sufficiently large.

Now we calculate varentropy with Equation (4) using ${\mathrm{f}}^{-1}\left(\mathrm{t}\right)={\displaystyle \frac{\ln (\mathrm{Z}\mathrm{t})}{-\mathsf{\alpha }}}$. It is straightforward to calculate ${\int }_0^{\mathsf{\rho }(\mathrm{x})}{\displaystyle \frac{\ln (\mathrm{Z}\mathrm{t})}{-\mathsf{\alpha }}}\mathrm{d}\mathrm{t}={\displaystyle \frac{\mathsf{\rho }(\mathrm{x})\ln (\mathrm{Z}\mathsf{\rho }\left(\mathrm{x}\right))-\mathsf{\rho }(\mathrm{x})}{-\mathsf{\alpha }}}$, so ${\mathrm{S}}_{\mathrm{V}}=-{\displaystyle \frac{\mathrm{A}}{\mathsf{\alpha }}}{\int }_0^{\mathrm{\infty }}\left[\mathsf{\rho }(\mathrm{x})\ln (\mathrm{Z}\mathsf{\rho }\left(\mathrm{x}\right))-\mathsf{\rho }(\mathrm{x})\right]\mathrm{d}\mathrm{x}+\mathrm{C}$. Let $\mathrm{C}=-{\displaystyle \frac{\mathrm{A}}{\mathsf{\alpha }}}{\int }_0^{\mathrm{\infty }}\mathsf{\rho }\left(\mathrm{x}\right)\mathrm{d}\mathrm{x}=-{\displaystyle \frac{\mathrm{A}}{\mathsf{\alpha }}}$ and $\mathrm{A}=1$, and we get

$${\mathrm{S}}_{\mathrm{V}}=-{\displaystyle \frac{1}{\mathsf{\alpha }}}{\int }_0^{\mathrm{\infty }}\mathsf{\rho }\ln \mathrm{Z}\mathsf{\rho }\mathrm{d}\mathrm{x},$$

which is different from Equation (2). There is no problem of improper use of logarithm because $\mathrm{Z}\mathsf{\rho }$ is dimensionless. The positive value of this expression can be seen by substituting $\mathsf{\rho }\left(\mathrm{x}\right)={\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-\mathsf{\alpha }\mathrm{x}}$ into the integral to obtain ${\mathrm{S}}_{\mathrm{V}}=-{\displaystyle \frac{1}{\mathsf{\alpha }}}{\int }_0^{\mathrm{\infty }}{\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-\mathsf{\alpha }\mathrm{x}}\left(-\mathsf{\alpha }\mathrm{x}\right)\mathrm{d}\mathrm{x}={\displaystyle \frac{1}{\mathsf{\alpha }}}>0$.

In order to show the role of the constant A to guarantee positive ${\mathrm{S}}_{\mathrm{V}}$, let us suppose an increasing exponential distribution $\mathsf{\rho }\left(\mathrm{x}\right)={\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-\mathsf{\alpha }\mathrm{x}}$ for $-\mathrm{\infty }<\mathrm{x}\le 0$ with $\mathsf{\alpha }<0$. After the same calculation as above, we reach ${\mathrm{S}}_{\mathrm{V}}={\displaystyle \frac{\mathrm{A}}{\mathsf{\alpha }}}$. As $\mathsf{\alpha }<0$, we can choose $\mathrm{A}=-1$ to have a positive entropy ${\mathrm{S}}_{\mathrm{V}}=-{\displaystyle \frac{1}{\mathsf{\alpha }}}$.

3.3. Stretched Exponential Distribution

We consider the continuous stretched exponential distribution $\rho \left(x\right)={\displaystyle \frac{1}{Z}}{e}^{-x^{\beta }}$ for positive $0<\mathrm{x}<\mathrm{\infty }$ and $\beta >0$, $Z$ being the normalization constant $Z={\int }_0^{\infty }{e}^{-x^{\beta }}dx$. Let us first calculate its entropy $S_{BS}$:

$$\mathrm{S}=-{\int }_0^{\mathrm{\infty }}{\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-{\mathrm{x}}^{\mathsf{\beta }}}\ln \left({\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-{\mathrm{x}}^{\mathsf{\beta }}}\right)\mathrm{d}\mathrm{x}={\displaystyle \frac{\ln \mathrm{Z}}{\mathrm{Z}}}{\int }_0^{\mathrm{\infty }}{\mathrm{e}}^{-{\mathrm{x}}^{\mathsf{\beta }}}\mathrm{d}\mathrm{x}+{\displaystyle \frac{1}{\mathrm{Z}}}{\int }_0^{\mathrm{\infty }}{\mathrm{x}}^{\mathsf{\beta }}{\mathrm{e}}^{-{\mathrm{x}}^{\mathsf{\beta }}}\mathrm{d}\mathrm{x}.$$

Considering the change in variable $t=x^{\beta }$ and the definition of Gamma function, we obtain $S={\displaystyle \frac{\ln Z}{Z\beta }}\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)+{\displaystyle \frac{1}{Z\beta }}\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}+1\right)$, which becomes, with the equality $\mathsf{\Gamma }\left(x+1\right)=x\mathsf{\Gamma }\left(x\right)$, $S={\displaystyle \frac{1}{Z\beta }}\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)[\ln Z+{\displaystyle \frac{1}{\beta }}]$. Considering the normalization constant $Z={\displaystyle \frac{1}{\beta }}\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)$, we have:

$$S={\displaystyle \frac{1}{\beta }}-\ln \beta +\ln \left(\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)\right),$$

which is negative whenever $\ln \beta >{\displaystyle \frac{1}{\beta }}+\ln (\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right))$.

Now let us see the varentropy of the continuous stretched exponential distribution ${\delta S}_V=A{\int }_0^{\infty }x\delta \rho \left(x\right)dx$. Introducing the change in variable $\mathrm{x}={\mathrm{t}}^{{\displaystyle \frac{1}{\mathsf{\beta }}}},$ we have $\mathsf{\delta }\mathsf{\rho }\left(\mathrm{x}\right)=\mathsf{\delta }\left({\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-\mathrm{t}}\right)=-{\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{e}}^{-\mathrm{t}}\mathsf{\delta }\mathrm{t}$ and ${\mathsf{\delta }\mathrm{S}}_{\mathrm{V}}=-{\displaystyle \frac{\mathrm{A}}{\mathrm{Z}}}{\int }_0^{\mathrm{\infty }}{\mathrm{t}}^{{\displaystyle \frac{1}{\mathsf{\beta }}}}{\mathrm{e}}^{-\mathrm{t}}\mathsf{\delta }\mathrm{t}\mathrm{d}\mathrm{x}=-{\displaystyle \frac{\mathrm{A}}{\mathrm{Z}}}{\int }_0^{\mathrm{\infty }}\mathsf{\delta }\left[{\int }_0^{\mathrm{t}}{\mathsf{\tau }}^{{\displaystyle \frac{1}{\mathsf{\beta }}}}{\mathrm{e}}^{-\mathsf{\tau }}\mathrm{d}\mathsf{\tau }+\mathrm{C}\right]\mathrm{d}\mathrm{x}=-{\displaystyle \frac{\mathrm{A}}{\mathrm{Z}}}\mathsf{\delta }\left[{\int }_0^{\mathrm{\infty }}\mathsf{\gamma }\left({\displaystyle \frac{1}{\mathsf{\beta }}}+1,\mathrm{t}\right)\mathrm{d}\mathrm{x}+\mathrm{C}\right]$, where $\mathrm{C}$ is a constant of integration and $\mathsf{\gamma }\left({\displaystyle \frac{1}{\mathsf{\beta }}}+1,\mathrm{t}\right)={\int }_0^{\mathrm{t}}{\mathsf{\tau }}^{{\displaystyle \frac{1}{\mathsf{\beta }}}}{\mathrm{e}}^{-\mathsf{\tau }}\mathrm{d}\mathsf{\tau }$ is the lower incomplete gamma function. As $\mathsf{\rho }\left(\mathrm{x}\right)$ is bijective, $t=x^{\beta }=\ln {\displaystyle \frac{1}{Z\rho (x)}}$, the varentropy reads

$$S_V=-{\displaystyle \frac{A}{Z}}\left[{\int }_0^{\infty }\gamma \left({\displaystyle \frac{1}{\beta }}+1,\ln {\displaystyle \frac{1}{Z\rho \left(x\right)}}\right)dx+C\right].$$

Let us choose $\mathrm{A}=-1$ and $C=0$, the stretched exponential varentropy reads

$$S_V={\displaystyle \frac{\beta }{\mathsf{\Gamma }\left(\frac{1}{\beta }\right)}}{\int }_0^{\infty }\gamma \left({\displaystyle \frac{1}{\beta }}+1,x^{\beta }\right)dx,$$

where we used $Z={\displaystyle \frac{1}{\beta }}\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)$, and ${\mathrm{S}}_{\mathrm{V}}\ge 0$ because $\beta >0$, $\mathsf{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)>0$, and $\gamma \left({\displaystyle \frac{1}{\beta }}+1,x^{\beta }\right)$> 0.

3.4. Continuous Normal Distribution

We consider the continuous normal distribution $\rho \left(x\right)={\displaystyle \frac{1}{Z}}{e}^{-{\displaystyle \frac{({x-\mu )}^2}{2{\sigma }^2}}}$ for any $x\in R$, where $\mathrm{Z}=\mathsf{\sigma }\sqrt{2\mathsf{\pi }}$ is the normalization constant, and $\mathsf{\mu }$ is the mean. Its BS entropy has been calculated and reads $\mathrm{S}=\mathit{ln}(\sigma \sqrt{2\pi e})$ [6], which is negative when $\sigma \sqrt{2\pi e}<1$.

Before calculating its varentropy, let us remind that the normal distribution is not a bijective function. According to Equation (5), we must separate the domain of $x$ into two parts, $-\mathrm{\infty }\le \mathrm{x}-\mathsf{\mu }\le 0$ and $0\le \mathrm{x}-\mathsf{\mu }\le \mathrm{\infty }$, in each of which the normal distribution is bijective. Let $\mathrm{y}=\mathrm{x}-\mathsf{\mu }$ and $\rho \left(y\right)={\displaystyle \frac{1}{Z}}{e}^{-{\displaystyle \frac{y^2}{2{\sigma }^2}}}$, we have ${\mathsf{\delta }\mathrm{S}}_{\mathrm{V}}=\mathrm{A}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}(\mathrm{y}+\mathsf{\mu })\mathsf{\delta }\mathsf{\rho }\left(\mathrm{y}\right)\mathrm{d}\mathrm{y}=\mathrm{A}{\int }_{-\mathrm{\infty }}^0\mathrm{y}\mathsf{\delta }\mathsf{\rho }\left(\mathrm{y}\right)\mathrm{d}\mathrm{y}+\mathrm{A}{\int }_0^{\mathrm{\infty }}\mathrm{y}\mathsf{\delta }\mathsf{\rho }\left(\mathrm{y}\right)\mathrm{d}\mathrm{y}+\mathrm{A}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\mathsf{\mu }\mathsf{\delta }\mathsf{\rho }\left(\mathrm{y}\right)\mathrm{d}\mathrm{y}$. The third term should be zero since ${\int }_{-\infty }^{\infty }\mu \delta \rho \left(y\right)=\mu \delta \left(1\right)=0$ due to the normalization ${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\mathsf{\rho }\left(\mathrm{y}\right)=1$.

Now let us make a change in variable $t={\left({\displaystyle \frac{y}{\sqrt{2}\sigma }}\right)}^2\ge 0$ or $y=\pm \sqrt{2}\sigma t^{1/2}$ and write $\rho \left(t\right)={\displaystyle \frac{1}{Z}}{e}^{-t}$. The first term of varentropy is $A{\int }_{-\infty }^{0}y\delta \rho \left(y\right)dy=-A{\int }_{\infty }^0\sqrt{2}\sigma t^{{\displaystyle \frac{1}{2}}}\delta \left({\displaystyle \frac{1}{Z}}{e}^{-t}\right)dy=-{\displaystyle \frac{A}{Z}}\delta {\int }_0^{\infty }\sqrt{2}\sigma \gamma \left({\displaystyle \frac{3}{2}},\ln {\displaystyle \frac{1}{Z\rho \left(y\right)}}\right)dx$ where we used $t=\ln {\displaystyle \frac{1}{Z\rho \left(y\right)}}$. The second term gives the same result as the first one. So ${\delta S}_V=-{\displaystyle \frac{2A}{\sqrt{\pi }}}\delta {\int }_0^{\infty }\gamma \left({\displaystyle \frac{3}{2}},\ln {\displaystyle \frac{1}{\sigma \sqrt{2\pi }\rho \left(x\right)}}\right)dx$, which implies $S_V=-{\displaystyle \frac{2A}{\sqrt{\pi }}}{\int }_0^{\infty }\gamma \left({\displaystyle \frac{3}{2}},\ln {\displaystyle \frac{1}{\sigma \sqrt{2\pi }\rho \left(y\right)}}\right)dy+C$. Let $\mathrm{A}=-1$ and $\mathrm{C}=0$, we get

$${\mathrm{S}}_{\mathrm{V}}={\displaystyle \frac{2}{\sqrt{\mathsf{\pi }}}}{\int }_0^{\mathrm{\infty }}\mathsf{\gamma }\left({\displaystyle \frac{3}{2}},{\displaystyle \frac{{\mathrm{y}}^2}{2{\mathsf{\sigma }}^2}}\right)\mathrm{d}\mathrm{y},$$

which is always positive and independent from $\mu$.

3.5. Power Law Distribution

A typical example of continuous power law is the Pareto law $\rho \left(x\right)={\displaystyle \frac{\beta }{x^{\beta +1}}}$, for $1<\beta <\infty$. The continuous Boltzmann–Gibbs entropy ${\mathrm{S}}_{\mathrm{B}\mathrm{G}}$ is calculated as follows [6]:

$$S=-{\int }_1^{\infty }\rho \left(x\right)\mathrm{l}\mathrm{n}\rho \left(x\right)dx=-\mathrm{l}\mathrm{n}\beta +1+{\displaystyle \frac{1}{\beta }}.$$

The negative value of BS entropy takes place in the interval $\mathrm{l}\mathrm{n}\beta >1+{\displaystyle \frac{1}{\beta }}$ or $\beta >3.59$.

Now let us calculate the varentropy for the power law distribution $\mathsf{\rho }\left(\mathrm{x}\right)={\displaystyle \frac{1}{\mathrm{Z}}}{\mathrm{x}}^{-{\displaystyle \frac{1}{\mathrm{b}}}}$. From the definition of varentropy, we write ${\delta S}_V=A\int {\left(Z\rho \right)}^{-b}\delta \rho dx=A\int \delta \left({\displaystyle \frac{Z^{-b}}{1-b}}{\rho }^{1-b}\right)dx=\delta \{{\displaystyle \frac{A}{1-b}}\int \left({Z^{-b}\rho }^{1-b}-m\right)dx\}$}. Let $A=1$, the continuous varentropy reads

$${\mathrm{S}}_{\mathrm{V}}=\int \mathsf{\rho }{\displaystyle \frac{{(\mathrm{Z}\mathsf{\rho })}^{-\mathrm{b}}-\mathrm{m}}{1-\mathrm{b}}}\mathrm{d}\mathrm{x},$$

where the function m is such that $\int \mathsf{\rho }\mathrm{m}(\mathrm{x})\mathrm{d}\mathrm{x}=\mathrm{C}$ is a constant of the variation, i.e., $\delta C=0$.

For Pareto PDF $\mathsf{\rho }\left(\mathrm{x}\right)={\displaystyle \frac{\mathsf{\beta }}{{\mathrm{x}}^{\mathsf{\beta }+1}}}$ with β $={\displaystyle \frac{1}{\mathrm{b}}}-1$, ${\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}=1$ and ${\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}=\mathrm{\infty }$, we obtain

$${\mathrm{S}}_{\mathrm{V}}={\displaystyle \frac{{\int }_1^{\mathrm{\infty }}{\mathsf{\beta }}^{-(\mathsf{\beta }+1)}{\mathsf{\rho }}^{\mathsf{\beta }/(\mathsf{\beta }+1)}\mathrm{d}\mathrm{x}-\mathrm{C}}{\mathsf{\beta }/(\mathsf{\beta }+1)}}={\displaystyle \frac{\mathsf{\beta }+1}{\mathsf{\beta }}}\left({\displaystyle \frac{\mathsf{\beta }}{\mathsf{\beta }-1}}-\mathrm{C}\right).$$

Let $\mathrm{C}=1$, we obtain

$${\mathrm{S}}_{\mathrm{V}}={\displaystyle \frac{\mathsf{\beta }+1}{\mathsf{\beta }(\mathsf{\beta }-1)}},$$

which is always positive for $1<\mathsf{\beta }<\mathrm{\infty }$ as plotted in Figure 1 as a function of $\mathsf{\beta }$.

4. Discussion

To summarize, despite some undesirable features, the continuous entropies, as a heritage of the long history of statistical mechanics and information theory, continue to be considered as a measure of the uncertainty in continuous random variables, sometimes with the help of relative entropies to avoid negative values [11,13,14,15]. We have proposed here an alternative measure called varentropy to avoid several undesirable features of continuous entropies. Examples of varentropy for several well-known continuous probabilities have been calculated from its variational definition, showing the following features with respect to continuous entropies.

• Varentropy $S_V$ is positive for the distributions studied in this work. The continuous entropies of these distributions can be negative.
• $S_V$ is zero for deterministic case as expected for a measure of probabilistic uncertainty, while continuous entropy goes to minus infinity.
• $S_V$ can avoid the improper use of logarithm $\mathrm{l}\mathrm{n}\rho$ because the probability density distribution is a dimensioned quantity. Other generalized continuous entropies [10,12] containing the term ${\sum }_i{\rho }_i^q$ have the same undesirable feature: loss of scale invariance for example.
• $S_V$ has a sound physical background because it is defined with a variational equation, which is just the statistical form of the first law of thermodynamics. Different from other entropies defined with given functional formulas, it has great flexibility to generate different functionals for different distributions.
• As each given formula of varentropy is maximized for its distribution, it is the optimal measure of the uncertainty of that distribution, meaning that its value is always the largest one among all the possible measures (Shannon, Tsallis, etc.) for the same distribution. A case study of this feature was presented in [19,21].

Further investigation is necessary to confirm these features of varentropy for other continuous distributions.

One should have noticed that, in this work, for varentropy to be positive, a constant A must be chosen according to the kind of distribution function. The existence of such a constant is quite natural in such a general definition of a quantity whose nature differs in different situations. Such a constant is also in the Boltzmann–Shannon formula, written as $S=-A{\sum }_{i=1}^W{p}_i\mathrm{l}\mathrm{n}{p}_i$. For a binary system ($W=2$), the choice of $A=1$ defines the unit of the information measure in bit. When the formula is used for ideal gas to measure the probabilistic uncertainty of the distribution of internal energy, we must write $A=k_B$, the Boltzmann constant, for $S$ to be the thermodynamic entropy of the gas. But it is impossible to choose a single constant $A$ to make continuous entropies always positive because, as discussed above in Section 3, they can be positive and negative for a given distribution, and sometimes go to minus infinity.

Finally, we would like to mention that we need information measures for continuous probability distributions in both classical and quantum physics. An example is the calculation of the path entropy [22,23] of random dynamics using path integral method [24] and the classical path probability [23] or the quantum propagator [24], which are both continuous exponential functions of the classical action [25,26]; hence, the use of the BS formula (or other generalized ones) risks giving negative values. The path entropy is an increasing function of time of random motion [23,25,26] and an important ingredient in the study of the irreversible processes in both classical and quantum world [22].