Information Geometric Duality of ϕ-Deformed Exponential Families

Abstract

In the world of generalized entropies—which, for example, play a role in physical systems with sub- and super-exponential phase space growth per degree of freedom—there are two ways for implementing constraints in the maximum entropy principle: linear and escort constraints. Both appear naturally in different contexts. Linear constraints appear, e.g., in physical systems, when additional information about the system is available through higher moments. Escort distributions appear naturally in the context of multifractals and information geometry. It was shown recently that there exists a fundamental duality that relates both approaches on the basis of the corresponding deformed logarithms (deformed-log duality). Here, we show that there exists another duality that arises in the context of information geometry, relating the Fisher information of $\varphi$-deformed exponential families that correspond to linear constraints (as studied by J.Naudts) to those that are based on escort constraints (as studied by S.-I. Amari). We explicitly demonstrate this information geometric duality for the case of $(c,d)$-entropy, which covers all situations that are compatible with the first three Shannon–Khinchin axioms and that include Shannon, Tsallis, Anteneodo–Plastino entropy, and many more as special cases. Finally, we discuss the relation between the deformed-log duality and the information geometric duality and mention that the escort distributions arising in these two dualities are generally different and only coincide for the case of the Tsallis deformation.

1. Introduction

Entropy is one word for several distinct concepts [1]. It was originally introduced in thermodynamics, then in statistical physics, information theory, and last in the context of statistical inference. One important application of entropy in statistical physics, and in statistical inference in general, is the maximum entropy principle, which allows us to estimate probability distribution functions from limited information sources, i.e., from data [2,3]. The formal concept of entropy was generalized to also account for power laws that occur frequently in complex systems [4]. Literally dozens of generalized entropies have been proposed in various contexts, such as relativity [5], multifractals [6], or black holes [7]; see [8] for an overview. All these generalized entropies, whenever they fulfil the first three Shannon–Khinchin axioms (and violate the composition axiom) are special cases of the $(c,d)$-entropy asymptotically [9]. Generalized entropies play a role in non-multinomial, sub-additive systems (whose phase space volume grows sub-exponentially with the degrees of freedom) [10,11] and in systems, whose phase space grows super-exponentially [12]. All generalized entropies, for sub-, and super-exponential systems, can be treated within a single, unifying framework [13].

With the advent of generalized entropies, depending on context, two types of constraint are used in the maximum entropy principle: traditional linear constraints (typically moments), $⟨E⟩={\sum }_i{p}_i{E}_i$, motivated by physical measurements, and the so-called escort constraints, ${⟨E⟩}_u={\sum }_{i}u\left(p_i\right)E_i/{\sum }_{i}u\left(p_i\right)$, where u is some nonlinear function. Originally, the latter were introduced with multifractals in mind [4]. Different types of constraint arise from different applications of relative entropy. While for physics-related contexts (such as thermodynamics), linear constraints are normally used, in other applications, such as non-linear dynamical systems or information geometry, it might be more natural to consider escort constraints. The question about their correct use and the appropriate form of constraints has caused a heated debate in the past decade [14,15,16,17,18,19]. To introduce escort distributions in the maximum entropy principle in a consistent way, two approaches have been discussed. The first [20] appears in the context of deformed entropies that are motivated by superstatistics [21]. It was later observed in [22] that this approach is linked to other deformed entropies with linear constraints through a fundamental duality (deformed-log duality), such that both entropies lead to the same functional form of MaxEnt distributions. The second way to obtain escort distributions was studied by Amari et al. and is motivated by information geometry and the theory of statistical estimation [23,24]. There, escort distributions represent natural coordinates on a statistical manifold [24,25].

In this paper, we show that there exists another duality relation between this information geometric approach with escort distributions and an approach that uses linear constraints. The relation can be given a precise information geometric meaning on the basis of the Fisher information. We show this within the framework of $\varphi$-deformations [26,27,28]. We establish the duality relation for both cases in the relevant information geometric quantities. As an example, we explicitly show the duality relation for the class of $(c,d)$-exponentials, introduced in [9,10]. Finally, we discuss the relation between the deformed-log duality and the information geometric duality and show that these have fundamental differences. Each type of duality is suitable for different applications. We hope that this paper helps to avoid confusion in the use of escort distributions in the various contexts.

Let us start with reviewing central concepts of (non-deformed) information geometry, in particular relative entropy and its relation to the exponential family of distributions through the maximum entropy principle. Let us consider a probability simplex, $S^n$, with n independent probabilities, $p_i$, and probability $p_0$. Its value is not independent, but determined by the normalization condition, $p_0=1-{\sum }_i{p}_i$. Further, consider a parametric family of distributions $\mathit{p}\left(\mathit{\theta }\right)$ with parameter vector $\mathit{\theta }\in \mathcal{M}$, where $\mathcal{M}$ is a parametric space. In this paper, we focus on probabilities over discrete sample spaces, only. For the continuous case see the mathematical formulation of Pistone and Sempi [29]. For the sake of simplicity, we sometimes do not explicitly write the parameter vector $\theta$ and consider $p_i$ as the independent parameters of the distribution. It is easy to show that the choice of $p_i$ determines the choice of the parameters ${\theta }_i$.

Relative entropy, or Kullback–Leibler divergence, is defined as:

$$D_{KL}(\mathit{p}\parallel \mathit{q})=\sum _i{p}_{i}log\left(p_i/q_i\right).$$

(1)

For the uniform distribution $\mathit{q}={\mathit{u}}_n$, i.e., $q_i=1/n$, we have:

$$D_{KL}(\mathit{p}\parallel {\mathit{u}}_n)=logn-S\left(\mathit{p}\right),$$

(2)

where $S\left(\mathit{p}\right)$ is the Shannon entropy, $S\left(\mathit{p}\right)=-{\sum }_i{p}_{i}log{p}_i$. Let us consider a set of linear constraints, ${\sum }_j{p}_j{E}_{ij}=⟨{\mathit{E}}_i⟩$, and denote the configuration vector as ${\mathit{E}}_i$. Shannon entropy is maximized under this set of linear constraints and the normalization condition, by functions belonging to the exponential family of probability distributions, which can be written as:

$$p_i\left(\mathit{\theta }\right)\equiv p({\mathit{E}}_i;\mathit{\theta })=exp\left(\mathsf{\Psi }\left(\mathit{\theta }\right)+\mathit{\theta }·{\mathit{E}}_i\right)=exp\left(\mathsf{\Psi }\left(\mathit{\theta }\right)+\sum _j{\theta }_j{E}_{ij}\right).$$

(3)

$\mathsf{\Psi }\left(\mathit{\theta }\right)$ guarantees normalization. Fisher information defines the metric on the parametric manifold $\mathcal{M}$ by taking two infinitesimally-separated points, ${\mathit{\theta }}_0$ and $\mathit{\theta }={\mathit{\theta }}_0+\delta \mathit{\theta }$, and by expanding $D_{KL}(\mathit{p}\left({\mathit{\theta }}_0\right)\parallel \mathit{p}\left(\mathit{\theta }\right))$,

$$g_{ij}^F\left(\mathit{\theta }\right)={\left.\frac{{\partial }^2{D}_{KL}(\mathit{p}\left({\mathit{\theta }}_0\right)\parallel \mathit{p}\left(\mathit{\theta }\right))}{\partial {\theta }_i\partial {\theta }_j}\right|}_{\mathit{\theta }={\mathit{\theta }}_0}.$$

(4)

For the exponential family of distributions, it is a well-known fact that Fisher information is equal to the inverse of the probability in Equation (3):

$$g_{ij}^F\left(\mathit{\theta }\right)\equiv g_{ij}^F\left(\mathit{p}\left(\mathit{\theta }\right)\right)=\frac{1}{p_i}{\delta }_{ij}+\frac{1}{p_0}.$$

(5)

2. Deformed Exponential Family

We briefly recall the definition of $\varphi$-deformed logarithms and exponentials as introduced by Naudts [26]. The deformed logarithm is defined as:

$${log}_{\varphi }\left(x\right)={\int }_1^x\mathrm{d}y\frac{1}{\varphi \left(y\right)},$$

(6)

for some positive, strictly-increasing function, $\varphi \left(x\right)$, defined on $(0,+\infty )$. Then, ${log}_{\varphi }$ is an increasing, concave function with ${log}_{\varphi }\left(1\right)=0$. ${log}_{\varphi }\left(x\right)$ is negative on $(0,1)$ and positive on $(1,+\infty )$. Naturally, the derivative of ${log}_{\varphi }\left(x\right)$ is $1/\varphi \left(x\right)$. The inverse function of ${log}_{\varphi }\left(x\right)$ exists; we denote it by ${exp}_{\varphi }\left(x\right)$. Finally, the $\varphi$-exponential family of probability distributions is defined as a generalization of Equation (3):

$$p_i\left(\mathit{\theta }\right)={exp}_{\varphi }\left(\mathsf{\Psi }\left(\mathit{\theta }\right)+\sum _j{\theta }_j{E}_{ij}\right).$$

(7)

We can express $\mathsf{\Psi }\left(\mathit{\theta }\right)$ in the form:

$$\mathsf{\Psi }\left(\mathit{\theta }\right)={log}_{\varphi }\left(p_i\left(\mathit{\theta }\right)\right)-\mathit{\theta }·{\mathit{E}}_i,$$

(8)

which allows us to introduce dual coordinates to $\mathit{\theta }$. This is nothing but the Legendre transform of $\mathsf{\Psi }\left(\mathit{\theta }\right)$, which is defined as:

$$\phi \left(\mathit{\eta }\right)=\mathit{\eta }·\mathit{\theta }-\mathsf{\Psi }\left(\mathit{\theta }\right),$$

(9)

where:

$$\mathit{\eta }=\nabla \mathsf{\Psi }\left(\theta \right).$$

(10)

Because:

$${\partial }_{{\theta }_j}{p}_i\left(\mathit{\theta }\right)={exp}_{\varphi }^{\prime }\left(\mathsf{\Psi }\left(\mathit{\theta }\right)+\mathit{\theta }·{\mathit{E}}_i\right)\left({\partial }_{{\theta }_j}\mathsf{\Psi }\left(\mathit{\theta }\right)+E_{ij}\right)$$

(11)

holds, and using ${\sum }_i{\partial }_{{\theta }_j}{p}_i\left(\mathit{\theta }\right)=0$, we obtain that:

$${\eta }_j=\frac{{\sum }_i{exp}_{\varphi }^{\prime }(\mathsf{\Psi }\left(\mathit{\theta }\right)+\mathit{\theta }·{\mathit{E}}_i)E_{ij}}{{\sum }_i{exp}_{\varphi }^{\prime }(\mathsf{\Psi }\left(\mathit{\theta }\right)+\mathit{\theta }{\mathit{E}}_i)}={\mathit{E}}_{\mathbf{j}}·{\mathbf{P}}^{\varphi },$$

(12)

where ${\mathit{P}}^{\varphi }$ is the so-called escort distribution. With ${exp}_{\varphi }^{\prime }\left({log}_{\varphi }\left(x\right)\right)=\varphi \left(x\right)$, the elements of ${\mathit{P}}^{\varphi }$ are given by:

$$P_j^{\varphi }=\frac{\varphi \left(p_j\right)}{{\sum }_i\varphi \left(p_i\right)}=\frac{\varphi \left(p_j\right)}{h_{\varphi }\left(\mathit{p}\right)},$$

(13)

where we define $h_{\varphi }\left(\mathit{p}\right)\equiv {\sum }_i\varphi \left(p_i\right)$. The Legendre transform provides a connection between the exponential family and the escort family of probability distributions, where the coordinates are obtained in the form of escort distributions. This generalizes the results for the ordinary exponential family of distributions, where the dual coordinates form a mixture family, which can be obtained as the superposition of the original distribution. The importance of dual coordinates in information geometry comes from the existence of a dually-flat geometry for the pair of coordinates. This means that there exist two affine connections with vanishing coefficients (Christoffel symbols). For the exponential family of distributions, the connection determined by the exponential distribution is called e-connection, and the dual connection leading to a mixture family is called m-connection [25]. For more details, see, e.g., [24]. We next look at generalizations of the Kullback–Leibler divergence and the Fisher information for the case of $\varphi$-deformations.

3. Deformed Divergences, Entropies, and Metrics

For the $\varphi$-deformed exponential family of distributions, we have to define the proper generalizations of the relevant quantities, such as the entropy, divergence, and metric. A natural approach is to start with the deformed Kullback–Leibler divergence, denoted by $D_{\varphi }(\mathit{p}\parallel \mathit{q})$. $\varphi$-entropy can then be defined as:

$$S_{\varphi }\left(\mathit{p}\right)\sim -D_{\varphi }(\mathit{p}\parallel {\mathit{u}}_n),$$

(14)

where ∼ means that the relation holds up to a multiplicative constant depending only on n. Similarly, the $\varphi$-deformed Fisher information is:

$$g_{\varphi ,ij}\left(\mathit{\theta }\right)={\left.\frac{{\partial }^2{D}_{\mathit{\varphi }}(p\left({\mathit{\theta }}_0\right)\parallel p\left(\mathit{\theta }\right))}{\partial {\theta }_i\partial {\theta }_j}\right|}_{\mathit{\theta }={\mathit{\theta }}_0}.$$

(15)

There is now more than one way to generalize the ordinary Kullback–Leibler divergence. The first is Csiszár’s divergence: [30]

$$I_f(\mathit{p}\parallel \mathit{q})=\sum _i{q}_{i}f(p_i/q_i),$$

(16)

where f is a convex function. For $f\left(x\right)=xlnx$, we obtain the Kullback–Leibler divergence. Note, however, that the related information geometry based on the generalized Fisher information is trivial, because we have:

$$g_{ij}^f\left(\mathit{p}\right)=f^{″}\left(1\right)g_{ij}^F\left(\mathit{p}\right),$$

(17)

i.e., the rescaled Fisher information metric; see [27]. The second possibility is to use the divergence of Bregman type, usually defined as:

$$D_f(\mathit{p}\parallel \mathit{q})=f\left(\mathit{p}\right)-f\left(\mathit{q}\right)-\nabla f\left(\mathit{q}\right)·(\mathit{p}-\mathit{q}),$$

(18)

where the symbol “·” denotes the scalar product. This type of divergence can be understood as the first-order Taylor expansion of f around $\mathit{q}$, evaluated at $\mathit{p}$. Let us next discuss the two possible types of the Bregman divergence, which naturally correspond to the $\varphi$-deformed family of distributions. For both, the $\varphi$-exponential family of distributions is obtained from the maximum entropy principle of the corresponding $\varphi$-entropy, however, under different constraints. Note that the maximum entropy principle is just a special version of the more general minimal relative entropy principle, which minimizes the divergence functional $D(\mathit{p}\parallel \mathit{q})$ w.r.t. $\mathit{p}$, for some given prior distribution $\mathit{q}$.

3.1. Linear Constraints: Divergence a là Naudts

One generalization of the Kullback–Leibler divergence was introduced by Naudts [26] by considering $f\left(\mathit{p}\right)={\sum }_i\left({\int }_1^{p_i}{log}_{\varphi }\left(x\right)\mathrm{d}x+(1-p_i)\right)$, which leads to:

$$D_{\varphi }^N(\mathit{p}\parallel \mathit{q})=\sum _j{\int }_{q_j}^{p_j}\mathrm{d}x\left({log}_{\varphi }\left(x\right)-{log}_{\varphi }\left(q_j\right)\right).$$

(19)

The corresponding entropy can be expressed as:

$$S_{\varphi }^N\left(\mathit{p}\right)=-\sum _j{\int }_0^{p_j}\mathrm{d}x{ln}_{\varphi }\left(x\right).$$

(20)

$S_{\varphi }^N\left(\mathit{p}\right)$ is maximized by the $\varphi$-exponential family of distributions under linear constraints. The Lagrange functional is:

$${\mathcal{L}}_{\varphi }\left(\mathit{p}\right)=S_{\varphi }^N\left(\mathit{p}\right)-\mathsf{\Psi }\sum _i{p}_i-\sum _j{\theta }_j\sum _i{p}_i{E}_{ij},$$

(21)

which leads to:

$$-{log}_{\varphi }\left(p_i\right)-\mathsf{\Psi }-\sum _j{\theta }_j{E}_{ij}=0,$$

(22)

and we get:

$$\mathsf{\Psi }\left(\mathit{\theta }\right)=-\sum _i{p}_i{log}_{\varphi }\left(p_i\right)-\sum _j{\theta }_j⟨E_j⟩=-⟨{log}_{\varphi }\left(\mathit{p}\right)⟩-\sum _j{\theta }_j⟨E_j⟩,$$

(23)

which is just Equation (8), averaged over the distribution $p_i$. Note that Equation (23) provides the connection to thermodynamics, because $\mathsf{\Psi }\left(\theta \right)$ is a so-called Massieu function. For a canonical ensemble, i.e., one constraint on the average energy $⟨E⟩$, parameter $\theta$ plays the role of an inverse temperature, and $\mathsf{\Psi }$ can be related to the free energy, $F\left(\theta \right)=\theta \mathsf{\Psi }\left(\theta \right)$. Thus, the term $⟨{log}_{\varphi }\left(\mathit{p}\right)⟩$ can be interpreted as the thermodynamic entropy, which is determined from Equation (23). This is a consequence of the Legendre structure of thermodynamics.

The corresponding MaxEnt distribution can be written as:

$$p_i\left(\mathit{\theta }\right)={exp}_{\varphi }\left(-⟨{log}_{\varphi }\left(\mathit{p}\right)⟩-\sum _j{\theta }_j\left(⟨E_j⟩-E_{ij}\right)\right)={exp}_{\varphi }\left(\mathsf{\Psi }\left(\mathit{\theta }\right)+\mathit{\theta }·{\mathit{E}}_i\right).$$

(24)

Finally, the Fisher information metric can be obtained in the following form:

$$g_{\varphi ,ij}^N\left(\mathit{p}\right)={log}_{\varphi }^{\prime }\left(p_i\right){\delta }_{ij}+{log}_{\varphi }^{\prime }\left(p_0\right)=\frac{1}{\varphi \left(p_i\right)}{\delta }_{ij}+\frac{1}{\varphi \left(p_0\right)}.$$

(25)

3.2. Escort Constraints: Divergence a là Amari

Amari et al. [23,24] used a different divergence introduced in [31], which is based on the choice of $f\left(\mathit{p}\right)={\sum }_i{P}_i^{\theta }{log}_{\varphi }\left(p_i\right)$. This choice is motivated by the fact that the corresponding entropy is just the dual function of $\mathsf{\Psi }\left(\mathit{\theta }\right)$, i.e., $\phi \left(\mathit{\eta }\right)$. This is easy to show, because:

$$\phi \left(\mathit{\eta }\right)=\mathit{\eta }·\mathit{\theta }-\mathsf{\Psi }\left(\mathit{\theta }\right)=\sum _j{P}_j^{\varphi }({\theta }_j{E}_{ij}-\mathsf{\Psi }\left(\mathit{\theta }\right))=\sum _j{P}_j^{\varphi }{log}_{\varphi }\left(p_j\right).$$

(26)

Thus, the divergence becomes:

$$D_{\varphi }^A(\mathit{p}\parallel \mathit{q})=\frac{1}{h_{\varphi }\left(\mathit{p}\right)}\sum _j\varphi \left(p_j\right)({log}_{\varphi }\left(p_j\right)-{log}_{\varphi }\left(q_j\right)),$$

(27)

and the corresponding entropy can be expressed from Equation (26) as:

$$S_{\varphi }^A\left(\mathit{p}\right)=-\frac{1}{h_{\varphi }\left(\mathit{p}\right)}\sum _j\varphi \left(p_j\right){log}_{\varphi }\left(p_j\right),$$

(28)

so it is a dual function of $\mathsf{\Psi }\left(\mathit{\theta }\right)$. For this reason, the entropy is called “canonical”, because it is obtained by the Legendre transform from the Massieu function $\mathsf{\Psi }$. Interestingly, the entropy is maximized by the $\varphi$-exponential family of distributions under escort constraints. The Lagrange function is:

$${\mathcal{L}}_{\varphi }\left(\mathit{p}\right)={\mathcal{S}}_{\varphi }^A\left(\mathit{p}\right)-\mathsf{\Psi }\sum _i{p}_i-\sum _j{\theta }_j\sum _i{P}_i^{\varphi }{E}_{ij}.$$

(29)

After a straightforward calculation, we get:

$$\mathsf{\Psi }\left(\mathit{\theta }\right)=-\sum _i\varphi \left(p_i\left(\mathit{\theta }\right)\right),$$

(30)

and the corresponding MaxEnt distribution can be expressed as:

$$p_i\left(\mathit{\theta }\right)={exp}_{\varphi }\left(-{⟨{log}_{\varphi }\left(\mathit{p}\right)⟩}_{\varphi }-\sum _j{\theta }_j\left({⟨E_j⟩}_{\varphi }-E_{ij}\right)\right)={exp}_{\varphi }\left(\mathsf{\Psi }\left(\mathit{\theta }\right)+\mathit{\theta }·{\mathit{E}}_i\right),$$

(31)

where:

$$\mathsf{\Psi }\left(\mathit{\theta }\right)=-{⟨{log}_{\varphi }\left(\mathit{p}\right)⟩}_{\varphi }-\sum _j{\theta }_j{⟨E_j⟩}_{\varphi }.$$

(32)

Here, ${⟨·⟩}_{\varphi }$ denotes the average under the escort probability measure, ${\mathit{P}}^{\varphi }$. Interestingly, in the escort constraints scenario, the “MaxEnt” entropy is the same as the “thermodynamic” entropy in the case of linear constraints. We call this entropy, $S_{\varphi }^A\left(\mathit{p}\right)$, the dual entropy. Finally, one obtains the corresponding metric:

$$g_{\varphi ,ij}^A\left(\mathit{p}\right)=-\frac{1}{h_{\varphi }\left(\mathit{p}\right)}\left(\frac{{log}_{\varphi }^{″}\left(p_i\right)}{{log}_{\varphi }^{\prime }\left(p_i\right)}{\delta }_{ij}+\frac{{log}_{\varphi }^{″}\left(p_0\right)}{{log}_{\varphi }^{\prime }\left(p_0\right)}\right)=\frac{1}{h_{\varphi }\left(\mathit{p}\right)}\left(\frac{{\varphi }^{\prime }\left(p_j\right)}{\varphi \left(p_j\right)}{\delta }_{ij}+\frac{{\varphi }^{\prime }\left(p_0\right)}{\varphi \left(p_0\right)}\right).$$

(33)

Note that the metric can be obtained from $\mathsf{\Psi }\left(\mathit{\theta }\right)$ as: $g_{\varphi ,ij}^A\left(\mathit{\theta }\right)=\frac{{\partial }^2\mathsf{\Psi }\left(\mathit{\theta }\right)}{\partial {\theta }_i\partial {\theta }_j}$, which is the consequence of the Legendre structure of escort coordinates [24]. For a summary of the $\varphi$-deformed divergence, entropy, and metric, see Table 1.

Table 1. $\varphi$-deformation of divergence, entropy, and Fisher information corresponding to the $\varphi$-exponential family under linear and escort constraints. For the ordinary logarithm, $\varphi \left(x\right)=x$, the two entropies become the Shannon entropy, and the divergence is Kullback–Leibler.

	$\mathit{\varphi }$-Deformation	Linear Constraints	Escort Constraints
divergence	$D_{\varphi }(\mathit{p}\parallel \mathit{q})$	${\sum }_j{\int }_{q_j}^{p_j}\mathrm{d}x\left({log}_{\varphi }\left(x\right)-{log}_{\varphi }\left(q_j\right)\right)$	$\frac{{\sum }_j\varphi \left(p_j\right)({log}_{\varphi }\left(p_j\right)-{log}_{\varphi }\left(q_j\right))}{{\sum }_k\varphi \left(p_k\right)}$
entropy	$S_{\varphi }\left(\mathit{p}\right)$	$-{\sum }_i{\int }_0^{p_i}{log}_{\varphi }\left(x\right)\mathrm{d}x$	$-{\sum }_i\varphi \left(p_i\right){log}_{\varphi }\left(p_i\right)/{\sum }_k\varphi \left(p_k\right)$
metric	$g_{ij}^{\varphi }\left(\mathit{p}\right)$	$\frac{1}{\varphi \left(p_i\right)}{\delta }_{ij}+\frac{1}{\varphi \left(p_0\right)}$	$\frac{1}{{\sum }_k\varphi \left(p_k\right)}\left(\frac{{\varphi }^{\prime }\left(p_i\right)}{\varphi \left(p_i\right)}{\delta }_{ij}+\frac{{\varphi }^{\prime }\left(p_0\right)}{\varphi \left(p_0\right)}\right)$

3.3. Cramér–Rao Bound of Naudts Type

One of the important applications of the Fisher metric is the so-called Cramér–Rao bound, which is the lower bound for the variance of an unbiased estimator. The generalization of the Cramér–Rao bound for two families of distributions was given in [26,27]. Assume these two families of distributions to be denoted by $\mathit{p}\left(\mathit{\theta }\right)$ and $\mathit{P}\left(\mathit{\theta }\right)$, with their corresponding expectation values, ${⟨·⟩}_{\mathit{p}\left(\mathit{\theta }\right)}$ and ${⟨·⟩}_{\mathit{P}\left(\mathit{\theta }\right)}$. Let $c_k$ denote the estimator of the family $\mathit{p}\left(\mathit{\theta }\right)$ that fulfills ${⟨c_k⟩}_{\mathit{p}\left(\theta \right)}=\frac{\partial }{\partial {\theta }_k}f\left(\mathit{\theta }\right)$, for some function f, and let us consider a mild regularity condition ${⟨\frac{1}{\mathit{P}\left(\mathit{\theta }\right)}\frac{\partial }{\partial {\theta }_k}\mathit{p}\left(\mathit{\theta }\right)⟩}_{\mathit{P}\left(\mathit{\theta }\right)}=0$. Then:

$$\frac{{⟨c_k{c}_l⟩}_{\mathbf{P}\left(\mathit{\theta }\right)}-{⟨c_k⟩}_{\mathbf{P}\left(\mathit{\theta }\right)}{⟨c_l⟩}_{\mathbf{P}\left(\mathit{\theta }\right)}}{{\left(\frac{{\partial }^{2}f\left(\mathit{\theta }\right)}{\partial {\theta }_k\partial {\theta }_l}\right)}^2}\ge \frac{1}{I_{kl}\left(\mathit{\theta }\right)},$$

(34)

where:

$$I_{kl}\left(\mathit{\theta }\right)=\sum _i\frac{1}{P_i\left(\mathit{\theta }\right)}\frac{\partial p_i\left(\mathit{\theta }\right)}{\partial {\theta }_k}\frac{\partial p_i\left(\mathit{\theta }\right)}{\partial {\theta }_l}.$$

(35)

If $\mathit{p}\left(\mathit{\theta }\right)={\mathit{p}}_{\varphi }\left(\mathit{\theta }\right)$ is the $\varphi$-exponential family of distributions, in Equation (34), equality holds for the escort distribution $\mathit{P}\left(\mathit{\theta }\right)={\mathit{P}}^{\varphi }\left(\mathit{\theta }\right)$, [28]. It is easy to see that for this case, i.e., for the $\varphi$-exponential family and the corresponding escort distribution, the following is true:

$$I_{\varphi ;kl}^N\left(\mathit{p}\right)=h_{\varphi }\left(\mathit{p}\right)g_{\varphi ;kl}^N\left(\mathit{p}\right).$$

(36)

This provides a connection between the Cramér–Rao bound and the $\varphi$-deformed Fisher metric. In the next section, we show that the Cramér–Rao bound can be also estimated for the case of the Fisher metric of the “Amari type”.

4. The Information Geometric “Amari–Naudts” Duality

In the previous section, we have seen that there are at least two natural ways to generalize divergence, such that the $\varphi$-exponential family of distributions maximizes the associated entropy functional, however under different constraint types. These two ways result in two different geometries on the parameter manifold. The relation between the metric $g_{\varphi ,ij}^A$ and $g_{\varphi ,ij}^N$ can be expressed by the operator, T:

$$g_{ij}^A\left(p\right)=T\left(g_{ij}^N\left(p\right)\right),$$

(37)

where

$$T\left(g\left(x\right)\right)=-N_g{\left(logg\left(x\right)\right)}^{\prime },$$

(38)

with the normalization factor, $N_g={\sum }_{i}1/g\left(p_i\right)$. Note that the operator acts locally on the elements of the metric. In order to establish the connection to the Cramér–Rao bound, let us focus on the transformation of $g^A$.

4.1. Cramér–Rao Bound of the Amari Type

The metric of the “Amari case” can be seen as a conformal transformation [32] of the metric that is obtained in the “Naudts case”, for a different deformation of the logarithm. Two metric tensors are connected by a conformal transformation if they have the same form, except for the global conformal factor, $\mathsf{\Omega }\left(p\right)$, which depends only on the point p. Our aim is to connect the Amari metric with the Cramér–Rao bound and obtain another type of bound for the estimates that are based on escort distributions. To this end, let us consider a general metric of the Naudts type, corresponding to $\chi$-deformation, and a metric of the Amari type, corresponding to $\xi$-deformation. They are connected through the conformal transformation, which acts globally on the whole metric. The relation can be expressed as:

$$g_{\chi ,ij}^N\left(\mathit{p}\right)=\mathsf{\Omega }\left(\mathit{p}\right)g_{\xi ,ij}^A\left(\mathit{p}\right).$$

(39)

By using previous results in this relation, we obtain:

$$\frac{1}{\chi \left(p_i\right)}=\frac{\mathsf{\Omega }\left(\mathit{p}\right)}{h_{\xi }\left(\mathit{p}\right)}\frac{{\xi }^{\prime }\left(p_i\right)}{\xi \left(p_i\right)},$$

(40)

from which we see that $\mathsf{\Omega }\left(\mathit{p}\right)=h_{\xi }\left(\mathit{p}\right)$ and ${log}_{\chi }\left(x\right)=log\left(\xi \left(x\right)\right)$, i.e.,

$$\xi \left(x\right)=exp\left({log}_{\chi }\left(x\right)\right)\Rightarrow {log}_{\xi }\left(x\right)={\int }_1^{x}exp(-{log}_{\chi }\left(y\right))\mathrm{d}y.$$

(41)

Note that ${log}_{\chi }$ might not be concave because:

$$\frac{{\mathrm{d}}^2}{\mathrm{d}{x}^2}{log}_{\chi }\left(x\right)=\frac{\xi \left(x\right){\xi }^{″}\left(x\right)-{\xi }^{\prime }{\left(x\right)}^2}{\xi {\left(x\right)}^2}.$$

(42)

Concavity exists, if ${\xi }^{″}\left(x\right)\le \frac{{\xi }^{\prime }{\left(x\right)}^2}{\xi \left(x\right)}$. To now make the connection with the Cramér–Rao bound, let us take $\chi \left(x\right)=\varphi \left(x\right)$, so $\xi \left(x\right)=exp{log}_{\varphi }\left(x\right)$, and:

$$I_{\varphi ;kl}^A\left(\mathit{p}\right)=h_{exp\left({log}_{\varphi }\right)}\left(\mathit{p}\right)g_{exp\left({log}_{\varphi }\right);kl}^A\left(\mathit{p}\right).$$

(43)

As a consequence, there exist two types of Cramér–Rao bound for a given escort distribution, which might be used to estimate the lower bound of the variance of an unbiased estimator, obtained from two types of Fisher information.

4.2. Example: Duality of $(c,d)$-Entropy

We demonstrate the “Amari–Naudts” duality on the general class of $(c,d)$-entropies [9,10], which include all deformations associated with statistical systems that fulfil the first three Shannon–Khinchin axioms. These include most of the popular deformations, including Tsallis q-exponentials [4] and stretched exponentials studied in connection with the entropies by Anteneodo and Plastino [33]. The generalized $(c,d)$-logarithm is defined as:

$${log}_{(c,d)}\left(x\right)=r-r{x}^{c-1}{\left(1-\frac{(1-(1-c\left)r\right)}{dr}logx\right)}^d,$$

(44)

where c and d are the scaling exponents [8,9] and r is a free scale parameter (that does not influence the asymptotic behavior). The associated $\varphi$-deformation is:

$${\varphi }_{(c,d)}\left(x\right)=\frac{x}{r-{log}_{c,d}\left(x\right)}\left(\frac{(-cr+r-1)log\left(x\right)+dr}{(c-1)\left(\right(c-1)r+1)log\left(x\right)+d}\right).$$

(45)

The inverse function of ${log}_{(c,d)}$, the deformed $(c,d)$-exponential, can be expressed in terms of the Lambert W-function, which is the solution of equation, $W\left(z\right)e^{W\left(z\right)}=z$. The deformed $(c,d)$-exponential is:

$${exp}_{(c,d)}\left(x\right)=exp\left(-\frac{d}{1-c}\left[W\left(B{(1-x/r)}^{1/d}\right)-W\left(B\right)\right]\right),$$

(46)

where $B=\frac{(1-c)r}{1-(1-c)r}exp\left(\frac{(1-c)r}{1-(1-c)r}\right)$. The corresponding entropy that is maximized by $(c,d)$-exponentials (see [8] for their properties) is the $(c,d)$-entropy:

$$S_{(c,d)}\left(\mathit{p}\right)=r{A}^{-d}{e}^A\sum _i\Gamma (1+d,A-cln{p}_i)-rc,$$

(47)

where $A=\frac{cdr}{1-(1-c)r}$. This is an entropy of “Naudts type”, since it is maximized with $(c,d)$-exponentials under linear constraints. We can immediately write the metric as:

$$\begin{array}{c}\hfill g_{(c,d),ij}^N\left(\mathit{p}\right)=\left(\frac{r-{log}_{c,d}\left(p_i\right)}{p_i}\left(\frac{(c-1)((c-1)r+1)log\left(p_i\right)+d}{(-cr+r-1)log\left(p_i\right)+dr}\right)\right){\delta }_{ij}\\ \hfill +\frac{r-{log}_{c,d}\left(p_0\right)}{p_0}\left(\frac{(c-1)((c-1)r+1)log\left(p_i\right)+d}{(-cr+r-1)log\left(p_0\right)+dr}\right).\end{array}$$

(48)

The corresponding entropy of “Amari type”, i.e., maximized with $(c,d)$-exponentials under escort constraints:

$$\frac{{\sum }_i{\varphi }_{(c,d)}\left(p_i\right)E_i}{{\sum }_j{\varphi }_{(c,d)}\left(p_j\right)}={⟨E⟩}_{(c,d)},$$

(49)

is:

$$S_{(c,d)}^A\left(\mathit{p}\right)=-\frac{1}{h_{(c,d)}\left(\mathbf{P}\right)}\sum _i\frac{p_i{log}_{c,d}{p}_i}{r-{log}_{c,d}\left(p_i\right)}\left(\frac{(-cr+r-1)log\left(p_i\right)+dr}{(c-1)((c-1)r+1)log\left(p_i\right)+d}\right),$$

(50)

and its metric is:

$$\begin{array}{c}\hfill g_{(c,d),ij}^A\left(\mathit{p}\right)=\frac{1}{p_i}\left(2-c-\frac{(d-1)\left(\right(c-1)r+1)}{((c-1)r+1)log\left(p_i\right)-dr}-\frac{{(c-1)}^{2}r+c-1}{(c-1)dr-cdr+(c-1)((c-1)r+1)log\left(p_i\right)+d+dr}\right){\delta }_{ij}\\ \hfill +\frac{1}{p_0}\left(2-c-\frac{(d-1)\left(\right(c-1)r+1)}{((c-1)r+1)log\left(p_0\right)-dr}-\frac{{(c-1)}^{2}r+c-1}{(c-1)dr-cdr+(c-1)((c-1)r+1)log\left(p_0\right)+d+dr}\right).\end{array}$$

(51)

The metric of the Amari type of the $(c,d)$-entropy was already discussed in [34] based on $(c,d)$-logarithms. However, as demonstrated above, the metric can be found without using the inverse $\varphi$-deformed logarithms, which in the case of $(c,d)$-logarithms lead to Lambert W-functions. The Fisher metric of Naudts and Amari type and the corresponding Cramér–Rao bound are shown in Figure 1. The scaling parameter is set (following [9]) to $r=1/(1-c+cd)$, for $d\le 0$, and $r=exp(-d)/(1-c)$, for $d<0$. The Fisher metric of both types is displayed in Figure 2 as a function of the parameters c and d for a given point, $P=(1/3,2/3)$. We see that both types of metric have a singularity for $(c,d)=(1,0)$. This point corresponds to distributions with compact support. For one-dimensional distributions, the singularity corresponds to the transition between distributions with support on the real line and distributions with support on a finite interval.

Figure 1. Fisher metric for $p=(p,1-p)$ corresponding to various $(c,d)$-deformations ($(c,d)=(1,1)$, $(1,1/2)$, $(1/2,0)$) for (a) the Naudts type, (b) the Amari type, and (c) the Cramér–Rao bound corresponding to the metric.

Figure 2. Fisher metric for $(c,d)$-deformations as a function of c and d of Naudts type (a) and Amari type (b). The metric is evaluated at a point $p=(1/3,2/3)$.

Interestingly, for $(c,d)=(q,0)$, the metric simplifies to:

$$g_{(q,0),ij}^A\left(\mathit{p}\right)=\frac{2-q}{p_i}{\delta }_{ij}+\frac{2-q}{p_0},$$

(52)

which corresponds to the Tsallis q-exponential family of distributions. Therefore, $g_{(q,0),ij}^A\left(p\right)$ is just a conformal transformation of the Fisher information metric for the exponential family of distributions, as shown in [24]. Note, that only for Tsallis q-exponentials, the relation between $S_q^N\left(p\right)$ and $S_q^A\left(p\right)$ can be expressed as (see also Table 2):

$$S_q^A\left(\mathit{p}\right)=f\left(S_{q^{\prime }}^N\left(\mathit{p}\right)\right),$$

(53)

where $f\left(x\right)=(2-q)/x$ and $q^{\prime }=2-q$. This is nothing but the well-known additive duality $q\leftrightarrow 2-q$ of Tsallis entropy [11]. Interestingly, q-escort distributions form a group with ${\varphi }_q\left({\varphi }_{q^{\prime }}\left(x\right)\right)=\left({\varphi }_{q·q^{\prime }}\left(x\right)\right)$ and ${\varphi }_q^{-1}\left(x\right)={\varphi }_{1/q}\left(x\right)$, where $q\leftrightarrow 1/q$ is the multiplicative duality [35].

Table 2. Two important special cases of $(c,d)$-deformations and related quantities: Power laws (Tsallis) and stretched exponentials.

	Tsallis q-Exponential [4]	Stretched $\mathit{\eta }$-Exponential [33]
$\varphi \left(x\right)$	$x^q$	$x\eta log{\left(x\right)}^{1-1/\eta }$
${log}_{\varphi }\left(x\right)$	$\frac{x^{1-q}-1}{1-q}$	$log{\left(x\right)}^{1/\eta }$
${exp}_{\varphi }\left(x\right)$	${\left(1+(1-q)x\right)}^{1/(1-q)}$	$exp\left(x^{\eta }\right)$
${\chi }_{\varphi }\left(x\right)$	$\frac{x}{q}$	$\frac{x\eta log\left(x\right)}{(\eta -1)+\eta log\left(x\right)}$
$S_{\varphi }^N\left(\mathit{p}\right)$	$\frac{1}{q-1}\left({\sum }_i\frac{p_i^{2-q}}{2-q}-1\right)$	${\sum }_i\Gamma \left(\frac{\eta +1}{\eta },-log{p}_i\right)$
$S_{\varphi }^A\left(\mathit{p}\right)$	$\frac{1}{q-1}\left(\frac{1}{{\sum }_i{p}_i^q}-1\right)$	$\frac{{\sum }_i{p}_{i}log{p}_i}{{\sum }_i{p}_i{(log{p}_i)}^{1-1/\eta }}$

This is not the case for more general deformations, because typically, the inverse does not belong to the class of escort distributions. Popular deformations belonging to the $(c,d)$-family, as the Tsallis q-exponential family or the stretched exponential family, are summarized in Table 2.

5. Connection to the Deformed-Log Duality

A different duality of entropies and their associated logarithms under linear and escort averages has been discussed in [22]. There, two approaches were discussed. The first uses the generalized entropy of trace form under linear constraints. It was denoted by:

$$S^{HT}\left(\mathit{p}\right)=\sum _i{s}^{HT}\left(p_i\right)=\sum _i{\int }_0^{p_i}{log}_{}^{HT}\left(x\right)\mathrm{d}x.$$

(54)

It corresponds to the Naudts case, ${log}^{HT}\left(x\right)={log}_{{\varphi }^{HT}}^N\left(x\right)$. The second approach, originally introduced by Tsallis and Souza [20], uses the trace form entropy:

$$S^{TS}\left(\mathit{p}\right)=\sum _i{s}^{TS}\left(p_i\right)=\sum _i{\int }_0^{p_i}{log}^{TS}\left(x\right)\mathrm{d}x.$$

(55)

Note that in [20], the notion of the deformed logarithm is not used (as in Equation (55)). However, it is again an entropy of the Naudts type with the deformed logarithm ${log}^{TS}\left(x\right)={log}_{{\varphi }^{TS}}^N\left(x\right)$. Equation (55) is maximized under the escort constraints:

$${⟨E⟩}_{TS}=\frac{{\sum }_{j}u\left(p_j\right)E_j}{{\sum }_{j}u\left(p_j\right)},$$

(56)

where $u\left(p_j\right)=p_j+\nu s_{TS}\left(p_j\right)$. The linear constraints are recovered for $\nu =0$. This form is dictated by the Shannon–Khinchin axioms, as discussed in the next section. Let us assume that the maximization of both approaches—Equation (54) under linear, and Equation (55) under escort constraints—leads to the same MaxEnt distribution. One can then show that there exists the following duality (deformed-log duality) between ${log}^{HT}\left(x\right)$ and ${log}^{TS}$(x):

$${log}^{TS}\left(x\right)=\frac{1}{\frac{1}{{log}^{HT}\left(x\right)}+\nu }.$$

(57)

Let us focus on specific $\varphi$-deformations, so that ${log}^{HT}\left(x\right)={log}_{\varphi }^{HT}\left(x\right)\equiv {log}_{\varphi }^N\left(x\right)$. Then, ${log}^{TS}\left(x\right)$ is also a $\varphi$-deformation, with:

$${\varphi }^{TS}\left(x\right)={\varphi }^{HT}\left(x\right){\left(1+\nu {log}_{\varphi }^{HT}\left(x\right)\right)}^2={\varphi }^{HT}\left(x\right){\left(1+\nu {\int }_1^x\frac{1}{{\varphi }^{HT}\left(y\right)}\mathrm{d}y\right)}^2.$$

(58)

It is straightforward to calculate the metric corresponding to the entropy $S^{TS}\left(\mathit{p}\right)$:

$$g_{\varphi ,ij}^{TS}\left(\mathit{p}\right)=\frac{1}{{\varphi }^{TS}\left(p_i\right)}{\delta }_{ij}+\frac{1}{{\varphi }^{TS}\left(p_0\right)}={\mathcal{T}}_{\nu }\left(g_{\varphi ,ij}^{HT}\left(\mathit{p}\right)\right),$$

where:

$${\mathcal{T}}_{\nu }\left(g\left(x\right)\right)=g\left(x\right){\left(1+\nu {\int }_1^{x}g\left(y\right)\mathrm{d}y\right)}^2.$$

(59)

Thus, the Tsallis–Souza approach results in yet another information matrix. We may also start from the other direction and look at the situation when the escort distribution for the information geometric approach is the same as the escort distribution for the Tsallis–Souza approach. In this case, we get:

$$\varphi \left(x\right)=x+\nu s^{TS}\left(x\right)=x+{\int }_0^x{\mathcal{T}}_{\nu }\left({log}_{\varphi }\left(y\right)\right)\mathrm{d}y.$$

(60)

We find that the entropy must be expressed as:

$$S_{\varphi }\left(\mathit{p}\right)=\sum _i\frac{\varphi \left(p_i\right)-p_i}{\nu }.$$

(61)

Note that for $\varphi \left(x\right)=x^q$ and $\nu =1-q$, we obtain the Tsallis entropy:

$$S_{\varphi }\left(\mathit{p}\right)=\sum _i\frac{p_i^q-p_i}{1-q},$$

(62)

which corresponds to $S^{TS}\left(\mathit{p}\right)$ for $q^{\prime }=2-q$, which is nothing but the mentioned Tsallis additive duality. It turns out that Tsallis entropy is the only case where the deformed-log duality and the information geometric duality result in the same class of functionals. In general, the two dualities have different escort distributions.

6. Discussion

We discussed the information geometric duality of entropies that are maximized by $\varphi$-exponential distributions under two types of constraint: linear constraints, which are known from contexts such as thermodynamics, and escort constraints, which appear naturally in the theory of statistical estimation and information geometry. This duality implies two different entropy functionals: $S^N\left(\mathit{p}\right)=-{\sum }_i{\int }_0^{p_i}\mathrm{d}x{log}_{\varphi }\left(x\right)$ and $S^A\left(\mathit{p}\right)=-{⟨{log}_{\varphi }\left(P\right)⟩}_{\varphi }$. For $\varphi \left(x\right)=x$, they both boil down to Shannon entropy. The connection between the entropy of Naudts type and the one of Amari type can be established through the corresponding Fisher information through the Cramér–Rao bound. Contrary to the deformed-log duality introduced in [22], the information theoretic duality introduced here cannot be established within the framework of $\varphi$-deformations, since $S^A\left(\mathit{p}\right)$ is not a trace form entropy. We demonstrated the duality between the Naudts approach with linear constraints and the Amari approach with escort constraints, within the example of $(c,d)$-entropies, which include a wide class of popular deformations, including Tsallis and Anteneodo–Plastino entropy as special cases. Finally, we compared the information geometric duality to the deformed-log duality and showed that they are fundamentally different and result in other types of Fisher information.

Let us now discuss the role of information geometric duality and possible applications in information theory and thermodynamics. Recall that the Shannon entropic functional is determined by the four Shannon–Khinchin (SK) axioms. In many contexts, at least three of the axioms should hold:

• (SK1) Entropy is a continuous function of the probabilities $p_i$ only and should not explicitly depend on any other parameters.
• (SK2) Entropy is maximal for the equi-distribution $p_i=1/W$.
• (SK3) Adding a state $W+1$ to a system with $p_{W+1}=0$ does not change the entropy of the system.

Originally, the Shannon–Khinchin axioms contain four axioms. The fourth describes the “composition rule” for entropy of a joint system $S(A+B)=S\left(A\right)+S\left(B\right|A)$). The only entropy satisfying all four SK axioms is Shannon entropy. However, Shannon entropy is not sufficient to describe the statistics of complex systems [10] and can lead to paradoxes in applications in thermodynamics [12]. Therefore, instead of imposing the fourth axiom in situations where it does not apply, it is convenient to consider a weaker requirement, such as generic scaling relations of entropy in the thermodynamic limit [9,13]. It is possible to show that the only type of duality satisfying the first three Shannon–Khinchin axioms is the deformed-log duality of [22]. Moreover, entropies that are neither trace-form, nor sum-form (sum-form entropies are in the form $S\left(\mathit{p}\right)=f\left({\sum }_{i}g\left(p_i\right)\right)$) might be problematic from the view of information theory and coding. For example, it is then not possible to introduce a conditional entropy consistently [36] because the corresponding conditional entropy cannot be properly defined. This is related to the fact that the Kolmogorov definition of conditional probability is not generally valid for escort distributions [37]. Additional issues arise from the theory of statistical estimation, since only entropies of the form $S\left(\mathit{p}\right)=f\left({\sum }_{i}g\left(p_i\right)\right)$, i.e., sum-form entropies, can fulfil the consistency axioms [38]. From this point of view, the deformed-log duality using the class of Tsallis–Souza escort distributions can play a role in thermodynamical applications [39], because the corresponding entropy fulfills the SK axioms. On the other hand, the importance of escort distributions considered by Amari and others is in the realm of information geometry (e.g., dually-flat geometry or generalized Cramér–Rao bound), and their applications in thermodynamics might be limited. Finally, for the case of the Tsallis q-deformation, both dualities, the information geometric and the deformed-log duality, reduce to the well-known additive duality $q\leftrightarrow 2-q$.