Generalized Legendre Transforms Have Roots in Information Geometry ^†

Abstract

Artstein-Avidan and Milman [Annals of mathematics (2009), (169):661–674] characterized invertible reverse-ordering transforms in the space of lower, semi-continuous, extended, real-valued convex functions as affine deformations of the ordinary Legendre transform. In this work, we first prove that all those generalized Legendre transforms of functions correspond to the ordinary Legendre transform of dually corresponding affine-deformed functions. In short, generalized convex conjugates are ordinary convex conjugates of dually affine-deformed functions. Second, we explain how these generalized Legendre transforms can be derived from the dual Hessian structures of information geometry.

1. Introduction

Let $\mathbb{R}$ denote the field of real numbers and $\overline{\mathbb{R}}=\mathbb{R}\cup \{\pm \infty \}$ be the extended real line. A m-variate extended function $F:{\mathbb{R}}^m\to \overline{\mathbb{R}}$ is proper when its efficient domain $\Theta =\mathrm{dom}\left(F\right):=\left\{\theta :F\right(\theta )<+\infty \}$ is non-empty and lower semi-continuous (lsc.) when its epigraph $\mathrm{epi}\left(F\right):=\left\{\right(\theta ,y):y\ge F(\theta ),\theta \in \mathrm{dom}(F\left)\right\}$ is closed with respect to the metric topology of ${\mathbb{R}}^m$. We denote by ${\Gamma }_0$ (shorcut for ${\Gamma }_0\left({\mathbb{R}}^m\right)$) the space of of proper, lower, semi-continuous, extended, real-valued convex functions.

Consider the Legendre–Fenchel transform [1] (LFT) $\mathcal{L}F$ of a function F:

$$\left(\mathcal{L}F\right)\left(\eta \right):=\underset{\theta \in {\mathbb{R}}^m}{sup}\left\{\langle \theta ,\eta \rangle -F\left(\theta \right)\right\},$$

(1)

where $\langle v,v^{\prime }\rangle :={\sum }_{i=1}^m{v}_i{v}_i^{\prime }$ denotes the Euclidean inner product. The function $F^{*}=\mathcal{L}F$ is called the convex conjugate function. Some examples of convex conjugates are reported in Appendix B.

The Legendre transform originated in the work of Adrien-Marie Legendre [2] circa 1805. Legendre considered variational calculus problems and introduced the variable transform. The Legendre transform was later formalized by Hamilton [3] and Gibbs [4] as a way to switch between equivalent descriptions of a system by exchanging variables like position ↔ momentum or energy ↔ entropy. This variable duality principle induced by the Legendre transform turned out to be fundamental in many scientific fields, like thermodynamics [5] for relating thermodynamic potentials, quantum field theory [6] for connecting Lagrangian and Hamiltonian formulations (and underlying effective actions), economics for relating expenditure with utility functions [7], and convex optimization [8], where it is the cornerstone of duality theory in convex analysis.

The Moreau–Fenchel–Rockafellar theorem [9] states that when a function $F\in {\Gamma }_0$, its biconjugate function ${\left(F^{*}\right)}^{*}$ coincides with the function F: ${F^{*}}^{*}=F$. That is, the Legendre transform is an involutive transform of ${\Gamma }_0$. For general lsc. functions F (possibly non-convex), it is known that the biconjugate function ${\left(F^{*}\right)}^{*}:=\mathcal{L}\left(\mathcal{L}F\right)$ lower bounds the function F by the largest possible lsc. convex function ${\left(F^{*}\right)}^{*}\le F$. This lower bound convexification property has been proven useful in machine learning [10].

This study was motivated by the fundamental result of Artstein-Avidan and Milman [11,12], who proved the following theorem:

Theorem 1 

([12], Theorem 7). Let $\mathcal{T}$ be an invertible transform such that the following are true:

• $F_1\le F_2\Rightarrow \mathcal{T}{F}_2\le \mathcal{T}{F}_1$;
• $\mathcal{T}{F}_1\le \mathcal{T}{F}_2\Rightarrow F_2\le F_1$.

Then, $\mathcal{T}$ is a generalized Legendre–Fenchel transform (GLFT), written canonically as

$$\left(\mathcal{T}F\right)\left(\eta \right)=\lambda \left(\mathcal{L}F\right)(E\eta +f)+\langle \eta ,g\rangle +h,$$

(2)

where $\lambda >0$, $E\in \mathrm{GL}\left({\mathbb{R}}^m\right)$ (general linear group), $f,g\in {\mathbb{R}}^m$, and $h\in \mathbb{R}$.

We note in passing that Fenchel [13,14] interpreted the graph of the Legendre transform of an m-variate function $F\in {\Gamma }_0$ as the polarity with respect to the paraboloid surface of ${\mathbb{R}}^{m+1}$

$$\mathcal{P}=\left\{\left(\theta ,y={\displaystyle \frac{1}{2}}\sum _{i=1}^m{\theta }_i^2\right):\theta \in {\mathbb{R}}^m\right\}\in {\mathbb{R}}^{m+1},$$

of the graph of F. This geometric polarity connection of the Legendre transform is all the more interesting since Böröczky and Schneider [15] characterized the duality of convex bodies in Euclidean spaces containing the origin in the interior into the same space [11].

In this work, we shall first prove that a generalized convex conjugate $\mathcal{T}F$ obtained by Equation (2) can always be expressed as the ordinary convex conjugate of a corresponding affine-deformed function. Namely, we shall show that

$$\left(\mathcal{T}F\right)\left(\eta \right)=\mathcal{L}\left(\lambda F(A\theta +b)+\langle \theta ,c\rangle +d)\right),$$

where $A\in \mathrm{GL}\left({\mathbb{R}}^m\right)$, $b,c\in {\mathbb{R}}^m$ and $d\in \mathbb{R}$ are defined according to $E,f,g,h$ (details reported in Theorem 2). This equivalence result allows us to interpret the origin of the generalized Legendre transforms from the viewpoint of information geometry [16,17] and untangle the various degrees of freedom used when defining generalized Legendre transforms in Section 3.

2. Generalized Legendre Transforms as Ordinary Legendre Transforms

Consider a parameter

$$P=(\lambda ,A,b,c,d)\in \mathbb{P}:={\mathbb{R}}_{>0}\times \mathrm{GL}\left({\mathbb{R}}^m\right)\times {\mathbb{R}}^m\times {\mathbb{R}}^m\times \mathbb{R},$$

and deform a function $F\left(\theta \right)$ by carrying out affine transformations of both the parameter argument and its output as follows:

$$F_P\left(\theta \right):=\lambda F(A\theta +b)+\langle \theta ,c\rangle +d.$$

(3)

Those affine deformations preserve convexity:

Property 1 

(Convexity-preserving affine deformations). Let $F\in {\Gamma }_0$. Then, $F_P$ belongs to ${\Gamma }_0$ for all $P\in \mathbb{P}$.

Proof. 

Let us check the convexity of $F_P$:

$$\alpha F_P\left({\theta }_1\right)+(1-\alpha )F_P\left({\theta }_2\right)=\lambda (\alpha F\left({\overline{\theta }}_1\right)+(1-\alpha )F\left({\overline{\theta }}_2\right))+\langle \alpha {\theta }_1+(1-\alpha ){\theta }_2),c\rangle +d,$$

where ${\overline{\theta }}_1:=A{\theta }_1+b$ and ${\overline{\theta }}_2:=A{\theta }_2+b$. Since F is convex, we have $\alpha F\left({\overline{\theta }}_1\right)+(1-\alpha )F\left({\overline{\theta }}_2\right)\le F(\alpha {\overline{\theta }}_1+(1-\alpha ){\overline{\theta }}_2)$. Let ${\overline{\theta }}_{\alpha }^{\prime }:=\alpha {\overline{\theta }}_1+(1-\alpha ){\overline{\theta }}_2=A{\overline{\theta }}_{\alpha }^{\prime }+b$ with ${\theta }_{\alpha }^{\prime }:=\alpha {\theta }_1+(1-\alpha ){\theta }_2$. We obtain

$$\begin{array}{ccc}\hfill \alpha F_P\left({\theta }_1\right)+(1-\alpha )F_P\left({\theta }_2\right)& \le & \underset{=F_P(\alpha {\theta }_1+(1-\alpha ){\theta }_2)}{\underbrace{\lambda F(A{\overline{\theta }}_{\alpha }^{\prime }+b)+\langle {\theta }_{\alpha }^{\prime },c\rangle +d}},\hfill \end{array}$$

hence proving that $F_P$ is convex. Since the lower semi-continuous property is ensured (i.e., the epigraph of $F_P$ is closed), we conclude that $F_P\in {\Gamma }_0$. □

Next, let us express the convex conjugate of a function $F_P$ according to the ordinary convex conjugate $F^{*}=\mathcal{L}F$:

Proposition 1 

(LFT of an affine-deformed function). The Legendre transform of $F_P$ with $P=(\lambda ,A,b,c,d)\in \mathbb{P}$ is

$${\left(F_P\right)}^{*}={\left(F^{*}\right)}_{P^{\diamond }}$$

where

$$P^{\diamond }:=\left(\lambda ,{\displaystyle \frac{1}{\lambda }}{A}^{-1},-{\displaystyle \frac{1}{\lambda }}{A}^{-1}c,-A^{-1}b,\langle b,A^{-1}c\rangle -d\right)\in \mathbb{P}.$$

(4)

That is, we have $\mathcal{L}\left(F_P\right)={\left(\mathcal{L}F\right)}_{P^{\diamond }}$.

Proof. 

Let ${\Gamma }_1\subset {\Gamma }_0$ be defined as the class of differentiable, strictly convex Legendre-type functions [18] (see Appendix A). In general, given a function $F\left(\theta \right)\in {\Gamma }_1$, we proceed as follows to calculate its convex conjugate. First, we find the inverse function ${(\nabla F)}^{-1}$ of its gradient $\nabla F$ to obtain the gradient of the convex conjugate $\nabla F^{*}={(\nabla F)}^{-1}$. Then, we have

$$F^{*}\left(\eta \right)=\langle \theta ,\eta \rangle -F\left(\theta \right)=\langle {(\nabla F)}^{-1}\left(\eta \right),\eta \rangle -F\left({(\nabla F)}^{-1}\left(\eta \right)\right).$$

For parameter $P=(\lambda ,A,b,c,d)$, consider the strictly convex and differentiable function $F_P\left(\theta \right)=\lambda F(A\theta +b)+\langle c,\theta \rangle +d$ for an invertible matrix $A\in \mathrm{GL}(d,\mathbb{R})$, vectors $b,c\in {\mathbb{R}}^d$, and scalars $d\in \mathbb{R}$ and $\lambda \in {\mathbb{R}}_{>0}$. The gradient of $F_P$ is

$$\eta =\nabla F_P\left(\theta \right)=\lambda A^{\top }\nabla F(A\theta +b)+c.$$

We denote by $G=F^{*}$ and $G_P={\left(F_P\right)}^{*}$ the Legendre convex conjugates of F and $F_P$, respectively. By solving the equation $\nabla F_P\left(\theta \right)=\eta$, we find the reciprocal gradient $\theta \left(\eta \right)=\nabla G_P\left(\eta \right)$:

$$\nabla G_P\left(\eta \right)=A^{-1}\nabla G\left({\displaystyle \frac{1}{\lambda }}{A}^{-\top }(\eta -c)\right)-b.$$

Therefore, the Legendre convex conjugate is obtained as follows:

$$\begin{array}{ccc}\hfill G_P\left(\eta \right)& =& 〈\eta ,\nabla G_P\left(\eta \right)〉-F_P(\nabla G_P\left(\eta \right)),\hfill \\ & =& {\lambda }^{\prime }G(A^{\prime }\eta +b^{\prime })+\langle c^{\prime },\eta \rangle +d^{\prime },\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill {\lambda }^{\prime }& =& \lambda ,\hfill \\ \hfill A^{\prime }& =& {\displaystyle \frac{1}{\lambda }}{A}^{-1},\hfill \\ \hfill b^{\prime }& =& -{\displaystyle \frac{1}{\lambda }}{A}^{-1}c,\hfill \\ \hfill c^{\prime }& =& -A^{-1}b,\hfill \\ \hfill d^{\prime }& =& \langle b,A^{-1}c\rangle -d.\hfill \end{array}$$

Hence, it follows that we have $P^{\diamond }=\left(\lambda ,{\displaystyle \frac{1}{\lambda }}{A}^{-1},-{\displaystyle \frac{1}{\lambda }}{A}^{-1}c,-A^{-1}b,\langle b,A^{-1}c\rangle -d\right)\in \mathbb{P}.$ □

Let us check that the (diamond) ⋄ operator on affine deformation parameters is an involution:

Proposition 2 

(⋄ involution). The parameter transformation $P^{\diamond }$ is an involution, where ${\left(P^{\diamond }\right)}^{\diamond }=P$.

Proof. 

Let $P=(\lambda ,A,b,c,d)$ and $P^{\diamond }=({\lambda }^{\prime },A^{\prime },b^{\prime },c^{\prime },d^{\prime })$ with

$$\begin{array}{ccc}\hfill {\lambda }^{\prime }& =& \lambda ,\hfill \\ \hfill A^{\prime }& =& {\displaystyle \frac{1}{\lambda }}{A}^{-1},\hfill \\ \hfill b^{\prime }& =& -{\displaystyle \frac{1}{\lambda }}{A}^{-1}c,\hfill \\ \hfill c^{\prime }& =& -A^{-1}b,\hfill \\ \hfill d^{\prime }& =& \langle b,A^{-1}c\rangle -d.\hfill \end{array}$$

We check that ${P^{\diamond }}^{\diamond }=({\lambda }^{″},A^{″},b^{″},c^{″},d^{″})=(\lambda ,A,b,c,d)=P$ component-wise as follows:

$$\begin{array}{cccccc}& {\lambda }^{″}\hfill & \hfill =& {\lambda }^{\prime }\hfill & \hfill =& \lambda ,\hfill \\ & A^{″}\hfill & \hfill =& {\displaystyle \frac{1}{\lambda }}{A^{\prime }}^{-1}\hfill & \hfill =& {\displaystyle \frac{1}{\lambda }}{\left({\displaystyle \frac{1}{\lambda }}{A}^{-1}\right)}^{-1}\hfill & \hfill =& A,\hfill \\ & b^{″}\hfill & \hfill =& -{\displaystyle \frac{1}{\lambda }}{A^{\prime }}^{-1}{c}^{\prime }\hfill & \hfill =& -{\displaystyle \frac{1}{\lambda }}{\left({\displaystyle \frac{1}{\lambda }}{A}^{-1}\right)}^{-1}(-A^{-1}b)\hfill & \hfill =& b,\hfill \\ & c^{″}\hfill & \hfill =& -{A^{\prime }}^{-1}{b}^{\prime }\hfill & \hfill =& -{\left({\displaystyle \frac{1}{\lambda }}{A}^{-1}\right)}^{-1}\left(-{\displaystyle \frac{1}{\lambda }}{A}^{-1}c\right)\hfill & \hfill =& c,\hfill \\ & d^{″}\hfill & \hfill =& \langle b^{\prime },{A^{\prime }}^{-1}{c}^{\prime }\rangle -d^{\prime }\hfill & \hfill =& 〈-{\displaystyle \frac{1}{\lambda }}{A}^{-1}c,{\displaystyle \frac{1}{\lambda }}{\left({\displaystyle \frac{1}{\lambda }}{A}^{-1}\right)}^{-1}(-A^{-1}b)〉-\langle b,A^{-1}c\rangle +d\hfill & \hfill =& d.\hfill \end{array}$$

□

The ⋄ involution confirms that the Legendre–Fenchel transform is an involution:

$${\left({\left(F_P\right)}^{*}\right)}^{*}=\mathcal{L}\left(\mathcal{L}{F}_P\right)=\mathcal{L}{F}_{P^{\diamond }}=F_{{\left(P^{\diamond }\right)}^{\diamond }}=F_P.$$

Let us now define the following notation to express the generalized Legendre convex conjugates:

Definition 1 

(Generalized Legendre–Fenchel convex conjugates [12]). Let ${\mathcal{L}}_{\lambda ,E,f,g,h}$ denote a generalized Legendre–Fenchel transform

$${\mathcal{L}}_{\lambda ,E,f,g,h}F:={\mathcal{L}}_{P}F:=\lambda \left(\mathcal{L}F\right)(E\eta +f)+\langle \eta ,g\rangle +h$$

for the parameter $P=(\lambda ,E,f,g,h)\in \mathbb{P}$.

Our result is that we can interpret those generalized Legendre–Fenchel transforms of Equation (2) as the ordinary Legendre transform on corresponding affine-deformed convex functions:

Theorem 2. 

For any $F\in {\Gamma }_0$, we have ${\mathcal{L}}_P\left(F\right):={\left(F^{*}\right)}_P=\mathcal{L}\left(F_{P^{\diamond }}\right)$.

Proof. 

By definition, ${\mathcal{L}}_{P}F:={\left(\mathcal{L}F\right)}_P$. Since $P={\left(P^{\diamond }\right)}^{\diamond }$ (Proposition 2), we have ${\left(\mathcal{L}F\right)}_P={\left(\mathcal{L}F\right)}_{{\left(P^{\diamond }\right)}^{\diamond }}$, and by using Proposition 1, we obtain ${\left(\mathcal{L}F\right)}_{{\left(P^{\diamond }\right)}^{\diamond }}=\mathcal{L}\left(F_{P^{\diamond }}\right)$. To summarize, we have

$${\mathcal{L}}_P\left(F\right)={\left(\mathcal{L}F\right)}_P={\left(\mathcal{L}F\right)}_{{\left(P^{\diamond }\right)}^{\diamond }}=\mathcal{L}\left(F_{P^{\diamond }}\right).$$

□

That is, in plain words, the Artstein-Avidan–Milman generalized Legendre transforms [11,12] are ordinary Legendre transforms on affine-deformed functions evaluated on affine deformed arguments.

Next, we describe an information-geometric interpretation of Theorem 2 which casts light on the origin and meanings of the degrees of freedom used to define generalized Legendre transforms. Thus, understanding the ordinary Legendre transform from the information-geometric viewpoint allows one to understand how to recover the generalized Legendre transforms formalized axiomatically by Artstein-Avidan–Milman [11,12].

3. An Information-Geometric Interpretation of Generalized Legendre Transforms

We can interpret the fact that generalized Legendre convex conjugates are affine-deformed convex conjugates from the lens of information geometry [16]. Consider a smooth, strictly convex function $F:{\mathbb{R}}^m\to \overline{\mathbb{R}}$ of the space ${\Gamma }_2$ of twice-differentiable lsc. Legendre-type convex functions (with ${\Gamma }_2\subset {\Gamma }_1\subset {\Gamma }_0$). This function $F\left(\theta \right)$ defines a dually flat m-dimensional space [16] $(M,g,\nabla ,{\nabla }^{*})$ which is a global chart manifold $M=\left\{p\right\}$ of points p equipped with dual Hessian structures [19] $(g,\nabla )$ and $(g,{\nabla }^{*})$. The primal Hessian structure is induced by a potential function $\psi$ on M with a torsion-free, flat, affine connection ∇ such that $\psi \left(p\right)=F\left(\theta \right(p\left)\right)$, where $(M,\theta (·\left)\right)$ is the primal ∇-affine coordinate system. The dual Hessian structure is induced by a dual potential function $\varphi$ on M with a dual torsion-free, flat, affine connection ${\nabla }^{*}$ such that $\varphi \left(p\right)=F^{*}\left(\eta \left(p\right)\right)$, where $(M,\eta (·\left)\right)$ is the dual ${\nabla }^{*}$-affine coordinate system. The Riemannian metric g can be expressed in the dual coordinate charts as $g\left(\theta \right)={\nabla }^{2}F\left(\theta \right)$ or $g\left(\eta \right)={\nabla }^2{F}^{*}\left(\eta \right)$, where $F^{*}$ is the convex conjugate. These two potential functions $\psi$ and $\varphi$ living on the manifold M satisfy the Fenchel–Young inequality:

$$\psi \left(\theta \left(p\right)\right)+\varphi \left(\eta \left(q\right)\right)\ge \sum _{i=1}^m{\theta }_i\left(p\right){\eta }_i\left(q\right),$$

(5)

with equality if and only if $p=q$ on M. The metric tensor g can be expressed as $g=\nabla d\psi$ or equivalently as $g={\nabla }^{*}d\varphi$, where d denotes the exterior derivative and $\psi$ and $\varphi$ are 0 forms.

Now, the ∇-affine coordinate system $\theta$ is defined up to an affine transformation. That is, if $\theta (·)$ is a ∇ coordinate system, then the coordinate system is $\overline{\theta }(·)=A\theta (·)+b$. However, once parameters A and b are fixed, it fully determines the dual ${\nabla }^{*}$ coordinate system $\eta (·)$.

The potential function $\varphi$ is also reconstructed by solving differential modulo equations an affine term $\langle c,\theta \rangle +d$ (e.g., see the proofs relying on Poincaré lemma in [16,19,20,21]). Fixing the degrees of freedom in the reconstruction of $\psi$ will also fix the corresponding affine term in $F^{*}$ which expresses $\psi$. The dual potential functions $\psi$ and $\varphi$ on the manifold M are related by the fiberwise Legendre transform [22] in geometric mechanics.

Thus, the information geometry of dually flat spaces allows one to explain the decoupling of the interactions of the dual set of parameters $(A,b,c,d)$ with $(E,f,g,h)$. Lastly, the scalar parameter $\lambda >0$ is the degree of freedom obtained from the fact that if $(M,g,\nabla ,{\nabla }^{*})$ is a dually flat space, then so is $(M,\lambda g,\nabla ,{\nabla }^{*})$.

The Fenchel–Young inequality induced by the dual potential functions of Equation (5) defines a canonical divergence on a dually flat space that we term the dually flat Hessian divergence:

$$D_{g,\nabla ,}(p:q)=\psi \left(\theta \left(p\right)\right)+\varphi \left(\eta \left(q\right)\right)-\sum _{i=1}^m{\theta }_i\left(p\right){\eta }_i\left(q\right)\ge 0.$$

We have the following reference duality [23]: $D_{g,\nabla }(q:p)=D_{g,{\nabla }^{*}}(p:q)$.

The dually flat Hessian divergence can be expressed using the dual coordinate systems as a corresponding Fenchel–Young divergence defined by

$$Y_{F,F^{*}}(\theta :{\eta }^{\prime }):=F\left(\theta \right)+F^{*}\left({\eta }^{\prime }\right)-\sum _{i=1}^m{\theta }_i{\eta }_i^{\prime }.$$

A Fenchel–Young divergence can also be expressed equivalently as dual Bregman divergences $B_F$ or $B_{F^{*}}$:

$$Y_{F,F^{*}}(\theta :{\eta }^{\prime })=B_F(\theta :{\theta }^{\prime })=B_{F^{*}}({\eta }^{\prime }:\eta ),$$

where ${\theta }^{\prime }=\nabla F^{*}\left({\eta }^{\prime }\right)$ and $\eta =\nabla F\left(\theta \right)$ with

$$B_F(\theta :{\theta }^{\prime })=F\left(\theta \right)-F\left({\theta }^{\prime }\right)-\langle \theta -{\theta }^{\prime },\nabla F\left({\theta }^{\prime }\right)\rangle .$$

Thus, the dually flat Hessian divergence can be expressed as dual Fenchel–Young divergences (using the fact that ${F^{*}}^{*}=F$) in the mixed $\theta$/$\eta$ coordinate systems as

$$\begin{array}{ccc}\hfill D_{g,\nabla ,}(p:q)& =& Y_{F,F^{*}}(\theta \left(p\right):\eta \left(q\right))=Y_{F^{*},F}(\eta \left(q\right):\theta \left(p\right)),\hfill \end{array}$$

(6)

$$\begin{array}{ccc}& =& {\displaystyle \frac{1}{\lambda }}{Y}_{F_P,F_{P^{\diamond }}^{*}}(\overline{\theta }\left(p\right):\overline{\eta }\left(q\right))={\displaystyle \frac{1}{\lambda }}{Y}_{F_{P^{\diamond }}^{*},F_P}(\overline{\eta }\left(q\right):\overline{\theta }\left(p\right)),\forall P\in \mathbb{P},\hfill \end{array}$$

(7)

This follows from the fact that $\psi \left(p\right)=F_P\left(\overline{\theta }\left(p\right)\right)=F\left(\theta \left(p\right)\right)$ and $\varphi \left(q\right)=F_{P^{\diamond }}^{*}\left(\overline{\eta }\left(q\right)\right)=F^{*}\left(\eta \left(q\right)\right)$ for any $P\in \mathbb{P}$.

Thus, we can define an equivalence relation ∼ between the functions of ${\Gamma }_2$: $F\sim \tilde{F}$ if and only if $\tilde{F}=F_P$ for some $P\in \mathbb{P}$. Furthermore, a distance invariant under the Legendre transform [24] can be defined in the moduli space ${\Gamma }_2/\sim$ of dually flat spaces.

Lastly, we may consider some curvilinear dual coordinate systems instead of the mutually orthogonal, dually affine coordinate systems. Let us transform the affine $\theta$ coordinate system into some arbitrary curvilinear coordinate system $\tilde{\theta }$. The dual affine $\eta$ coordinate system transforms correspondingly into the curvilinear coordinate system $\tilde{\eta }$. The underlying geometric structures have been called the $(u,v)$ structures by Amari [16,25] or the $(\rho ,\tau )$ structures by Zhang [23]. The dual Bregman divergences or, equivalently, the dual Fenchel–Young divergences of the underlying dually flat space need to be addressed as undeforming arguments [26,27] $\tilde{\theta }$ or $\tilde{\eta }$. Since the inverse transform $\tilde{\theta }\to \theta$ can be interpreted as a representation function, the Bregman divergences addressed as undeforming $\tilde{\theta }$ arguments were called representational Bregman divergences in [26]. For example, the $\beta$ divergences are Bregman divergences [28,29,30] which can be transformed by a nonlinear change of coordinates into $\alpha$ divergences [31]. Although the $\alpha$ divergences are Bregman divergences for any $\alpha$ on positive measures [26,27], $\alpha$ divergences are not Bregman divergences anymore when constrained on the probability simplex (except for the $\alpha =\pm 1$ case).

Remark 1. 

In information geometry [16], and more generally in Hessian geometry [19], a Legendre-type $C^3$ convex function $F\left(\theta \right)$ of ${\Gamma }_3$ induces a dually flat space or manifold. The convex function $F\left(\theta \right)$ (potential function) may stem from several modeling sources. We may consider the following settings to build dually flat spaces:

• The cumulant function of a natural exponential family [16,32]: Let $(\mathcal{X},\Omega ,\mu )$ be a measurable space with a sample space $\mathcal{X}$, σ algebra Ω, and a positive measure μ (e.g., counting or Lebesgue measure). An exponential family $\mathcal{E}$ is a set of probability measures $\mathcal{E}=\left\{P_{\theta }\left(x\right)=exp(\langle x,\theta \rangle -F\left(\theta \right))d\mu \left(x\right):\theta \in \Theta \right\}$, where

  $$\Theta =\{\theta :F(\theta )=log\int exp(\langle x,\theta \rangle )d\mu (x)<\infty \},$$

  is the natural parameter space. $Z\left(\theta \right)=exp\left(F\right(\theta \left)\right)$ is strictly log-convex [32], and hence $F\left(\theta \right)=logZ\left(\theta \right)$ is strictly convex (and differentiable). In this case, the convex conjugate amounts to the negative Shannon or differential entropy [33] $F^{*}\left(\eta \right)=-S\left(p_{\theta }\right)=\int p_{\theta }\left(x\right)log{p}_{\theta }\left(x\right)d\mu \left(x\right)$, where $S\left(p\right)=-\int p\left(x\right)logp\left(x\right)d\mu \left(x\right)$ is the Shannon entropy (when μ is the counting measure) or the differential entropy (when μ is the Lebesgue measure).
• The negative Shannon entropy of a mixture family [16]: A mixture family $\mathcal{M}$ is a set of probability measures

  $$\mathcal{M}=\left\{m_{\theta }(x)=\sum _{i=1}^{d-1}{\theta }_i{p}_i(x)+(1-\sum _{i=1}^{d-1}{\theta }_i)p_0(x)\right\},$$

  parameterized by a normalized positive weight vector $\theta \in \Theta ={\Delta }_d$ (the standard $(d-1)$-dimensional simplex) such that the functions $1,p_0\left(x\right),\dots ,p_{d-1}\left(x\right)$ are linearly independent. The function $F\left(\theta \right)=-S\left(m_{\theta }\right)$ is strictly convex and differentiable (see [34] for a proof). In this case, the convex conjugate $F^{*}\left(\eta \right)=S^{\times }(p_0,m_{\theta })$ is the cross-entropy of $p_0\left(x\right)$ with the mixture $m_{\theta }\left(x\right)$, where $S^{\times }(p,q)=-\int p\left(x\right)logq\left(x\right)d\mu \left(x\right)$.
• The characteristic function of a regular cone [19] (i.e., convex and pointed cone): Let $\mathcal{K}$ be a regular cone and ${\mathcal{K}}^{*}={\cap }_{\theta \in \mathcal{K}}\{\eta :\langle \theta ,\eta \rangle \ge 0\}$ be its dual cone. We define $F\left(\theta \right)=log{\chi }_{\mathcal{K}}\left(\theta \right)$, where ${\chi }_{\mathcal{K}}\left(\theta \right)$ is the characteristic function ${\chi }_{\mathcal{K}}\left(\theta \right)={\int }_{{\mathcal{K}}^{*}}exp(-\langle \theta ,\eta \rangle )d\eta$. One can further build generalized Wishart exponential families on the cones [35]. (Note that Massieu [36] introduced the concept of characteristic functions and their convex conjugates in thermodynamics in the 19th century.) See also the barrier functions (logarithms of characteristic functions) on convex cones for interior point methods [37].
• The Guillemin potential of a (Delzant) polytope [38] which recovers the negative Shannon entropy when considering the standard simplex.

4. Conclusions

To summarize, we first proved that the generalized Legendre transforms obtained from the reverse-ordering involutive transform axiomatization of [11,12] can be interpreted as the ordinary Legendre transform of corresponding affine-deformed functions. Second, we explained how these generalized Legendre transforms are merely different expressions of the well-known geometric Legendre transforms on dually flat spaces (Hessian manifolds with global charts).

Figure 1 summarizes the various classes of convex functions used in this paper, with some of their key properties used to define information-geometric structures. In particular, the class of functions ${\Gamma }_3$ is thrice-differentiable Legendre-type lsc. convex functions (with ${\Gamma }_3\subset {\Gamma }_2\subset {\Gamma }_1\subset {\Gamma }_0$), which allows one to construct $\alpha$ geometry [16] from the Amari–Chentsov cubic tensor [17].