Lagrangian Submanifolds of Symplectic Structures Induced by Divergence Functions

Abstract

Divergence functions play a relevant role in Information Geometry as they allow for the introduction of a Riemannian metric and a dual connection structure on a finite dimensional manifold of probability distributions. They also allow to define, in a canonical way, a symplectic structure on the square of the above manifold of probability distributions, a property that has received less attention in the literature until recent contributions. In this paper, we hint at a possible application: we study Lagrangian submanifolds of this symplectic structure and show that they are useful for describing the manifold of solutions of the Maximum Entropy principle.

1. Introduction

Information Geometry [1,2] provides a sound and fruitful framework for interpreting statistics using classical differential geometry notions [3]. A principal object in Information Geometry is the notion of contrast or divergence function, which (informally speaking) measures the degree of separation between two probability distributions [4,5,6]. The main thrust of divergence functions is that they allow to define a Riemannian structure on a finite dimensional submanifold M of probability distributions endowed with a dual coordinate system, with far reaching implications. A less-studied spin off of contrast function is the possibility of introducing a symplectic structure on the square of M by the pull-back of the canonical symplectic structure defined on the cotangent bundle $T^{\ast }M$. This procedure was introduced in 1995 in the pioneering paper [7], suggesting that symplectic geometry may have a natural role to play in statistics. In recent times there has been a renewed interest in possible applications of the symplectic structures introduced, as in [7] for example, to studying the analogies with the discrete Lagrangian mechanics (see in [8]) or the relations with completely integrable systems of Hamiltonian mechanics (see in [9,10]).

In this paper, we try to look at a possible role for Lagrangian submanifolds of the above-discussed symplectic structure on $M^2$ in the case that M is an exponential family$M(h,k)$. Exponential families are prototypical examples of finite dimensional manifolds admitting a dually flat canonical structure defined by the canonical divergence, and they play a relevant role in information geometry and statistics [1,2]. For our argument, their importance is due to the fact that they represent the manifold of solutions of the variational problem associated to the Maximum Entropy Principle (MEP) with linear constraints ([11,12]). In some applications to statistical mechanics, e.g., in the descriptions of phase transitions in Ising spin systems, MEP with nonlinear constraints is considered, see, e.g., in [13,14,15]. In this case, the set of possible solutions has a richer structure, which is well captured by a Lagrangian submanifold of $T^{\ast }M(h,k)$. In this work, we are concerned with the Lagrangian submanifolds defined in the square of $M(h,k)$ via the canonical pull-back hinted at above.

The structure of the paper is as follows. In Section 2, we recall the needed tools of Symplectic Geometry, and in Section 2.1 we review the canonical pull-back construction via divergence function construction exposed in [7]. In Section 3, we consider the special case of exponential families associated with MEP with nonlinear constraints.

2. Synopsis of Symplectic Geometry

We briefly recall the basic facts of symplectic geometry that are necessary for introducing our argument referring to classical textbooks for the proof of the results. A symplectic manifold $(M,\omega )$ is a smooth even-dimensional manifold M equipped with a non-degenerate, closed two-form $\omega$ ($d\omega =0$, where d is the external derivation operator). A submanifold L of M is a Lagrangian submanifold if $2dimL=dimM$ and the two-form restricted to L is vanishing, ${\omega }_{|L}=0$. A prototypical example of symplectic manifold is the cotangent bundle $T^{\ast }S$ of a manifold S. If $x=(x^1,\dots ,x^n)$ are local coordinates on S, and $(x,\lambda )$ are local coordinates on $T^{\ast }S$, then the Liouville one-form ${\theta }_c$ on $T^{\ast }S$ has the local expression ${\theta }_c={\lambda }_{i}d{x}^i$ (summation over repeated indices is understood) and the symplectic two form is

$$\omega =d{\theta }_c=d{\lambda }_i\wedge d{x}^i.$$

(1)

A classical theorem of Darboux says that every symplectic manifold $(M,\omega )$ admits an atlas of local coordinates $(x,\lambda )$ such that locally $\omega$ has the representation (1). A relevant example of Lagrangian submanifold of $T^{\ast }S$ is the graph of the differential of a function $g:S\to \mathbb{R}$, that is,

$$L_g=\{(x,\lambda (x))\in T^{\ast }S:\lambda (x)=dg(x),x\in S\}.$$

Note that $L_g$ is a n-dimensional submanifold which is transversal to the fibers of the fibration $\pi :T^{\ast }S\to S$, that is, its tangent bundle $T{L}_g$ is transversal to the vertical bundle $kerT\pi$.

According to a theorem of Maslov–Hormander ([16,17]), a general (i.e. not necessarily trasversal) Lagrangian submanifold of $T^{\ast }S$ can be locally described as the graph of a smooth function G depending on extra parameters. Let us sketch briefly this construction along the lines of the works in [18,19].

Let U be a k-dimensional manifold called supplementary manifold, and let $G:S\times U\to \mathbb{R}$ be a smooth function whose representation in a local chart is $G(x,u)$. We define the critical set of G as (we use the notation ${(G_x)}_i=\partial G/\partial x^i$ and ${(G_{xy})}_{ij}={\partial }^{2}G/\partial x^i\partial y^j$) for partial derivatives)

$$\mathcal{E}=\{(x,u):G_u(x,u)=0\}.$$

(2)

If $d{G}_u$ has maximal rank over $\mathcal{E}$, that is,

$$\mathrm{rk}d{G}_u=\mathrm{rk}(G_{xu}{G}_{uu})=k\mathrm{for}\mathrm{all}(x,u)\in \mathcal{E}$$

(3)

then G is called Morse family and the following ${\Lambda }_G$ is a Lagrangian submanifold of $T^{\ast }S$,

$${\Lambda }_G=\{(x,G_x(x,u))\in T^{\ast }S\mathrm{where}(x,u)\in \mathcal{E}\}.$$

(4)

If there are no extra parameters $k=0$, then ${\Lambda }_G$ is the graph of a differential and thus ${\Lambda }_G$ is a transversal submanifold. Note that the above rank condition (3) can be satisfied if the square submatrix $G_{uu}$ has maximal rank, i.e., $det{G}_{uu}\ne 0$ on $\mathcal{E}$. In this case, by the implicit function theorem there exist a locally defined function $u=u(x)$ such that $\mathcal{E}$ is the graph of u and setting $\widehat{G}(x)=G(x,u(x))$ we have that

$${\widehat{G}}_x(x)=G_x(x,u(x))+G_u(x,u(x))u_x(x)=G_x(x,u(x))\mathrm{for}\mathrm{all}(x,u)\in \mathcal{E}.$$

Therefore, where $det{G}_{uu}\ne 0$ on $\mathcal{E}$, all the parameters u can be eliminated and ${\Lambda }_{\widehat{G}}$ is locally transversal to the fibers. The set of points of S where $det{G}_{uu}(x,u)=0$ for $(x,u)\in \mathcal{E}$ is called the caustic of ${\Lambda }_G$. These are the points where the Lagrangian submanifold is tangent to the fibers of $\pi :T^{\ast }S\to S$ and trasversality is lost.

2.1. Symplectic Structures Defined by Divergence Functions

Given a smooth n-dimensional manifold, M, let us denote with $M^2=M\times M$ the square of M and with ${\Delta }_M\subset M^2$ the diagonal of $M^2$. We will use local coordinates $x=(x^1,\dots ,x^n)$ on M and $(x,y)=(x^1,\dots ,x^n,y^1,\dots ,y^n)$ on $M^2$.

Let $D:M^2\to [0,+\infty )$ be a smooth non-negative function whose representation in a local chart is $D(x,y)\ge 0$. We use the notations

$${(D_x)}_i=\frac{\partial D}{\partial x^i},{(D_y)}_j=\frac{\partial D}{\partial y^j},{(D_{yx})}_{ji}=\frac{\partial }{\partial y^j}(\frac{\partial D}{\partial x^i})=-{\varphi }_{ji}$$

for first and second order derivatives of D. The function D is a yoke (see [7] ) if the following conditions hold and D is a divergence (see [8]) if $(\mathrm{iii})$ below holds on the whole $M^2$.

(i)

$D=0$ only on ${\Delta }_M$

(ii)

$D_x=0$ and $D_y=0$ on ${\Delta }_M$

(iii)

$\varphi =-D_{xy}$ is positive definite on ${\Delta }_M$

thus points of ${\Delta }_M$ are minima of D. A divergence function act as a pseudo-distance but it does not satisfy the symmetry nor the triangle inequality conditions. In [7], the following fibered map $F_D:M^2\to T^{\ast }M$ over M is considered, whose representation in a local chart is

$$F_D(x,y)=(x,D_x(x,y)).$$

(5)

By condition (iii) above there exist a neighborhood W of ${\Delta }_M$, where $F_D$ has a smooth inverse

$$F_D^{-1}(x,\lambda )=(x,y(x,\lambda )).$$

Using the local diffeomorphism $F_D$ a symplectic structure $(W,{\omega }_D)$ is defined in [7] via the pull-back ${\omega }_D=F_D^{*}\omega$ of the canonical two form (1) on $T^{\ast }M$. The local form of ${\omega }_D$ can be computed as follows,

$${\omega }_D=F_D^{*}\omega =F_D^{*}(d{\theta }_c)=d(F_D^{*}{\theta }_c)=d\left({(D_x)}_{i}d{x}^i)\right)$$

(6)

thus (see Section 3.2 in [7])

$${\omega }_D=\frac{{\partial }^{2}D}{\partial x^j\partial x^i}d{x}^j\wedge d{x}^i+\frac{{\partial }^{2}D}{\partial y^j\partial x^i}d{y}^j\wedge d{x}^i=-{\varphi }_{ji}d{y}^j\wedge d{x}^i$$

because the first term ${\partial }^{2}D/\partial x^j\partial x^i$ is symmetric in the $i,j$ indices. For the applications that we have in mind of the above theory, we will assume in (iii) above that $-D_{yx}$ is positive definite on the whole $M^2$ so that $F_D$ is a global diffeomorphism.

Simple examples of Lagrangian submanifolds of $M^2$ with respect to ${\omega }_D$ are (with a little abuse of notation) the n-dimensional submanifolds $M_x=M\times \{y\}\approx M$, which are also transversal to the fibers of ${\pi }_1:M^2\to M$, ${\pi }_1(x,y)=x$. Moreover, as ${\omega }_D(u,u)=0$, ${\Delta }_M$ is also a Lagrangian submanifold.

Note also that (6) implies that $F_D$ is a symplectomorphism, thus $L=F_D^{-1}(\Lambda )$ is a Lagrangian submanifold of $M^2$ whenever $\Lambda \subset T^{\ast }M$ is a Lagrangian submanifold. In this paper, we will be mainly concerned with the study of Lagrangian submanifolds of $M^2$ defined in this way.

In the following Section 2.2, we will compute the above introduced objects for the relevant case of exponential families of probability distributions and canonical divergence.

In [7], the Hamiltonian $H:T^{\ast }M\to [0,+\infty )$ associated to a divergence function is defined as $H=D\circ F_D^{-1}$ and locally it has the form

$$H(x,\lambda )=D(x,y(x,\lambda )).$$

(7)

2.2. Canonical Divergence and Exponential Families

In this section, we recall the basic definitions of exponential family and canonical divergence, as described, e.g., in [1,2]. Let $(X,\mathcal{B},dx)$ be a probability space, where X may be a discrete set or $X={\mathbb{R}}^k$. We stipulate that in case of a discrete set the integrals over X with respect to the measure $dx$ are substituted by summations. Let

$$\mathcal{P}(X)=\{p:X\to [0,+\infty ),p(x)\ge 0,{\int }_{X}pdx=1\}$$

and suppose that $q\in \mathcal{P}(X)$ for suitable k, where $q(x)=e^{k(x)}>0$. Consider n independent observables

$$h:X\to {\mathbb{R}}^n,\mathrm{rk}dh(x)=n\forall x\in X$$

and define the related free energy $\psi :\Theta \subset {\mathbb{R}}^n\to \mathbb{R}$ as (here $\theta ·h={\theta }^i{h}_i$)

$$e^{\psi (\theta )}={\int }_X{e}^{\theta ·h(x)+k(x)}dx.$$

(8)

The n real numbers ${\theta }^i$ are called canonical parameters. They define uniquely a probability distribution $p(·;\theta )$ which belongs to the exponential family defined by $h,k$,

$$M(h,k)=\{p(x;\theta )=e^{\theta ·h(x)+k(x)-\psi (\theta )},\theta \in \Theta \}\subset \mathcal{P}(X).$$

(9)

The relevant fact is that $M(h,k)$ is a n-dimensional submanifold of the infinite dimensional set $\mathcal{P}(X)$ and that the canonical parameters $\theta$ are local coordinates. Note that $q\in M(h,k)$ as $\psi (0)=0$ and $q(x)=p(x;0)$. Another system of local coordinates is provided by the so-called expectation parameters defined by

$$\eta ={\psi }_{\theta }(\theta )={\mathbb{E}}_{p_{\theta }}\left[h\right]={\int }_{X}h(x)p(x;\theta )dx.$$

As $\psi$ is a convex function, the gradient map ${\psi }_{\theta }(\theta )=\eta$ is globally invertible with inverse $\theta =\widehat{\theta }(\eta )$, which is also a gradient map $\widehat{\theta }(\eta )={\phi }_{\eta }(\eta )$, where

$$\phi (\eta )=\widehat{\theta }(\eta )·\eta -\psi (\widehat{\theta }(\eta ))$$

(10)

is the Legendre transform of $\psi$ (see, e.g., in [1]). We will denote with $p(x;\eta )$ the point in $M(h,k)$ associated to $\eta$. The Kullback–Leibler divergence is defined for general $(p,\tilde{p})$ in $\mathcal{P}{(X)}^2$ as

$$D_{KL}(p,\tilde{p})={\int }_{X}p(x)log\frac{p(x)}{\tilde{p}(x)}dx.$$

The restriction of $D_{KL}$ to $M{(h,k)}^2$, the square of $M(h,k)$, $D_{KL}:M{(h,k)}^2\to [0,+\infty )$ is called canonical divergence. It can be shown (see in [1]) that when $M(h,k)$ is referred to the coordinates $(\eta ,\theta )$, $D_{KL}$ has the local representation

$$D(\eta ,\theta )=\phi (\eta )+\psi (\theta )-\eta ·\theta .$$

(11)

Note that as $p(·;\theta )=q$ for $\theta =0$

$$D_{KL}(p,q)=D_{KL}(p(·;\eta ),p(·;0))=\phi (\eta )+\psi (0)-\eta ·0=\phi (\eta ).$$

(12)

A key object is the map $F_D$ introduced in (5) associated to $M(h,k)$ and the canonical divergence (11). It has the local form in coordinates $(\eta ,\theta )$, see (5) and (11),

$$F_D(\eta ,\theta )=(\eta ,D_{\eta })=(\eta ,{\phi }_{\eta }(\eta )-\theta ),$$

(13)

with the explicit inverse, using local coordinates $(\eta ,\lambda )$ in $T^{\ast }M(h,k)$,

$$F_D^{-1}(\eta ,\lambda )=(\eta ,\theta (\eta ,\lambda ))=(\eta ,{\phi }_{\eta }(\eta )-\lambda )=(\eta ,\widehat{\theta }(\eta )-\lambda ).$$

(14)

A simple but elegant result of the above-introduced framework is the following.

Proposition 1.

Let ${\Lambda }_G$ be a Lagrangian submanifold of $T^{\ast }M(h,k)$ described by the Morse family $G(\eta ,u)$ as in (4). Then, $L_S=F_D^{-1}({\Lambda }_G)$ is a Lagrangian submanifold of $M{(h,k)}^2$ described by the Morse family $S(\eta ,u)=\phi (\eta )-G(\eta ,u)$.

Proof. 

From (4) we have that $\lambda =G_{\eta }(\eta ,u)$ on ${\Lambda }_G$ and from (14)

$$F_D^{-1}({\Lambda }_G)=\{(\eta ,\theta )=(\eta ,{\phi }_{\eta }(\eta )-G_{\eta }(\eta ,u))=(\eta ,S_{\eta }(\eta ,u)),(\eta ,u)\in \mathcal{E}\}$$

where $S(\eta ,u)=\phi (\eta )-G(\eta ,u)$. Moreover, as $S_u(\eta ,u)=G_u(\eta ,u)$ the critical set $\mathcal{E}$ in (2) is the same. □

As a consequence of the above proposition, if ${\Lambda }_G$ is transversal to the fibers of $T^{\ast }M(h,k)$ (no extra parameters u), then its image in $M{(h,k)}^2$ is transversal to the fibers of ${\pi }_1$.

Another interesting consequence is that the zero section of the cotangent bundle $T^{\ast }M(h,k)$, locally represented as $Z=\{(\eta ,0):\eta \in E\}$, is mapped by $F_D^{-1}$ into

$$Z_0=F_D^{-1}(Z)=\{(\eta ,\widehat{\theta }(\eta )):\eta \in E\}$$

which is contained into $D^{-1}(0)$, the zero-level set of the canonical divergence. Indeed, from (10) and (11) we have that

$$D(\eta ,\widehat{\theta }(\eta ))=\phi (\eta )+\psi (\widehat{\theta }(\eta ))-\eta ·\widehat{\theta }(\eta )=\phi (\eta )-\phi (\eta )\equiv 0$$

(15)

thus $Z_0\subset D^{-1}(0)$ in the general case and $Z_0=D^{-1}(0)$ if $n=1$. For later use, we compute from (7) the Hamiltonian associated to the canonical divergence

$$H(\eta ,\lambda )=D\circ F_D^{-1}(\eta ,\lambda )=\phi (\eta )+\psi (\widehat{\theta }(\eta )-\lambda )-\eta ·(\widehat{\theta }(\eta )-\lambda ).$$

We set for the sake of simplicity $\widehat{\theta }(\eta )=\widehat{\theta }$ and we compute from (8) the free energy $\psi (\widehat{\theta }(\eta )-\lambda )$

$$\begin{array}{ccc}\hfill e^{\psi (\widehat{\theta }-\lambda )}& =& {\int }_X{e}^{(\widehat{\theta }-\lambda )·h+k}dx={\int }_X{e}^{(\widehat{\theta }-\lambda )·h+k+\psi (\widehat{\theta })-\psi (\widehat{\theta })}dx\hfill \\ & =& e^{\psi (\widehat{\theta })}{\int }_X{e}^{-\lambda ·h}{e}^{\widehat{\theta }·h+k-\psi (\widehat{\theta })}dx=e^{\psi (\widehat{\theta })}{\mathbb{E}}_{p_{\widehat{\theta }}}\left[e^{-\lambda ·h}\right].\hfill \end{array}$$

(16)

Using (15) and (16), the Hamiltonian can be written using relation (10) as

$$\begin{array}{ccc}\hfill H(\eta ,\lambda )& =& \phi (\eta )+\psi (\widehat{\theta })+ln{\mathbb{E}}_{p_{\widehat{\theta }}}\left[e^{-\lambda ·h}\right]-\eta ·\widehat{\theta }+\eta ·\lambda \hfill \\ & =& ln{\mathbb{E}}_{p_{\widehat{\theta }}}\left[e^{-\lambda ·h}\right]+\eta ·\lambda .\hfill \end{array}$$

(17)

It is interesting to investigate more in detail the structure of the Lagrangian submanifold $L_S=F_D^{-1}({\Lambda }_G)\subset M{(h,k)}^2$ by studying the form of the two probability distributions $F_D^{-1}(\eta ,\lambda )=(\eta ,\widehat{\theta }-\lambda )$ in $L_S$ associated to the coordinates respectively $\eta$ and $\widehat{\theta }-\lambda$. We compute from (9)

$$p(x;\eta )=e^{\widehat{\theta }·h(x)+k(x)-\psi (\widehat{\theta })}$$

and using (17)

$$\begin{array}{ccc}\hfill p(x;\widehat{\theta }-\lambda )& =& e^{(\widehat{\theta }-\lambda )·h+k-\psi (\widehat{\theta }-\lambda )}\hfill \\ & =& e^{\widehat{\theta }·h-\lambda ·h+k-\psi (\widehat{\theta })-ln{\mathbb{E}}_{p_{\widehat{\theta }}}\left[e^{-\lambda ·h}\right]}\hfill \\ & =& p(x;\eta )\frac{e^{-\lambda ·h(x)}}{{\mathbb{E}}_{p_{\widehat{\theta }}}\left[e^{-\lambda ·h}\right]}.\hfill \end{array}$$

(18)

Note that setting

$$p(x;\lambda )=\frac{e^{-\lambda ·h(x)}}{Z(\lambda )}=\frac{e^{-\lambda ·h(x)}}{{\int }_X{e}^{\lambda ·h(x)dx}}$$

relation (18) can be given the form

$$p(x;\widehat{\theta }-\lambda )=\frac{p(x;\eta )e^{-\lambda ·h(x)}}{{\int }_{X}p(x;\eta )e^{-\lambda ·h(x)}dx}=\frac{p(x;\eta )p(x;\lambda )}{{\int }_{X}p(x;\eta )p(x;\lambda )dx}.$$

(19)

We will give an interpretation of this relation in the case of discrete probability distributions in Section 3.2 below.

3. Application to Maximum Entropy Principle with Nonlinear Constraints and Phase Transitions

A relevant application of the above-introduced framework concerns the use of the Maximum Entropy Principle with nonlinear constraints. Let us consider a physical system X whose description is given in terms of a probability distribution $q\in \mathcal{P}(X)$. The Maximum Entropy Principle (E.T. Jaynes, see in [11,12]) is a general inference procedure that allows to update an initial probability distribution q on the basis of subsequent information on the system represented by the average values ${\mathbb{E}}_p\left[h\right]$ of some observables h of interest for the system. The sought distribution p is the one that minimizes the relative entropy $D_{KL}(p,q)$ on the set of the distributions which satisfy the constraints on ${\mathbb{E}}_p\left[h\right]$. From a mathematical point of view, we are faced with a constrained extremization problem to be solved below using the Lagrange multipliers method.

We will see that the set of solutions for different values of the constraints defines a Lagrangian submanifold of a cotangent space of a manifold $M(h,k)$. We are interested in describing the corresponding Lagrangian submanifold in $M{(h,k)}^2$.

This section has a pedagogical character, so for the sake of simplicity we will avoid technicalities and assume that $X=\{1,\dots ,n\}$ is a discrete space and that there is only one observable of interest defined by assigning $h=(h_1,\dots ,h_n)$. The case of k observables can be dealt with along the same lines with no extra effort. The case of a continuous space $X\subset {\mathbb{R}}^n$ presents more technical difficulties and it is considered in [20].

Let $q_i=e^{k_i}\in \mathcal{P}(X)$ be the a priori distribution describing X. The Kullback–Leibler divergence is called relative entropy in this setting and has the form

$$D(p,q)=\sum _i{p}_{i}ln\frac{p_i}{q_i}.$$

Let $f:\mathbb{R}\to \mathbb{R}$ be a smooth globally non-invertible function (think for example of a cubic $f(x)=x(x^2-a^2)$ for $a\in \mathbb{R}$, see Figure 1 below). We look for the minima of D on the set of $p\in \mathcal{P}(X)$ that satisfy the nonlinear constraint on p in the form $g:{\mathbb{R}}_{+}^n\to \mathbb{R},g(p)=y$ that is

$$g(p)=f({\mathbb{E}}_p\left[h\right])=f(\sum _{i=1}^n{h}_i{p}_i)=y.$$

(20)

The choice of this type of constraints is motivated by classical applications in statistical physics. For example in the Ising model in the Curie–Weiss (mean field) approximation the average energy of the spin lattice is a quadratic function of the average magnetization ${\mathbb{E}}_p\left[s\right]$, see [14,15]. We have that

$$dg(p)=f^{\prime }({\mathbb{E}}_p\left[h\right])h=(f^{\prime }({\mathbb{E}}_p\left[h\right])h_1,\dots ,f^{\prime }({\mathbb{E}}_p\left[h\right])h_n).$$

(21)

Note that we do not take into account at this stage of the procedure the normalization constraint stipulating that we will enforce it by dividing any candidate extremum point $\widehat{p}$ by ${\sum }_i{\widehat{p}}_i$. After introducing the Lagrange function where $\lambda$ is the Lagrange multiplier associated to the constraint (20)

$$G(y,p,\lambda )=D(p,q)-\lambda (f({\mathbb{E}}_p\left[h\right])-y)$$

(22)

we see that the candidate extrema are the solutions $(p,\lambda )$ for given y of (here $i=1,\dots ,n$)

$${(G_p)}_i=ln\frac{p_i}{q_i}+1-\lambda f^{\prime }({\mathbb{E}}_p\left[h\right])h_i=0,G_{\lambda }=f({\mathbb{E}}_p\left[h\right])-y=0$$

(23)

that is, setting $q_i=e^{k_i}$, we have to face a trascendental equation for the unnormalized probability

$$p_i=c{e}^{\lambda f^{\prime }({\mathbb{E}}_p\left[h\right])h_i+k_i},f({\mathbb{E}}_p\left[h\right])=y.$$

(24)

After normalization, (24)₁ becomes

$$p_i=e^{\lambda f^{\prime }({\mathbb{E}}_p\left[h\right])h_i+k_i-\psi (\lambda ,p)},e^{\psi (\lambda ,p)}=\sum e^{\lambda f^{\prime }({\mathbb{E}}_p\left[h\right])h_i+k_i}.$$

(25)

Let us denote with $f^{\leftarrow }(y)\subset \mathbb{R}$ the set of pre-images of y along f (see, e.g., Figure 1 below)

$$f^{\leftarrow }(y)=\{\eta \in \mathbb{R}:f(\eta )=y\}=\{{\eta }_1,\dots ,{\eta }_{\alpha },\dots ,{\eta }_A\},{\eta }_{\alpha }={\eta }_{\alpha }(y)$$

(26)

where we have supposed that, for every y, $f^{\leftarrow }(y)$ is a finite set of cardinality $A(y)<+\infty$. The crux is that we can substitute the constraint $f({\mathbb{E}}_p\left[h\right])=y$ in (24)₂ with the following equivalent one

$$f({\mathbb{E}}_p\left[h\right]))=y⟺{\mathbb{E}}_p\left[h\right]\in f^{\leftarrow }(y)$$

therefore we can describe the—possibly non-unique—solution (25) of the extremum problem (23) as

$$p_i^{\alpha }=e^{\lambda f^{\prime }({\eta }_{\alpha })h_i+k_i-\psi (\lambda ,\alpha )},e^{\psi (\lambda ,\alpha )}=\sum e^{\lambda f^{\prime }({\eta }_{\alpha })h_i+k_i}$$

(27)

where $\alpha =1,\dots ,A(y)$, showing that the candidate solution belongs to an exponential family $M(h,k)$. Note that in Information Geometry, the critical points of the MEP extremum problem are computed as geodesic projections over a submanifold which is an exponential family and multiplicity of solutions are related to the non-uniqueness of the geodesic projection, see in [1,15].

Figure 1. Plot of $y=f(\eta )=\eta ({\eta }^2-a^2)$. Points $b,c$ correspond to points where $f^{\prime }(\eta )=0$.

Note that where $f^{\prime }({\eta }_{\alpha })\ne 0$ setting $\lambda f^{\prime }({\eta }_{\alpha })={\theta }_{\alpha }$ the solution (27) can be given the standard form (see in [1,14]) of MEP solution

$${\widehat{p}}_i=e^{\widehat{\theta }{h}_i+k_i-\psi (\widehat{\theta })},\widehat{\theta }(\eta )={\phi }_{\eta }(\eta )$$

with linear constraint $\mathbb{E}\left[h\right]={\eta }_{\alpha }$, hence (25) becomes

$$p_i^{\alpha }=e^{{\theta }_{\alpha }{h}_i+k_i-\psi ({\theta }_{\alpha })},e^{\psi ({\theta }_{\alpha })}=\sum e^{{\theta }_{\alpha }{h}_i+k_i}.$$

(28)

The multipliers ${\theta }_{\alpha }=\widehat{\theta }({\eta }_{\alpha }(y))$, $\alpha =1,\dots ,A(y)$ are uniquely determined (see (10)) by the equation

$${\psi }_{\theta }(\theta )=\eta \mathrm{i}.\mathrm{e}.\widehat{\theta }(\eta )={\phi }_{\eta }(\eta )$$

(29)

for $\eta ={\eta }_{\alpha }(y)$ and accordingly we can compute the multipliers $\lambda$ as

$${\lambda }_{\alpha }(y)=\frac{\widehat{\theta }({\eta }_{\alpha }(y))}{f^{\prime }({\eta }_{\alpha }(y))}.$$

(30)

Note that the solution to our constrained extremization problem (28) has the form of a curved exponential family (see [1]) with respect to the discrete parameter $\alpha$. We will see in the next Section 3.1 that the framework of Lagrangian submanifold is useful to describe the global picture of the solutions in case of multiple solutions.

3.1. The Global Picture via Lagrange Submanifold

If we set in the Lagrange function (22) $(p,\lambda )=u$, we see that for $G(y,u)$ the set of points $(y,u)$ satisfying the first order necessary condition for unconstrained extremum (23) is the critical set

$$\mathcal{E}=\{(y,u):G_u(y,u)=0\}.$$

We can check if the Lagrange function $G(y,u)$ defines a Morse family using the rank condition (3)

$$\mathrm{rk}\left(G_{yu}{G}_{uu}\right)=n+1\mathrm{for}\mathrm{all}(y,u)\in \mathcal{E}$$

where in this case

$$\left(G_{yu}{G}_{uu}\right)=\left(\begin{array}{ccc}0& G_{pp}& -d{g}^T\\ 1& -dg& 0\end{array}\right)$$

(31)

and $G_{pp}$ is the n-dimensional Hessian matrix (here ${\delta }_{ij}$ is Kronecker symbol)

$${(G_{pp})}_{ij}={(D_{pp})}_{ij}-\lambda f^{\prime \prime }({\mathbb{E}}_p\left[h\right])h_i{h}_j=\frac{{\delta }_{ij}}{p_i}-\lambda f^{\prime \prime }({\mathbb{E}}_p\left[h\right])h_i{h}_j.$$

(32)

If $G(y,u)$ is a Morse family, then by Maslov–Hormander theorem

$${\Lambda }_G=\{(y,G_y)\mathrm{where}(y,u)\in \mathcal{E}\}$$

(33)

is a Lagrangian submanifold of $T^{\ast }\mathbb{R}$. We claim that (33) provides a global description of the set of solutions (28). We have seen in Section 1 that a sufficient condition for the elimination of all extra parameters u is that $G_{uu}$ has maximal rank for all $(y,u)\in \mathcal{E}$. A criterion for this is given by the following classical result in constrained optimization theory, here adapted to our notations, which express the second order sufficient condition for maxima or minima (see in [14,21] for the proof).

Proposition 2.

If the symmetric matrix $G_{pp}$ in (32) is (positive or negative) definite on $kerdg$ for $(y,u)\in \mathcal{E}$, then the square matrix $G_{uu}$ in (31) has maximal rank.

From (21), we have that for $(y,u)\in \mathcal{E}$

$$kerdg(p)=\{u\in {\mathbb{R}}^n:f^{\prime }({\eta }_{\alpha })h·u=0\}$$

and from (32), that

$$G_{pp}u·u=\left(\sum _i\frac{u_i^2}{p_i}\right)-\lambda f^{″}({\eta }_{\alpha }){(h·u)}^2.$$

It is straightforward to derive from the above relations that the two cases below hold

$$\left\{\begin{array}{c}{f}^{\prime }({\eta }_{\alpha })\ne 0\Rightarrow kerdg(p)=\{u:h·u=0\}\Rightarrow G_{pp}u·u>0\forall u\ne 0,\hfill \\ \\ f^{\prime }({\eta }_{\alpha })=0\Rightarrow kerdg(p)={\mathbb{R}}^n\Rightarrow G_{pp}u·u\in \mathbb{R}.\hfill \end{array}\right.$$

Therefore, at points $(y,u)\in \mathcal{E}$ where $f^{\prime }({\eta }_{\alpha })\ne 0$ the Lagrangian submanifold ${\Lambda }_G$ in (33) is transversal. At points in $\mathcal{E}$ where $f^{\prime }({\eta }_{\alpha })=0$, we have $dg=f^{\prime }({\eta }_{\alpha })h=0$, see (21), thus transversality is lost as—see the form of $G_{uu}$ in (31)—for these points

$$det{G}_{uu}(p,\lambda )=0,and(y,u)\in \mathcal{E}.$$

We remark that the above introduced framework is able to give the global description of the set of solutions (28), (30) in terms of the Lagrangian submanifold locally described as

$${\Lambda }_f^{(y)}=\{(y,G_y)=\left(y,\lambda ({\eta }_{\alpha }(y))\right)=(y,\frac{\widehat{\theta }({\eta }_{\alpha }(y))}{f^{\prime }({\eta }_{\alpha }(y))})\}\subset T^{\ast }{\mathbb{R}}_y$$

(34)

where $\lambda ({\eta }_{\alpha }(y))$ is given by (30). If we consider $f:E\subset {\mathbb{R}}_{\eta }\to {\mathbb{R}}_y$, $y=f(\eta )$ as a local change of coordinates on $M(h,k)$ (since f is locally invertible where $f^{\prime }(\eta )\ne 0$) it is easy to prove that

Proposition 3.

The submanifold ${\Lambda }_f^{(y)}\subset T^{\ast }{\mathbb{R}}_y$ in (34) is the image ${\Lambda }_f^{(y)}=T^{\ast }f({\Lambda }_f)$ of

$${\Lambda }_f=\{(\eta ,{\widehat{\theta }}_{\alpha }(\eta )):\eta \in E\}\subset T^{\ast }M(h,k)$$

(35)

where ${\widehat{\theta }}_{\alpha }(\eta )$ is the multiplier in (29) associated to the constraint ${\mathbb{E}}_p\left[h\right]={\eta }_{\alpha }$ and ${\eta }_{\alpha }\in I(\eta )=f^{\leftarrow }(f(\eta ))$.

Proof. 

If $y=f(\eta )$ is the local change of coordinates in $M(h,k)$, then the tangent map $Tf:T{\mathbb{R}}_{\eta }\to T{\mathbb{R}}_y$ has the local form $(y,\dot{y})=Tf(\eta ,\dot{\eta })=(f(\eta ),f^{\prime }(\eta )\dot{\eta })$ and the cotangent map $T^{\ast }f:T^{\ast }{\mathbb{R}}_{\eta }\to T^{\ast }{\mathbb{R}}_y$ has the local form

$$(y,\lambda )=T^{\ast }f(\eta ,\beta )=\left(f(\eta ),\frac{\beta }{f^{\prime }(\eta )}\right)$$

if we want that the Liouville one-form (see above (1)) has the same canonical form ${\theta }_c=\lambda dy=\beta d\eta$ in the two coordinate charts. See, e.g., in [19] for a proof of this last classical result of differential geometry. □

We want to study the Lagrangian submanifold ${\Lambda }_f$ defined in (35) and its image $L_f=F_D^{-1}({\Lambda }_f)\subset M{(h,k)}^2$, where $F_D^{-1}$ is defined in (14), whose local expression is

$$L_f=\{(\eta ,\widehat{\theta }(\eta )-\widehat{\theta }({\eta }_{\alpha })):\eta \in E\}.$$

(36)

First we consider the case that f is a globally invertible function. In this case, $I(\eta )=f^{\leftarrow }(f(\eta ))=\{\eta \}$ and $\widehat{\theta }(\eta )={\phi }_{\eta }(\eta )$. The Lagrangian submanifold ${\Lambda }_f$ in (35) is the graph of the differential ${\phi }_{\eta }(\eta )$ and it is transversal, see Figure 2a. Moreover, see below (9), if $\eta ={\eta }_0={\mathbb{E}}_q\left[h\right]$ then $\widehat{\theta }({\eta }_0)=0$. As ${\psi }_{\theta }(\theta )=\eta$ is invertible with inverse $\theta =\widehat{\theta }(\eta )$, we have

$$\frac{d\widehat{\theta }}{d\eta }(\eta )={(\frac{d^2\psi }{d{\theta }^2})}^{-1}={\mathrm{var}}_{\widehat{p}}(h)={\mathbb{E}}_{\widehat{p}}\left[h^2\right]-{\eta }^2>0$$

and $\widehat{\theta }(\eta )$ is a monotonically increasing function, see Figure 2a. Its image (36) is $L_f=M(h,k)\times \{0\}$, see Figure 2b.

Figure 2. The case of a transversal Lagrangian submanifold.

If we consider a globally non invertible function f as the one depicted in Figure 1, then $I(\eta )$ contains multiple points and ${\Lambda }_f$ is non transversal at points where $f^{\prime }(\eta )=0$, see Figure 3a. The corresponding image $L_f$ has multiple branches and it is not a manifold at points $(b,c)$ where transversality fails, see Figure 3b).

Figure 3. The case of a folded, i.e., non transversal Lagrangian submanifold.

3.2. Probability Distributions in $L_f$

In this section, we study the structure of the probability distributions in $L_f$. In the local coordinate systems $(\eta ,\theta )$ of $M{(h,k)}^2$, $\eta$ and $\widehat{\theta }(\eta )$ describe the same probability distribution that we write for brevity as $p_i(\eta )=p_i(\widehat{\theta })$. Therefore, the probability distributions in $L_f$ in (36) associated to $\eta$ and $\widehat{\theta }(\eta )-\widehat{\theta }({\eta }_{\alpha })$ are, respectively,

$$p_i(\eta )=e^{\widehat{\theta }{h}_i-k_i-\psi (\widehat{\theta })}$$

(37)

and, see (18),

$$p_i(\widehat{\theta }-\widehat{\theta }({\eta }_{\alpha }))=p_i(\eta )\frac{e^{-\widehat{\theta }({\eta }_{\alpha })h_i}}{{\sum }_i{p}_i(\eta )e^{-\widehat{\theta }({\eta }_{\alpha })h_i}}.$$

(38)

Setting

$${\tilde{p}}_i({\eta }_{\alpha })=\frac{e^{-\widehat{\theta }({\eta }_{\alpha })h_i}}{Z(\lambda )},Z(\lambda )=\sum _i{e}^{-\widehat{\theta }({\eta }_{\alpha })h_i},$$

the above (38) can be rewritten as the discrete version of (19), that is,

$$p_i(\widehat{\theta }-\widehat{\theta }({\eta }_{\alpha }))=\frac{p_i(\eta ){\tilde{p}}_i({\eta }_{\alpha })}{{\sum }_i{p}_i(\eta ){\tilde{p}}_i({\eta }_{\alpha })}.$$

(39)

This last formula can be interpreted as follows; let A and B be two independent random variables $A,B$: $\Omega \to X$, where $X=\{1,\dots ,n\}$ is the discrete state space, described by the probability distributions $p_i$ and ${\tilde{p}}_i$, respectively (for example, A and B describe two dices with n faces). Then, ${\sum }_i{p}_i{\tilde{p}}_i$ is the probability that A and B are found in the same state and

$$Prob(A=i,B=i|A=B)=\frac{p_i{\tilde{p}}_i}{{\sum }_i{p}_i{\tilde{p}}_i}$$

in (39) is the conditional probability that A and B are found in the state i provided that they are found in the same state. Note that for $p_i(\eta )$ in (37) we have $e^{k_i}=q_i$, thus (37) can be rewritten as

$$p_i(\eta )=q_i\frac{e^{\widehat{\theta }{h}_i}}{{\sum }_i{q}_i{e}^{\widehat{\theta }{h}_i}}=\frac{q_i\tilde{p}(\widehat{\theta })}{{\sum }_i{q}_i{\tilde{p}}_i(\widehat{\theta })}$$

and (39) above is equal to

$$p_i(\widehat{\theta }-\widehat{\theta }({\eta }_{\alpha }))=\frac{q_i{p}_i{\tilde{p}}_i}{{\sum }_i{q}_i{p}_i{\tilde{p}}_i}=Prob(A=i,B=i,C=i|A=B=C)$$

where $A,B,C$ are described by $q_i$, $p_i={\tilde{p}}_i(\widehat{\theta }(\eta ))$, ${\tilde{p}}_i={\tilde{p}}_i(\widehat{\theta }({\eta }_{\alpha }))$.

4. Discussion

Canonical coordinates $\eta$ and $\theta$ associated to an exponential family $M(h,k)$ are dually flat coordinates with respect to the duality defined by the canonical divergence. With respect to these coordinates, a generalization of the Pitagorean theorem is proved in Information Geometry which provides a generalized formulation of the Maximum Entropy Principle with linear constraints as a geodesic projection problem (see [2]). Multiplicity of the solutions $\widehat{\theta }(\eta )$ of the Maximum Entropy problem are due to the non uniqueness of the projection. In this paper, we have shown that the set of couples $(\eta ,\widehat{\theta }(\eta ))$ defines a transversal Lagrangian submanifold $\Lambda$ of $T^{\ast }M(h,k)$, and we have seen with an example that if nonlinear constraints are considered the set of possible multiple solutions to the Maximum Entropy problem is globally described by a folded (i.e., a possibly non-trasversal) Lagrangian submanifold ${\Lambda }_f$. We have computed their pull-back to the square manifold $M{(h,k)}^2$ via the map $F_D^{-1}$. We think that this framework offers a complementary view to the generalized Pitagorean Theorem. We plan to address in a subsequent paper a generalization of the theory presented here to a more general form of nonlinear constraint.