Affine Calculus for Constrained Minima of the Kullback–Leibler Divergence

Abstract

The non-parametric version of Amari’s dually affine Information Geometry provides a practical calculus to perform computations of interest in statistical machine learning. The method uses the notion of a statistical bundle, a mathematical structure that includes both probability densities and random variables to capture the spirit of Fisherian statistics. We focus on computations involving a constrained minimization of the Kullback–Leibler divergence. We show how to obtain neat and principled versions of known computations in applications such as mean-field approximation, adversarial generative models, and variational Bayes.

1. Introduction and Notations

Many modern Artificial Intelligence (AI) and machine learning (ML) algorithms are based on non-parametric statistical methods and optimization algorithms based on the minimization of a divergence measure between probability functions. In particular, one computes the gradient of a function defined on the probability simplex; then, the learning uses a gradient ascent technique. Such a basic approach is illustrated, for example, in the textbook [1] (Ch. 18).

In most papers, ordinary convex calculus tools on the open probability simplex provide the relevant derivatives and gradients. The relation between the analytic computations and their statistical meaning is not exposed. This paper focuses on the derivative and gradient computations by providing the geometric framework called Information Geometry (IG). This geometry differs from the usual convex analysis because its devices have a direct statistical meaning. For example, the velocity of a one-dimensional parametric curve $\theta \mapsto p\left(\theta \right)$ in the open probability simplex is defined to be the Fisher’s score ${\displaystyle \frac{d}{d\theta }}logp\left(\theta \right)$ instead of the ordinary derivative ${\displaystyle \frac{d}{d\theta }}p\left(\theta \right)$. Generally speaking, IG is a geometric interpretation of Fisherian inference ([1], Ch. 5).

Amari’s Information Geometry (IG) [2,3,4] has been successfully applied to modern AI algorithms; see, for example, [5]. Here, we use the non-parametric version of IG of [6,7]. This version is non-parametric because the basic set of states is the open probability simplex, it is affine, as it satisfies a generalization of the classical Weyl’s axioms [8]. Moreover, it is dually affine in the sense already defined in Amari contributions because the covariance bilinear operator appears as a duality pairing in the vector space of coordinates.

The specific applications we will consider as examples come from the literature in statistical ML, particularly those that involve the constrained minimization of the Kullback–Leibler divergence (KL-divergence). Indeed, our main result in Section 2 is a form of the total gradient of the KL-divergence as expressed in the dually affine geometry. Namely, we consider symmetric divergences [9], generative adversarial networks [10], mixed entropy and transport optimization [11,12], and variational Bayes [13,14].

Non-parametric IG stands in general sample spaces and under various functional assumptions. One option, among many, is the use of Orlicz spaces [15]; see [6,16]. In this paper, we are not interested in discussing the functional setup. Still, we are interested in presenting the peculiar affine calculus of positive probability functions on a finite state space $\Omega$ in a geometric language compatible with the infinite-dimensional theory [17]. Such a calculus provides principled definitions of a curve’s velocity, the scalar field’s gradient, and the gradient flow.

1.1. Prerequisites

Below, we provide a schematic summary of the theory. For complete details, we refer to previous presentations in [7,18].

Let $\Omega$ be a finite sample space. We look at the open simplex as the maximal exponential model denoted as $\mathcal{E}\left(\Omega \right)$. In fact, we present every couple of positive probability functions on $\Omega$, say $p,q$, in the form inspired by Statistical Physics [19]:

$$q={\mathrm{e}}^{v-\Psi }·p,$$

(1)

where p represents a ground state, q is a perturbation of the ground state, v is a random variable, $\Psi$ is a normalizing constant, and $\psi ={\kappa }_p\left(v\right)=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right]$ is the cumulant function.

The random variable v depends on p and q up to a constant. If we specify ${\mathbb{E}}_p\left[v\right]=0$ in Equation (1), then a straightforward computation gives

$$\begin{gathered} \begin{array}{}v=log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{q}{p}}\right],q={\mathrm{e}}^{v-{\kappa }_p\left(v\right)}·p,& \\ {\kappa }_p\left(v\right)=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right]={\mathbb{E}}_p\left[log{\displaystyle \frac{p}{q}}\right]=D\left(p\parallel q\right),& \\ \end{array} \end{gathered}$$

where $D\left(p\parallel q\right)$ is the KL-divergence. Regarding the entropy,

$$D\left(p\parallel q\right)={\mathbb{E}}_p\left[log{\displaystyle \frac{p}{q}}\right]={\mathbb{E}}_p\left[logp\right]-{\mathbb{E}}_p\left[logq\right]=-\mathcal{H}\left(p\right)+\mathcal{H}\left(p,q\right).$$

If we specify ${\mathbb{E}}_q\left[v\right]=0$ in Equation (1), an analogous computation gives

$$\begin{gathered} \begin{array}{}v=log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_q\left[log{\displaystyle \frac{q}{p}}\right],q={\mathrm{e}}^{v-{\kappa }_p\left(v\right)}·p,& \\ {\kappa }_p\left(v\right)=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right]=-{\mathbb{E}}_q\left[log{\displaystyle \frac{p}{q}}\right]=-D\left(q\parallel p\right).& \\ \end{array} \end{gathered}$$

A vector bundle is a collection of vector spaces, and each vector space is called a fiber of the bundle. For example, the tangent bundle collects all tangent vectors at each point in differential geometry. In Fisher’s statistics of the open probability simplex, one considers the vector space of all Fisher’s scores of one-dimensional models through the probability function q. Inspired by this last example, we call the statistical bundle the vector bundle $S\mathcal{E}\left(\Omega \right)$ of all couples $(q,v)$ of a positive probability function q and a q-centered random variable, ${\mathbb{E}}_q\left[v\right]=0$,

$$S\mathcal{E}\left(\Omega \right)=\left\{(q,v)|q\in \mathcal{E}\left(\Omega \right),{\mathbb{E}}_q\left[v\right]=0\right\}.$$

Each fiber $S_q\mathcal{E}\left(\Omega \right)$ is a Euclidean space for the covariance inner product ${〈v,w〉}_q={\mathbb{E}}_q\left[vw\right]$.

The covariance inner product is both a Riemannian metric and a duality pairing. The metric interpretation leads to the Riemannian version of IG. The duality pairing interpretation leads to our dually affine IG. Because of that, we want to distinguish between the fibers $S_p\mathcal{E}\left(\Omega \right)$ and the dual fibers ${{}^{*}S}_p\mathcal{E}\left(\Omega \right)$. The first bundle is called exponential bundle, while the second bundle is called mixture bundle. We use the notation

$${{}^{*}S}_p\mathcal{E}\left(\Omega \right)\times S_p\mathcal{E}\left(\Omega \right)\ni (v,w)\mapsto {〈v,w〉}_p,p\in \mathcal{E}\left(\Omega \right).$$

In our setup, all the vector spaces of random variables are finite-dimensional; hence, the fibers $S_p\mathcal{E}\left(\Omega \right)$ and ${{}^{*}S}_p\mathcal{E}\left(\Omega \right)$ are equal vector spaces. However, it is a useful distinction, as it will be apparent in the discussion of parallel transports below.

The definition of the statistical bundle aims to capture an essential mechanism of Fisher’s approach to statistics ([1], Ch. 4). Suppose $t\mapsto q\left(t\right)\in \mathcal{E}\left(\Omega \right)$ is a one-dimensional statistical model. In that case, the Fisher’s score is $t\mapsto {\displaystyle \frac{d}{dt}}logq\left(t\right)=\stackrel{\star }{q}\left(t\right)$, and $t\mapsto (q\left(t\right),\stackrel{\star }{q}\left(t\right))\in S\mathcal{E}\left(\Omega \right)$ is the lift of the curve to the statistical bundle.

Dually affine geometry follows from the definition of two parallel transports on the fibers and two affine charts. The parallel transports act between the fibers

$$\begin{array}{c}{S}_q\mathcal{E}\left(\Omega \right)\ni v\mapsto {{}^{\mathrm{e}}\mathbb{U}}_q^{r}v=v-{\mathbb{E}}_r\left[v\right]\in S_r\mathcal{E}\left(\Omega \right)\mathit{is}\mathit{the}\mathit{exponential}\mathit{transport},\end{array}$$

(2)

$$\begin{array}{c}{{}^{*}S}_q\mathcal{E}\left(\Omega \right)\ni w\mapsto {{}^{\mathrm{m}}\mathbb{U}}_q^{r}w={\displaystyle \frac{q}{r}}v\in {{}^{*}S}_r\mathcal{E}\left(\Omega \right)\mathit{is}\mathit{the}\mathit{mixture}\mathit{transport}.\end{array}$$

(3)

It is easy to check that the transports are duals of each other:

$$\begin{array}{c}{〈v,{{}^{\mathrm{e}}\mathbb{U}}_r^{q}w〉}_q={〈{{}^{\mathrm{m}}\mathbb{U}}_q^{r}v,w〉}_r\mathit{is}\mathit{the}{\mathit{transport}}^{\prime }\mathit{s}\mathit{duality},\end{array}$$

(4)

$$\begin{array}{c}{〈v,w〉}_q={〈{{}^{\mathrm{m}}\mathbb{U}}_q^{r}v,{{}^{\mathrm{e}}\mathbb{U}}_q^{r}w〉}_r\mathit{is}\mathit{the}\mathit{inner}\mathit{product}\mathit{push}.\end{array}$$

(5)

The affine charts that define the two dual affine geometries by mapping the base set to a vector space of coordinates are

$$\begin{array}{c}q\mapsto s_p\left(q\right)=log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{q}{p}}\right]\in S_p\mathcal{E}\left(\Omega \right)\mathit{is}\mathit{the}\mathit{exponential}\mathit{chart}\end{array}$$

(6)

$$\begin{array}{c}q\mapsto {\eta }_p\left(q\right)={\displaystyle \frac{q}{p}}-1\in {{}^{*}S}_p\mathcal{E}\left(\Omega \right)\mathit{is}\mathit{the}\mathit{mixture}\mathit{chart},\end{array}$$

(7)

and the geometries defined by the two atlases are affine because the parallelogram law holds in both cases:

$$\begin{gathered} \begin{array}{}{s}_p\left(q\right)+{{}^{\mathrm{e}}\mathbb{U}}_q^p{s}_q\left(r\right)=s_p\left(r\right)& \\ {\eta }_p\left(q\right)+{{}^{\mathrm{m}}\mathbb{U}}_q^p{\eta }_q\left(r\right)={\eta }_p\left(r\right)& \\ \end{array} \end{gathered}$$

The inverse of the exponential chart is a non-parametric exponential family ([1], Ch. 5), and the known mechanisms of the cumulant function provide a fundamental calculus tool [20]. If $K_p$ is the restriction of ${\kappa }_p$ to $S_p\mathcal{E}\left(\Omega \right)$ and $v=s_p\left(q\right)$ then $q=s_p^{-1}\left(v\right)=e_p\left(v\right)={\mathrm{e}}^{v-K_p\left(v\right)}·p$,

$$\begin{array}{c}{K}_p\left(v\right)=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right]\mathit{cumulant}\mathit{function}\end{array}$$

(8)

$$\begin{array}{c}D\left(p\parallel e_p\left(v\right)\right)=K_p\left(v\right)\mathit{cumulant}\mathit{function}\mathit{express}\mathit{the}\mathit{KL}\text{-}\mathit{divergence},\end{array}$$

(9)

$$\begin{array}{c}d{K}_p\left(v\right)\left[h\right]={\mathbb{E}}_{e_p\left(v\right)}\left[h\right]\mathit{is}\mathit{the}\mathit{derivative}\mathit{of}{\mathit{K}}_{\mathit{p}}\mathit{in}\mathit{the}\mathit{direction}\mathit{h},\end{array}$$

(10)

$$\begin{array}{c}{d}^{2}K\left(v\right)[h,k]={Cov}_{e_p\left(v\right)}\left(h,k\right)\mathit{is}\mathit{the}\mathit{second}\mathit{derivative}\mathit{of}{\mathit{K}}_{\mathit{p}}\mathit{in}\mathit{the}\mathit{directions}\mathit{h},\mathit{k}.\end{array}$$

(11)

Equations (10) and (11) are the non-parametric version of the well-known properties of the derivative of the cumulant function in exponential models; see ([1], § 5.5) and [20].

We can now show that Fisher’s score is a velocity in the technical sense of a velocity computed in the moving frame of both charts. If $t\mapsto q\left(t\right)\in \mathcal{E}\left(\Omega \right)$ is a smooth curve, and $\Phi :\mathcal{E}\left(\Omega \right)\to \mathbb{R}$ is a smooth mapping,

$$\begin{array}{c}\stackrel{\star }{q}\left(t\right)={\left.{\displaystyle \frac{d}{dt}}{s}_p\left(q\left(t\right)\right)\right|}_{p=q\left(t\right)}={\left.{\displaystyle \frac{d}{dt}}{\eta }_p\left(q\left(t\right)\right)\right|}_{p=q\left(t\right)}={\displaystyle \frac{d}{dt}}logq\left(t\right)\mathit{is}\mathit{the}\mathit{velocity},\end{array}$$

(12)

$$\begin{array}{c}{\displaystyle \frac{d}{dt}}\Phi \left(q\left(t\right)\right)={〈grad\Phi \left(q\left(t\right)\right),\stackrel{\star }{q}\left(t\right)〉}_{q\left(t\right)}\mathit{is}\mathit{the}\mathit{gradient}.\end{array}$$

(13)

The squared norm of the velocity (12),

$${〈\stackrel{\star }{q}\left(t\right),\stackrel{\star }{q}\left(t\right)〉}_{q\left(t\right)}={\mathbb{E}}_{q\left(t\right)}\left[{\left({\displaystyle \frac{d}{dt}}logq\left(t\right)\right)}^2\right]=\sum {\displaystyle \frac{{\dot{q}\left(t\right)}^2}{q\left(t\right)}},$$

is the Fisher information that appeared first in the classical Cramer–Rao lower bound.

The gradient defined in Equation (13) is frequently called the natural gradient in the IG literature, following the use introduced in the case of parametric models by Amari [3]. In Riemannian geometry [17,21], the metric acts as a duality pairing, and the definition of the gradient is similar to Equation (13). The classic example of the computation of the gradient is the gradient of the expected value as a function of the probability function,

$${\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q\left(t\right)}\left[u\right]={\displaystyle \frac{d}{dt}}\sum u{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}q\left(t\right)={〈u-{\mathbb{E}}_{q\left(t\right)}\left[u\right],\stackrel{\star }{q}\left(t\right)〉}_{q\left(t\right)},$$

so that $grad{\mathbb{E}}_q\left[u\right]=u-{\mathbb{E}}_q\left[u\right]$.

The gradient of $\Phi$ gives the velocities of curves “orthogonal” to the surfaces of constant $\Phi$-value, that is, the curves of steepest ascent. The solutions of the equation $grad\Phi =0$ are the stationary points of $\Phi$, and an equation of the form

$$\stackrel{\star }{q}\left(t\right)=ϵ\left(t\right)grad\Phi \left(q\left(t\right)\right)$$

is a gradient flow equation.

In conclusion, we review the derivation of a function f between two maximal exponential models using the mixture charts Equation (7). The expressions of f and its derivative $df$ in the charts centered, respectively, at $p_1$ and $p_2$, are

$$\begin{array}{cc}\begin{array}{c}\hfill \mathcal{E}\left({\mu }_1\right)\underrightarrow{f}\end{array}\hfill & \mathcal{E}\left({\mu }_2\right)\\ n_{p_1}^{{-}_1}\uparrow \hfill & \downarrow {\eta }_{p2}\\ S_{p_1}\mathcal{E}\left({\mu }_1\right)\underset{f_{p_1·p_2}}{\to }\hfill & S_{p_2}\mathcal{E}\left({\mu }_2\right)\end{array}\mathrm{and}\begin{array}{cc}\begin{array}{c}\hfill S_{q_1}\mathcal{E}\left({\mu }_1\right)\underrightarrow{df\left(q_1\right)}\end{array}\hfill & S_{f\left(q_1\right)}\mathcal{E}\left({\mu }_2\right)\\ {{}^{\mathrm{m}}\mathbb{U}}_{p_1}^{q_1}\uparrow \hfill & \downarrow {{}^{\mathrm{m}}\mathbb{U}}_{f\left(p_1\right)}^{q_2}\\ S_{p_1}\mathcal{E}\left({\mu }_1\right)\underset{d{f}_{p_1·p_2\left({\mathrm{\eta }}_{p_1}\right(q_1)}}{\to }\hfill & S_{p_2}\mathcal{E}\left({\mu }_2\right)\end{array}$$

It follows that the computation of the derivative from its expression is

$$df\left(q\right)\left[\stackrel{\star }{q}\right]={{}^{\mathrm{m}}\mathbb{U}}_{p_2}^{q}d{f}_{p_1,p_2}\left({\eta }_{p_1}\left(q\right)\right)\left[{{}^{\mathrm{m}}\mathbb{U}}_q^{p_1}\stackrel{\star }{q}\right].$$

(14)

1.2. Summary of Content

In the following sections, we give both new results and new versions of the known results. The aim is to show the interest of the non-parametric dually affine IG in computing the gradient flow of a constrained KL-divergence.

In Section 2, we show how to use the statistical bundle formalism to compute derivatives of functions defined on the open probability simplex and how to compute natural gradients and total natural gradients of the KL-divergence, the cross entropy, the entropy, and the Jensen–Shannon divergence.

In Section 3, we apply the general computations of the previous section to independence models and marginal conditional probabilities in a factorial product setting. The dually affine methodology methodically reproduces known computations and suggests neat variations of potential interest. In particular, Section 3.5 contains a fully worked example of the derivation of a gradient flow equation of interest in approximate Bayes computations.

2. Total Natural Gradient of the KL-Divergence

The KL-divergence ([5], Ch. 3) as a function of two variables is

$$D:\mathcal{E}\left(\Omega \right)\times \mathcal{E}\left(\Omega \right)\ni (q,r)\mapsto D\left(q\parallel r\right)={\mathbb{E}}_q\left[log{\displaystyle \frac{q}{r}}\right].$$

(15)

The computation of the total derivative is well-known in Information Theory. However, we provide proof in the affine setting, expressing the result in the affine charts.

In the exponential chart at p and in the mixture chart at p, the expressions of the probability functions q and r are, respectively,

$$q=e_p\left(v\right)={\mathrm{e}}^{v-K_p\left(v\right)}·p,r={\eta }_p^{-1}\left(w\right)=(1+w)·p,$$

(16)

By plugging (16) into (15) and using Equation (10), one sees that the expressions of the partial KL-divergences are, respectively,

$$\begin{gathered} \begin{array}{c}D\left(e_p\left(v\right)\parallel r\right)={\mathbb{E}}_{e_p\left(v\right)}\left[v-K_p\left(v\right)\right]-{\mathbb{E}}_{e_p\left(v\right)}\left[log{\displaystyle \frac{r}{p}}\right]=\hfill \\ {\mathbb{E}}_{e_p\left(v\right)}\left[v-K_p\left(v\right)\right]-{\mathbb{E}}_{e_p\left(v\right)}\left[s_p\left(v\right)\right]-D\left(p\parallel r\right)=\\ \hfill d{K}_p\left(v\right)\left[v\right]-K_p\left(v\right)-d{K}_p\left(v\right)\left[s_p\left(r\right)\right]-D\left(p\parallel r\right),& \\ \end{array} \end{gathered}$$

(17)

and

$$D\left(q\parallel {\eta }_p^{-1}\left(w\right)\right)=D\left(q\parallel p\right)-{\mathbb{E}}_q\left[log(1+w)\right].$$

(18)

Notice that the peculiar choice of the charts in the combination exponential for the first variable and mixture for the second variable is inessential in the finite state space case because any other choice will produce the same final result in the computation of the total natural gradient. However, it is consistent with the dual affine setting, in which two connections exist between one space and its dual. However, the expression of the KL-divergence using the exponential chart in both variables is interesting because, in such a case, the resulting expression is equal to the Bregman divergence of the cumulant function $K_p$,

$$D\left(e_p\left(v\right)\parallel e_p\left(w\right)\right)=K_p\left(w\right)-K_p\left(v\right)-d{K}_p\left(v\right)[w-v],$$

which, in turn, is the second remainder in the Taylor expansion. For example, one closed form is

$$D\left(e_p\left(v\right)\parallel e_p\left(w\right)\right)={\int }_0^1{\int }_0^1{d}^{2}K(v+st(w-v))[w-v,w-v]sdsdt.$$

If $q(s,t)=e_p(v+st(w-v))\propto q^{1-st}{r}^{st}$, then by Equation (11),

$$D\left(q\parallel r\right)={\int \int }_0^1{Var}_{q(s,t)}\left(log{\displaystyle \frac{r}{q}}\right)sdsdt.$$

2.1. Total Natural Gradient of the KL-Divergence

We compute our gradients in the duality induced on each fiber by the covariance; hence, the total natural gradient of the KL-divergence has two components implicitly defined by

$${\displaystyle \frac{d}{dt}}D\left(q\left(t\right)\parallel r\left(t\right)\right)={〈\stackrel{\star }{q}\left(t\right),{grad}_{1}D\left(q\left(t\right)\parallel r\left(t\right)\right)〉}_{q\left(t\right)}+{〈{grad}_{2}D\left(q\left(t\right)\parallel r\left(t\right)\right),\stackrel{\star }{r}\left(t\right)〉}_{r\left(t\right)},$$

(19)

where ${grad}_{1}D\left(q\parallel r\right)$ is a random variable in the fiber at Q, while ${grad}_{2}D\left(q\parallel r\right)$ is a random variable in the fiber at r. The adjective total refers to the fact that D is a function of two variables.

Proposition 1.

The total natural gradient of the KL-divergence is

$$(q,r)\mapsto gradD\left(q\parallel r\right)=\left(-s_q\left(r\right),-{\eta }_r\left(q\right)\right)\in S\mathcal{E}\left(\Omega \right)\times {}^{*}S\mathcal{E}\left(\Omega \right).$$

(20)

That is, more explicitly, for each smooth couple of curves $t\mapsto q\left(t\right)$ and $t\mapsto r\left(t\right)$, Equation (19) becomes

$${\displaystyle \frac{d}{dt}}D\left(q\left(t\right)\parallel r\left(t\right)\right)=-{〈\stackrel{\star }{q}\left(t\right),s_{q\left(t\right)}\left(r\left(t\right)\right)〉}_{q\left(t\right)}-{〈{\eta }_{r\left(t\right)}\left(q\left(t\right)\right),\stackrel{\star }{r}\left(t\right)〉}_{r\left(t\right)}.$$

Proof. 

From Equation (11), the derivative at $v\in S_p\mathcal{E}\left(\Omega \right)$ of Equation (17) in the direction $h={{}^{\mathrm{e}}\mathbb{U}}_q^p\stackrel{\star }{q}$ is

$$\begin{gathered} \begin{array}{c}{d}^2{K}_p\left(v\right)[v,h]-d{K}_p\left(v\right)\left[h\right]+d{K}_p\left(v\right)\left[h\right]-d^2{K}_p\left(v\right)[s_p\left(r\right),h]=\hfill & \\ {Cov}_q\left(s_p\left(q\right)-s_p\left(r\right),h\right)={Cov}_q\left(log{\displaystyle \frac{q}{r}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{q}{r}}\right],h\right)=\\ \hfill {\mathbb{E}}_q\left[\left(log{\displaystyle \frac{q}{r}}-{\mathbb{E}}_q\left[log{\displaystyle \frac{q}{r}}\right]\right)\left(h-{\mathbb{E}}_q\left[h\right]\right)\right]={〈-s_q\left(r\right),\stackrel{\star }{q}〉}_q.\end{array} \end{gathered}$$

The derivative at $w\in {{}^{*}S}_p\mathcal{E}\left(\Omega \right)$ of Equation (18) in the direction $k={{}^{\mathrm{m}}\mathbb{U}}_r^p\stackrel{\star }{r}$ is

$$-{\mathbb{E}}_q\left[{\displaystyle \frac{p}{r}}k\right]=-{\mathbb{E}}_r\left[{\displaystyle \frac{q}{r}}\stackrel{\star }{r}\right]={〈-{\eta }_r\left(q\right),\stackrel{\star }{r}〉}_r.$$

□

The gradient computation forms the corresponding gradient flow equation, whose discretization provides basic optimization algorithms. Here are two basic examples.

Given $r\in \mathcal{E}\left(\Omega \right)$, the solution of the gradient flow equation

$$\stackrel{\star }{q}\left(t\right)=-{grad}_{1}D\left(q\left(t\right)\parallel r\right)=s_q\left(r\right),q\left(0\right)=q_0,$$

is the exponential family

$$t\mapsto q\left(t\right)={\mathrm{e}}^{{\mathrm{e}}^{-t}{v}_0-K_r\left({\mathrm{e}}^{-t}{v}_0\right)}·r,v_0=s_r\left(q_0\right).$$

(21)

In fact, the LHS of Equation (21) is given by

$$\begin{gathered} \begin{array}{c}logq\left(t\right)={\mathrm{e}}^{-t}{v}_0-K_r\left({\mathrm{e}}^{-t}{v}_0\right)+logr,& \\ \stackrel{\star }{q}\left(t\right)=-{\mathrm{e}}^{-t}{v}_0+d{K}_r\left({\mathrm{e}}^{-t}{v}_0\right)\left[{\mathrm{e}}^{-t}{v}_0\right],\end{array} \end{gathered}$$

while the RHS is

$$\begin{array}{c}{s}_{q\left(t\right)}\left(r\right)=-{\mathrm{e}}^{-t}{v}_0+K_r\left({\mathrm{e}}^{-t}{v}_0\right)-{\mathbb{E}}_{q\left(t\right)}\left[-{\mathrm{e}}^{-t}{v}_0+K_r\left({\mathrm{e}}^{-t}{v}_0\right)\right]=\hfill \\ \hfill -{\mathrm{e}}^{-t}{v}_0+{\mathbb{E}}_{q\left(t\right)}\left[{\mathrm{e}}^{-t}{v}_0\right].\end{array}$$

The conclusion follows from Equation (10).

Given $q\in \mathcal{E}\left(\Omega \right)$, the solution of the gradient flow equation

$$\stackrel{\star }{r}\left(t\right)=-{grad}_{2}D\left(q\parallel r\left(t\right)\right)={\eta }_{r\left(t\right)}\left(q\right),r\left(0\right)=r_0,$$

is the mixture family

$$t\mapsto r\left(t\right)={\mathrm{e}}^{-t}{r}_0+(1-{\mathrm{e}}^{-t})q.$$

(22)

The LHS of Equation (22) is

$$\stackrel{\star }{r}\left(t\right)={\displaystyle \frac{\dot{r}\left(t\right)}{r\left(t\right)}}={\displaystyle \frac{-{\mathrm{e}}^{-t}{r}_0+{\mathrm{e}}^{-t}q}{{\mathrm{e}}^{-t}{r}_0+(1-{\mathrm{e}}^{-t})q}}={\displaystyle \frac{q-r_0}{r_0+({\mathrm{e}}^t-1)q}},$$

while the RHS is

$${\eta }_{r\left(t\right)}\left(q\right)={\displaystyle \frac{q}{{\mathrm{e}}^{-t}{r}_0+(1-{\mathrm{e}}^{-t})q}}-1={\displaystyle \frac{q-{\mathrm{e}}^{-t}{r}_0+(1-{\mathrm{e}}^{-t})q}{{\mathrm{e}}^{-t}{r}_0+(1-{\mathrm{e}}^{-t})q}}={\displaystyle \frac{q-r_0}{r_0+({\mathrm{e}}^t-1)q}}.$$

Notice that in both cases, the t parameter appears in the solution in exponential form. Other forms of the temperature parameter will follow from a weighted form of the gradient flow equation.

2.2. Natural Gradient of the Entropy and Total Natural Gradient of the Cross Entropy

The KL-divergence equals the cross entropy minus the entropy,

$$D\left(q\parallel r\right)={\mathbb{E}}_q\left[-logr\right]-{\mathbb{E}}_q\left[-logq\right]=\mathcal{H}\left(q,r\right)-\mathcal{H}\left(q\right).$$

In the exponential chart at p for the first variable, the cross entropy is

$$\begin{gathered} \begin{array}{c}\mathcal{H}\left(s_p^{-1}\left(v\right),r\right)=\hfill & \\ {\mathbb{E}}_{e_p\left(v\right)}\left[-logr\right]={\mathbb{E}}_{e_p\left(v\right)}\left[-logr-{\mathbb{E}}_p\left[-logr\right]\right]+\mathcal{H}\left(p,r\right)=\\ \hfill d{K}_p\left(v\right)[-logr-{\mathbb{E}}_p\left[-logr\right]]+\mathcal{H}\left(p,r\right),\end{array} \end{gathered}$$

(23)

with derivative at v in the direction h

$$\begin{gathered} \begin{array}{c}{d}^2{K}_p\left(v\right)[-logr-{\mathbb{E}}_p\left[-logr\right],h]={Cov}_q\left(-logr,h\right)=\hfill & \\ \hfill {\mathbb{E}}_q\left[\left(-logr-\mathcal{H}\left(q,r\right)\right)\left(h-{\mathbb{E}}_q\left[h\right]\right)\right]={〈\stackrel{\star }{q},-logr-\mathcal{H}\left(q,r\right)〉}_q.\end{array} \end{gathered}$$

In the mixture chart at p for the second variable

$$\mathcal{H}\left(q,{\eta }_p^{-1}\left(w\right)\right)={\mathbb{E}}_q\left[-log\left((1+w)·p\right)\right]={\mathbb{E}}_q\left[-log\left(1+w\right)\right]+\mathcal{H}\left(q,p\right),$$

(24)

with derivative at w in the direction k,

$${\mathbb{E}}_q\left[-{(1+w)}^{-1}k\right]={\mathbb{E}}_r\left[-{\displaystyle \frac{q}{r}}{{}^{\mathrm{m}}\mathbb{U}}_p^{r}k\right]={\mathbb{E}}_r\left[-\left({\displaystyle \frac{q}{r}}-1\right)\stackrel{\star }{r}\right]={〈{\eta }_r\left(q\right),\stackrel{\star }{r}〉}_r.$$

Proposition 2.

The total natural gradient of the cross entropy is

$$grad\mathcal{H}\left(q,r\right)=(-logr-\mathcal{H}\left(q,r\right),{\eta }_r\left(q\right))$$

and the natural gradient of the entropy is

$$grad\mathcal{H}\left(q\right)=-logq-\mathcal{H}\left(q\right).$$

(25)

Proof. 

The first statement follows from Equations (23) and (24). From the decomposition $\mathcal{H}\left(q\right)=\mathcal{H}\left(q,r\right)-D\left(q\parallel r\right)$, we find the gradient of the entropy,

$$grad\mathcal{H}\left(q\right)=-logr-\mathcal{H}\left(q,r\right)+s_q\left(r\right)=-logq-\mathcal{H}\left(q\right).$$

□

2.3. Total Natural Gradient of the Jensen–Shannon Divergence

The Jensen–Shannon divergence [9] is

$$\begin{gathered} \begin{array}{c}JS\left(q,r\right)={\displaystyle \frac{1}{2}}D\left(q\parallel {\displaystyle \frac{1}{2}}(q+r)\right)+{\displaystyle \frac{1}{2}}D\left(r\parallel {\displaystyle \frac{1}{2}}(q+r)\right)=\hfill & \\ \hfill \mathcal{H}\left({\displaystyle \frac{1}{2}}(q+r)\right)-{\displaystyle \frac{1}{2}}\mathcal{H}\left(q\right)-{\displaystyle \frac{1}{2}}\mathcal{H}\left(r\right).\end{array} \end{gathered}$$

(26)

It is the minimum value of the function

$$\varphi :p\mapsto {\displaystyle \frac{1}{2}}\left(D\left(q\parallel p\right)+D\left(r\parallel p\right)\right).$$

In fact,

$$grad\varphi \left(p\right)=-{\displaystyle \frac{1}{2}}\left({\eta }_p\left(q\right)+{\eta }_p\left(r\right)\right)={\displaystyle \frac{1}{2}}\left(-{\displaystyle \frac{q}{p}}+1-{\displaystyle \frac{r}{p}}+1\right)=-{\displaystyle \frac{\frac{1}{2}(q+r)}{p}}+1$$

which vanishes for $p={\displaystyle \frac{1}{2}}(q+r)$.

Let us compute the derivative of $f:q\mapsto {\displaystyle \frac{1}{2}}(q+r)$. The mixture expression of f at p according to Equation (7) is the affine function

$$f_p\left(v\right)={\eta }_p\circ f\circ {\eta }_p^{-1}\left(v\right)={\displaystyle \frac{\frac{1}{2}(1+v)·p+r}{p}}-1={\displaystyle \frac{1}{2}}v+{\displaystyle \frac{1}{2}}{\eta }_p\left(r\right),$$

so that the derivative in the direction h is $d{f}_p\left(v\right)\left[h\right]=h/2$.

The push-back, according to the mixture transport Equation (3), is

$$df\left(q\right)\left[\stackrel{\star }{q}\right]={{}^{\mathrm{m}}\mathbb{U}}_p^{{\displaystyle \frac{1}{2}}(q+r)}{\displaystyle \frac{1}{2}}{{}^{\mathrm{m}}\mathbb{U}}_q^p\stackrel{\star }{q}={\displaystyle \frac{p}{\frac{1}{2}(q+r)}}{\displaystyle \frac{1}{2}}{\displaystyle \frac{q}{p}}\stackrel{\star }{q}={\displaystyle \frac{1}{2}}{\displaystyle \frac{q}{\frac{1}{2}(q+r)}}\stackrel{\star }{q}={\displaystyle \frac{1}{2}}{{}^{\mathrm{m}}\mathbb{U}}_q^{{\displaystyle \frac{1}{2}}(q+r)}\stackrel{\star }{q}.$$

(27)

We now compute the gradient of $q\mapsto JS\left(q,r\right)$ of Equation (26), using the total natural gradient of the KH-divergence of Proposition 1, the derivative Equation (27), and the duality of parallel transports Equation (4):

$$\begin{array}{c}grad(q\mapsto JS\left(q,r\right))=\hfill \\ {\displaystyle \frac{1}{2}}\left(-s_q\left({\displaystyle \frac{1}{2}}(q+r)\right)-{\displaystyle \frac{1}{2}}{{}^{\mathrm{e}}\mathbb{U}}_{{\displaystyle \frac{1}{2}}(q+r)}^q{\eta }_{{\displaystyle \frac{1}{2}}(q+r)}\left(q\right)\right)+{\displaystyle \frac{1}{2}}\left(-{\displaystyle \frac{1}{2}}{{}^{\mathrm{e}}\mathbb{U}}_{{\displaystyle \frac{1}{2}}(q+r)}^q{\eta }_{{\displaystyle \frac{1}{2}}(q+r)}\left(r\right)\right)=\\ {\displaystyle \frac{1}{2}}\left(-s_q\left({\displaystyle \frac{1}{2}}(q+r)\right)-{\displaystyle \frac{1}{2}}{{}^{\mathrm{e}}\mathbb{U}}_{{\displaystyle \frac{1}{2}}(q+r)}^q\left({\eta }_{{\displaystyle \frac{1}{2}}(q+r)}\left(q\right)+{\eta }_{{\displaystyle \frac{1}{2}}(q+r)}\left(r\right)\right)\right)=\\ {\displaystyle \frac{1}{2}}\left(-s_q\left({\displaystyle \frac{1}{2}}(q+r)\right)-{\displaystyle \frac{1}{2}}{{}^{\mathrm{e}}\mathbb{U}}_{{\displaystyle \frac{1}{2}}(q+r)}^q\left({\displaystyle \frac{q}{\frac{1}{2}(q+r)}}-1+{\displaystyle \frac{r}{\frac{1}{2}(q+r)}}-1\right)\right)=\\ \hfill -{\displaystyle \frac{1}{2}}{s}_q\left({\displaystyle \frac{1}{2}}(q+r)\right).\end{array}$$

It is also instructive to use the expression of the Jensen–Shannon divergence in terms of entropies. From Equation (25),

$$\begin{array}{c}grad(q\mapsto JS\left(q,r\right))=\hfill \\ {\displaystyle \frac{1}{2}}{{}^{\mathrm{e}}\mathbb{U}}_{{\displaystyle \frac{1}{2}}(q+r)}^q\left(-log{\displaystyle \frac{1}{2}}(q+r)-\mathcal{H}\left({\displaystyle \frac{1}{2}}(q+r)\right)\right)-{\displaystyle \frac{1}{2}}\left(-logq-\mathcal{H}\left(q\right)\right)=\\ -{\displaystyle \frac{1}{2}}\left(log{\displaystyle \frac{1}{2}}(q+r)-{\mathbb{E}}_q\left[log{\displaystyle \frac{1}{2}}(q+r)\right]+logq-{\mathbb{E}}_q\left[logq\right]\right)=\\ \hfill -{\displaystyle \frac{1}{2}}{s}_q\left({\displaystyle \frac{1}{2}}(q+r)\right).\end{array}$$

3. Product Sample Space

This section uses $\Omega ={\Omega }_1\times {\Omega }_2$ as a factorial sample space. For each $r\in \mathcal{E}\left(\Omega \right)$, the margins are $r_1\in \mathcal{E}\left({\Omega }_1\right)$ and $r_2\in \mathcal{E}\left({\Omega }_2\right)$. In the mean-field assumption, the model equals the tensor product of the margins,

$$\overline{r}=r_1\otimes r_2\in \mathcal{E}\left({\Omega }_1\right)\otimes \mathcal{E}\left({\Omega }_2\right)\subset \mathcal{E}\left(\Omega \right).$$

The velocities are, respectively,

$$\begin{array}{c}\stackrel{\star }{r}(x,y;t)={\displaystyle \frac{d}{dt}}logr(x,y;t),\end{array}$$

(28)

$$\begin{array}{c}\stackrel{\star }{\overline{r}}(x,y;t)={\displaystyle \frac{d}{dt}}log\overline{r}(x,y;t)={\stackrel{\star }{r}}_1(x;t)+{\stackrel{\star }{r}}_2(y;t).\end{array}$$

(29)

Below, we will discuss the optimality of a mean-field approximation.

3.1. Product Sample Space: Marginalization

The (first) marginalization is

$${\Pi }_1:\mathcal{E}\left({\Omega }_1\times {\Omega }_2\right)\ni r\mapsto r_1\in \mathcal{E}\left({\Omega }_1\right),r_1\left(x\right)=\sum _{b}r(x,b).$$

(30)

We will compute the bundle derivative of Equation (30) following the scheme of Equation (14).

Proposition 3.

The derivative $d{\Pi }_1$ of the marginalization Equation (30) is

$$d{\Pi }_1:{}^{*}S\mathcal{E}\left(\Omega \right)\ni (r,\stackrel{\star }{r})\mapsto \left(r_1,{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)\right)\in {}^{*}S\mathcal{E}\left({\Omega }_1\right).$$

Proof. 

In the mixture chart centered at $p_1\otimes p_2$ and $p_1$, respectively, the expression of the marginalization is

$$\begin{gathered} \begin{array}{c}{\eta }_{p_1}\circ {\Pi }_1\circ {\eta }_{p_1\otimes p_2}^{-1}\left(v\right)={\displaystyle \frac{{\Pi }_1\circ {\eta }_{p_1\otimes p_2}^{-1}\left(v\right)}{p_1}}-1=\hfill & \\ \hfill {\displaystyle \frac{{\sum }_b(1+v(·,b))·p_1{p}_2\left(b\right)}{p_1}}-1=\sum _{b}v(·,b)p_2\left(b\right).\end{array} \end{gathered}$$

(31)

Note that the expression in Equation (31) is linear. Hence, the derivative at v in the direction h is $x\mapsto {\sum }_{b}h(·,b)p_2\left(b\right)$ with $h={{}^{\mathrm{m}}\mathbb{U}}_r^{p_1\otimes p_2}\stackrel{\star }{r}$ so that the bundle derivative is

$$\begin{gathered} \begin{array}{c}d{\Pi }_1\left(r\right)\left[\stackrel{\star }{r}\right]={{}^{\mathrm{m}}\mathbb{U}}_{p_1}^{r_1}\sum _b{\displaystyle \frac{r(·,b)}{p_1{p}_2\left(b\right)}}\stackrel{\star }{q}(·,b)p_2\left(b\right)=\hfill & \\ \hfill {\displaystyle \frac{p_1}{r_1}}\sum _b{\displaystyle \frac{r(·,b)}{p_1{p}_2\left(b\right)}}\stackrel{\star }{q}(·,b)p_2\left(b\right)=\sum _b\stackrel{\star }{q}(·,b)r_{1|2}(·|b)={\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right).\end{array} \end{gathered}$$

□

There is an interesting relation between conditional expectation and mixture transport. The conditional expectation commutes with the mixture transports,

$${{}^{\mathrm{m}}\mathbb{U}}_{r_1}^{q_1}{\mathbb{E}}_r\left({{}^{\mathrm{m}}\mathbb{U}}_q^{r}v|{\Pi }_1\right)={\mathbb{E}}_q\left(v|{\Pi }_1\right),v\in {{}^{*}S}_q\mathcal{E}\left(\Omega \right).$$

It is a way to express Bayes’ theorem for conditional expectations. For all $\varphi$,

$$\begin{gathered} \begin{array}{c}{\mathbb{E}}_q\left[{{}^{\mathrm{m}}\mathbb{U}}_{r_1}^{q_1}{\mathbb{E}}_r\left({{}^{\mathrm{m}}\mathbb{U}}_q^{r}v|{\Pi }_1\right)\varphi \left({\Pi }_1\right)\right]={\mathbb{E}}_{q_1}\left[{\displaystyle \frac{r_1}{q_1}}{\mathbb{E}}_r\left({\displaystyle \frac{q}{r}}v|{\Pi }_1\right)\varphi \left({\Pi }_1\right)\right]=\hfill & \\ \hfill {\mathbb{E}}_{r_1}\left[{\mathbb{E}}_r\left({\displaystyle \frac{q}{r}}v|{\Pi }_1\right)\varphi \left({\Pi }_1\right)\right]={\mathbb{E}}_r\left[{\mathbb{E}}_r\left({\displaystyle \frac{q}{r}}v|{\Pi }_1\right)\varphi \left({\Pi }_1\right)\right]={\mathbb{E}}_r\left[{\displaystyle \frac{q}{r}}v\Phi \left({\Pi }_1\right)\right]={\mathbb{E}}_q\left[v\varphi \left({\Pi }_1\right)\right].\end{array} \end{gathered}$$

3.2. Product Sample Space: Mean-Field Approximation

The derivative of the joint marginalization

$$\Pi :\mathcal{E}\left({\Omega }_1\times {\Omega }_2\right)\ni r\mapsto r_1{\Pi }_1\left(r\right)\otimes {\Pi }_2\left(r\right)\in \mathcal{E}\left({\Omega }_1\times {\Omega }_2\right),$$

follows from the derivative of the marginalization in Equation (29).

Proposition 4.

The derivative $d\Pi$ of the joint marginalization in  Section 3.2  is

$$d\Pi :(r,\stackrel{\star }{r})\mapsto \left(r_1\otimes r_2,{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)\right).$$

(32)

Proof. 

Compose the partial derivatives with the mapping $({\stackrel{\star }{r}}_1,{\stackrel{\star }{r}}_2)\mapsto {\stackrel{\star }{r}}_1\otimes {\stackrel{\star }{r}}_2$ □

The decomposition of the velocity $velocityr$ according to Equation (32) provides a better decomposition than ${\stackrel{\star }{r}}_1+{\stackrel{\star }{r}}_2$ of Equations (28) and (29) and provides a definition of the mean-field approximation. In the language of ANOVA decomposition of statistical interactions, the derivative part in Equation (32) is the sum of the simple effects of the velocity,

$$\stackrel{\star }{q}={\stackrel{\star }{q}}_1+{\stackrel{\star }{q}}_2+{\stackrel{\star }{q}}_{12},$$

where $q_i={\mathbb{E}}_q\left(\stackrel{\star }{q}|{\Pi }_i\right)$, $i=1,2$, and the last term is the interaction, the q-orthogonal residual. See [22] for a discussion of the ANOVA decomposition in the context of the statistical bundle.

The equation for the total natural gradient of the KL-divergence and the computation of the derivative above provide the natural gradients of the divergence between the joint probability function and the mean-field approximation. In information theory [23], the KL-divergence in Equation (34) is called mutual information.

Proposition 5.

The natural gradients of the divergences of a joint distribution r and its mean-field approximation $\Pi \left(r\right)$ are

$$\begin{array}{c}gradD\left(\Pi \left(r\right)\parallel r\right)={\mathbb{E}}_{r_1\otimes r_2}\left(s_r(r_1\otimes r_2)|{\Pi }_1\right)+{\mathbb{E}}_{r_1\otimes r_2}\left(s_r(r_1\otimes r_2)|{\Pi }_2\right)-{\eta }_r(r_1\otimes r_2).\end{array}$$

(33)

$$\begin{array}{c}gradD\left(r\parallel \Pi \left(r\right)\right)=-s_r(r_1\otimes r_2)+{\mathbb{E}}_r\left({\eta }_r(r_1\otimes r_2)|{\Pi }_1\right)+{\mathbb{E}}_r\left({\eta }_r(r_1\otimes r_2)|{\Pi }_2\right).\end{array}$$

(34)

The conditional terms in Equation (33) depend on the mean-field model; hence, we could express them as a disintegration of r. For example,

$$\begin{array}{c}{\mathbb{E}}_{r_1\otimes r_2}\left(s_r(r_1\otimes r_2)|{\Pi }_1=x\right)=\hfill \\ \sum _y{r}_2\left(y\right)log{\displaystyle \frac{r_1\left(x\right)r_2\left(y\right)}{r(x,y)}}-\sum _{xy}{r}_1\left(x\right)r_2\left(y\right)log{\displaystyle \frac{r_1\left(x\right)r_2\left(y\right)}{r(x,y)}}=\\ \hfill \mathcal{H}\left(r_2,r_{2|1}(·|x)\right)-\sum _x{r}_1\left(x\right)\mathcal{H}\left(r_2,r_{2|1}(·|x)\right),\end{array}$$

where the last term is the conditional entropy $\mathcal{H}\left({\Pi }_2|{\Pi }_1\right)$ under r.

Proof of Equation (33).

We find the natural gradient of $r\mapsto D\left(\Pi \left(r\right)\parallel r\right)$ by computing with Equations (20) and (32) the variation along a smooth curve $t\mapsto r\left(t\right)\in \mathcal{E}\left({\Omega }_1\times {\Omega }_2\right)$ such that $r\left(0\right)=r$ and $\stackrel{\star }{r}\left(0\right)=\stackrel{\star }{r}$. It holds that

$$\begin{array}{c}{\left.{\displaystyle \frac{d}{dt}}D\left(\Pi \left(r\right(t\left)\right)\parallel r\left(t\right)\right)\right|}_{t-0}=-{〈s_{\Pi \left(r\right)}\left(r\right),d\Pi \left(r\right)\left[\stackrel{\star }{r}\right]〉}_{\Pi \left(r\right)}-{〈{\eta }_r(\Pi \left(r\right)),\stackrel{\star }{r}〉}_r=\hfill \\ \hfill -{〈s_{r_1\otimes r_2}\left(r\right),{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)〉}_{r_1\otimes r_2}-{〈{\eta }_r(r_1\otimes r_2),\stackrel{\star }{r}〉}_r\end{array}$$

We want to present the first term of the RHS as an inner product at r applied to $\stackrel{\star }{r}$. Let us push the inner product from $r_1\otimes r_2$ to r with Equation (5). It holds that

$$\begin{gathered} \begin{array}{c}-{〈s_{r_1\otimes r_2}\left(r\right),({\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right))〉}_{r_1\otimes r_2}=\hfill & \\ -{〈{{}^{\mathrm{e}}\mathbb{U}}_{r_1\otimes r_2}^r{s}_{r_1\otimes r_2}\left(r\right),{{}^{\mathrm{m}}\mathbb{U}}_{r_1\otimes r_2}^r({\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right))〉}_r=\\ {〈s_r(r_1\otimes r_2),{\displaystyle \frac{r_1\otimes r_2}{r}}({\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right))〉}_r=\\ {\mathbb{E}}_r\left[s_r(r_1\otimes r_2){\displaystyle \frac{r_1\otimes r_2}{r}}({\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right))\right]=\\ {\mathbb{E}}_r\left[\left({\mathbb{E}}_r\left(s_r(r_1\otimes r_2){\displaystyle \frac{r_1\otimes r_2}{r}}|{\Pi }_1\right)+{\mathbb{E}}_r\left(s_r(r_1\otimes r_2){\displaystyle \frac{r_1\otimes r_2}{r}}|{\Pi }_2\right)\right)\stackrel{\star }{r}\right]=\\ \hfill {〈{\mathbb{E}}_{r_1\otimes r_2}\left(s_r(r_1\otimes r_2)|{\Pi }_1\right)+{\mathbb{E}}_{r_1\otimes r_2}\left(s_r(r_1\otimes r_2)|{\Pi }_2\right),\stackrel{\star }{r}〉}_r.\end{array} \end{gathered}$$

The last equality follows from

$${\mathbb{E}}_r\left(s_r(r_1\otimes r_2){\displaystyle \frac{r_1\otimes r_2}{r}}|{\Pi }_i\right)={\mathbb{E}}_{r_1\otimes r_2}\left(s_r(r_1\otimes r_2)|{\Pi }_i\right),i=1,2.$$

□

Proof of Equation (34).

$$\begin{array}{c}{\left.{\displaystyle \frac{d}{dt}}D\left(r\left(t\right)\parallel \Pi \left(r\right(t\left)\right)\right)\right|}_{t=0}=-{〈s_r(r_1\otimes r_2),\stackrel{\star }{r}〉}_r-{〈{\eta }_{r_1\otimes r_2}\left(r\right),d\Pi \left(r\right)\left[\stackrel{\star }{r}\right]〉}_{r_1\otimes r_2}=\hfill \\ \hfill -{〈s_r(r_1\otimes r_2),\stackrel{\star }{r}〉}_r-{〈{\eta }_{r_1\otimes r_2}\left(r\right),{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)〉}_{r_1\otimes r_2}\end{array}$$

and compute the second term as

$$\begin{array}{c}-{〈{\eta }_{r_1\otimes r_2}\left(r\right),{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)〉}_{r_1\otimes r_2}=\hfill \\ -{〈{{}^{\mathrm{m}}\mathbb{U}}_{r_1\otimes r_2}^r{\eta }_{r_1\otimes r_2}\left(r\right),{{}^{\mathrm{e}}\mathbb{U}}_{r_1\otimes r_2}^r\left({\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)\right)〉}_r=\\ {〈{\eta }_r(r_1\otimes r_2),{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)〉}_r=\\ {\mathbb{E}}_r\left[{\eta }_r(r_1\otimes r_2)\left({\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_1\right)+{\mathbb{E}}_r\left(\stackrel{\star }{r}|{\Pi }_2\right)\right)\right]=\\ {\mathbb{E}}_r\left[\left({\mathbb{E}}_r\left({\eta }_r(r_1\otimes r_2)|{\Pi }_1\right)+{\mathbb{E}}_r\left({\eta }_r(r_1\otimes r_2)|{\Pi }_2\right)\right)\stackrel{\star }{r}\right]=\\ \hfill {〈{\mathbb{E}}_r\left({\eta }_r(r_1\otimes r_2)|{\Pi }_1\right)+{\mathbb{E}}_r\left({\eta }_r(r_1\otimes r_2)|{\Pi }_2\right),\stackrel{\star }{r}〉}_r.\end{array}$$

Equation (34) follows. □

3.3. Product Sample Space: Kantorovich and Scrödinger

If $\Pi$ denotes the joint marginalization, the set of transport plans with margins $q_1$ and $q_2$ is

$$\Gamma (q_1,q_2)={\Phi }^{-1}(q_1\otimes q_2)=\left\{q|\Pi \left(q\right)=q_1\otimes q_2\right\}.$$

Here, we deal with a classical topic with considerable literature. We mention only the monograph of ref. [12] and, from the Information Geometry perspective, ref. [11,22].

Let us consider first the Kantorovich problem. Given the cost function (i.e., potential function)

$$U:\Omega \to \mathbb{R},$$

and a curve $t\mapsto q\left(t\right)\in \gamma (q_1,q_2)$, we want to minimize the cost

$$S\left(t\right)={\mathbb{E}}_{q\left(t\right)}\left[U\right].$$

As ${\mathbb{E}}_{q\left(t\right)}\left[\varphi \left({\Pi }_i\right)\right]={\mathbb{E}}_{q_i}\left[\varphi \right]$ for all $\varphi$,

$$0={\displaystyle \frac{d}{dt}}{〈U-{\mathbb{E}}_{q\left(t\right)}\left[U\right],\stackrel{\star }{q}\left(t\right)〉}_{q\left(t\right)},$$

so that ${\mathbb{E}}_{q\left(t\right)}\left(\stackrel{\star }{q}\left(t\right)|{\Pi }_i\right)=0$. The velocity of a curve in the transport plans is an interaction. Now, the derivative of the cost is

$${\displaystyle \frac{d}{dt}}S\left(q\left(t\right)\right)={〈U-{\mathbb{E}}_{q\left(t\right)}\left[U\right],\stackrel{\star }{q}\left(t\right)〉}_{q\left(t\right)}.$$

(35)

From the interaction property of $\stackrel{\star }{q}\left(t\right)$, it follows that if the ANOVA decomposition

$$U={\mathbb{E}}_q\left[U\right]+(u_1({\Pi }_1;q)+u_2({\Pi }_2;q))+u_{12}({\Pi }_1,{\Pi }_2;q)$$

holds, then

$${\displaystyle \frac{d}{dt}}S\left(q\left(t\right)\right)={〈u_{12}({\Pi }_1,{\Pi }_2;q\left(t\right)),\stackrel{\star }{q}\left(t\right)〉}_{q\left(t\right)}.$$

The Scrödinger problem is similar. Given the cost function (i.e., potential function)

$$U:\Omega \to \mathbb{R},$$

consider the exponential perturbation of the mean-field probability function

$$exp\left(-{\displaystyle \frac{U}{ϵ}}-\varphi \left(ϵ\right)\right)·q_1\otimes q_2.$$

(36)

The parameter $ϵ>0$ is called temperature, and the normalizing constant is

$$\psi \left(ϵ\right)=log{\mathbb{E}}_{q_1\otimes q_2}\left[{\mathrm{e}}^{-U/ϵ}\right].$$

The KL-divergence of q relative to the perturbed probability function of Equation (36) is

$$\begin{gathered} \begin{array}{c}{S}_{ϵ}\left(q\right)=D\left(q\parallel {\mathrm{e}}^{-U/ϵ-\varphi \left(ϵ\right)}·(q_1\otimes q_2)\right)=\hfill & \\ {\mathbb{E}}_q\left[log{\displaystyle \frac{q}{log{q}_1\otimes q_2}}\right]+{ϵ}^{-1}{\mathbb{E}}_q\left[U\right]+\psi \left(ϵ\right)=\\ \hfill {ϵ}^{-1}\left(S\left(q\right)+ϵD\left(q\parallel q_1\otimes q_2\right)+ϵ\varphi \left(ϵ\right)\right).\end{array} \end{gathered}$$

The gradient of $q\mapsto S_{ϵ}\left(q\right)$ is, from Equations (35) and (34),

$$\begin{array}{c}grad{S}_{ϵ}\left(q\right)=\hfill \\ \hfill {ϵ}^{-1}\left(U-{\mathbb{E}}_q\left[U\right]-s_q(q_1\otimes q_2)+{\mathbb{E}}_q\left({\eta }_q(q_1\otimes q_2)|{\Pi }_1\right)+{\mathbb{E}}_q\left({\eta }_q(q_1\otimes q_2)|{\Pi }_2\right)\right).\end{array}$$

Only the interaction part is relevant in the constrained problem $q\in \Gamma (q_1,q_2)$, and the interaction kills the two conditional expectations, which leaves

$$\left(U-{\mathbb{E}}_q\left[U\right]\right){)}_{12;q}-{\left(s_q(q_1\otimes q_2)\right)}_{12;q}.$$

We refer to [22] for a method to compute the interaction part of a random variable.

3.4. Product Sample Space: Conditional Probability Function

When the sample space is a product, $\Omega ={\Omega }_1\times {\Omega }_2$, we can represent each probability function in the maximal exponential model via conditioning on one margin,

$$\begin{array}{cc}\hfill \mathcal{E}\left({\Omega }_1\times {\Omega }_2\right)& =\left\{q=q_{1|2}·q_2|q_{1|2}(·|y)\in \mathcal{E}\left({\Omega }_1\right),y\in {\Omega }_2,q_2\in \mathcal{E}\left({\Omega }_2\right)\right\}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & =\left\{q=q_{2|1}·q_1|q_{2|1}(·|x)\in \mathcal{E}\left({\Omega }_2\right),x\in {\Omega }_1,q_1\in \mathcal{E}\left({\Omega }_1\right)\right\}.\hfill \end{array}$$

(38)

The two representations are

$$\mathcal{E}{\left({\Omega }_1\right)}^{\otimes {\Omega }_2}\times \mathcal{E}\left({\Omega }_2\right)\leftrightarrow \mathcal{E}\left({\Omega }_1\times {\Omega }_2\right)\leftrightarrow \mathcal{E}\left({\Omega }_1\right)\times \mathcal{E}{\left({\Omega }_2\right)}^{\otimes {\Omega }_1}.$$

(39)

Following the approach of [10], ([5], Ch. 11), we look at the transition mapping

$${\Omega }_2\ni y\mapsto q_{1|2}(·|y)\in \mathcal{E}\left({\Omega }_1\right)$$

as a family of probability functions representing alternative probability models. The other transition mapping

$${\Omega }_1\ni x\mapsto q_{2|1}(·|x)\in \mathcal{E}\left({\Omega }_2\right)$$

is the discriminator, that is, $q_{2|1}\left(y\right|x)$ is the probability that the sample x comes from $q_{1|2}(·|y)$.

The right-to-left second mapping in Equation (39),

$$B:\mathcal{E}\left({\Omega }_1\right)\times \mathcal{E}{\left({\Omega }_2\right)}^{\otimes {\Omega }_1}\ni \left(q_1,q_{2|1}(·|x),x\in {\Omega }_1\right)\mapsto q=q_{2|1}·q_1\in \mathcal{E}\left({\Omega }_1\times {\Omega }_2\right),$$

(40)

maps the vector of the 1-margin and the set of alternative probability functions to the joint probability function. The kinematics of Equation (40), that is, the computation of velocities, is

$$\stackrel{\star }{q}(x,y;t)={\stackrel{\star }{q}}_1(x;t)+{\stackrel{\star }{q}}_{2|1}\left(y\right|x;t),$$

Hence, the total derivative of B is

$$dB(q_1,q_{2|1}(·|x),x\in {\Omega }_1)[{\stackrel{\star }{q}}_1,{\stackrel{\star }{q}}_{2|1}(·|x),x\in {\Omega }_1]:(x,y)\mapsto {\stackrel{\star }{q}}_1\left(x\right)+{\stackrel{\star }{q}}_{2|1}\left(y\right|x).$$

The transposed total derivative is defined by

$${〈v,dB(q_1,q_{2|1})[{\stackrel{\star }{q}}_1,{\stackrel{\star }{q}}_{2|1}]〉}_{B(q_1,q_{2|1})}={〈dB{(q_1,q_{2|1})}^{*}\left[v\right],({\stackrel{\star }{q}}_1,{\stackrel{\star }{q}}_{2|1})〉}_{(q_1,q_{2|1})},$$

that is,

$$\begin{array}{c}\sum _{x,y}v(x,y)\left({\stackrel{\star }{q}}_1\left(x\right)+{\stackrel{\star }{q}}_{2|1}\left(y\right|x)\right)q_1\left(x\right)q_{2|1}\left(y\right|x)=\hfill \\ \sum _{x,y}v(x,y){\stackrel{\star }{q}}_1\left(x\right)q_1\left(x\right)q_{2|1}\left(y\right|x)+\sum _{x,y}v(x,y){\stackrel{\star }{q}}_{2|1}\left(y\right|x)q_1\left(x\right)q_{2|1}\left(y\right|x)=\\ \sum _x\left(\sum _{y}v(x,y)q_{2|1}\left(y\right|x)\right){\stackrel{\star }{q}}_1\left(x\right)q_1\left(x\right)+\sum _x\sum _y{q}_1\left(x\right)v(x,y){\stackrel{\star }{q}}_{2|1}\left(y\right|x)q_{2|1}\left(y\right|x)=\\ {〈{\mathbb{E}}_q\left(v|{\Pi }_1\right),{\stackrel{\star }{q}}_1〉}_{q_1}+\sum _x{q}_1\left(x\right)\sum _{y}v(x,y){\stackrel{\star }{q}}_{2|1}\left(y\right|x)q_{2|1}\left(y\right|x)=\\ \hfill {〈{\mathbb{E}}_q\left(v|{\Pi }_1\right),{\stackrel{\star }{q}}_1〉}_{q_1}+\sum _x{〈q_1\left(x\right)\left(v(x,·)-{\mathbb{E}}_{q_{2|1}(·|x)}\left[v(x,·)\right]\right),{\stackrel{\star }{q}}_{2|1}\left(y\right|x)〉}_{q_{2|1}(·|x)}.\end{array}$$

In conclusion, the transposed total derivative is

$$dB{(q_1,q_{2|1})}^{*}\left[v\right]:(x,y)\mapsto \left({\mathbb{E}}_q\left(v|{\Pi }_1\right),q_1\left(x\right)\left(v(x,·)-{\mathbb{E}}_{q_{2|1}(·|x)}\left[v(x,·)\right]\right),x\in {\Omega }_1\right).$$

It is interesting to derive $dB$ in the mixture atlas. The mixture expression of B with respect to $(p_1,p_2^{\otimes {\Omega }_1})$ and $p_1\otimes p_2$ is

$$\begin{array}{c}{B}_{p_1,p_2}:(v_1,v_{2|1}(·|x),x\in {\Omega }_1)\mapsto \hfill \\ (x,y)\mapsto (1+v_1\left(x\right))p_1\left(x\right)(1+v_{2|1}\left(y\right|x))p_2\left(y\right)\mapsto \\ \hfill (1+v_1)(1+v_{2|1})-1\end{array}$$

and the total derivative in the directions $h_1$, $h_{2|1}$ is

$$\begin{array}{c}d{B}_{p_1,p_2}(v_1,v_{2|1}(·|z),z\in {\Omega }_1)[h_1,h_{2|1}(·|z),z\in {\Omega }_1]=\hfill \\ \hfill (1+v_{2|1})h_1+\sum _z(1+v_1\left(z\right))h_{2|1}(·|z).\end{array}$$

The push-back of the total derivative expression to the statistical bundles uses the equations

$$1+v_1=q_1/p_1,1+v_{2|1}=q_{2|1}/p_2,h_1={{}^{\mathrm{m}}\mathbb{U}}_{q_1}^{p_1}{\stackrel{\star }{q}}_1,h_{2|1}(·|z)={{}^{\mathrm{m}}\mathbb{U}}_{q_{2|1}(·|z)}^{p_2}{\stackrel{\star }{q}}_{2|1}(·|z)$$

to obtain

$$\begin{array}{c}dB(q_1,q_{2|1}(·|z),z\in {\Omega }_1)[{\stackrel{\star }{q}}_1,{\stackrel{\star }{q}}_{2|1}(·|z),z\in {\Omega }_1]=\hfill \\ {{}^{\mathrm{m}}\mathbb{U}}_{p_1\otimes p_2}^{q_{2|1}·q_1}\left({\displaystyle \frac{q_{2|1}}{p_2}}{{}^{\mathrm{m}}\mathbb{U}}_{q_1}^{p_1}{\stackrel{\star }{q}}_1+\sum _z{\displaystyle \frac{q_1\left(z\right)}{p_1\left(z\right)}}{{}^{\mathrm{m}}\mathbb{U}}_{q_{2|1}(·|z)}^{p_2}{\stackrel{\star }{q}}_{2|1}(·|z)\right)=\\ \hfill {\displaystyle \frac{p_1\otimes p_2}{q_{2|1}·q_1}}\left({\displaystyle \frac{q_{2|1}}{p_2}}{\displaystyle \frac{q_1}{p_1}}{\stackrel{\star }{q}}_1+\sum _z{\displaystyle \frac{q_1\left(z\right)}{p_1\left(z\right)}}{\displaystyle \frac{q_{2|1}(·|z)}{p_2}}{\stackrel{\star }{q}}_{2|1}(·|z)\right)\end{array}$$

In our affine language, we repeat computations in [10]. We especially derive the natural gradient of a composite function by the equation

$$grad\Phi \circ B=d{B}^{*}grad\Phi \left(B\right).$$

For a given target $p\in \mathcal{E}\left({\Omega }_1\times {\Omega }_2\right)$, we express the KL-divergence as a function of B in Equation (40),

$$\begin{array}{c}{K}_q:(r_1,r_{2|1}(·|x),x\in {\Omega }_1)\mapsto D\left(p\parallel B(q_1,q_{2|1}(·|x),x\in {\Omega }_1)\right)=\hfill \\ \hfill \sum _{x,y}p(x,y)log{\displaystyle \frac{p(x,y)}{q_1\left(x\right)q_{2|1}\left(y\right|x)}}.\end{array}$$

We have

$$grad{K}_p\left(B(q_1,q_{2|1})\right)=-{\eta }_{q_1·q_{2|1}}\left(p\right)=1-{\displaystyle \frac{p}{q_1·q_{2|1}}}.$$

The first component of $grad{K}_p\circ B$ is

$$\begin{gathered} \begin{array}{c}{[dB{(q_1,q_{2|1})}^{*}grad{K}_p\left(B(q_1,q_{2|1})\right)]}_1:x\mapsto \hfill & \\ \hfill \sum _z\left(1-{\displaystyle \frac{p(x,z)}{q_1\left(x\right)q_{2|1}\left(z\right|x)}}\right)q_{2|1}\left(z\right|x)=\sum _z{\displaystyle \frac{q_1\left(x\right)q_{2|1}\left(z\right|x)-p(x,z)}{q_1\left(x\right)}}=1-{\displaystyle \frac{p_1\left(x\right)}{q_1\left(x\right)}},\end{array} \end{gathered}$$

(41)

so that ${[grad{K}_p\circ B(q_1,q_{2|1})]}_1=-{\eta }_{q_1}\left(p_1\right)$. The x-component is

$$\begin{gathered} \begin{array}{c}{[dB{(q_1,q_{2|1})}^{*}grad{K}_p\left(B(q_1,q_{2|1})\right)]}_x:y\mapsto \hfill & \\ {q}_1\left(x\right)\left(1-{\displaystyle \frac{p(x,y)}{q_1\left(x\right)q_{2|1}\left(y\right|x)}}\right)-q_1\left(x\right)\sum _y\left(1-{\displaystyle \frac{p(x,y)}{q_1\left(x\right)q_{2|1}\left(y\right|x)}}\right)q_{2|1}\left(y\right|x)=\\ \hfill -{\displaystyle \frac{p(x,y)}{q_{2|1}\left(y\right|x)}}+p_1\left(x\right),\end{array} \end{gathered}$$

(42)

so that ${[dB{(q_1,q_{2|1})}^{*}grad{K}_p\left(B(q_1,q_{2|1})\right)]}_x=-p_1\left(x\right){\eta }_{q_{2|1}(·|x)}\left(p_{2|1}(·|x)\right)$.

We now assume a target probability function $g\in \mathcal{E}\left({\Omega }_1\right)$ and consider the probability function on the product sample space where all the model probability functions equal g, and the discriminator is uniform, say $p={\displaystyle \frac{1}{m}}g$, $m=\#{\Omega }_2$.

In this case, Equation (41) becomes

$$grad{K}_p\circ B(q_1,q_{2|1}){]}_1=-{\eta }_{q_1}\left(g\right)$$

and Equation (42) becomes

$$grad{K}_p\circ B(q_1,q_{2|1}){]}_x=-q_1\left(x\right)\left(1-{\displaystyle \frac{g\left(x\right)/m}{q_1\left(x\right)q_{2|1}\left(y\right|x)}}\right)$$

3.5. Variational Bayes

We revisit and develop some computations of ([13], § 2.2). We keep the same notation as above so that Bayes’ formula is

$$q_{2|1}\left(y\right|x)={\displaystyle \frac{q_{1|2}\left(x\right|y)q_2\left(y\right)}{q_1\left(x\right)}}={\displaystyle \frac{q_{1|2}\left(x\right|y)q_2\left(y\right)}{{\sum }_y{q}_{1|2}\left(x\right|y)q_2\left(y\right)}},$$

where x is a sample value and y is a latent variable value.

For a fixed $x\in {\Omega }_1$, we look for a r in some model $\mathcal{M}\subset \mathcal{E}\left({\Omega }_2\right)$ in order to approximate the conditional $q_{2|1}(·|x)$. If $q\mapsto \mathcal{L}(r,x)$ satisfies

$$log{q}_1\left(x\right)=D\left(r\parallel q_{2|1}(·|x)\right)+\mathcal{L}(r,x),$$

then

$$\begin{array}{c}\mathcal{L}(r,x)=log{q}_1\left(x\right)-\sum _{y}r\left(y\right)log{\displaystyle \frac{r\left(y\right)}{q_{2|1}\left(y\right|x)}}=\hfill \\ \sum _{y}r\left(y\right)\left(log{q}_1\left(x\right)-log{\displaystyle \frac{r\left(y\right)}{q_{2|1}\left(y\right|x)}}\right)=\sum _{y}r\left(y\right)log{\displaystyle \frac{q_{2|1}\left(y\right|x)q_1\left(x\right)}{r\left(y\right)}}=\\ \sum _{y}r\left(y\right)log{\displaystyle \frac{q_{12}(x,y)}{r\left(y\right)}}=\sum _{y}r\left(y\right)log{\displaystyle \frac{q_2\left(y\right)}{r\left(y\right)}}+\sum _{y}r\left(y\right)log{\displaystyle \frac{q_{12}(x,y)}{q_2\left(y\right)}}=\\ \hfill -D\left(r\parallel q_2\right)+\sum _{y}r\left(y\right)log{q}_{1|2}\left(x\right|y).\end{array}$$

The so-called variational lower bound follows from $D\left(r\parallel q_{2|1}(·|x)\right)\ge 0$,

$$\begin{array}{c}log{q}_1\left(x\right)=D\left(r\parallel q_{2|1}(·|x)\right)-D\left(r\parallel q_2\right)+{\mathbb{E}}_r\left[log{q}_{1|2}\left(x\right|·)\right]\ge \hfill \\ \hfill -D\left(r\parallel q_2\right)+{\mathbb{E}}_r\left[log{q}_{1|2}\left(x\right|·)\right]=\mathcal{L}(r;x).\end{array}$$

for all $r\in \mathcal{M}$. The bound is exact because $r\left(y\right)=q_{2|1}\left(y\right|x)$ for all y if, and only if, $D\left(r\parallel q_{2|1}(·|x)\right)=0$ The lower-bound variation along a curve $t\mapsto r\left(t\right)\in \mathcal{M}$ is

$$\begin{gathered} \begin{array}{c}{\displaystyle \frac{d}{dt}}\mathcal{L}(r\left(t\right);x)=\hfill & \\ {〈\stackrel{\star }{r}\left(t\right),s_{r\left(t\right)}\left(q_2\right)〉}_{r\left(t\right)}+{〈\stackrel{\star }{r}\left(t\right),log{q}_{1|2}\left(x\right|·)-{\mathbb{E}}_{r\left(t\right)}\left[log{q}_{1|2}\left(x\right|·)\right]〉}_{r\left(t\right)}=\\ {〈\stackrel{\star }{r}\left(t\right),log{\displaystyle \frac{q_2}{r\left(t\right)}}+log{q}_{1|2}\left(x\right|·)-{\mathbb{E}}_{r\left(t\right)}\left[log{\displaystyle \frac{q_2}{r}}+log{q}_{1|2}\left(x\right|·)\right]〉}_{r\left(t\right)}=\\ \hfill {〈\stackrel{\star }{r}\left(t\right),log{\displaystyle \frac{q_{12}(x,·)}{r\left(t\right)}}-{\mathbb{E}}_{r\left(t\right)}\left[log{\displaystyle \frac{q_{12}(x,·)}{r\left(t\right)}}\right]〉}_{r\left(t\right)}.\end{array} \end{gathered}$$

(43)

If the model $\mathcal{M}$ is an exponential tilting of the margin $q_2$,

$$r=e_{q_2}(\theta ·u)={\mathrm{e}}^{\theta ·u-\psi \left(\theta \right)}·q_2,$$

where $\theta \in \Theta \subset {\mathbb{R}}^d$ is a vector parameter, and u is the vector of sufficient statistics of the exponential family with ${\mathbb{E}}_{q_2}\left[u\right]=0$, then the velocity in Equation (43) becomes

$$\stackrel{\star }{r}\left(t\right)=\dot{\theta }\left(t\right)·(u-\nabla \psi \left(\theta \left(t\right)\right)=\dot{\theta }\left(t\right)·(u-{\mathbb{E}}_{r\left(t\right)}\left[u\right])=\dot{\theta }\left(t\right)·{{}^{\mathrm{e}}\mathbb{U}}_{q_2}^{r\left(t\right)}u,$$

and the gradient in Equation (43) becomes

$$\begin{array}{c}log{\displaystyle \frac{q_{12}(x,·)}{r\left(t\right)}}-{\mathbb{E}}_{r\left(t\right)}\left[log{\displaystyle \frac{q_{12}(x,·)}{r\left(t\right)}}\right]=\hfill \\ log{\displaystyle \frac{q_{12}(x,·)}{q_2}}-(\theta \left(t\right)·u-\psi \left(u\right))-{\mathbb{E}}_{r\left(t\right)}\left[log{\displaystyle \frac{q_{12}(x,·)}{q_2}}-(\theta \left(t\right)·u-\psi \left(u\right))\right]=\\ log{q}_{1|2}\left(x\right|·)-\theta \left(t\right)·u-{\mathbb{E}}_{r\left(t\right)}\left[log{q}_{1|2}\left(x\right|·)-\theta \left(t\right)·u\right]=\\ \hfill {{}^{\mathrm{e}}\mathbb{U}}_{q_2}^{r\left(t\right)}\left(log{q}_{1|2}\left(x\right|·)-{\mathbb{E}}_{q_2}\left[log{q}_{1|2}\left(x\right|·)\right]-\theta \left(t\right)·u\right).\end{array}$$

Using repeatedly Equation (11), we find the derivative of the lower bound in Equation (43),

$$\begin{array}{c}{\displaystyle \frac{d}{dt}}\mathcal{L}(r\left(t\right);x)=\hfill \\ {〈\dot{\theta }\left(t\right)·{{}^{\mathrm{e}}\mathbb{U}}_{q_2}^{r\left(t\right)}u,{{}^{\mathrm{e}}\mathbb{U}}_{q_2}^{r\left(t\right)}\left(log{q}_{1|2}\left(x\right|·)-{\mathbb{E}}_{q_2}\left[log{q}_{1|2}\left(x\right|·)\right]-\theta \left(t\right)·u\right)〉}_{r\left(t\right)}=\\ \sum _{i=1}^d{\dot{\theta }}_i\left(t\right)\left({Cov}_{r\left(t\right)}\left(u_i,log{q}_{1|2}\left(x\right|·)\right)-\sum _{j=1}^d{\theta }_j\left(t\right){Cov}_{r\left(t\right)}\left(u_i,u_j\right)\right)=\\ \hfill \dot{\theta }\left(t\right)·\left({Cov}_{r\left(t\right)}\left(u,log{q}_{1|2}\left(x\right|·)\right)-Hess\psi \left(\theta \left(t\right)\right)\theta \left(t\right)\right).\end{array}$$

In conclusion, the gradient flow equation for the maximization of the lower bound under the model $\mathcal{M}$ is

$$\dot{\theta }=-Hess\psi \left(\theta \left(t\right)\right)\theta \left(t\right)+{Cov}_{e_{q_2}(\theta \left(t\right)·u)}\left(u,log{q}_{1|2}\left(x\right|·)\right).$$

(44)

As a sanity check, assume that the model is exact for the given x,

$$q_{2|1}\left(y\right|x)={\mathrm{e}}^{\overline{\theta }·u\left(y\right)-\psi \left(\overline{\theta }\right)}·q_2\left(y\right).$$

Hence,

$$log{q}_{1|2}\left(x\right|y)=log{\mathrm{e}}^{\overline{\theta }·u\left(y\right)-\psi \left(\overline{\theta }\right)}{q}_2\left(y\right){\displaystyle \frac{q_1\left(x\right)}{q_2\left(y\right)}}=\overline{\theta }·u=\psi \left(\overline{\theta }\right)+log{q}_1\left(x\right),$$

where the last two terms do not depend on y so that

$${Cov}_{e_{q_2}(\theta ·u}\left(u,\overline{\theta }·u\right)=Hess\psi \left(\theta \right)\overline{\theta }$$

and Equation (44) becomes

$$\dot{\theta }=-Hess\psi \left(\theta \left(t\right)\right)(\theta \left(t\right)-\overline{\theta }).$$

The solution of the gradient flow Equation (44) requires the ability to compute the covariance for the current model distribution. We do not discuss the numerical and simulation issues related to the implementation here.

4. Discussion

In this paper, we have shown how the dually affine formalism for the open probability simplex provides a system of affine charts in which the statistical notion of Fisher’s score becomes the moving-chart affine velocity, and the natural gradient becomes the gradient. The construction applies to standard computations of interest for statistical machine learning. In particular, we have discussed the neat form of the total gradient of the KL-divergence and its applications in a factorial sample space, such as mean-field approximation and Bayes computations.

This approach is helpful because the unique features of the Fisherian approach to statistics, such as Fisher’s score, maximum likelihood, and Fisher’s information, are formalized as an affine calculus so that all the statistical tools are available in this more extensive theory. Moreover, this setting potentially unifies the formalisms of Statistics, Optimal Transport, and Statistical Physics, examples being the affine modeling of Optimal Transport [24] and the second-order methods of optimization [25].

We have not considered the implementation of the formal gradient flow equation as a practical learning algorithm. Such further development is currently outside the scope of this piece of research. It would require the numerical analysis of the continuous equation and the search for sampling versions of the expectation operators. We hope this note will prompt further research. On the abstract side, topics worth studying seem to be the cases of continuous sample space as in [16], or Gaussian models as in [26].