Information Geometry Description of Inferential Scattering

Abstract

We investigate the geometrical structure underlying the notion of inferential scattering, which was formulated by E. T. Jaynes in the 1980s using the language of equilibrium statistical mechanics. We show that inferential scattering can be naturally defined on a dually flat Riemannian manifold equipped with dual coordinate systems, a differential-geometric structure that occupies a central place in Information Geometry. We find that the controlled evolution of the system on the dually flat manifold can be expressed as the horizontal lift of an integrable connection.

1. Introduction

The notion of inferential scattering was exposed by E.T. Jaynes in several papers [1,2] whose content has remained little known in the scientific literature with respect to more celebrated works of this author. Jaynes’ original exposition refers to equilibrium statistical mechanics. It is well known that the probabilistic formalism of equilibrium statistical mechanics is described by an exponential family manifold which is an instance of a dually flat manifold [3]; the main aim of this work is to rephrase the content of the notion of inferential scattering in this enlarged setting of dually flat manifolds to better appreciate its geometric structure and possibly expand its range of applications. Dually flat manifolds are a generalization of Riemannian manifolds, and they occupy a central place in Information Geometry. They have found multiple applications in the description of neural networks [4], machine learning (especially Restricted Boltzmann Machines [5]), statistical estimation in exponential and mixture families, optimization algorithms like mirror descent and EM [6], the description of hierarchical systems [7], non-equilibrium stochastic thermodynamics [8].

In a dually flat manifold M, there are two local coordinate systems, $\theta =\theta \left(p\right)$ and $\eta =\eta \left(p\right)$, which describe the same point $p\in M$. The transition maps $\eta \circ {\theta }^{-1}$ and $\theta \circ {\eta }^{-1}$ are related by a Legendre transform so that $\eta \left(\theta \right)=\nabla \psi \left(\theta \right)$ with inverse $\theta \left(\eta \right)=\nabla \phi \left(\eta \right)$, where ∇ denotes the gradient and $\phi ,\psi$ are called potentials. In the case of an exponential manifold, there is a natural statistical mechanics interpretation of $\theta$ as control variables and $\eta$ as effect variables; see Section 4. Jaynes’ paper [1] explains how to realize the variation of control variables $\theta$ (thermal control), which corresponds to a variation in the parameters of a statistical model. The tangents of the above introduced gradient maps give the linear variation $d\eta$ in the effect variables due to a variation $d\theta$ in control variables and vice versa.

A first consequence of the use of a dually flat manifold equipped with a dual coordinate systems is that the controlled evolution of the point giving the statistical state of the system can be understood in terms of horizontal lift with respect to a couple of integrable connections (dually orthogonal foliations) which are characteristics of the dually flat manifold structure. In this way, we can extend Jaynes’ formula describing the infinitesimal system evolution to finite evolution (see Section 4 and Section 5 below).

The phenomenon of inferential scattering arises when particular kinds of variations $d\eta$ and $d\theta$ are considered, precisely the ones corresponding to the mixed controlled evolution of the system, see Section 5. In this case, the variation $d\eta$ in effect variables can be written as an infinite sum of contributions. In Jaynes’ paper [1], this perturbative expansion of $d\eta$ is called “generalized scattering” by analogy with the physical phenomenon of the multiple scattering of a wave emanating from a point source A in presence of an obstacle C: the intensity of the wave at a point B in space is the infinite sum of the contribution of the wave arriving from A plus the contribution of the wave originated in A and reflected by C plus the one of the wave reflected from C back in A and then in B and so on with all the higher order multiple reflections (Born series; see [9]). A second consequence of this analysis of Jaynes’ result is that the scattering phenomenon is present also in the abstract setting of a dually flat manifold, where it establishes a relation between the dual coordinate systems $\theta$ and $\eta$.

We conclude with the plan of the paper: In Section 2 we give a cursory overview of the differential geometry notion of dually flat manifold and mutually dual foliations. In Section 3 we describe the paradigmatic case of the exponential family manifold. In Section 4 we investigate the geometric structure of the notion of forward and backward control and, in Section 5, that of inferential scattering.

2. Dually Flat Manifolds

We briefly recall the notions of differential geometry useful for introducing the definition of a dually flat manifold, a central object in Information Geometry. We keep the exposition to a minimum, without proofs, essentially following [3]. Other important introductory references are [6,10,11].

Let $(M,g)$ be a n-dimensional smooth manifold with g a Riemannian metric. In a local chart $\zeta$ the coordinates of a point $p\in M$ are $\zeta \left(p\right)=(x_1,\dots x_n)$, and the canonical basis is ${\partial }_i={\displaystyle \frac{\partial }{\partial x_i}}$. We equip M with a linear connection ∇ whose expression in the local chart is given by the Christoffel symbols ${\mathrm{\Gamma }}_{ij}^k$ as

$${\nabla }_{{\partial }_i}{\partial }_j={\mathrm{\Gamma }}_{ij}^k{\partial }_k,{\mathrm{\Gamma }}_{ij,k}=g({\nabla }_{{\partial }_i}{\partial }_j,{\partial }_k)=g_{lk}{\mathrm{\Gamma }}_{ij}^l.$$

Throughout the paper we adopt the convention that repeated indices are summed. The torsion $\tau$ of the connection ∇ is defined as

$$\tau (X,Y)={\nabla }_{X}Y-{\nabla }_{Y}X-[X,Y]$$

where $X,Y\in \mathfrak{X}\left(M\right)$ are vector fields on M and $[X,Y]$ is the Lie bracket. The connection ∇ is said to be compatible with the metric g or simply metric if

$$X·g(X,Y)=g({\nabla }_{X}Y,Z)+g(X,{\nabla }_{Y}Z)\forall X,Y,Z\in \mathfrak{X}\left(M\right).$$

(1)

If the connection is metric, the parallel transport defined by ∇ is an isometry, preserving the length and the angle between vectors. A celebrated theorem in differential geometry states that in a Riemannian manifold there exists a unique metric connection which is torsion-free $\tau =0$, the so-called Levi-Civita connection $\widehat{\nabla }$. For later use we introduce the curvature defined by the connection ∇

$$R(X,Y,Z)={\nabla }_X\left({\nabla }_{Y}Z\right)-{\nabla }_Y\left({\nabla }_{X}Z\right)-{\nabla }_{[X,Y]}Z.$$

The connection ∇ is flat if the associated curvature is identically zero. This is a coordinate-free condition, and the Christoffel symbols are zero in every coordinate chart. Moreover, there exists an affine coordinate system with the property that every canonical vector $\partial /\partial x_i$ of the basis can be parallel transported along the coordinate line $x_i$ so each coordinate line is a geodesic.

The above notion (1) of compatibility between the metric g and a connection ∇ is generalized in Information Geometry using the following

Definition 1.

The connections ∇ and ${\nabla }^{*}$ are called conjugate or dual with respect to the metric in $(M,g)$ if

$$X·g(X,Y)=g({\nabla }_{X}Y,Z)+g(X,{\nabla }_Y^{*}Z)\forall X,Y,Z\in \mathfrak{X}\left(M\right).$$

One can show [3] that the connection ${\nabla }^{*}$ dual to ∇ is unique. Moreover, in [3] it is proved that if $\nabla ,{\nabla }^{*}$ are dual and both torsion-free, then $R=0$ if and only if $R^{*}=0$. This motivate the following

Definition 2.

A Riemannian manifold $(M,g,\nabla ,{\nabla }^{*})$ equipped with two connections, which are dual and torsion-free, is called dually flat when $R=R^{*}=0$, i.e., both connections are flat.

As a consequence, there exists on M two affine coordinate systems, $\theta$ and $\eta$, with the property that the vectors ${\partial }^i={\displaystyle \frac{\partial }{\partial {\eta }^i}}$ of the basis are ∇-parallel transported along the coordinate lines ${\eta }^i$ and the vectors ${\partial }_i={\displaystyle \frac{\partial }{\partial {\theta }_i}}$ are ${\nabla }^{*}$-parallel transported along the coordinate lines ${\theta }_i$. The coordinates $\theta$ and $\eta$ are called dual or bi-orthogonal because it holds that $g({\partial }_i,{\partial }^j)={\delta }_i^j$ where $\delta$ is the Kronecker symbol. Note that the two basis are orthogonal each other but the vectors of each base need not to be orthogonal. In greater detail, if we consider the representation of the metric g in the two basis

$$g^{ij}=g({\partial }^i,{\partial }^j),g_{ij}=g({\partial }_i,{\partial }_j)$$

it holds that the two matrices are in the relation $g^{ij}{g}_{jk}={\delta }_k^i$. Finally, let $\theta =\theta \left(\eta \right)$ and $\eta =\eta \left(\theta \right)$ be the transition maps connecting the representation of a point $p\in M$ in the two coordinate charts. Note that they are the inverse of each other. Using the orthogonality condition $g^{ij}{g}_{jk}={\delta }_k^i$, one can prove that their tangent maps coincide with the representation of the metric g in the two coordinate basis; that is

$$g^{ij}\left(\eta \right)={\displaystyle \frac{\partial {\theta }_i\left(\eta \right)}{\partial {\eta }^j}},g_{ij}\left(\theta \right)={\displaystyle \frac{\partial {\eta }^i\left(\theta \right)}{\partial {\theta }_j}}.$$

(2)

We now show that

Proposition 1.

The change of coordinate $\eta =\eta \left(\theta \right)$ and $\theta =\theta \left(\eta \right)$ can be represented using the so-called potential scalar functions $\phi \left(\eta \right)$ and $\psi \left(\theta \right)$ in the sense that

$${\theta }_i\left(\eta \right)={\displaystyle \frac{\partial \phi \left(\eta \right)}{\partial {\eta }^i}},{\eta }^i\left(\theta \right)={\displaystyle \frac{\partial \psi \left(\theta \right)}{\partial {\theta }_i}},i=1,\dots ,n.$$

(3)

Proof. 

We prove the first relation of (3), the proof for the second follows the same pattern. We need to prove that the one form $\omega ={\theta }_i\left(\eta \right)d{\eta }^i$ is exact; that is, there exists a scalar function $\phi \left(\eta \right)$, such that $d\phi =\omega$. A necessary condition (and also sufficient if the manifold is simply connected as we will assume to be the case) is that $d\omega =0$. Then we need to check that

$$d\omega =d\left({\theta }_i\left(\eta \right)d{\eta }^i\right)={\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}d{\eta }^j\wedge d{\eta }^i=\left({\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}-{\displaystyle \frac{\partial {\theta }_j}{\partial {\eta }^i}}\right)d{\eta }^j\otimes d{\eta }^i=0$$

that is

$${\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}-{\displaystyle \frac{\partial {\theta }_j}{\partial {\eta }^i}}=0\forall i,j=1,\dots n$$

Now, using (2), the above relation becomes $g^{ij}=g^{ji}$, which is true. □

Note that the relations (2) and (3) together imply that the potentials $\phi \left(\eta \right)$ and $\psi \left(\theta \right)$ are convex functions. Indeed we have

$${\displaystyle \frac{{\partial }^2\phi \left(\eta \right)}{\partial {\eta }^j\partial {\eta }^i}}={\displaystyle \frac{\partial {\theta }_i}{\partial {\eta }^j}}=g^{ij}\left(\eta \right)\in Sy{m}^{+}\left(n\right),{\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_j\partial {\theta }_i}}={\displaystyle \frac{\partial {\eta }^i}{\partial {\theta }_j}}=g_{ij}\left(\theta \right)\in Sy{m}^{+}\left(n\right)$$

(4)

i.e., the Hessian matrix of $\phi$ and $\psi$ is positive definite. Since the metric g is the Hessian of a function, the dually flat manifold is an example of a Hessian manifold. We do not pursue this line of investigation; see [10] for further details.

Note that the existence of a dual coordinate system induces stringent conditions on the metric g: indeed by deriving (2) and using Schwartz condition on mixed second order derivative, we obtain

$${\displaystyle \frac{\partial g^{ij}}{\partial {\eta }^k}}={\displaystyle \frac{\partial g^{ik}}{\partial {\eta }^j}}\mathrm{and}{\displaystyle \frac{\partial g_{ij}}{\partial {\theta }_k}}={\displaystyle \frac{\partial g_{ik}}{\partial {\theta }_j}}.$$

Mutually Dual Foliations of a Dually Flat Space

A remarkable fact of dually flat manifolds is that they can admit a couple of orthogonal foliations. Let us consider an n-dimensional dually flat manifold $(M,g,\nabla ,{\nabla }^{*})$ and let $\eta \left(p\right)=\eta$, $\theta \left(p\right)=\theta$, $p\in M$ be several bi-orthogonal affine coordinate systems. Given a k with $1<k<n$, we partition the coordinates in two subsets

$$\eta =({\eta }^A,{\eta }^B)=({\eta }^1,\dots ,{\eta }^k,{\eta }^{k+1},\dots ,{\eta }^n)\mathrm{and}\theta =({\theta }_A,{\theta }_B)=({\theta }_1,\dots ,{\theta }_k,{\theta }_{k+1},\dots ,{\theta }_n)$$

and we consider the coordinate projections ${\pi }_A\left(\eta \right)={\eta }^A$ and ${\pi }_B\left(\theta \right)={\theta }_B$. For every ${\overline{\eta }}^A$ and ${\overline{\theta }}_B$, we define the submanifolds of M

$$E\left({\overline{\eta }}^A\right)=\{p\in M:{\pi }_A\circ \eta \left(p\right)={\overline{\eta }}^A)\}Q\left({\overline{\theta }}_B\right)=\{p\in M:{\pi }_B\circ \theta \left(p\right)={\overline{\theta }}_B)\}$$

(5)

In [3], it is proven that since M is dually flat and the subsets $E\left({\overline{\eta }}^A\right)$ and $Q\left({\overline{\theta }}_B\right)$ are defined as affine subspaces of an affine coordinate system, they are totally geodesic submanifolds with respect to the ∇ and ${\nabla }^{*}$ connection, respectively; that is, it holds that a ∇-geodesic (respectively, ${\nabla }^{*}$-geodesic), which is tangent to the submanifold at a point, remains tangent to the submanifold. Moreover, since their tangent spaces can be described as

$$T_{p}E\left({\overline{\eta }}_A\right)=<{\partial }^{k+1},\dots ,{\partial }^n>,T_{p}Q\left({\overline{\theta }}_B\right)=<{\partial }_1,\dots ,{\partial }_k>$$

the two submanifolds are mutually orthogonal by the condition $g({\partial }_i,{\partial }^j)={\delta }_i^j$ and complementary in the sense that their direct sum of subspaces gives the tangent space $T_{p}M$. See Figure 1 below. If we introduce the so-called mixed coordinate system

$$m\left(p\right)=({\eta }^A\left(p\right),{\theta }_B\left(p\right))=({\pi }_A\left(\eta \left(p\right)\right),{\pi }_B\left(\theta \left(p\right)\right))$$

the metric g admits the block decomposition

$$g\left(m\left(p\right)\right)=g^{ab}d{\eta }^a\otimes d{\eta }^b+g_{cd}d{\theta }_c\otimes d{\theta }_d,a,b\in \{1,\dots ,k\},c,d\in \{k+1,\dots ,n\}.$$

So the complementary projections ${\pi }_A$ and ${\pi }_B$ define a couple of mutually dual foliation of M, whose leaves are orthogonal submanifolds of constant dimension (respectively, $\dim E\left({\overline{\eta }}^A\right)=n-k$, and $\dim Q\left({\overline{\theta }}_B\right)=k$) and totally geodesic with respect to the ∇ and ${\nabla }^{*}$ connections. This partition of the space is called k-cut of a dual coordinate system in [6]. The notion of mutually dual foliations has important applications on Information Geometry due to the fact that a version of the Pythagorean Theorem applies (see again, [3]).

Figure 1. Cartoon picture of a mutually dual foliation in a dually flat space.

Figure 1. Cartoon picture of a mutually dual foliation in a dually flat space.

3. Exponential Family Manifold

We describe an example of dually flat manifold which occupies a central place in Information Geometry. We start with the definition of statistical model. Let $(X,\mathcal{B},d\mu )$ be a measure space, where the set X can be numerable or not; $\mathcal{B}$ is a $\sigma$-algebra over X; and $d\mu$ is a measure on X. The infinite dimensional space of all the positive probability densities over X is

$$\mathcal{P}\left(X\right)=\{p\left(x\right)\ge 0a.e.\mathrm{on}X,\mathrm{and}{\int }_{X}p\left(x\right)d\mu =1\}.$$

Given an open convex set $U\subset {\mathbb{R}}^n$ with coordinates $\theta =({\theta }_1,\dots ,{\theta }_n)$, we consider the subset of $\mathcal{P}\left(X\right)$ of probability densities parameterized by the parameters $\theta \in U$

$$S=\{p_{\theta }=p(x,\theta ),\theta \in U\}\subset \mathcal{P}\left(X\right).$$

The set S has the structure of a n-dimensional submanifold of $\mathcal{P}\left(X\right)$ if: (1) the map $\theta \mapsto p_{\theta }$ is injective and (2) the n base functions $p_i=\partial p(x,\theta )/\partial {\theta }_i$ are linearly independent as functions over X. In this case we call S a n-dimensional statistical model. Usually, linear independence condition is checked using the equivalent score function basis functions $l_i=\partial l/\partial {\theta }_i$, where $l=\ln p$ is the likelihood of p.

An exponential family is the n-dimensional statistical manifold

$$\mathcal{E}=\{p(x,\theta )=e^{\theta ·h\left(x\right)+k\left(x\right)-\psi \left(\theta \right)},\theta \in U\}\subset \mathcal{P}\left(X\right)$$

(6)

where $h:X\to {\mathbb{R}}^n$ is a map of maximum rank n whose components $h_i$, $i=1,\dots ,n$ are called feature functions or observables, k is a positive function such that $e^{k\left(x\right)}d\mu$ is a probability density over X and the function $\psi \left(\theta \right)$

$$\psi \left(\theta \right)=\ln {\int }_X{e}^{\theta ·h\left(x\right)+k\left(x\right)}d\mu$$

is the cumulant or moment generating function, called free energy in statistical mechanics. A direct computation shows that the averages of $h_i$ can be written as

$${\eta }^i={\mathbb{E}}_{p_{\theta }}\left(h_i\right)={\displaystyle \frac{\partial \psi \left(\theta \right)}{\partial {\theta }_i}},i=1,\dots ,n$$

(7)

and that the covariances are

$$\mathrm{cov}(h_i,h_j)={\mathbb{E}}_{p_{\theta }}\left(h_i{h}_j\right)-{\mathbb{E}}_{p_{\theta }}\left(h_i\right){\mathbb{E}}_{p_{\theta }}\left(h_j\right)={\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_i\partial {\theta }_j}}.$$

Using the score vectors $l_i=h_i-\partial \psi /\partial {\theta }_i$ associated to the exponential model, one can show that the Hessian of the cumulant function coincides with the Fisher Riemannian metric $g_{ij}$ [3]; that is

$$g_{ij}\left(\theta \right)={\mathbb{E}}_{p_{\theta }}\left(l_i{l}_j\right)={\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_i\partial {\theta }_j}}\left(\theta \right),$$

which is positive definite by the maximum rank hypothesis on h. So $(\mathcal{E},g)$ is a Riemannian manifold. To show that $\mathcal{E}$ is dually flat manifold one needs to introduce Amari $\alpha$-connection with Christoffel symbols:

$${\mathrm{\Gamma }}_{ij,k}^{\left(\alpha \right)}=\mathbb{E}[({\partial }_i{\partial }_{j}l+{\displaystyle \frac{1-\alpha }{2}}{\partial }_{i}l{\partial }_{j}l){\partial }_{k}l],\alpha \in [0,1]$$

where $l=\ln p(x,\theta )$. By direct computation, one shows that ${\mathrm{\Gamma }}^{\left(1\right)}$ (resp. ${\mathrm{\Gamma }}^{(-1)}$) vanish in $\theta$ (resp. $\eta )$ coordinates, so $\theta$ and $\eta$ are global affine dual coordinates and the curvature $R^{\left(1\right)}$ and $R^{(-1)}$ both vanish, making $\mathcal{E}$ dually flat. Note that a connection can be locally flat (zero curvature) without admitting a global affine coordinate system. For an exponential family manifold the Christoffel symbols vanish on the entire domains of the $\theta$ and $\eta$ coordinates, which are convex open sets, so the flat structure is global. For more details, see [3].

4. Controlled Evolution in a Dually Flat Manifold

In this Section we show how the dual coordinate system $\eta ,\theta$ can be used to describe the controlled evolution of a point on a dually flat manifold. It is well known that the statistical description of a thermodynamic system is given by an exponential family (6) whose feature functions $h\left(x\right)\in {\mathbb{R}}^n$ are the physical observables of interest for the macroscopic description of the system given in terms of their average values $\eta \in {\mathbb{R}}^n$. The associated canonical parameters $\theta =\theta \left(\eta \right)$ are interpreted as generalized inverse temperatures. This is a generalization of the fact that if $h\left(x\right)\in \mathbb{R}$ represents the energy of the system in the microscopic phase-space configuration $x\in X$, its average $\eta$ is the internal energy of the system in contact with an heat bath at the inverse temperature $\theta \left(\eta \right)$. The heat bath allows one to control the (inverse) temperature $\theta$ of the system. This brief review of equilibrium statistical mechanics terminology motivates the name of (thermal) control coordinates for $\theta \in {\mathbb{R}}^n$ and of effect coordinate for $\eta \in {\mathbb{R}}^n$.

The equilibrium statistical mechanics formalism is an example of an exponential family manifold, which is an instance of a dually flat manifold; we are interested in the following two problems formulated for a dually flat n-dimensional Riemannian manifold $(M,g)$ equipped with a dual coordinate systems $(\eta ,\theta )$ (we mutuate the terminology of forward and backward problem, which is current in the applications of Information Geometry; from [12]):

(1)

Forward problem: given a path in control variables $\theta \left(t\right)\in {\mathbb{R}}^n$ to determine the corresponding evolution $\eta \left(t\right)\in {\mathbb{R}}^n$ in effect space;

(2)

Backward problem: given an observed evolution $\eta \left(t\right)$ in effect space, to determine the corresponding control or cause path $\theta \left(t\right)$.

Since $\eta$ and $\theta$ are local affine coordinates of a dually flat manifold, the above problems have a solution using (3)

$$\mathrm{forward} \mathrm{problem}:\mathrm{given}\theta \left(t\right),\mathrm{the} \mathrm{evolution} \mathrm{is}\eta \left(t\right)=\nabla \psi \left(\theta \right(t\left)\right)$$

(8)

and

$$\mathrm{backward} \mathrm{problem}:\mathrm{given}\eta \left(t\right),\mathrm{the} \mathrm{evolution} \mathrm{is}\theta \left(t\right)=\nabla \phi \left(\eta \right(t\left)\right).$$

(9)

In the following we will consider two important classes of forward and backward problems.

4.1. Forward Problem: System in a Heat Bath

In [1] the following instance of a forward problem for a control path referred to a k-cut of a mutually dual foliation (see Section Mutually Dual Foliations of a Dually Flat Space) is considered: let $t\mapsto ({\theta }_A\left(t\right),{\overline{\theta }}_B$) be given; that is, the system is contained in a heat bath, which realizes the constraint ${\theta }_B\left(t\right)={\overline{\theta }}_B$. Using (8) we have, adopting for simplicity the shorthand block notation, ${\psi }_{AB}={\partial }_{AB}\psi$

$$\left\{\begin{array}{c}{\dot{\eta }}^A\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }_A\psi ({\theta }_A\left(t\right),{\overline{\theta }}_B)\right)={\partial }_{AA}\psi {\dot{\theta }}_A={\psi }_{AA}{\dot{\theta }}_A,\hfill \\ \\ {\dot{\eta }}_B\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }_B\psi ({\theta }_A\left(t\right),{\overline{\theta }}_B)\right)={\psi }_{BA}{\dot{\theta }}_A\hfill \end{array}\right.$$

and since ${\psi }_{AA}\in Mat\left(k\right)$ is invertible, we can write ${\dot{\theta }}_A={\psi }_{AA}^{-1}{\dot{\eta }}^A$ and obtain the result

$${\dot{\eta }}^B={\psi }_{BA}{\psi }_{AA}^{-1}{\dot{\eta }}^A={\mathrm{\Gamma }}_{BA}({\theta }_A\left(t\right),{\overline{\theta }}_B){\dot{\eta }}^A,{\dot{\eta }}^A={\psi }_{AA}{\dot{\theta }}_A.$$

(10)

So, if the system is contained in a B-type heath bath ${\theta }_B\left(t\right)={\overline{\theta }}_B$, then the evolution in the ${\eta }^B$ effect variables can be reconstructed from the evolution of the ${\eta }^A$ variables using (10)₁.

This result has an interesting geometrical interpretation: if we consider the local fibration $\eta \circ {\pi }_A\left(p\right)={\eta }^A$, with fibers $E\left({\overline{\eta }}^A\right)$ (see (5)), the orthogonal submanifold $Q\left({\overline{\theta }}_B\right)$ defines an orthogonal complement $H(\eta \circ {\pi }_A)$ to the tangent space to the fibers $V(\eta \circ {\pi }_A)=\ker T(\eta \circ {\pi }_A)$ and, hence, $Q\left({\overline{\theta }}_B\right)$ is an integrable orthogonal Ehresmann connection, i.e, a section of the fiber map $\eta \circ {\pi }_A$ and the ${\mathrm{\Gamma }}_{BA}$ are the coefficients of the Ehresmann connection

$$\omega =d{\eta }^B-{\mathrm{\Gamma }}_{BA}d{\eta }^A,\ker \omega =H(\eta \circ {\pi }_A)$$

see, e.g., [13,14]. Therefore (10) can be seen as the horizontal lift of the path ${\eta }^A\left(t\right)$ in base space

$${\dot{\eta }}^B=\mathrm{hor}\left({\dot{\eta }}^A\left(t\right)\right)={\mathrm{\Gamma }}_{BA}\left(\theta \left(t\right)\right){\dot{\eta }}^A\left(t\right)\otimes {\partial }^B$$

where $\mathrm{hor}:{\mathbb{R}}^k\to H(\eta \circ {\pi }_A)=TQ\left({\theta }_B\right)\subset T_{p}M$ is the horizontal lift operator whose local expression is as follows: let $u={\dot{\eta }}^A{\partial }^A$ be a base vector, then

$$\mathrm{hor}\left(u\right)={\dot{\eta }}^a\left({\partial }^A+{\mathrm{\Gamma }}_{BA}{\partial }^B\right),\omega \left(\mathrm{hor}\left(u\right)\right)=0.$$

In the case of an exponential family manifold, the connection coefficients ${\mathrm{\Gamma }}_{BA}$ have an important interpretation in statistical theory

$${\mathrm{\Gamma }}_{BA}\left(\theta \right)={\psi }_{BA}{\psi }_{AA}^{-1}=cov(h_A,h_B)cov{(h_A,h_A)}^{-1}$$

where $cov(h_A,h_B)$ is the matrices of the covariances between $h_A$ and $h_B$ observables and $cov(h_A,h_A)$ is the square matrix of the variance and covariance between $h_A$-type observables. This shows that when the ${\theta }_B$ parameter are kept constant, the inference on the ${\eta }^B$ variables given the evolution of the ${\eta }^A$ variables is the linear predictor of the B random variables in terms of the A random variables; see [15].

The dual of the above system evolution is given by the following:

4.2. Backward Problem: System Insensitive to the Control in the A Variables

Given an evolution in the effect space of the type $\eta \left(t\right)=({\overline{\eta }}^A,{\eta }^B\left(t\right))$ to determine the corresponding time evolution in the cause parameter $\theta \left(t\right)$. Using the same argument as above we have from (8)

$$\left\{\begin{array}{c}{\dot{\eta }}^A\left(t\right)={\displaystyle \frac{d}{dt}}({\partial }_A\psi ({\theta }_A\left(t\right),{\theta }_B\left(t\right))={\psi }_{AA}{\dot{\theta }}_A+{\psi }_{AB}{\dot{\theta }}_B=0,\hfill \\ \\ {\dot{\eta }}^B\left(t\right)={\displaystyle \frac{d}{dt}}({\partial }_B\psi ({\theta }_A\left(t\right),{\theta }_B\left(t\right))={\psi }_{BA}{\dot{\theta }}_A+{\psi }_{BB}{\dot{\theta }}_B.\hfill \end{array}\right.$$

Therefore the constraint in effect space induces the following constraint in control space

$${\dot{\theta }}_A=-{\psi }_{AA}^{-1}{\psi }_{AB}{\dot{\theta }}_B=-{\mathrm{\Gamma }}_{AB}^T({\theta }_A\left(t\right),{\theta }_B\left(t\right)){\dot{\theta }}_B$$

(11)

where ${\mathrm{\Gamma }}_{AB}^T$ is the transpose of the matrix ${\mathrm{\Gamma }}_{BA}$. Note that if ${\dot{\theta }}_B\left(t\right)\ne 0$ and in the one-dimensional case ${\theta }_A,{\theta }_B\in \mathbb{R}$, the above Equation (11) can be given the simpler form of an ODE for the function ${\theta }_A={\theta }_A\left({\theta }_B\right)$ as

$${\displaystyle \frac{d{\theta }_A}{d{\theta }_B}}=-{\mathrm{\Gamma }}_{AB}^T({\theta }_A,{\theta }_B).$$

For the sake of completeness, we derive the above result (11) using the backward problem formulation in (9). Adopting the shorthand notation ${\partial }^{AB}\phi ={\phi }^{AB}$, we have

$$\left\{\begin{array}{c}{\dot{\theta }}_A\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }^A\phi ({\overline{\eta }}^A\left(t\right),{\eta }^B)\right)={\partial }^{AB}\phi {\dot{\eta }}^B={\phi }^{AB}{\dot{\eta }}^B,\hfill \\ \\ {\dot{\theta }}_B\left(t\right)={\displaystyle \frac{d}{dt}}\left({\partial }^B\phi ({\overline{\eta }}^A\left(t\right),{\eta }_B)\right)={\phi }^{BB}{\dot{\eta }}^B\hfill \end{array}\right.$$

and since ${\phi }^{BB}\in Mat(n-k)$ is invertible, we obtain

$${\dot{\theta }}_A={\phi }^{AB}{\left({\phi }^{BB}\right)}^{-1}{\dot{\theta }}_B={\mathrm{\Gamma }}^{AB}{\dot{\theta }}_B$$

which coincides with the above result (11) because

$${\mathrm{\Gamma }}^{AB}={\phi }^{AB}{\left({\phi }^{BB}\right)}^{-1}=-{\psi }_{AA}^{-1}{\psi }_{AB}=-{\mathrm{\Gamma }}_{AB}^T.$$

This last equality can be easily checked by expanding in block representation the identity

$$g^{ij}{g}_{jk}={\phi }^{ij}{\psi }_{jk}={\delta }_k^i.$$

Example: Forward Problem for the Gaussian Exponential Manifold

Before turning to the mixed control problem, we give a simple example of the theory developed so far. It is well known that the Gaussian probability density is the maximal entropy density under the constraint of fixed value for the first two moments. Therefore, we can consider for $x\in X=\mathbb{R}$ the feature functions $h_1\left(x\right)=x$ and $h_2\left(x\right)=x^2$. The probability density defines an exponential family of the form

$$p({\theta }_1,{\theta }_2,x)=e^{{\theta }_{1}x+{\theta }_2{x}^2-\psi ({\theta }_1,{\theta }_2)}$$

where

$$\psi ({\theta }_1,{\theta }_2)=\ln {\int }_{\mathbb{R}}{e}^{{\theta }_{1}x+{\theta }_2{x}^2}dx=-{\displaystyle \frac{{\theta }_1^2}{4{\theta }_2}}+{\displaystyle \frac{1}{2}}\ln (-{\theta }_2)+{\displaystyle \frac{1}{2}}\ln \pi ,{\theta }_2<0.$$

We consider the following forward control problem: to determine the evolution of ${\eta }^B={\eta }^2={\mathbb{E}}_{p_{\theta }}\left[x^2\right]$ given the evolution ${\eta }^A={\eta }^1={\mathbb{E}}_{p_{\theta }}\left[x\right]$ under the control path $({\theta }_A\left(t\right),{\overline{\theta }}_B)=({\theta }_1\left(t\right),{\overline{\theta }}_2)$. This can be done using (10) by computing the Ehresmann connection coefficient

$${\mathrm{\Gamma }}_{BA}={\mathrm{\Gamma }}_{21}={\displaystyle \frac{{\psi }_{12}}{{\psi }_{11}}}={\displaystyle \frac{{\psi }_{{\theta }_1{\theta }_2}}{{\psi }_{{\theta }_1{\theta }_1}}}=-{\displaystyle \frac{{\theta }_1}{{\theta }_2}}=-{\displaystyle \frac{{\theta }_1\left(t\right)}{{\overline{\theta }}_2}}$$

therefore, the horizontal lift equation is

$${\dot{\eta }}^2\left(t\right)={\mathrm{\Gamma }}_{21}\left(t\right){\dot{\eta }}^1\left(t\right)=-{\displaystyle \frac{{\theta }_1\left(t\right)}{{\overline{\theta }}_2}}{\dot{\eta }}^1\left(t\right),{\eta }^2\left(0\right)={\eta }_0^2$$

(12)

which can directly integrate

$${\eta }^2\left(t\right)=-{\displaystyle \frac{1}{{\overline{\theta }}_2}}{\int }_0^t{\theta }_1\left(t\right){\dot{\eta }}^1\left(t\right)dt+{\eta }_0^2.$$

If ${\dot{\eta }}^1\left(t\right)\ne 0$, ${\eta }^1\left(t\right)$ can be locally inverted in $t=t\left({\eta }^1\right)$ and we can express the horizontal lift (12) in the form

$${\displaystyle \frac{d{\eta }^2}{d{\eta }^1}}={\mathrm{\Gamma }}_{21}\left(t\right)={\mathrm{\Gamma }}_{21}\left(t\left({\eta }^1\right)\right)$$

so that

$${\eta }^2={\int }_{{\eta }_0^1}^{{\eta }^1}{\mathrm{\Gamma }}_{21}\left(t\left({\eta }^1\right)\right)d{\eta }^1+{\eta }_0^2=-{\displaystyle \frac{1}{{\overline{\theta }}_2}}{\int }_{{\eta }_0^1}^{{\eta }^1}{\theta }_1\left(t\left({\eta }^1\right)\right)d{\eta }^1+{\eta }_0^2.$$

As an example, consider the evolution ${\eta }^1\left(t\right)=t$, with inverse $t\left({\eta }^1\right)={\eta }^1$. We want to determine the control path ${\theta }_1\left(t\left({\eta }^1\right)\right)={\theta }_1\left({\eta }^1\right)$ so that the variance is a given function $var\left(h_1\right)={\eta }^2-{\left({\eta }^1\right)}^2=f\left({\eta }^1\right)$. We have the functional equation

$$f\left({\eta }^1\right)={\eta }^2-{\left({\eta }^1\right)}^2=-{\displaystyle \frac{1}{{\overline{\theta }}_2}}{\int }_{{\eta }_0^1}^{{\eta }^1}{\theta }_1\left({\eta }^1\right)d{\eta }^1+{\eta }_0^2-{\left({\eta }^1\right)}^2$$

which gives, by derivation,

$${\theta }_1\left({\eta }^1\right)=(-{\overline{\theta }}_2)(2{\eta }^1+f^{\prime }\left({\eta }^1\right)).$$

5. Mixed Control Problem and Inferential Scattering

The notion of inferential scattering was exposed in a couple of papers by E.T. Jaynes [1,2]. Jaynes’ original exposition refers to equilibrium statistical mechanics. In this setting, the results of the above Section describe the inference on the evolution of the average value of a group of random variables (B) given the evolution of the average value of another group (A) of random variables under the hypothesis that the random variables are described by an exponential family and a constraint is in force. It can be seen as an inference result.

To expose the notion of inferential scattering we need to introduce the notion of $(k,l)$-cut in a n-dimensional dual coordinate system of a dually flat manifold: let $(l,k)$ with $1<l<k<n$ and consider the splitting of the coordinates

$$\eta =({\eta }^a,{\eta }^c,{\eta }^B),\theta =({\theta }_a,{\theta }_c,{\theta }_B),a=1,\dots ,l,c=l+1,\dots ,k,B=k+1,\dots ,n.$$

In practice, we split the A block variables in $A=(a,c)$. Now we can consider the following instance of mixed control problem.

5.1. Mixed Control Problem for a System in a B-Type Heat Bath and Insensitive to the Control in the c Variables

This is a combination of the two above cases. We want to determine the evolution in the ${\eta }^B$ variables under the controlled evolution ${\theta }_a\left(t\right)$ knowing that we have the constraint ${\eta }^c\left(t\right)={\overline{\eta }}^c$ in the effect space and ${\theta }_B\left(t\right)={\overline{\theta }}_B$ in cause space. Reasoning as above, we consider the system of evolution equations

$$\left\{\begin{array}{c}{\dot{\eta }}^a={\displaystyle \frac{d}{dt}}({\partial }_a\psi ({\theta }_a(t),{\theta }_c(t),{\overline{\theta }}_B)={\psi }_{aa}{\dot{\theta }}_a+{\psi }_{ac}{\dot{\theta }}_c,\hfill \\ {\dot{\eta }}^c={\displaystyle \frac{d}{dt}}({\partial }_c\psi ({\theta }_a(t),{\theta }_c(t){\overline{\theta }}_B)={\psi }_{ca}{\dot{\theta }}_a+{\psi }_{cc}{\dot{\theta }}_c=0,\hfill \\ {\dot{\eta }}^B={\displaystyle \frac{d}{dt}}({\partial }_B\psi ({\theta }_a(t),{\theta }_c(t),{\overline{\theta }}_B)={\psi }_{Ba}{\dot{\theta }}_a+{\psi }_{Bc}{\dot{\theta }}_c.\hfill \end{array}\right.$$

From the second equation (note that ${\psi }_{cc}=g_{cc}$—see (4)—hence, it is invertible). we get

$${\dot{\theta }}_c=-{\psi }_{cc}^{-1}{\psi }_{ca}{\dot{\theta }}_a$$

(13)

and we substitute ${\dot{\theta }}_c$ in the first and third equation to obtain

$$\left\{\begin{array}{c}{\dot{\eta }}^a={\psi }_{aa}{\dot{\theta }}_a-{\psi }_{ac}{\psi }_{cc}^{-1}{\psi }_{ca}{\dot{\theta }}_a={\psi }_{aa}(\mathbb{I}-{\psi }_{aa}^{-1}{\psi }_{ac}{\psi }_{cc}^{-1}{\psi }_{ca}){\dot{\theta }}_a={\psi }_{aa}(\mathbb{I}-R(a,c)){\dot{\theta }}_a\hfill \\ {\dot{\eta }}^B={\psi }_{Ba}{\dot{\theta }}_a-{\psi }_{Bc}{\psi }_{cc}^{-1}{\psi }_{ca}{\dot{\theta }}_a=({\psi }_{Ba}-{\psi }_{Bc}{\psi }_{cc}^{-1}{\psi }_{ca}){\dot{\theta }}_a\hfill \end{array}\right.$$

(14)

where $\mathbb{I}$ is the identity matrix and we have introduced the correlation coefficient matrix

$$R(a,c)={\psi }_{aa}^{-1}{\psi }_{ac}{\psi }_{cc}^{-1}{\psi }_{ca}.$$

(15)

The last equality in (14)₁ can be inverted

$${\dot{\theta }}_a={(\mathbb{I}-R(a,c))}^{-1}{\psi }_{aa}^{-1}{\dot{\eta }}^a$$

and substituted in (14)₂ to finally obtain

$${\dot{\eta }}^B=({\psi }_{Ba}-{\psi }_{Bc}{\psi }_{cc}^{-1}{\psi }_{ca}){(\mathbb{I}-R(a,c))}^{-1}{\psi }_{aa}^{-1}{\dot{\eta }}^a={\mathcal{G}}_{Ba}{\dot{\eta }}^a.$$

(16)

This is our inference result on the evolution of the ${\eta }^B$ variables given the mixed (forward and backward) control. Note that ${\mathcal{G}}_{Ba}={\mathcal{G}}_{Ba}({\theta }_a\left(t\right),{\theta }_c\left(t\right),{\overline{\theta }}_B))$ but the evolution in the controlled variables ${\theta }_a,{\theta }_c$ is subject to the Equation (13), which enforces the constraint in effect space ${\eta }^c={\overline{\eta }}^c$.

5.2. The Inferential Scattering Formula

Since $R(a,c)$ in (15) is the analogous of the correlation coefficient of two random variables

$$\rho (A,B)={\displaystyle \frac{cov{(A,B)}^2}{var\left(A\right)var\left(B\right)}},0<\rho (A,B)<1$$

it reasonable to assume that the matrix $R(a,c)$ has spectral radius less than one. Under this assumption, the following Taylor expansion hold

$${(\mathbb{I}-R(a,c))}^{-1}=\mathbb{I}+R(a,c)+R{(a,c)}^2+R{(a,c)}^3+\dots$$

(17)

If we substitute (17) in (16), we readily obtain the following expansion

$${\dot{\eta }}^B=({\mathrm{\Gamma }}_{Ba}-{\mathrm{\Gamma }}_{Bc}{\mathrm{\Gamma }}_{ca}+{\mathrm{\Gamma }}_{Ba}{\mathrm{\Gamma }}_{ac}{\mathrm{\Gamma }}_{ca}-{\mathrm{\Gamma }}_{Bc}{\mathrm{\Gamma }}_{ca}{\mathrm{\Gamma }}_{ac}{\mathrm{\Gamma }}_{ca}+\dots ){\dot{\eta }}^a$$

(18)

where we have set

$${\mathrm{\Gamma }}_{Ba}={\psi }_{Ba}{\psi }_{aa}^{-1},{\mathrm{\Gamma }}_{Bc}={\psi }_{Bc}{\psi }_{cc}^{-1},{\mathrm{\Gamma }}_{ca}={\psi }_{ca}{\psi }_{aa}^{-1},{\mathrm{\Gamma }}_{ac}={\psi }_{ac}{\psi }_{cc}^{-1}.$$

The above expansion Formula (18) has the following interpretation (see [1]): the term ${\mathrm{\Gamma }}_{Ba}d{\eta }^a$ is the inferred (infinitesimal) variation of the B observable induced by a variation $d{\eta }^a$ of the a observables. The term ${\mathrm{\Gamma }}_{Bc}{\mathrm{\Gamma }}_{ca}d{\eta }^a$ is the inferred variation in B as induced by a variation $d{\eta }_c={\mathrm{\Gamma }}_{ca}d{\eta }^a$ of the c observables, which is, in turn, induced by the variation $d{\eta }^a$. All the other terms in (18) have a similar interpretation. This result has been called inferential scattering formula in [2] by analogy with the Born series expansion of the operator, which describes the multiple scattering phenomenon [9]: a wave can reach a point B in space directly from source a (${\mathrm{\Gamma }}_{Ba}$ term) or after having been reflected by an obstacle in c (${\mathrm{\Gamma }}_{Bc}{\mathrm{\Gamma }}_{ca}$ term) or after having been reflected back in a from c and then arriving at B (${\mathrm{\Gamma }}_{Ba}{\mathrm{\Gamma }}_{ac}{\mathrm{\Gamma }}_{ca}$ term). See Figure 2 below. Note that the first term in (18) ${\mathrm{\Gamma }}_{Ba}$ coincides with the one derived before in (10) in case of forward control. Of course the size of the higher order correction terms after ${\mathrm{\Gamma }}_{Ba}$ depend on the norm of the correlation matrix $R(a,c)$.

5.3. A Mixed Control Example: Neural Network of Two Neurons

The following toy model (see, e.g., [6]) describes the neural activity of a set of two neurons using two discrete random variables, which can take the value 1 (active) or 0 (silent). The state space of the network is $X=\left\{\right(0,0),(0,1),(1,0),(1,1\left)\right\}$. Given $x=(x_1,x_2)\in X$, let $p\left(x\right)$ be the probability that the network is in the state x. As observables we consider $h_i\left(x\right)=x_i$, $i=1,2$, which describe the state of the two neurons and $h_3\left(x\right)=x_1{x}_2$. The exponential manifold is comprised of discrete probability distributions of the form

$$p\left(x\right)=e^{\theta ·h-\psi \left(\theta \right)}=e^{{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_1{x}_2-\psi \left(\theta \right)}$$

where

$$\psi \left(\theta \right)=\ln \left(\sum _{x\in X}{e}^{{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_1{x}_2}\right)=\ln \left(1+e_1^{\theta }+e_2^{\theta }+e^{{\theta }_1+{\theta }_2+{\theta }_3}\right)=\ln Z\left(\theta \right)$$

The associated effect space coordinates

$${\eta }^i={\mathbb{E}}_{p_{\theta }}\left[x_i\right]={\displaystyle \frac{\partial \psi }{\partial {\theta }_i}},i=1,2{\eta }^3={\mathbb{E}}_{p_{\theta }}\left[x_1{x}_2\right]={\displaystyle \frac{\partial \psi }{\partial {\theta }_3}}$$

are called firing rate of neurons $i=1,2$ and joint firing rate. A direct computation shows that

$${\eta }^i={\displaystyle \frac{e^{{\theta }_i}+e^{{\theta }_1+{\theta }_2+{\theta }_3}}{Z\left(\theta \right)}}=\mathrm{Prob}(x_i=1),i=1,2,{\eta }^3={\displaystyle \frac{e^{{\theta }_1+{\theta }_2+{\theta }_3}}{Z\left(\theta \right)}}=\mathrm{Prob}(x_1{x}_2=1).$$

The Riemannian metric matrix $g\in Mat\left(3\right)$ can also be computed directly as

$$g_{ij}\left(\theta \right)={\displaystyle \frac{{\partial }^2\psi \left(\theta \right)}{\partial {\theta }_i\partial {\theta }_j}}.$$

We investigate the mixed control problem (see Section 5) for this system. We consider the $(2,1)$-cut in the 3-dimensional dual coordinate system defined by the following identification

$$({\eta }^1,{\eta }^2,{\eta }^3)=({\eta }^a,{\eta }^c,{\eta }^B),({\theta }_1,{\theta }_2,{\theta }_3)=({\theta }_a,{\theta }_c,{\theta }_B).$$

We want to determine the evolution in the ${\eta }^B={\eta }^3$ variable (joint firing rate of neuron 1 and 2) in terms of the evolution of ${\eta }^a={\eta }^1$ (firing rate of neuron 1) under the control ${\theta }_B\left(t\right)={\theta }_3\left(t\right)$ knowing that we have the constraint ${\eta }^c\left(t\right)={\eta }^2\left(t\right)={\overline{\eta }}^2$ (firing rate of neuron 2 is insensitive to the control) in the effect space. To this we can use the perturbation expansion (18)

$$d{\eta }^3=({\mathrm{\Gamma }}_{31}-{\mathrm{\Gamma }}_{32}{\mathrm{\Gamma }}_{21}+{\mathrm{\Gamma }}_{31}{\mathrm{\Gamma }}_{12}{\mathrm{\Gamma }}_{21}-{\mathrm{\Gamma }}_{32}{\mathrm{\Gamma }}_{21}{\mathrm{\Gamma }}_{12}{\mathrm{\Gamma }}_{21}+\dots )d{\eta }^1$$

where

$${\mathrm{\Gamma }}_{31}={\displaystyle \frac{{\psi }_{31}}{{\psi }_{11}}},{\mathrm{\Gamma }}_{32}={\displaystyle \frac{{\psi }_{32}}{{\psi }_{22}}},{\mathrm{\Gamma }}_{21}={\displaystyle \frac{{\psi }_{21}}{{\psi }_{11}}}{\mathrm{\Gamma }}_{12}={\displaystyle \frac{{\psi }_{12}}{{\psi }_{22}}}$$

and

$$R(1,2)={\mathrm{\Gamma }}_{12}{\mathrm{\Gamma }}_{21}={\displaystyle \frac{{\psi }_{12}{\psi }_{21}}{{\psi }_{11}{\psi }_{22}}}={\displaystyle \frac{{\psi }_{12}^2}{{\psi }_{11}{\psi }_{22}}}={\displaystyle \frac{cov{(x_1,x_2)}^2}{var\left(x_1\right)var\left(x_2\right)}}={\rho }^2(x_1,x_2)$$

where $\rho (x_1,x_2)\in [0,1]$ is Pearson’s correlation coefficient, which is a measure of the strength of the linear correlation (normalized covariance) between the two variables. So the perturbation parameter in the Born series expansion (17) is the correlation coefficient. The explicit expression (setting ${\theta }_1=x$, ${\theta }_2=y$ and ${\theta }_3=z$ for simplicity) is

$$R(1,2)(x,y,z)=({\displaystyle \frac{1}{1+e^x}}-{\displaystyle \frac{1}{e^{x+z}}})({\displaystyle \frac{1}{1+e^y}}-{\displaystyle \frac{1}{e^{y+z}}}).$$

Note that the two control variables $(x,y)$ are not completely free because relation (13), which enforces the constraint ${\eta }^3={\eta }^c={\overline{\eta }}^c$ holds. In our notation, after a straightforward computation, (13) reads

$$\dot{y}=-{\displaystyle \frac{{\psi }_{12}}{{\psi }_{22}}}\dot{x}=\left({\displaystyle \frac{1}{1+e^x}}-{\displaystyle \frac{1}{1+e^{x+\overline{z}}}}\right)\dot{x}=-{\mathrm{\Gamma }}_{12}(x,\overline{z})\dot{x}$$

where we have set $z={\theta }_3={\overline{\theta }}_3=\overline{z}$. If $\dot{x}\ne 0$ the above equation can be readily integrated

$${\displaystyle \frac{dy}{dx}}={\mathrm{\Gamma }}_{12}(x,\overline{z})\mathrm{hence}y\left(x\right)=\int {\mathrm{\Gamma }}_{12}(x,\overline{z})dx$$

to give

$$y(x,\overline{z})=2[\mathrm{artanh}(1+2e^{x+\overline{z}})-\mathrm{artanh}(1+2e^x)].$$

Therefore, in the mixed control case, the correlation coefficient $R(1,2)$ is a function of ${\theta }_1=x$ only, of the form

$$R_{\overline{z}}(1,2)\left(x\right)=R(1,2)(x,y(x,\overline{z}),\overline{z}).$$

A plot of $R(1,2)$ is in Figure 3. The figure helps to visualize the size of the perturbation expansion parameter $R(1,2)$. This simple example is only suggestive of the possible applications of the notion of inferential scattering.

6. Discussion

The primary aim of this paper was to embed the notion of inferential scattering as exposed by E.T. Jaynes for a thermodynamical system into the language and the concepts of Information Geometry. The key geometrical object turns out to be a dually flat manifold with its coupled bi-orthogonal coordinate systems $\eta$ and $\theta$. These coordinates have a simple and appealing interpretation in the statistical mechanics language of (thermal) control and effect variables.

We have shown that the forward and backward control problems can be seen as a horizontal lift (parallel transport) with respect to an integrable orthogonal Ehresmann connection, which is globally defined due to the dually flat nature of the ambient manifold. In this way the controlled evolution of the system in effect space is globally defined providing a generalization of the original formulation for infinitesimal displacements contained in the Jaynes papers. We stress that this notion of controlled evolution has a potentially wide application range being a form of inference, hence applicable to every statistical system with insufficient information. We have presented a couple of examples that highlight the use of the globally defined horizontal lift equation to describe the controlled evolution of the dually flat manifold, one dealing with a bi-dimensional exponential family manifold and one with a tri-dimensional exponential family describing a network of two neurons, both outside the classical Statistical Mechanics framework.

Moreover, we have shown that Jaynes’ notion of inferential scattering, which is the computation of the infinitesimal variation in effect variables subject to a mixed control, can be computed as a sequence of horizontal lift operations with respect to different fibrations. Even if the actual computation is quite involved (see Formula (18) above), in principle, it can be carried over and give the finite system evolution.

The geometrical analysis presented in this paper could be further investigated; for example, one could analyze its formulation in the larger setting of Hessian manifolds and the possible developments using a weaker setting for the Legendre transform. However, we hope that the material presented here can renew interest in the scientific community for the original contribution of E.T. Jaynes and its potential applications.