Examples of the Application of Nonparametric Information Geometry to Statistical Physics

Abstract

We review a nonparametric version of Amari’s information geometry in which the set of positive probability densities on a given sample space is endowed with an atlas of charts to form a differentiable manifold modeled on Orlicz Banach spaces. This nonparametric setting is used to discuss the setting of typical problems in machine learning and statistical physics, such as black-box optimization, Kullback-Leibler divergence, Boltzmann-Gibbs entropy and the Boltzmann equation.

1. Introduction

Information geometry was developed in the seminal monograph by Amari and Nagaoka [1], where previous—essentially metric—descriptions of probabilistic and statistics concepts are extended in the direction of differential geometry, including the fundamental treatment of differential connections. The differential geometry involved in their construction is finite dimensional, and the formalism is based on coordinate systems. Following a suggestion by Phil Dawid in [2,3,4], a particular nonparametric version of the Amari-Nagaoka theory was developed in a series of papers [5,6,7,8,9,10,11,12,13,14], where the set ${\mathcal{P}}_{>}$ of all strictly positive probability densities of a measure space is shown to be a Banach manifold (as defined in [15,16,17]) modeled on an Orlicz Banach space, see [18] (Ch II).

Specifically, Gibbs densities, $q={\mathrm{e}}^{u-K_p\left(u\right)}·p$, ${error}_p\left[u\right]=0$, are represented by the chart, $s_p:q\mapsto u$. Because of the exponential form, the random variable, u, is required to belong to an exponential Orlicz space, which is similar to ordinary Lebesgue spaces, but lacks some important features of these spaces, such as reflexivity and separability. On the other side, the nonparametric setting emphasizes in a nice way the fact that statistical manifolds are actually affine manifolds with a Hessian structure, cfr. [19].

Such a formalism has been frequently criticized as unnecessarily involved to be of use in practical applications and, also, as lacking really new results with respect to the Amari-Nagaoka theory. However, it should be observed that most applications in statistical physics, such as the Boltzmann equation theory [20], are intrinsically nonparametric. I like to quote here a line by Serge Lang in [17] (p. vi): “One major function of finding proofs valid in the infinite dimensional case is to provide proofs which are especially natural and simple in the finite dimensional case.” A good example is the use here of a different Banach space for each chart in the atlas defining our manifold, with a notable advantage in the interpretation of the results.

If $p\left(t\right)$, $t\in I$, is a curve in the manifold of positive densities, the Fisher score, $\delta p\left(t\right)=\frac{d}{dt}ln\left(p\left(t\right)\right)$, is a parameterized family of random variables, such that ${error}_{p\left(t\right)}\left[\delta p\left(t\right)\right]=0$, $t\in I$. The Fisher score provides the correct notion of the velocity of a statistical curve, while the set of all Fisher scores, i.e., the vector space of all random variables, u, such that ${error}_p\left[u\right]=0$, suggest the form of the tangent space at p. We attach to each density $p\in {\mathcal{P}}_{>}$ a vector space of random variables whose expected value with respect to p is zero to form the linear fiber of a vector bundle on the statistical manifold. This vector space can be either an Orlicz space, denoted here $B_p$ or $*B_p$, or a Hilbert space $H_p=L_0^2\left(p\right)$. The purpose of this paper is to show that this mathematical formalism can be rigorously defined in such a way as to allow for the treatment of the Boltzmann equation as an evolution equation on the statistical manifold.

This paper is organized as follows. Section 2 and Section 3 are a review of the basic material on statistical exponential manifolds with some emphasis on the functional analytic setting and on second order structures. Section 4 contains a discussion of examples of the application to the differential geometry of expected values, Kullback-Leibler divergence, Boltzmann-Gibbs entropy and the Boltzmann equation. Section 5 presents some topics that would require further study, together with references to some lines of current research.

2. Model Spaces

Given a σ-finite measure space, $(\Omega ,\mathcal{F},\mu )$, we denote by ${\mathcal{P}}_{>}$ the set of all densities that are positive μ-a.s, by ${\mathcal{P}}_{\ge }$ the set of all densities, by ${\mathcal{P}}_1$ the set of measurable functions f with $\int fd\mu =1$. In the finite state space case, ${\mathcal{P}}_1$ is an affine subspace, ${\mathcal{P}}_{\ge }$ is the simplex and ${\mathcal{P}}_{>}$ its topological interior. We summarize below the basic notations and results. Missing proofs are to be found, e.g., in [10] and in [18] (Ch II).

If both ϕ and ${\varphi }_{*}$ are monotone, continuous functions on ${\mathbb{R}}_{\ge }$ onto itself, such that ${\varphi }^{-1}={\varphi }_{*}$, we call the pair:

$$\Phi \left(x\right)={\int }_0^{\left|x\right|}\varphi \left(u\right)du,{\Phi }_{*}\left(y\right)={\int }_0^{\left|y\right|}{\varphi }_{*}\left(v\right)dv$$

(1)

a Young pair. Each Young pair satisfies the Young inequality:

$$\left|xy\right|\le \Phi \left(x\right)+{\Phi }_{*}\left(y\right)$$

(2)

with equality, if and only if $y=\varphi \left(x\right)$. The relation in a Young pair is symmetric, and either element is called a Young function. We will use the following Young pairs:

$$\begin{array}{ccccc}& {\varphi }_{*}& \varphi ={\varphi }_{*}^{-1}& {\Phi }_{*}& \Phi \\ \text{(a)}\hfill & ln\left(1+u\right)& {\mathrm{e}}^v-1& (1+\left|x\right|)ln\left(1+\left|x\right|\right)-\left|x\right|& {\mathrm{e}}^{\left|y\right|}-1-\left|y\right|\\ \text{(b)}\hfill & {sinh}^{-1}u& sinhv& \left|x\right|{sinh}^{-1}\left|x\right|-\sqrt{1+x^2}+1& coshy-1\end{array}$$

(3)

Let us derive a few elementary, but crucial, inequalities. If $x\ge 0$:

$${\Phi }_{*}^{\text{(a)}}\left(x\right)={\int }_0^x\frac{x-u}{1+u}du,{\Phi }_{*}^{\text{(b)}}\left(x\right)={\int }_0^x\frac{x-u}{\sqrt{1+u^2}}du$$

(4)

hence, as $\sqrt{1+u^2}\le 1+u\le \sqrt{2}\sqrt{1+u^2}$, if $u\ge 0$, for all real x, we have:

$${\Phi }_{*}^{\text{(a)}}\left(x\right)\le {\Phi }_{*}^{\text{(b)}}\left(x\right)\le \sqrt{2}{\Phi }_{*}^{\text{(a)}}\left(x\right)$$

(5)

From Equation (4), we have for $a>1$:

$${\Phi }_{*}^{\text{(a)}}\left(ax\right)=a^2{\int }_0^x\frac{x-v}{1+av}dv\le a^2{\Phi }_{*}^{\text{(a)}}\left(x\right),{\Phi }_{*}^{\text{(b)}}\left(ax\right)=a^2{\int }_0^x\frac{x-v}{\sqrt{1+a^2{v}^2}}dv\le a^2{\Phi }_{*}^{\text{(b)}}\left(x\right)$$

(6)

In a similar way, from:

$${\Phi }^{\text{(a)}}\left(y\right)={\int }_0^y(y-v){\mathrm{e}}^{v}dv,{\Phi }^{\text{(b)}}\left(y\right)={\int }_0^y(y-v)coshvdv$$

(7)

and $coshv\le {\mathrm{e}}^v\le 2coshv$, if $v\ge 0$ we have a relation similar to Equation (5), that is, for all y:

$${\Phi }^{\text{(}b)}\left(y\right)\le {\Phi }^{\text{(}a)}\left(y\right)\le 2{\Phi }^{\text{(}b)}\left(y\right)$$

(8)

Property Equation (6) does not hold in this case, i.e., $\Phi \left(ax\right)/\Phi \left(x\right)$, $a>1$, is unbounded for $x\to \infty$. Such a type of inequality is called a ${\Delta }_2$-condition and has a crucial role in the theory of Orlicz spaces; see [18] (Th. 8.14).

If Φ is any Young function, a real random variable, u, belongs to the Orlicz space, $L^{\Phi }\left(p\right)$, if ${error}_p\left[\Phi \left(\alpha v\right)\right]<+\infty$ for some $\alpha >0$. A norm is obtained by defining the set, $\left\{v:{error}_p\left[\Phi \left(v\right)\right]\le 1\right\}$, to be the closed unit ball. It follows that the open unit ball consists of those u’s, such that $\alpha u$ is in the closed unit ball for some $\alpha >1$. The corresponding norm, ${\parallel ·\parallel }_{\Phi ,p}$, is called the Luxemburg norm and defines a Banach space; see [18] (Th 7.7). From Equation (8) and Equation (5) follows that cases (a) and (b) in Equation (3) define equal vector spaces with equivalent norms; see [10] (Lemma 1). Therefore, we drop any mention of them.

The Young function, $cosh-1$, has been chosen here, because the condition ${error}_p\left[\Phi \left(\alpha v\right)\right]<+\infty$ is clearly equivalent to ${error}_p\left[{\mathrm{e}}^{tv}\right]<+\infty$ for $t\in [-\alpha ,\alpha ]$, that is, the random variable, u, has a Laplace transform around zero. The case of a moment-generating function defined on all of the real line is special and defines a notable subspace of the Orlicz space. The use of such a space has been proposed by [21].

There are technical issues in working with Orlicz spaces, such as $L^{(cosh-1)}\left(p\right)$, in particular, the regularity of its unit sphere $S_{cosh-1}=\left\{u:{∥u∥}_{(cosh-1),p}=1\right\}$. In fact, while ${error}_p\left[coshu-1\right]=1$ implies $u\in S_{cosh-1}$, the latter implies only ${error}_p\left[coshu-1\right]\le 1$. Subspaces of $L^{\Phi }\left(p\right)$, where it cannot happen at the same time ${∥(cosh-1),p∥}_{,}u=1$ and ${error}_p\left[coshu-1\right]<1$ are of special interest. In general, the sphere, $S_{cosh-1}$, is not smooth; see an example in [14] ( Example 3).

Because the functions, Φ and ${\Phi }_{*}$, are a Young pair, for each $u\in L^{\Phi }\left(p\right)$ and $v\in L^{{\Phi }_{*}}\left(p\right)$, such that ${∥u∥}_{\Phi ,p},{∥v∥}_{{\Phi }_{*},p}\le 1$, we have from the Young inequality Equation (2), $\left|{error}_p\left[uv\right]\right|\le 2$; hence:

$$L^{{\Phi }_{*}}\left(p\right)\times L^{\Phi }\left(p\right)\ni (v,u)\mapsto {〈u,v〉}_p={error}_p\left[uv\right]$$

(9)

is a duality pairing, $\left|{〈u,v〉}_p\right|\le 2{∥u∥}_{{\Phi }_{*},p}{∥v∥}_{\Phi ,p}$. It is a classical result that in our case Equation (3), the space, $L^{{\Phi }_{*}}\left(p\right)$, is separable and its dual space is $L^{\Phi }\left(p\right)$, the duality pairing being $(u,v)\mapsto {〈u,v〉}_p$. This duality extends to a continuous chain of spaces:

$$L^{\Phi }\left(p\right)\to L^a\left(p\right)\to L^b\left(p\right)\to L^{{\Phi }_{*}}\left(p\right),1<b\le 2,\frac{1}{a}+\frac{1}{b}=1$$

(10)

where → denotes continuous injection.

2.1. Cumulant Generating Functional

Let $p\in {\mathcal{P}}_{>}$ be given. The following theorem has been proven in [9] (Ch 2); see also [10].

Proposition 1

• For $a\ge 1$, $n=0,1,\cdots$ and $u\in L^{\Phi }\left(p\right)$:

  $${\lambda }_{a,n}\left(u\right):\left(w_1,\cdots ,w_n\right)\mapsto {\displaystyle \frac{w_1}{a}}\cdots {\displaystyle \frac{w_n}{a}}{\mathrm{e}}^{\frac{u}{a}}$$

  (11)

  is a continuous, symmetric, n-multi-linear map from $L^{\Phi }\left(p\right)$ to $L^a\left(p\right)$.
• $v\mapsto {\sum }_{n=0}^{\infty }\frac{1}{n!}{\left({\displaystyle \frac{v}{a}}\right)}^n$ is a power series from $L^{\Phi }\left(p\right)$ to $L^a\left(p\right)$, with radius of convergence $\ge 1$.
• The superposition mapping, $v\mapsto {\mathrm{e}}^{v/a}$, is an analytic function from the open unit ball of $L^{\Phi }\left(p\right)$ to $L^a\left(p\right)$.

The previous theorem provides an improvement upon the original construction of [5].

Definition 1 Let $\Phi =cosh-1$ and $B_p=L_0^{\Phi }\left(p\right)=\left\{u\in L_0^{\Phi }\left(p\right):{error}_p\left[u\right]=0\right\}$, $p\in {\mathcal{P}}_{>}$. The moment generating functional is $M_p:L^{\Phi }\left(p\right)\ni u\mapsto {error}_p\left[{\mathrm{e}}^u\right]\in {\mathbb{R}}_{>}\cup \left\{+\infty \right\}$. The cumulant generating functional is $K_p:B_p\ni u\mapsto log{M}_p\left(u\right)\in {\mathbb{R}}_{>}\cup \left\{+\infty \right\}$.

The moment-generating functional is the partition functional (normalizing factor) of the Gibbs model, $({\mathrm{e}}^u/M_p\left(u\right))·p\in {\mathcal{P}}_{>}$, if $u\in L^{\Phi }\left(p\right)$, $M_p\left(u\right)<+\infty$. The same model is written ${\mathrm{e}}^{u-K_p\left(u\right)}·p$, if, moreover, ${error}_p\left[u\right]=0$.

Proposition 2

• $K_p\left(0\right)=0$; otherwise, for each $u\ne 0$, $K_p\left(u\right)>0$.
• $K_p$ is convex and lower semi-continuous, and its proper domain is a convex set that contains the open unit ball of $B_p$; in particular, the interior of the proper domain is a non-empty open convex set denoted ${\mathcal{S}}_p$.
• $K_p$ is infinitely Gâteaux-differentiable in the interior of its proper domain.
• $K_p$ is bounded, infinitely Fréchet-differentiable and analytic on the open unit ball of $B_p$.

Other properties of the functional $K_p$ are described below, as they relate directly to the exponential manifold.

3. Exponential Manifold

The set of positive densities, ${\mathcal{P}}_{>}$, around a given $p\in {\mathcal{P}}_{>}$ is modeled by the subspace of centered random variables in the Orlicz space, $L^{\Phi }\left(p\right)$. Hence, it is crucial to discuss the isomorphism of the model spaces for different p’s.

Definition 2 (Maximal exponential model: [10] (Def 20)). For $p\in {\mathcal{P}}_{>}$, let ${\mathcal{S}}_p$ be the topological interior of the proper domain of the cumulant functional, $K_p:B_p$. The maximal exponential model at p is:

$$\mathcal{E}\left(p\right)=\left\{{\mathrm{e}}^{u-K_p\left(u\right)}·p:u\in {\mathcal{S}}_p\right\}$$

(12)

It is important to observe that $q\in \mathcal{E}\left(p\right)$ is equivalent to $p\in \mathcal{E}\left(q\right)$, as is proven in Proposition 3 below.

Definition 3 (Connected densities). Densities $p,q\in {\mathcal{P}}_{>}$ are connected by an open exponential arc, $p⌣q$, if there exists an open exponential family containing both, i.e., if for a neighborhood I of $[0,1]$:

$$\int p^{1-t}{q}^{t}d\mu ={error}_p\left[{\left(\frac{q}{p}\right)}^t\right]={error}_q\left[{\left(\frac{p}{q}\right)}^{1-t}\right]<+\infty ,t\in I$$

(13)

The following example is of interest for the applications in Section 4. Let $f_0$ be the standard normal density on ${\mathbb{R}}^N$ and f, a density, $f\left(x\right)\propto (1+{\left|x\right|}^a)f_0\left(x\right)$, $a>0$. Then, $\int {(1+{\left|x\right|}^a)}^t{f}_0\left(x\right)dx<+\infty$ for all real t, hence $f_0⌣f$.

Proposition 3 (Characterization of a maximal exponential model: ([10], Th 19 and 21)). The following statements are equivalent:

• $p,q\in {\mathcal{P}}_{>}$ are connected by an open exponential arc, $p⌣q$;
• $q\in \mathcal{E}\left(p\right)$;
• $\mathcal{E}\left(p\right)=\mathcal{E}\left(q\right)$;
• $log\frac{q}{p}$ belongs to both $L^{\Phi }\left(p\right)$ and $L^{\Phi }\left(q\right)$.

If it holds, then:

5.

$L^{\Phi }\left(p\right)$ and $L^{\Phi }\left(q\right)$ are equal as vector spaces and their norms are equivalent.

We can now define the exponential manifold as follows.

Definition 4 (Exponential manifold: [5,7,9,10]). For each $p\in {\mathcal{P}}_{>}$, define the charts:

$$s_p:\mathcal{E}\left(p\right)\ni q\mapsto ln\left(\frac{q}{p}\right)-{error}_p\left[ln\left(\frac{q}{p}\right)\right]\in {\mathcal{S}}_p\subset B_{,}$$

(14)

with inverse

$$s_p^{-1}=e_p:{\mathcal{S}}_p\ni u\mapsto {\mathrm{e}}^{u-K_p\left(u\right)}·p\in \mathcal{E}\left(p\right)\subset {\mathcal{P}}_{>}$$

(15)

The atlas, $\left\{s_p:{\mathcal{S}}_p:p\in {\mathcal{P}}_{>}\right\}$, is affine and defines the exponential (statistical) manifold${\mathcal{P}}_{>}$.

The affine manifold we have defined has a simple and natural structure because of Proposition 3. The domains, $\mathcal{E}\left(p\right)$, $\mathcal{E}\left(q\right)$, of the charts, $s_p$, $s_q$, are either disjoint or equal when $p⌣q$:

$$\mathcal{E}\left(p\right){\left[r\right]}^{s_p}@=\left[d\right]{\mathcal{S}}_p{\left[d\right]}_{s_q\circ s_p^{-1}}{\left[r\right]}^I{B}_p{\left[d\right]}^{d(s_q\circ s_p^{-1})}{\left[r\right]}^I{L}^{\Phi }\left(p\right)@=\left[d\right]\mathcal{E}\left(q\right){\left[r\right]}_{s_q}{\mathcal{S}}_q{\left[r\right]}_I{B}_q{\left[r\right]}_I{L}^{\Phi }\left(q\right)$$

(16)

For ease of reference, various results from [5,7,9,10] are collected in the following proposition. We assume $q={\mathrm{e}}^{u-K_p\left(u\right)}·p\in \mathcal{E}\left(p\right)$. It should be noted that $K_p\left(u\right)={error}_p\left[ln\left(p/q\right)\right]$ is the expression in the chart centered at p of the Kullback-Leibler divergence, $D\left(p\parallel q\right)$.

Proposition 4.

• The first three derivatives of $K_p$ on ${\mathcal{S}}_p$ are:

  $$\begin{array}{cc}\hfill d{K}_p\left(u\right)v& ={error}_q\left[v\right]\hfill \end{array}$$

  (17)

  $$\begin{array}{cc}\hfill d^2{K}_p\left(u\right)(v_1,v_2)& ={Cov}_q\left(v_1,v_2\right)\hfill \end{array}$$

  (18)

  $$\begin{array}{cc}\hfill d^3{K}_p\left(u\right)(v_1,v_2,v_3)& ={Cov}_q(v_1,v_2,v_3)\hfill \end{array}$$

  (19)
• The random variable, $\frac{q}{p}-1$, belongs to $*B_p$ and:

  $$d{K}_p\left(u\right)v={error}_p\left[\left(\frac{q}{p}-1\right)v\right]$$

  (20)

  In other words, the gradient of $K_p$ at u is identified with an element of the predual space of $B_p$, viz. $*B_p=L_0^{{\Phi }_{*}}\left(p\right)$, denoted by $\nabla K_p\left(u\right)=e^{u-K_p\left(u\right)}-1=\frac{q}{p}-1$.
• The mapping, ${\mathcal{S}}_p\ni u\mapsto \nabla K_p\left(u\right)\in *B_p$, is monotonic:

  $${〈\nabla K_p\left(u\right)-\nabla K_p\left(v\right),u-v〉}_p>0,u\ne v$$

  (21)

  in particular, one-to-one.
• The weak derivative of the map, ${\mathcal{S}}_p\ni u\mapsto \nabla K_p\left(u\right)\in *B_p$, at u applied to $w\in B_p$ is given by:

  $$d\left(\nabla K_p\left(u\right)\right)w=\frac{q}{p}\left(w-{error}_q\left[w\right]\right)$$

  (22)

  and it is one-to-one at each point.
• The mapping, $\text{m}{\mathbb{U}}_p^q:v\mapsto \frac{p}{q}v$, is an isomorphism of $*B_p$ onto $*B_q$. It is called the mixture transport or m-transport.
• $q/p\in L^{{\Phi }_{*}}\left(p\right)$.
• $D\left(q\parallel p\right)=D{K}_p\left(u\right)u-K_p\left(u\right)$ with $q={\mathrm{e}}^{u-K_p\left(u\right)}p$, in particular $-D\left(q\parallel p\right)<+\infty$.
• $B_q$ is defined by an orthogonality property:

  $$B_q=L_0^{\Phi }\left(q\right)=\left\{u\in L^{\Phi }\left(p\right):{error}_p\left[u\frac{q}{p}\right]=0\right\}$$

  (23)
• The mapping, $\text{e}{\mathbb{U}}_p^q:u\mapsto u-{error}_q\left[u\right]$, is an isomorphism of $B_p$ onto $B_q$. It is called the exponential transport or e-transport.

3.1. Tangent Bundle

Our discussion of the tangent bundle of the exponential manifold is based on the concept of the velocity of a curve as in [16] (§3.3), and it is mainly intended to underline its statistical interpretation, which is obtained by identifying curves with one-parameter statistical models. For a statistical model $p\left(t\right)$, $t\in I$, the random variable, $\dot{p}\left(t\right)/p\left(t\right)$, the Fisher score, has zero expectation with respect to $p\left(t\right)$, and its meaning in the exponential manifold is velocity. If $p\left(t\right)={\mathrm{e}}^{tv-\psi \left(t\right)}·p$, $v\in L^{\Phi }\left(p\right)$, is an exponential family, then $\dot{p}\left(t\right)/p\left(t\right)=v-{error}_{p\left(t\right)}\left[v\right]\in B_{p\left(t\right)}$; see [22] on exponential families.

Let $p(·):I\to \mathcal{E}\left(p\right)$, I the open real interval containing zero. In the chart centered at p, the curve is $u(·):I\to B_p$, where $p\left(t\right)={\mathrm{e}}^{u\left(t\right)-K_p\left(u\left(t\right)\right)}·p$. The transition maps of the exponential manifold are:

$$s_q\circ e_p:{\mathcal{S}}_p\ni u\mapsto s_q({\mathrm{e}}^{u-K_p\left(u\right)}·p)=u-{error}_q\left[u\right]+ln\left(\frac{p}{q}\right)-{error}_q\left[ln\left(\frac{p}{q}\right)\right]\in {\mathcal{S}}_q={\mathcal{S}}_p$$

(24)

with derivative:

$$d_v{s}_q\circ s_p^{-1}\left(u\right)=v-{error}_q\left[v\right]=\text{e}{\mathbb{U}}_p^{q}v,v\in B_p$$

(25)

Definition 5 (Velocity field of a curve).

• Assume $t\mapsto u\left(t\right)=s_p\left(p\left(t\right)\right)$ is differentiable with derivative $\dot{u}\left(t\right)$. Define:

  $$\delta p\left(t\right)=\text{e}{\mathbb{U}}_p^{p\left(t\right)}\dot{u}\left(t\right)=\dot{u}\left(t\right)-{error}_{p\left(t\right)}\left[\dot{u}\left(t\right)\right]=\frac{d}{dt}(u\left(t\right)-K_p\left(u\left(t\right)\right)=\frac{d}{dt}ln\left(\frac{p\left(t\right)}{p}\right)=\frac{\frac{d}{dt}p\left(t\right)}{p\left(t\right)}$$

  (26)

  Note that $\delta p$ does not depend on the chart $s_p$ and that the derivative of $t\mapsto p\left(t\right)$ in the last term of the equation is computed in $L^{{\Phi }_{*}}\left(p\right)$. The curve $t\mapsto \left(p\right(t),\delta p(t\left)\right)$ is the velocity field of the curve.
• On the set $\left\{(p,v):p\in {\mathcal{P}}_{>},v\in B_p\right\}$, the charts:

  $$s_p:\left\{(q,w):q\in \mathcal{E}\left(p\right),w\in B_q\right\}\ni (q,w)\mapsto (s_p\left(q\right),\text{e}{\mathbb{U}}_q^{p}w)\in {\mathcal{S}}_p\times B_p\subset B_p\times B_p$$

  (27)

  define the tangent bundle, $T{\mathcal{P}}_{>}$. The isomorphism $w\mapsto \text{e}{\mathbb{U}}_p^{q}w=w-{error}_p\left[w\right]=d(s_q\circ s_p^{-1})\left(u\right)w$ of Proposition 4(9) is the (exponential) parallel transport.

Let $E:\mathcal{E}\left(p\right)\to \mathbb{R}$ be a $C^1$ function. Then, $E_p=E\circ e_p:{\mathcal{S}}_p\to \mathbb{R}$ is differentiable and:

$$\frac{d}{dt}E\left(p\left(t\right)\right)=\frac{d}{dt}{E}_p\left(u\left(t\right)\right)=d{E}_p\left(u\left(t\right)\right)\dot{u}\left(t\right)=d{E}_p\left(u\left(t\right)\right)\text{e}{\mathbb{U}}_{p\left(t\right)}^p\delta p\left(t\right)$$

(28)

Proposition 5 (Covariant derivative of a real function).

• As $v\mapsto d{E}_p\left(u\right)v$ is a linear operator on $B_p$, $w\mapsto d{E}_p\left(u\right)\text{e}{\mathbb{U}}_{e_p\left(u\right)}^{p}w$ is a linear operator on $B_{e_p\left(u\right)}$, which does not depend on p.
• If G is a vector field in $T{\mathcal{P}}_{>}$, the covariant derivative $D_{G}E$ is:

  $$D_{G}E\left(q\right)=d{E}_p\left(s_p\left(q\right)\right)\text{e}{\mathbb{U}}_{e_p\left(u\right)}^{p}w=d{E}_q\left(0\right)w,w=G\left(q\right)$$

  (29)
• Assume moreover that $d{E}_p\left(u\right)\in {B_p}^{*}$ can be identified with an element, $\nabla E_p\left(u\right)\in *B_p$, by:

  $$d{E}_p\left(u\right)w={error}_p\left[\nabla E_p\left(u\right)w\right],w\in B_p$$

  (30)

  Then, for $u=e_p\left(q\right)$:

  $$D_{G}E\left(q\right)=d{E}_p\left(u\right)\text{e}{\mathbb{U}}_q^{p}G\left(q\right)={error}_q\left[\text{m}{\mathbb{U}}_p^q\nabla E_p\left(u\right)G\left(q\right)\right]$$

  (31)

  We define the covariant gradient , ${\nabla }_{G}E\left(q\right)$, by $D_{G}E\left(q\right)={error}_q\left[{\nabla }_{G}E\left(q\right)G\left(q\right)\right]$.

Proof.

• Assume $u_1=s_{p_1}\left(q\right)=s_{p_1}\circ e_{p_2}\left(u_2\right)$, so that $E\left(q\right)=E_{p_1}\left(u_1\right)=E_{p_2}\left(u_2\right)=E_{p_1}(s_{p_1}\circ e_{p_2}\left(u_2\right))$ and:

  $$d{E}_{p_2}\left(u_2\right)\text{e}{\mathbb{U}}_q^{p_2}w=d{E}_{p_1}\circ s_{p_1}\circ e_{p_2}\left(u_2\right)\text{e}{\mathbb{U}}_q^{p_2}w=d{E}_{p_1}\left(u_1\right)\text{e}{\mathbb{U}}_{p_1}^{p_2}\text{e}{\mathbb{U}}_q^{p_2}w=d{E}_{p_1}\left(u_1\right)\text{e}{\mathbb{U}}_q^{p_1}w$$

  (32)
• Compute the derivative of $t\mapsto E\circ p\left(t\right)$ when $\delta p\left(t\right)=G\left(p\right(t\left)\right)$.
• It is a computation based on:

  $${error}_p\left[\nabla E_p\left(u\right)(G\left(p\right)-{error}_p\left[G\left(q\right)\right])\right]={error}_q\left[\frac{p}{q}\nabla E_p\left(u\right)(G\left(q\right)-{error}_p\left[G\left(q\right)\right])\right]={error}_q\left[\frac{p}{q}\nabla E_p\left(u\right)(G\left(q\right)\right]$$

  (33)

☐

Definition 6. Let $F,G:\mathcal{E}\left(p\right)$ be vector fields of $T{\mathcal{P}}_{>}$. In the chart at p, $F_p\left(u\right)=\text{e}{\mathbb{U}}_{e_p\left(u\right)}^{p}F\circ e_p\left(u\right)$, $u\in {\mathcal{S}}_p$ has differential $B_p:v\mapsto d{F}_p\left(u\right)v\in B_p$. The e-covariant derivative is the vector field defined by $D_{G}F\left(q\right)=\text{e}{\mathbb{U}}_p^{q}d{F}_p\left(s_p\left(q\right)\right)\text{e}{\mathbb{U}}_q^{p}w$, $w=G\left(q\right)$, and this definition does not depend on p.

3.2. Pretangent Bundle

Because of the lack of reflexivity of the exponential Orlicz space, we are forced to distinguish between the dual tangent bundle ${\left(T{\mathcal{P}}_{>}\right)}^{*}=\left\{(p,v):p\in {\mathcal{P}}_{>},v\in {\left(B_p\right)}^{*}\right\}$ and a pretangent bundle.

Definition 7. The set, $\left\{(q,v):q\in {\mathcal{P}}_{>},v\in *B_q\right\}$, together with the charts:

$$*s_p:\left\{(q,v):q\in \mathcal{E}\left(p\right),v\in *B_q\right\}\ni (q,v)\mapsto \left(s_p\left(q\right),\text{m}{\mathbb{U}}_q^{p}v\right)$$

(34)

are the pretangent bundle, $*T{\mathcal{P}}_{>}$.

The pretangent bundle is actually the tangent bundle of the mixture manifold (cfr. [1]) on ${\mathcal{P}}_1=\left\{f\in L^1\left(\mu \right):\int fd\mu =1\right\}$, whose charts are of the form ${\eta }_p\left(q\right)=q/p-1\in *B_p$. For each $p\in {\mathcal{P}}_{>}$, consider the set:

$$*{\mathcal{U}}_p=\left\{q\in {\mathcal{P}}_1:q/p\in L^{{\Phi }_{*}}\left(p\right)\right\}$$

(35)

and the mapping:

$${\eta }_p:*{\mathcal{U}}_p\ni q\mapsto {\eta }_p\left(q\right)=q/p-1\in *B_p.$$

(36)

Characterize $*{\mathcal{U}}_p$ as the set of q’s of finite Kullback-Leibler divergence from p.

Proposition 6 ([9], Proposition 30). Let $p\in {\mathcal{P}}_{>}$ and $q\in {\mathcal{P}}_1$. Define $\tilde{q}=\left|q\right|/\int \left|q\right|d\mu$. Then, $D\left(\tilde{q}\parallel p\right)<+\infty$, if and only if $q/p\in L^{{\Phi }_{*}}\left(p\right)$.

Proof. The second derivative of ${\Phi }_{*}\left(x\right)=(1+x)ln\left(1+x\right)-x$, $x>0$, is $1/(1+x)$, while the second derivative of $xln\left(x\right)$ is $1/x$. The function $xln\left(x\right)$ is more convex than ${\Phi }_{*}\left(x\right)$ as $0<1/(1+x)<1/x$. The two functions have parallel tangents at $x>0$ if $ln\left(1+x\right)=ln\left(x\right)+1$, that is, at $\overline{x}=1/(\mathrm{e}-1)$. At this point, the difference of the values is:

$${\Phi }_{*}\left(\overline{x}\right)-\overline{x}ln\left(\overline{x}\right)=1-ln\left(\mathrm{e}-1\right)$$

(37)

In conclusion, we have the inequalities:

$${\Phi }_{*}\left(x\right)\le xln\left(x\right)+1-ln\left(\mathrm{e}-1\right)<xlnx+1,x\ge 0$$

(38)

If $D\left(\tilde{q}\parallel p\right)<+\infty$, then:

$$+\infty >\int ln\left(\frac{\tilde{q}}{p}\right)\tilde{q}d\mu ={error}_p\left[\frac{\tilde{q}}{p}ln\left(\frac{\tilde{q}}{p}\right)\right]>{error}_p\left[{\Phi }_{*}\left(\frac{\tilde{q}}{p}\right)\right]-1={error}_p\left[{\Phi }_{*}\left({\left(\int \left|q\right|d\mu \right)}^{-1}\frac{q}{p}\right)\right]-1$$

(39)

so that $q/p\in L^{{\Phi }_{*}}\left(p\right)$.

Assume now $q/p\in L^{{\Phi }_{*}}\left(p\right)$, or, equivalently, $\tilde{q}/p\in L^{{\Phi }_{*}}\left(p\right)$. As $x{ln}^{+}\left(x\right)\le (1+x)ln\left(x\right)$ for $x\le 0$, we have:

$$+\infty >{error}_p\left[{\varphi }_{*}\left(\frac{\tilde{q}}{p}\right)\right]={error}_p\left[\left(1+\frac{\tilde{q}}{p}\right)ln\left(1+\frac{\tilde{q}}{p}\right)\right]-1\ge {error}_p\left[\frac{\tilde{q}}{p}{ln}^{+}\left(\frac{\tilde{q}}{p}\right)\right]-1$$

(40)

which, in turn, implies that:

$$D\left(\tilde{q}\parallel p\right)<{error}_p\left[\frac{\tilde{q}}{p}{ln}^{+}\left(\frac{\tilde{q}}{p}\right)\right]$$

(41)

is finite. ☐

The covariant gradient defined in Proposition 3(3) is a vector field of the pretangent bundle. Note that the injection, ${\mathcal{P}}_{>}\hookrightarrow {\mathcal{P}}_1$, is represented in the charts centered at p by $u\mapsto {\mathrm{e}}^{u-K_p\left(u\right)}·p-1$. We do not further discuss here the mixture manifold and refer to [10] (Section 5) for further information on this topic.

Let F be a vector field of the pretangent bundle, $*T{\mathcal{P}}_{>}$. The chart centered at p$\mathcal{E}\left(p\right)\ni q\mapsto F\left(q\right)$ is represented by:

$$F_p\left(u\right)=\text{m}{\mathbb{U}}_{e_p\left(u\right)}^{p}F\circ e_p\left(u\right)\in *B_p,u\in {\mathcal{S}}_p$$

(42)

If $F_p$ is of class $C^1$ with derivative $d{F}_p\left(u\right)\in L(B_p,*B_p)$, for each differentiable curve $t\mapsto p\left(t\right)={\mathrm{e}}^{u\left(t\right)-K_p\left(u\left(t\right)\right)}·p$:

$$\frac{d}{dt}{F}_p\left(p\left(t\right)\right)=d{F}_p\left(u\left(t\right)\right)\dot{u}\left(t\right)=d{F}_p\left(u\left(t\right)\right)\text{e}{\mathbb{U}}_{p\left(t\right)}^p\delta p\left(t\right)\in *B_p$$

(43)

For each $q=e_p\left(u\right)\in \mathcal{E}\left(p\right)$, $w\in *B_q$, $\text{m}{\mathbb{U}}_p^{q}d{F}_p\left(u\right)\text{e}{\mathbb{U}}_q^{p}w\in *B_q$ does not depend on p.

Definition 8 (Covariant derivative in $*T{\mathcal{P}}_{>}$.). Let F be a vector field of the pretangent bundle, $*T{\mathcal{P}}_{>}$, and G a vector field in the tangent bundle, ${\mathcal{P}}_{>}$, both of class $C^1$ on $\mathcal{E}\left(p\right)$. The covariant derivative is:

$$D_{G}F\left(q\right)=d\text{m}{\mathbb{U}}_{e_q\left(u\right)}^{q}F\circ e_q\left(0\right)w,w=G\left(q\right)$$

(44)

The tangent and pretangent bundle can be coupled to produce the new frame bundle:

$$(*T\times T){\mathcal{P}}_{>}=\left\{(p,v,w):p\in {\mathcal{P}}_{>},v\in B_p,w\in *B_p\right\}$$

(45)

with the duality coupling:

$$(*T\times T){\mathcal{P}}_{>}\ni (p,v,w)\mapsto {〈v,w〉}_p={error}_p\left[vw\right]={error}_q\left[\text{m}{\mathbb{U}}_p^{q}v\text{e}{\mathbb{U}}_p^{q}w\right],p⌣q$$

(46)

Proposition 7 (Covariant derivative of the duality coupling) . Let F be a vector field of $*T{\mathcal{P}}_{>}$, $G,H$ vector fields of $T{\mathcal{P}}_{>}$, all of class $C^1$ on a maximal exponential model $\mathcal{E}$. Then:

$$D_H〈F,G〉=〈D_{H}F,G〉+〈F,D_{H}G〉$$

(47)

Proof. Consider the real function $\mathcal{E}\ni q\mapsto 〈F,G〉\left(q\right)={error}_q\left[F\left(q\right)G\left(q\right)\right]$ in the chart centered at any $p\in \mathcal{E}$:

$${\mathcal{S}}_p\ni u\mapsto {error}_q\left[F\left(q\right)G\left(q\right)\right]={error}_p\left[\text{m}{\mathbb{U}}_q^{p}F\circ e_p\left(u\right)\text{e}{\mathbb{U}}_q^{p}G\circ e_p\left(u\right)\right]={error}_p\left[F_p\left(u\right)G_p\left(u\right)\right]$$

(48)

and compute its derivative. ☐

3.3. The Hilbert Bundle

The duality on $(*T\times T){\mathcal{P}}_{>}$ is reminiscent of a Riemannian metric, but it is not, because we do not have a Riemannian manifold unless the state space is finite. However, we can push on the analogy, by constructing a Hilbert bundle. As $L^{\Phi }\left(p\right)\subset L^2\left(p\right)\subset L^{{\Phi }_{*}}\left(p\right)$, $p\in {\mathcal{P}}_{>}$, we have $B_p\subset H_p\subset *B_p$, $L_0^2\left(p\right)=H_p$ being the fiber at p. The Hilbert bundle:

$$H{\mathcal{P}}_{>}=\left\{(p,v):p\in {\mathcal{P}}_{>},v\in H_p\right\}$$

(49)

is provided with an atlas of charts by using the isometries, ${\mathbb{U}}_p^q:H_p\to H_q$, which result from the pull-back of the metric connection on the sphere $S_{\mu }=\left\{f\in L^2\left(\mu \right):\int f^{2}d\mu =1\right\}$; see [6,8,23] and [14] (Section 4).

Proposition 8 (Isometric transport: ([14], Proposition 13)).

• For all $p,q\in {\mathcal{P}}_{>}$, the mapping:

  $${\mathbb{U}}_p^q:v\mapsto \sqrt{\frac{p}{q}}u-{\left(1+{error}_q\left[\sqrt{\frac{p}{q}}\right]\right)}^{-1}\left(1+\sqrt{\frac{p}{q}}\right){error}_q\left[\sqrt{\frac{p}{q}}v\right]$$

  (50)

  is an isometry of $H_p{\mathcal{P}}_{>}$ onto $H_q{\mathcal{P}}_{>}$.
• ${\mathbb{U}}_q^p\circ {\mathbb{U}}_p^{q}u=u$, $u\in H_p{\mathcal{P}}_{>}$ and ${\left({\mathbb{U}}_p^q\right)}^t={\mathbb{U}}_q^p$.

Note that ${\mathbb{U}}_q^r{\mathbb{U}}_p^q\ne {\mathbb{U}}_p^r$.

Definition 9 (Hilbert bundle). The charts:

$$2s_p:\left\{(q,v):q\in \mathcal{E}\left(p\right),v\in H_q\right\}\ni (q,v)\mapsto \left(s_p\left(q\right),{\mathbb{U}}_q^{p}v\right)\in {\mathcal{S}}_p\times H_p\subset B_p\times H_p$$

(51)

form an atlas on $H{\mathcal{P}}_{>}$.

Let $t\mapsto p\left(t\right)$ be a $C^1$ curve in $\mathcal{E}\left(p\right)$, $p=p\left(0\right)$, $u\left(t\right)=s_p\left(p\left(t\right)\right)$ and $F:\mathcal{E}\left(p\right)$ a $C^1$ vector field in $H{\mathcal{P}}_{>}$. In the chart centered at p, we have $F_p\left(u\left(t\right)\right)={\mathbb{U}}_{p\left(t\right)}^p(F\circ e_p)\left(u\left(t\right)\right)$. A computation shows that:

$$\begin{array}{cc}\hfill {\left.\frac{d}{dt}{F}_p\left(t\right)\right|}_{t=0}& ={\left.\frac{d}{dt}{\mathbb{U}}_{p\left(t\right)}^p(F\circ e_p)\left(u\left(t\right)\right)\right|}_{t=0}\hfill \end{array}$$

$$\begin{array}{cc}& =d{F}_p\left(0\right)\delta p\left(0\right)+\frac{1}{2}{F}_p\left(0\right)\delta p\left(0\right)-{error}_p\left[d{F}_p\left(0\right)\delta p\left(0\right)+\frac{1}{2}{F}_p\left(0\right)\delta p\left(0\right)\right]\hfill \end{array}$$

(52)

which could be used as a nonparametric definition of the metric connection; see [14] (Section 4.4) and [23] .

Our results on parallel transports and connections are a development, not yet complete, of previous work on statistical bundles in [6,8,14,23].

3.4. The Second Tangent Bundle

We briefly discuss here the second order structure, i.e., the tangent bundle of tangent bundle $T{\mathcal{P}}_{>}$. Let $F:I\ni t\mapsto \left(p\right(t),V(t\left)\right)$ be a $C^1$ curve in the tangent bundle, $T{\mathcal{P}}_{>}$. In the chart centered at p, we have:

$$F_p\left(t\right)=(s_p\left(p\left(t\right)\right),\text{e}{\mathbb{U}}_{p\left(t\right)}^{p}V\left(t\right))=(u\left(t\right),V_p\left(t\right))$$

(53)

where $p\left(t\right)={\mathrm{e}}^{u\left(t\right)-K_p\left(u\left(t\right)\right)}·p$ and $V\left(t\right)=V_p\left(t\right)-{error}_{p\left(t\right)}\left[V_p\left(t\right)\right]=V_p\left(t\right)-d{K}_p\left(u\left(t\right)\right)\left(V_p\left(t\right)\right)$. It follows that $t\mapsto V\left(t\right)$ is differentiable in $L^{\Phi }\left(p\right)$, with derivative:

$$\dot{V}\left(t\right)={\dot{V}}_p\left(t\right)-d{K}_p\left(u\left(t\right)\right)\left({\dot{V}}_p\left(t\right)\right)-d^2\left(p\left(t\right)\right)(V_p\left(t\right),\dot{u}\left(t\right))=\text{e}{\mathbb{U}}_p^{p\left(t\right)}{\dot{V}}_p\left(t\right)-{Cov}_{p\left(t\right)}\left(V_p\left(t\right),\dot{u}\left(t\right)\right)$$

(54)

hence:

$$\text{e}{\mathbb{U}}_p^{p\left(t\right)}{\dot{V}}_p\left(t\right)=\dot{V}\left(t\right)+{error}_{p\left(t\right)}\left[V\left(t\right)\right]\delta p\left(t\right)$$

(55)

It follows, in particular, that ${error}_{p\left(t\right)}\left[\dot{V}\left(t\right)\right]=-{error}_{p\left(t\right)}\left[V\left(t\right)\delta p\left(t\right)\right]$ and $\text{e}{\mathbb{U}}_p^{p\left(t\right)}{\dot{V}}_p\left(t\right)=\dot{V}\left(t\right)-{error}_{p\left(t\right)}\left[\dot{V}\left(t\right)\right]$. Note that the left-end side is not a transport, but an extension of the transport, precisely the projection, ${\Pi }^{p\left(t\right)}:L^{\Phi }\left(p\right)\to B_{p\left(t\right)}$. It follows from ${\dot{F}}_p\left(t\right)=\left(\dot{u}\left(t\right),{\dot{V}}_p\left(t\right)\right)$ that the velocity vector is:

$$\delta (p,V)\left(t\right)=\left(\delta p\left(t\right),\text{e}{\mathbb{U}}_p^{p\left(t\right)}{\dot{V}}_p\left(t\right)\right)=\left(\delta p\left(t\right),{\Pi }^{p\left(t\right)}\dot{V}\left(t\right)\right)$$

(56)

The Equation (55) in the case $V\left(t\right)=\left(\delta p\right)\left(t\right)$ gives:

$${\Pi }^{p\left(t\right)}\dot{\left(\delta p\right)}\left(t\right)=\dot{\left(\delta p\right)}\left(t\right)+{error}_{p\left(t\right)}\left[\delta p{\left(t\right)}^2\right]=\dot{\left(\delta p\right)}\left(t\right)+I\left(p\left(t\right)\right)$$

(57)

where we have denoted by $I\left(p\left(t\right)\right)={error}_{p\left(t\right)}\left[\frac{d}{dt}ln{\left(p\left(t\right)\right)}^2\right]$ the Fisher information. In this case, we can write:

$$\delta (p,\delta p)\left(t\right)=(\delta p\left(t\right),\dot{\left(\delta p\right)}\left(t\right)+I\left(p\left(t\right)\right))$$

(58)

4. Applications

In this section, we consider a typical set of examples where the nonparametric framework is applicable.

4.1. Expected Value

Let $f\in L^{\Phi }\left(p\right)$, $f_0=f-{error}_p\left[f\right]\in B_p$, and consider its expected value as a function of the the density, $q\in \mathcal{E}\left(p\right)$:

$$E:\mathcal{E}\left(p\right)\ni q\mapsto {error}_q\left[f\right]={error}_q\left[f_0\right]+{error}_p\left[f\right]$$

(59)

which is sometimes called the relaxed version of f in optimization theory, where it is convenient to regularize the problem of finding ${max}_{x}f\left(x\right)$ by extending it to the problem of finding ${max}_{q}E\left(q\right)$. As ${error}_q\left[f\right]<maxf$, $q\in {\mathcal{P}}_{>}$, unless f is μ-a.s.constant; relaxed optimization can produce a maximizing sequence only.

The information geometric study of the relaxed mapping can be based on the notion of natural gradient as defined in a seminal paper by Amari [24], and it is currently used for optimization, see, e.g., [25,26,27,28,29,30,31]. The covariant derivative of a real function is the nonparametric counterpart of Amari’s natural gradient.

From the properties of $K_p$ in Equation (17) and (18) of Proposition 4, we obtain the representation $E_p\left(u\right)=E\circ s_p^{-1}\left(u\right)$ of the function in Equation (59) in the chart centered at p, $E_p\left(u\right)=d{K}_p\left(u\right)\left(f_0\right)+{error}_p\left[f\right]$, whose differential in the direction v is $d{E}_p\left(u\right)v=d^2{K}_p\left(u\right)(f_0,v)={Cov}_q\left(f,v\right)$. The covariant derivative at $(q.w)\in T{B}_q$ is computed from Definition 5(2) as:

$$d{E}_p\left(u\right)\text{e}{\mathbb{U}}_q^{p}w={Cov}_q\left(f,\text{e}{\mathbb{U}}_q^{p}w\right)={error}_q\left[(f-{error}_q\left[f\right])(w-{error}_p\left[w\right])\right]={error}_q\left[(f-{error}_q\left[f\right])w\right]$$

(60)

hence, $D_{G}E\left(p\right)={error}_p\left[(f-{error}_p\left[f\right])G\left(p\right)\right]$ with gradient ${\nabla }_{G}E\left(q\right)=f-{error}_q\left[f\right]$ in the duality on $*B_q\times B_q$. Note that the gradient is never zero unless f is constant and that the covariant derivative is zero for each vector field G, which is uncorrelated with f.

We now compute the covariant derivative of the of the gradient in order to obtain the Hessian of the function, E. Consider the gradient vector field $F\left(q\right)=f-{error}_q\left[f\right]\in *T{\mathcal{P}}_{>}$. The gradient flow is:

$$\delta p\left(t\right)=\frac{d}{dt}ln\left(p\left(t\right)\right)=f-{error}_{p\left(t\right)}\left[f\right]$$

(61)

whose unique solution is the exponential family, $p\left(t\right)\propto {\mathrm{e}}^{tf}·p\left(0\right)$. In fact, the gradient is actually the e-transport of $f_0$, $F\left(p\right)=\text{e}{\mathbb{U}}_p^{e_p\left(u\right)}{f}_0$, and the exponential family is the exponential curve of the e-transport.

Let us discuss the differentiability of the gradient. In the chart centered at p, the gradient is represented as:

$$F_p\left(u\right)=\text{m}{\mathbb{U}}_{e_p\left(u\right)}^p[f_0-d{K}_p\left(u\right)\left(f_0\right)]=\text{m}{\mathbb{U}}_{e_p\left(u\right)}^p\text{e}{\mathbb{U}}_p^{e_p\left(u\right)}{f}_0$$

(62)

Let us first compute the differential of $u\mapsto {〈F_p\left(u\right),w〉}_p$, $w\in B_p$, in the direction $v\in B_p$, i.e., the weak differential:

$$\begin{array}{c}{d}_v{〈F_p\left(u\right),w〉}_p=d_v{〈\text{m}{\mathbb{U}}_{e_p\left(u\right)}^p\text{e}{\mathbb{U}}_p^{e_p\left(u\right)}{f}_0,w〉}_p={Cov}_{e_p\left(u\right)}\left(f_0,w\right)=\hfill \\ \hfill d_v{d}^2{K}_p\left(u\right)(f_o,w)=d^3{K}_p\left(u\right)(f_0,w,v)={Cov}_{e_p\left(u\right)}(f_0,w,v)\end{array}$$

(63)

where we have used Proposition 4. At $u=0$:

$$d_v{〈F_p\left(0\right),w〉}_p={error}_p\left[f_{0}wv\right]={error}_p\left[\left(f_{0}v-{error}_p\left[f_{0}v\right]\right)w\right]={〈f_{0}v-{error}_p\left[f_{0}v\right],w〉}_p$$

(64)

The product $f_{0}G\left(p\right)$ belongs to $*B_p$. In fact:

$${error}_p\left[{\Phi }_{*}\left(f_{0}G\left(p\right)\right)\right]={error}_p\left[f_0^2{\int }_0^{\left|G\left(p\right)\right|}\frac{\left|G\left(p\right)\right|-u}{1+\left|f_0\right|u}du\right]\le \frac{1}{2}{error}_p\left[f_0^{2}G{\left(p\right)}^2\right]<+\infty$$

(65)

If $D_G\nabla E$ exists in $*{\mathcal{P}}_{>}$ as a Fréchet derivative, then:

$$D_G\nabla E\left(p\right)=f_{0}G\left(p\right)-{error}_p\left[f_{0}G\left(p\right)\right]$$

(66)

The differentiability in Orlicz spaces of superposition operators is discussed in detail in [32].

4.2. Kullback-Leibler Divergence

If $\mathcal{E}$ is a maximal exponential model, the mapping:

$$\mathcal{E}\times \mathcal{E}\ni (q_1,q_2)\mapsto D\left(q_1\parallel q_2\right)={error}_{q_1}\left[ln\left(\frac{q_1}{q_2}\right)\right]$$

(67)

is represented in the charts centered at p by:

$$E_p:{\mathcal{S}}_p\times {\mathcal{S}}_p\ni (u_1,u_2)\mapsto d{K}_p\left(u_1\right)(u_1-u_2)-(K_p\left(u_1\right)-K_p\left(u_2\right))$$

(68)

Hence, from Proposition 2(4), it is $C^{\infty }$ jointly in both variables and, moreover, analytic:

$$E_p(u_1,u_2)=\sum _{n\ge 2}\frac{1}{n!}{d}^n{K}_p\left(u_1\right){(u_1-u_2)}^{\circ n},{∥u_1-u_2∥}_{\Phi ,p}<1$$

(69)

This regularity result is to be compared with what is available when the restriction, $q_1⌣q_2$, is removed, i.e., the semi-continuity [33].

The (partial) derivative of $u_2\mapsto E_p(u_1,u_2)$ in the direction $v_2\in B_p$ is:

$$d_2{E}_p(u_1,u_2)v_2=-d{K}_p\left(u_1\right)v_2+d{K}_p\left(u_2\right)v_2={error}_{q_2}\left[v_2\right]-{error}_{q_1}\left[v_2\right]$$

(70)

If $v_2=\text{e}{\mathbb{U}}_q^{p}w$, we have ${error}_{q_2}\left[v_2\right]-{error}_{q_1}\left[v_2\right]={error}_{q_2}\left[w\right]-{error}_{q_1}\left[w\right]$, and the covariant derivative of the partial functional $q\mapsto D\left(q_1\parallel q\right)$ is:

$$D_{2,w}D\left(q_1\parallel q\right)={error}_q\left[w\right]-{error}_{q_1}\left[w\right]={error}_q\left[\left(1-\frac{q_1}{q}\right)w\right],{\nabla }_{q}D\left(q_1\parallel q\right)=1-\frac{q_1}{q}$$

(71)

The second mixed derivative of $E_p$ is:

$$d_1{d}_2{E}_p(u_1,u_2)(v_1,v_2)=-d^2{K}_p\left(u_1\right)(v_1,v_2)=-{Cov}_{q_1}\left(v_1,v_2\right)$$

(72)

Equivalently, we consider the mapping, $q_1\mapsto D_{2,w}D\left(q_1\parallel q\right)$, in the chart $u_1\mapsto {error}_q\left[w\right]-{error}_{q_1}\left[w\right]$, to obtain:

$${\left.D_{1,w_1}{D}_{2,w_2}D\left(q_1\parallel q_2\right)\right|}_{q_1=q_2=q}=-{error}_q\left[w_1,w_2\right]$$

(73)

4.3. Boltzmann-Gibbs Entropy

While our discussion of the Kullback-Leibler divergence in the previous Section 4.2 does not require any special assumption, but the restriction of its domain to a maximal exponential model, in the present discussion of the Boltzmann-Gibbs entropy, a further restriction is required. If $p,q$ belong to the same maximal exponential model, $p⌣q$, then, from $q={\mathrm{e}}^{u-K_p\left(u\right)}·p$ with $u\in B_p$, we obtain $lnq-lnp\in L^{\Phi }\left(p\right)$, so that $lnq\in L^{\Phi }\left(p\right)$, if and only if $lnp\in L^{\Phi }\left(p\right)$.

We study the Boltzmann-Gibbs entropy $E\left(q\right)={error}_q\left[ln\left(q\right)\right]$ on a maximal exponential model $q\in \mathcal{E}$, such that for at least one, and, hence, for all, $p\in \mathcal{E}$, it holds $ln\left(p\right)\in L^{\Phi }\left(p\right)$, i.e., $\int \left(p^{1+\alpha }+p^{1-\alpha }\right)d\mu <+\infty$ for some $\alpha >0$. This is, for example, the case when the reference measure is finite and p is constant. Another notable example is the Gaussian case, i.e., the sample space is ${\mathbb{R}}^n$ endowed with the Lebesgue measure and $p\left(x\right)\propto exp-1/{2\left|x\right|}^2$. In fact ${\int cosh\left(\alpha \right|x|}^2)exp\left(-1/{2\left|x\right|}^2\right)dx<+\infty$ for $0<\alpha <1/2$.

Under our assumption, the Boltzmann-Gibbs entropy is a smooth function. As:

$$ln\left(q\right)=u-K_p\left(u\right)+ln\left(p\right)=u-K_p\left(u\right)+(ln\left(p\right)-E\left(p\right))+E\left(p\right)\in L^{\Phi }\left(p\right)$$

(74)

the representation in the chart centered at p is:

$$E_p\left(u\right)={error}_{e_p\left(u\right)}\left[u-K_p\left(u\right)+ln\left(p\right)\right]=d{K}_p\left(u\right)\left[u+(ln\left(p\right)-E\left(p\right))\right]-K_p\left(u\right)+E\left(p\right)$$

(75)

Hence, it is a $C^{\infty }$ real function. The derivative in the direction v equals:

$$d{E}_p\left(u\right)v=d^2{K}_p\left(u\right)\left(u+(ln\left(p\right)-E\left(p\right)),v\right)={Cov}_q\left(u+ln\left(p\right),v\right)$$

(76)

in particular:

$$d{E}_p\left(0\right)v={error}_p\left[(ln\left(p\right)-E\left(p\right))v\right]={〈ln\left(p\right)-E\left(p\right),v〉}_p$$

(77)

The value of the covariant derivative $D_{G}E$ at q and $G\left(q\right)=w$ is:

$$d{E}_p\left(u\right)\text{e}{\mathbb{U}}_q^{p}w={Cov}_q\left(u+ln\left(p\right),w\right)={error}_q\left[((ln\left(q\right)+K_p\left(u\right))w\right]={error}_q\left[(ln\left(q\right)-E\left(q\right))w\right]$$

(78)

The gradient $\nabla E\left(q\right)\in {\left(B_q\right)}^{*}$, $D_{G}E\left(q\right)={〈\nabla E\left(q\right),G\left(q\right)〉}_q$ is identified with a random variable in $B_q\subset *B_q$, and:

$$\begin{array}{cc}\hfill F\left(q\right)& =ln\left(q\right)-E\left(q\right)\hfill \end{array}$$

$$\begin{array}{cc}& =u-K_p\left(u\right)+ln\left(p\right)-{error}_q\left[u-K_p\left(u\right)+ln\left(p\right)\right]\hfill \end{array}$$

$$\begin{array}{cc}& =(u+ln\left(p\right)-E\left(p\right))-d{K}_p\left(u\right)(u+ln\left(p\right)-E\left(p\right))\hfill \end{array}$$

$$\begin{array}{cc}& =\text{e}{\mathbb{U}}_p^q(u+ln\left(p\right)-E\left(p\right))\in B_q\hfill \end{array}$$

(79)

is a vector field in the tangent bundle, $T\mathcal{E}$, hence a vector field in the Hilbert bundle, $H\mathcal{E}$, and in the pretangent bundle, $*T\mathcal{E}$.

The equation, $\nabla E\left(q\right)=0$, implies $q=E\left(q\right)$, hence constant. The Boltzmann-Gibbs entropy is increasing along the vector field $G\in T\mathcal{E}$ if ${error}_q\left[(ln\left(q\right)-E\left(q\right))G\left(q\right)\right]={Cov}_q\left(ln\left(q\right),G\left(q\right)\right)>0$. The exponential family tangent at p to $\nabla E\left(p\right)$ is $p\left(t\right)\propto {\mathrm{e}}^{tln\left(p\right)}·p=p^{1+t}$. The gradient flow equation is $\delta q\left(t\right)=\nabla E\left(q\right(t\left)\right)$, that is:

$$\frac{d}{dt}ln\left(q\left(t\right)\right)=lnq\left(t\right)-E\left(q\left(t\right)\right)$$

(80)

In the pretangent bundle, the action of the dual exponential transport, ${\left(\text{e}{\mathbb{U}}_q^p\right)}^{*}$, is identified with $\text{m}{\mathbb{U}}_q^p$. It follows that the representation of the gradient in the chart centered at p is:

$$\begin{array}{cc}\hfill F_p\left(u\right)& ={\mathrm{e}}^{u-K_p\left(u\right)}\left[(u+ln\left(p\right)-E\left(p\right))-d{K}_p\left(u\right)(u+ln\left(p\right)-E\left(p\right))\right]\hfill \end{array}$$

$$\begin{array}{cc}& =\text{m}{\mathbb{U}}_{e_p\left(u\right)}^p\text{e}{\mathbb{U}}_p^{e_p\left(u\right)}[(u+ln\left(p\right)-E\left(p\right))\hfill \end{array}$$

(81)

Let us assume $u\mapsto F_p\left(u\right)$ is (strongly) differentiable, and let us compute the derivative by the product rule. As $u\mapsto F_p\left(u\right)$ can be seen locally as the product of an analytic mapping, $u\mapsto {\mathrm{e}}^{u-K_p\left(u\right)}$, with values in $L^a\left(p\right)$, $a>1$, because of Proposition 1, while the second factor is an analytic function with values in $L^{\Phi }\left(p\right)\subset {\cap }_{a>1}{L}^a\left(p\right)$, we can compute its differential in the direction, $v\in B_p$, as the product of two functions in the Fréchet space ${\cap }_{a>1}{L}^a\left(p\right)$ as:

$$\begin{array}{c}d{\left(\nabla E\right)}_p\left(u\right)v={\mathrm{e}}^{u-K_p\left(u\right)}\times \hfill \\ \left[(v-d{K}_p\left(u\right)v)\left[(u+ln\left(p\right)-E\left(p\right))-d{K}_p\left(u\right)(u+ln\left(p\right)-E\left(p\right))\right]\right.\\ \left.+v-d^2{K}_p\left(u\right)(u+ln\left(p\right)-E\left(p\right),v)-d{K}_p\left(u\right)v\right]=\\ \hfill \frac{q}{p}\left[(v-{error}_q\left[v\right])(ln\left(q\right)-E\left(q\right))+v-{error}_q\left[v\right]-{Cov}_q\left(ln\left(q\right),v\right)\right]\end{array}$$

(82)

in particular, for $u=0$:

$$\begin{array}{cc}\hfill d{\left(\nabla E\right)}_p\left(0\right)v& =(ln\left(p\right)-E\left(p\right)+1)v-{error}_p\left[ln\left(p\right)v\right]\hfill \end{array}$$

$$\begin{array}{cc}& =(\nabla E\left(p\right)+1)v-{error}_p\left[\nabla E\left(p\right)v\right]\hfill \end{array}$$

(83)

The covariant derivative of the gradient, $\nabla E$, of the Boltzmann-Gibbs entropy in the pretangent bundle, $*T\mathcal{E}$, is:

$$\begin{array}{cc}\hfill D_G\left(\nabla E\right)\left(p\right)& =(ln\left(p\right)-E\left(p\right)+1)G\left(p\right)+{error}_p\left[ln\left(p\right)G\left(p\right)\right]\hfill \end{array}$$

$$\begin{array}{cc}& =(\nabla E\left(p\right)+1)G\left(p\right)+{error}_p\left[\nabla E\left(p\right)G\left(p\right)\right],p\in \mathcal{E}\hfill \end{array}$$

(84)

The existence of the covariant derivative implies $ln\left(p\right)G\left(p\right)\in L^{{\Phi }_{*}}\left(p\right)$, $p\in \mathcal{E}$. We do not discuss here the existence problem.

The computation of the covariant derivative of the same gradient in the tangent bundle, $T\mathcal{E}$, would be:

$$\begin{array}{cc}\hfill {\overline{F}}_p\left(u\right)& =\text{e}{\mathbb{U}}_q^p(ln\left(q\right)-E\left(q\right))=ln\left(q\right)-{error}_p\left[ln\left(q\right)\right]=u+ln\left(p\right)-E\left(p\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill d{\overline{F}}_p\left(u\right)v& =v\hfill \end{array}$$

(85)

but we cannot suggest any use for this computation.

4.4. Boltzmann Equation

Orlicz spaces as a setting for the Boltzmann equation have been recently discussed in [34], while the use of exponential manifolds has been suggested in [14] (Example 11). Here we further work out this framework for a space-homogeneous Boltzmann operator with angular collision kernel $B(z,x)=\left|x^{\prime }z\right|$; see the presentation in [20]. In order to avoid a clash with the notations used in other parts of this paper, we use v and w to denote velocities in ${\mathbb{R}}^3$ in place of the more common couple, v and $v_{*}$, and the velocities after collision are denoted by $v_x$ and $w_x$ instead of $v^{\prime }$, $v_{*}^{\prime }$, $x\in S^2$ being a unit vector.

Let $v,w\in {\mathbb{R}}^3$ be the velocities of two particles, and $\overline{v},\overline{w}$ be the velocities after an elastic collision, i.e.,

$$v+w=\overline{v}+\overline{w},{\left|v\right|}^2+{\left|w\right|}^2={\left|\overline{v}\right|}^2+{\left|\overline{w}\right|}^2$$

(86)

Using Equation (86), we derive from the development of ${\left|v+w\right|}^2={\left|\overline{v}+\overline{w}\right|}^2$ that $v·w=\overline{v}·\overline{w}$. The four vectors, $v,w,\overline{v},\overline{w}$, all lie on a circle with center $z=(v+w)/2=(\overline{v}+\overline{w})/2$. In fact, the four vectors and z lie on the same plane, because $v-z=-(w-z)$, $\overline{v}-z=-(\overline{w}-z)$, and moreover, ${\left|v-z\right|}^2={\left|\overline{v}-z\right|}^2$. As $v,w,\overline{v},\overline{w}$ form a rectangle, we can denote by x the common unit vector of the parallel sides, $\overline{w}-w$ and $v-\overline{v}$, and write $\overline{w}-w=v-\overline{v}$ as the orthogonal projection of $v-w$ on x. Given the unit vector $x\in S_2=\left\{x\in {\mathbb{R}}^3:x^{\prime }x=1\right\}$, the collision transformation $(v,w)\mapsto (\overline{v},\overline{w})=(v_x,w_x)$ is linear and represented by a ${\mathbb{R}}^{(3+3)\times (3+3)}$ matrix:

$$A_x=\left[\begin{array}{cc}(I-{\Pi }_x)& {\Pi }_x\\ {\Pi }_x& (I-{\Pi }_x)\end{array}\right],\left\{\begin{array}{cc}\hfill v_x& =v-x{x}^{\prime }(v-w)=(I-x{x}^{\prime })v+x{x}^{\prime }w\hfill \\ \hfill w_x& =w+x{x}^{\prime }(v-w)=x{x}^{\prime }v+(I-x{x}^{\prime })w\hfill \end{array}\right.$$

(87)

where ${}^{\prime }$ denotes the transposed vector.

Given any $x\in S_2$, we have $A_x=A_{-x}$. If $v,w,v_x,w_x$ are as in Equation (87), then the elastic collision invariants of Equation (86) hold, $v+w=v_x+w_x$, ${\left|v\right|}^2+{\left|w\right|}^2={\left|v_x\right|}^2+{\left|w_x\right|}^2$. The components in the direction x are exchanged, $x{x}^{\prime }{v}_x=x{x}^{\prime }w$ and $x{x}^{\prime }{w}_x=x{x}^{\prime }v$, while the orthogonal components are conserved.

Let σ be the uniform probability on $S^2$. For each positive function, $g:{\mathbb{R}}^3\times {\mathbb{R}}^3$, the integral, ${\int }_{S_2}g(v_x,w_x)\sigma \left(dx\right)$, depends on the collision invariants only. In fact:

$$\begin{array}{cc}\hfill v_x& =\frac{v+w}{2}+\frac{\left|v-w\right|}{2}y\hfill \end{array}$$

(88)

$$\begin{array}{cc}\hfill w_x& =\frac{v+w}{2}-\frac{\left|v-w\right|}{2}y\hfill \end{array}$$

(89)

where the unit vector $y=\widehat{v_x-w_x}=(I-2x{x}^{\prime })\widehat{v-w}$ and all other terms depend on the collision invariants, in particular, ${\left|v-w\right|}^2=2({\left|v\right|}^2+{\left|w\right|}^2)-{\left|v+w\right|}^2$.

On the sample space $({\mathbb{R}}^3,dv)$, let $f_0$ be the standard normal density viz. the Maxwell distribution of velocities. As $A_x{A}_x=I_6$ the identity matrix on ${\mathbb{R}}^6$, in particular $\left|det{A}_x\right|=1$, we have:

$$A_x(V,W)=(V_x,W_x)\sim (V,W)$$

(90)

if $(V,W)\sim \text{N}(0_6,I_6)$. We can give the previous remarks a more probabilistic form as follows.

Proposition 9. Let $f_0$ be the density of the standard normal N$(0_3,I_3)$.

• If $(V,W)\sim f_0\otimes f_0$, then ${\int }_{S^2}g(V_x,W_x)\sigma \left(dx\right)$ is the conditional expectation of $g(V,W)$, given $V+W$ and ${\left|V\right|}^2+{\left|W\right|}^2$.
• Assume $(V,W)\sim f$, $f\in \mathcal{E}\left(f_0\otimes f_0\right)$; then ${\int }_{S^2}f\circ A_x\sigma \left(dx\right)\in \mathcal{E}\left(f_0\otimes f_0\right)$ and:

  $$error\left(g(V,W)\left|V+W,{\left|V\right|}^2+{\left|W\right|}^2\right.\right)=\frac{{\int }_{S^2}g(V_x,W_x)f(V_x,W_x)\sigma \left(dx\right)}{{\int }_{S^2}f(V_x,W_x)\sigma \left(dx\right)}$$

  (91)

Proof. 1. The random variable, ${\int }_{S^2}g(V_x,W_x)\sigma \left(dx\right)={\int }_{S^2}g\circ A_x(V,W)\sigma \left(dx\right)$, is a function $\tilde{g}(m_1(V,W),m_2(V,W))$ with $m_1(V,W)=V+W$ and $m_2(V,W)={\left|V\right|}^2+{\left|W\right|}^2$. For all $h_1:{\mathbb{R}}^3$, $h_2:{\mathbb{R}}^3$:

$$\begin{array}{c}error\left(\left({\int }_{S^2}g\circ A_x(V,W)\sigma \left(dx\right)\right)h_1\left(m(V,W)h_2\left(m_2(V,W)\right)\right)\right)=\hfill \\ \hfill error\left(g(V,W)h_1\left(m(V,W)h_2\left(m_2(V,W)\right)\right)\right)\end{array}$$

(92)

because of $A_x(V,W)\sim (V,W)$ and $m_1\circ A_x=m_1$, $m_2\circ A_x=m_2$.

2. We use Proposition 3. If $f\in \mathcal{E}\left(f_0\otimes f_0\right)$, then:

$$f={\mathrm{e}}^{u-K_0\left(u\right)}·f_0\otimes f_0,u\in {\mathcal{S}}_{f_0\otimes f_0}$$

(93)

and there exists a neighborhood, I, of $[0,1]$, where the one dimensional exponential family:

$$f_t={\mathrm{e}}^{tu-K_0\left(tu\right)}·f_0\otimes f_0,t\in I$$

(94)

exists. To show ${error}_{f_0\otimes f_0}\left[{(f/f_0\otimes f_0)}^t\right]<+\infty$ for $t\in I$, it is enough to consider the convex cases, $t<0$ and $t>1$. We have:

$${\int }_{S_2}f\circ A_x\sigma \left(dx\right)={\int }_{S_2}{\mathrm{e}}^{fu\circ A_x-K_0\left(u\right)}\sigma \left(dx\right)·f_0\otimes f_0$$

(95)

and in the convex cases:

$$\begin{array}{c}{error}_{f_0\otimes f_0}\left[{\left(\frac{{\int }_{S_2}f\circ A_x\sigma \left(dx\right)}{f_0\otimes f_0}\right)}^t\right]={error}_{f_0\otimes f_0}\left[{\left({\int }_{S_2}{\mathrm{e}}^{u\circ A_x-K_0\left(u\right)}\sigma \left(dx\right)\right)}^t\right]\le \hfill \\ \hfill {error}_{f_0\otimes f_0}\left[{\int }_{S_2}{\mathrm{e}}^{tu\circ A_x-t{K}_0\left(u\right)}\sigma \left(dx\right)\right]={error}_{f_0\otimes f_0}\left[{\mathrm{e}}^{tu-t{K}_0\left(u\right)}\right]={\mathrm{e}}^{K_0\left(tu\right)-t{K}_0\left(u\right)}\end{array}$$

(96)

The last equation is Bayes’ formula for conditional expectation. ☐

Definition 10. For each element of the maximal exponential model containing $f_0$, $f\in \mathcal{E}\left(f_0\right)$, the Boltzmann operator is:

$$\begin{array}{c}Q\left(f\right)\left(v\right)=\hfill \\ \hfill {\int }_{{\mathbb{R}}^3}{\int }_{S^2}(f(v-x{x}^{\prime }(v-w))f(w+x{x}^{\prime }(v-w))-f\left(v\right)f\left(w\right))\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)dw\end{array}$$

(97)

In our definition, we have restricted the domain of the Boltzmann operator to a maximal exponential model containing the standard normal density in order to fit into our framework and be able to prove the smoothness of the operator. The maximal exponential model $\mathcal{E}\left(f_0\right)$ contains all normal densities, $f\sim \text{N}(\mu ,\Sigma )$. It has other peculiar properties.

As $f\in \mathcal{E}\left(f_0\right)$, $f={\mathrm{e}}^{u-K_0\left(u\right)}·f_0$; u belongs to the interior of the proper domain of $K_0$, $u\in {\mathcal{S}}_{f_0}\subset B_{f_0}$. It follows from Proposition 3 that we have the equality and isomorphism of the Banach spaces, $L^{\Phi }\left(f\right)$ and $L^{\Phi }\left(f_0\right)$. For the random variable, $V_a:v\mapsto {\left|v\right|}^a$, it holds $V_{\alpha }\in L^{\Phi }\left(f_0\right)=L^{\Phi }\left(f\right)$ for all $a\in [1,2]$. In fact:

$${error}_{f_0}\left[cosh\left(\alpha V_a\right)\right]={\left(2\pi \right)}^{-3/2}{\int }_{{\mathbb{R}}^3}cosh\left(\alpha {\left|v\right|}^a\right)exp\left(-{\left|v\right|}^2/2\right)dv$$

(98)

is finite for all α if $a\in [0,2[$ and for $\alpha <1/2$ if $a=2$. In particular, it follows that $V_1\left(v\right)=\left|v\right|$ has finite moments with respect to f, $\int {\left|v\right|}^{n}f\left(v\right)dv<+\infty$, $n=1,2,\cdots$.

As $x^{\prime }(v-w)=-x^{\prime }(v_x-w_x)$, the measure, $\left|x^{\prime }(v-w)\right|dvdw$, is invariant under the transformation $A_x$ and the measure $f\left(v_x\right)f\left(w_x\right)\left|x^{\prime }(v-w)\right|dvdw$ is the image of $f\left(v\right)f\left(w\right)\left|x^{\prime }(v-w)\right|dvdw$ under $A_x$. Other properties are obtained in the proof of the following proposition.

Proposition 10. Let $f_0\left(v\right)={\left(2\pi \right)}^{-3/2}exp\left(-{\left|v\right|}^2/2\right)$ and $f\in \mathcal{E}\left(f_0\right)$; then, $Q\left(f\right)/f\in *B_f$. Then, $f\mapsto Q\left(f\right)/f$ is a vector field in the pretangent bundle, $*T\mathcal{E}\left(f_0\right)$, called the Boltzmann field.

Proof. Let us consider first the second part of the Boltzmann operator:

$$Q_{-}\left(f\right)\left(v\right)={\int }_{{\mathbb{R}}^3}{\int }_{S^2}f\left(v\right)f\left(w\right)\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)dw=f\left(v\right){\int }_{{\mathbb{R}}^3}f\left(w\right)\left({\int }_{S^2}\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)\right)dw$$

(99)

Note that from inequality Equation (6):

$${\Phi }_{*}\left({\int }_{S^2}\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)\right)={\Phi }_{*}\left(\left|v-w\right|{\int }_{S^2}\left|x_1\right|\sigma \left(dx\right)\right)\le {\left({\int }_{S^2}\left|x_1\right|\sigma \left(dx\right)\right)}^2{\Phi }_{*}\left(\left|v-w\right|\right)$$

(100)

We prove $Q_{-}\left(f\right)/f\in L^{{\Phi }_{*}}\left(f\right)$:

$$\begin{array}{cc}\hfill {error}_f\left[{\Phi }_{*}\left(\frac{Q_{-}\left(f\right)}{f}\right)\right]& ={\int }_{{\mathbb{R}}^3}dvf\left(v\right){\Phi }_{*}\left({\int }_{{\mathbb{R}}^3}f\left(w\right)\left({\int }_{S^2}\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)\right)dw\right)\hfill \end{array}$$

$$\begin{array}{cc}& \le {\int }_{{\mathbb{R}}^3}dvf\left(v\right){\int }_{{\mathbb{R}}^3}dwf\left(w\right){\Phi }_{*}\left({\int }_{S^2}\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)\right)\hfill \end{array}$$

$$\begin{array}{cc}& ={\int }_{{\mathbb{R}}^3}dvf\left(v\right){\int }_{{\mathbb{R}}^3}dwf\left(w\right){\Phi }_{*}\left(b\left|v-w\right|\right)\hfill \end{array}$$

$$\begin{array}{cc}& \le \frac{b^2}{2}{\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}dvdwf\left(v\right)f\left(w\right){\left|v-w\right|}^2\hfill \end{array}$$

(101)

which is finite as ${\left|v-w\right|}^2\le 2({\left|u\right|}^2+{\left|v\right|}^2)$.

We consider now the first part of the Boltzmann operator:

$$\begin{array}{cc}\hfill Q_{+}\left(f\right)\left(v\right)& ={\int }_{{\mathbb{R}}^3}{\int }_{S^2}f(v-x{x}^{\prime }(v-w))f(w+x{x}^{\prime }(v-w))\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)dw\hfill \end{array}$$

$$\begin{array}{cc}& ={\int }_{{\mathbb{R}}^3}{\int }_{S^2}f\left(v_x\right)f\left(w_x\right)\left|x^{\prime }(v-w)\right|\sigma \left(dx\right)dw\hfill \end{array}$$

(102)

We want to prove that $Q_{+}\left(f\right)/f\in L^{{\Phi }_{*}}\left(f\right)$ or, equivalently, $Q_{+}\left(f\right)/f_0\in L^{{\Phi }_{*}}\left(f_0\right)$. As $f\in \mathcal{E}\left(f_0\right)$, we can write f as $f={\mathrm{e}}^{u-K_0\left(u\right)}·f_0$, where $u\in B_{f_0}$, so that:

$$Q_{+}\left(f\right)\left(w\right)=f_0\left(w\right){\int }_{S^2}\sigma \left(dx\right){\int }_{{\mathbb{R}}^3}dv{f}_0\left(v\right){\mathrm{e}}^{u\left(v_x\right)+u\left(w_x\right)-2K_0\left(u\right)}\left|x^{\prime }(v-w)\right|$$

(103)

and:

$$\begin{array}{cc}\hfill {\Phi }_{*}\left(\frac{Q_{+}\left(f\right)\left(w\right)}{f_0\left(w\right)}\right)& \le {\int }_{S^2}\sigma \left(dx\right){\int }_{{\mathbb{R}}^3}dv{f}_0\left(v\right){\Phi }_{*}\left({\mathrm{e}}^{u\left(v_x\right)+u\left(w_x\right)-2K_0\left(u\right)}\left|x^{\prime }(v-w)\right|\right)\hfill \end{array}$$

$$\begin{array}{cc}& \le {\int }_{S^2}\sigma \left(dx\right){\int }_{{\mathbb{R}}^3}dv{f}_0\left(v\right)L\left(\left|x^{\prime }(v-w)\right|\right){\Phi }_{*}\left({\mathrm{e}}^{u\left(v_x\right)+u\left(w_x\right)-2K_0\left(u\right)}\right)\hfill \end{array}$$

(104)

where $L\left(a\right)=a\vee a^2$. It follows:

$$\begin{array}{c}{\Phi }_{*}\left(\frac{Q_{+}\left(f\right)\left(w\right)}{f_0\left(w\right)}\right)\hfill \\ \hfill \le {\int }_{S^2}\sigma \left(dx\right){\int }_{{\mathbb{R}}^3}dv{f}_0\left(v\right)L\left(\left|x^{\prime }(v-w)\right|\right)\left((u\left(v_x\right)+u\left(w_x\right)-2K_0\left(u\right)){\mathrm{e}}^{u\left(v_x\right)+u\left(w_x\right)-2K_0\left(u\right)}+1\right)\end{array}$$

(105)

and:

$$\begin{array}{c}{error}_{f_0}\left[{\Phi }_{*}\left(\frac{Q_{+}\left(f\right)}{f_0}\right)\right]\hfill \\ \le \underset{{\mathbb{R}}^3}{\int \int }dvdw{f}_0\left(v\right)f_0\left(w\right)L\left(\left|x^{\prime }(v-w)\right|\right)\left((u\left(v\right)+u\left(w\right)-2K_0\left(u\right)){\mathrm{e}}^{u\left(v\right)+u\left(w\right)-2K_0\left(u\right)}+1\right)\\ \le \underset{{\mathbb{R}}^3}{\int \int }dvdwf\left(v\right)f\left(w\right)L\left(\left|x^{\prime }(v-w)\right|\right)\left(u\left(v\right)+u\left(w\right)-2K_0\left(u\right)\right)\\ \hfill +\underset{{\mathbb{R}}^3}{\int \int }dvdw{f}_0\left(v\right)f_0\left(w\right)L\left(\left|x^{\prime }(v-w)\right|\right)\end{array}$$

(106)

where both terms are finite.

Finally, the integral of the Boltzmann operator is zero:

$$\begin{array}{c}{\int }_{{\mathbb{R}}^3}Q\left(f\right)\left(v\right)dv=\hfill \\ {\int }_{S^2}{\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}(f\left(v_x\right)f\left(w_x\right)-f\left(v\right)f\left(w\right))\left|x^{\prime }(v-w)\right|dwdvdx=\\ {\int }_{S^2}{\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}f\left(v_x\right)f\left(w_x\right)\left|x^{\prime }(v_x-w_x)\right|d{w}_{x}d{v}_{x}dx-\\ \hfill {\int }_{S^2}{\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}f\left(v\right)f\left(w\right))\left|x^{\prime }(v-w)\right|dwdvdx=0\end{array}$$

(107)

☐

The smoothness of the Boltzmann field can be studied by carefully analyzing the structure of the operator as a superposition of:

(1)

Product: $\mathcal{E}\left(f_0\right)\ni f\mapsto f\otimes f\in \mathcal{E}\left(f_0\otimes f_0\right)$;

(2)

Interaction: $\mathcal{E}\left(f_0\otimes f_0\right)\ni f\otimes f\mapsto g=Bf\otimes f\in \mathcal{E}\left(f_0\otimes f_0\right)$;

(3)

Conditioning: $\mathcal{E}\left(f_0\otimes f_0\right)\ni g\mapsto {\int }_{S_2}g\circ A_x\sigma \left(dx\right)\in \mathcal{E}\left(f_0\otimes f_0\right)$;

(4)

Marginalization.

The single operations of the chain are discussed in [7]. We do not do this analysis here and conclude the section by rephrasing in our language Maxwell’s weak form [20] (I.2.3) of the Boltzmann operator.

Proposition 11. Let $f\in \mathcal{E}\left(f_0\right)$ and $g\in L^{\Phi }\left(f\right)$. Then, $Ag$ defined by:

$$Ag(v,w)={\int }_{S^2}\frac{1}{2}(g\left(v_x\right)+g\left(w_x\right))\sigma \left(dx\right)-\frac{1}{2}(g\left(v\right)+g\left(w\right))$$

(108)

belongs to $L^{\Phi }(f\otimes f)$ and:

$${〈g,Q\left(f\right)/f〉}_f={error}_{f\otimes f}\left[Ag\right]$$

(109)

Especially, if $f={\mathrm{e}}^{u-K_0\left(u\right)}·f_0$:

$${〈u,Q\left(f\right)/f〉}_f={error}_{f\otimes f}\left[A\left(\frac{f}{f_0}\right)\right]$$

(110)

5. Conclusions and Discussion

We have shown that a careful consideration of the relevant functional analysis allows us to discuss some basic features of statistical models of interest in statistical physics in the framework of the nonparametric information geometry based on Orlicz spaces. In particular, we have defined the exponential statistical manifold and its vector bundles, namely, the tangent bundle, the pretangent bundle and the Hilbert bundle. Partial results are obtained on connections, which is a topic considered by many authors at the very core of statistical manifolds theory.

For example, the Boltzmann equation takes the form of an evolution equation for the Boltzmann field:

$$\delta f_t=\frac{Q\left(f_t\right)}{f_t},\delta f\in T\mathcal{E}\left(f_0\right),\frac{Q\left(f\right)}{f}\in *T\mathcal{E}\left(f_0\right)$$

(111)

and we can compute the covariant derivative of Boltzmann-Gibbs entropy along the Boltzmann field $D_{Q\left(f\right)/f}E\left(f\right)={〈Q\left(f\right)/f,ln\left(f\right)-E\left(f\right)〉}_f$ with Proposition 11, cfr. [20] (Ch. 3). Our treatment of Boltzmann-Gibbs entropy and Boltzmann equation does not add any new result, but our aim is to transform a generic geometric intuition about the geometry of probability densities into a formal geometrical methodology.

A number of issues remain open, in particular, the proper topological setting of the second order structures and the proper definition of sub-manifold, an important topic that is not mentioned at all in this paper.

In the case of the pretangent bundle, we have been able to show that it is actually the tangent bundle of an extension of the exponential manifold, the mixture manifold, $*T{\mathcal{P}}_{>}\to T{\mathcal{P}}_1$. The construction of an extended manifold whose tangent space would extend the Hilbert bundle, $H{\mathcal{P}}_{>}$, has been the object of much research. In some sense, the answer is known because of the embedding $p\mapsto \sqrt{p}$ that maps positive densities, ${\mathcal{P}}_{>}$, into the unit sphere $S_{\mu }$, but a proper definition of the charts is difficult in this setting.

It has been suggested to use functions called deformed exponentials to mimic the theory of exponential families; see the monograph [35] and also [12,14] (Section 5). An example of deformed exponential is:

$${exp}_d\left(u\right)={\left(\frac{1}{2}u+\sqrt{1+\frac{1}{4}{u}^2}\right)}^2$$

which is a special case of the class introduced in [36,37]. See [38] for an example of application.

The function, ${exp}_d$, maps $\mathbb{R}$ onto ${\mathbb{R}}_{>}$, is increasing, convex and:

$${\Phi }_d\left(u\right)=\frac{1}{2}({exp}_d\left(u\right)+{exp}_d(-u))-1=\frac{1}{2}{u}^2$$

(112)

The Young conjugate is ${\Phi }_{d,*}={\Phi }_d$, and the Orlicz space is $L^{{\Phi }_d}\left(p\right)=L^2\left(p\right)$. A nonparametric exponential family around the positive density, p, was defined by [39] to be:

$$q={exp}_d\left(u-K_p\left(u\right)+{ln}_{d}p\right)$$

(113)

where:

$${ln}_d\left(v\right)={exp}_d^{-1}\left(v\right)=v^{1/2}-v^{-1/2}$$

(114)

If we assume ${error}_{\overline{p}}\left[u\right]=0$, where $\overline{p}$ is a suitable density associated with p, then:

$$K_p\left(u\right)={error}_{\overline{p}}\left[{ln}_{d}p-{ln}_{d}q\right]$$

(115)

An account of this research in progress will be published elsewhere.

There are other approaches to nonparametric information geometry that are not based on the notion of the exponential family. We refer in particular to [40].