Reconsideration of Information-Theoretic Principles—Perspective from the Dual Probability Distribution

Abstract

We reconsider information-theoretic principles, such as the maximum entropy principle/minimum Massieu potential principle, from the perspective of the dual probability distribution. This is introduced through Sanov’s Lemma for the multinomial distribution. The dual correspondence becomes asymptotically manifest. The Massieu potential is rewritten as the Kullback–Leibler divergence between the dual probability distribution and the dual reference distribution. Similarly, the dual potential is rewritten as the cumulant generating function with respect to the dual reference distribution. This perspective gives us new insight into information-theoretic principles. As the dual probability distribution naturally arises in data sampling, we anticipate that this new perspective will play a significant role in data analysis.

1. Introduction

In information geometry, probability distributions are treated as points in a parameter space, and the ‘distance’ between them is treated as divergence [1,2]. The well-known Kullback–Leibler (KL) divergence/relative entropy is not symmetric with respect to the exchange of two probability distributions, meaning it does not satisfy the axiom of distance.

Symmetry plays a significant role in physics. Attempts to introduce symmetric divergences, such as the Jensen–Shannon divergence [3], are significant. However, Amari et al. focused on the hidden dual structure behind asymmetric divergences instead, revealing the dual flat structure in the Pythagorean theorem [1,2].

The dual probability distributions, also known as ‘statistically dual distributions’ [4], refer to a pair of distributions in which the roles of the parameter and the stochastic variable are interchanged. For example, in a canonical distribution $\rho \left(E\left(x\right)|\beta \right)=\exp \left[-\beta E\left(x\right)\right]$ with a constant temperature $T$, the parameter is the inverse temperature $\beta ={\displaystyle \frac{1}{kT}}$, and the stochastic variable is the energy $E\left(x\right)$ of a state $x$. We can consider its dual distribution $\rho \left(\beta \left(y\right)|E\right)$, where these roles are swapped: the energy $E$ is treated as the parameter and the inverse temperature $\beta \left(y\right)$ of a state as the stochastic variable $y$. At the same time, the temperature in a canonical distribution is usually a fixed number that does not fluctuate. In contrast, in a real system, in the case of a finite heat bath, its temperature can fluctuate. In this case, the parameter is given as the average of the fluctuating temperature. The dual probability distribution accounts for the variability of the parameter that is fixed in the thermodynamic limit.

In Section 2, we provide a brief review of the information geometry of the exponential family on the n-dimensional simplex $S_n$, with a given reference distribution. The Massieu potential [5] becomes the cumulant generating function with respect to the reference distribution. The dual parameter vector is generated as an average of the Massieu potential via Legendre transformation, and the dual potential becomes the KL divergence between the probability distribution and the reference distribution.

In Section 3, we introduce the concept of a dual probability distribution. According to Sanov’s Lemma [6], the dual potential can be interpreted as an exponent of the sampling distribution generated by the reference distribution. The integral representation of this exponent introduces new stochastic variables that follow a dual probability distribution. This yields a dual-symmetric correspondence asymptotically. The dual potential can be rewritten as a cumulant generating function with respect to the dual reference distribution. The parameter vector can be generated as an average from the dual potential using the inverse Legendre transformation. The Massieu potential can be expressed as the KL divergence between the dual probability distribution and the dual reference distribution.

Our derivation is based on Stirling’s approximation. Therefore, the Massieu/dual potential requires modification for finite $N$. Upon estimating the correction term, we found that the $O(\log N/N)$ term arising from Stirling’s formula is negligible, and that the correction to the Massieu/dual potential is a convex function of order $O(1/N)$.

In Section 4, we reconsider the maximum entropy principle and its dual in the context of the dual probability distribution. The maximum entropy principle is derived asymptotically from the Pythagorean theorem based on the orthogonality between the canonical geodesic and the iso-energy surface. For the dual principle, the minimum Massieu potential principle, we introduce an explicit dual canonical distribution. The exponent of this distribution is expressed by one dual parameter and a dual energy function. This principle is also derived asymptotically from the Pythagorean theorem based on the orthogonality between the dual canonical geodesic and the dual-iso-energy surface.

We expect new insights to play an important role in data analysis, given that dual probability distributions naturally arise in data sampling. Specifically, we intend to apply our argument to the analysis of time series for simple chaotic systems [7], the probability distribution of which evolves according to a gradient flow equation [8,9,10]. These decay modes are known as the Ruelle–Pollicott resonances. Although originally introduced in connection with decay in classical chaotic systems, it has recently attracted attention in relation to irreversibility in quantum many-body chaos [11].

In Section 5, we present the results of applying our discussion to a simple chaotic map system. We simulate the time evolution for several initial states. In particular, we focus on the time evolution of the Massieu potential to clarify its information-geometric meaning.

In Section 6, we summarize these duality correspondences and comment.

2. Exponential Family in $\mathit{n}$-Dimensional Simplex

2.1. Exponential Family in $n$-Dimensional Simplex

The exponential family is a class of probability distributions commonly encountered in information geometry. For simplicity, we will mainly deal with $S_n$ in this section. In data analysis, a probability distribution is represented as a histogram of empirical data. In practice, any probability distribution can be approximated by $S_n$.

A probability distribution in $S_n$ is written as the following representation of the exponential family in information geometry [1,2],

$$p_x\left(\theta \right)=\mathrm{e}\mathrm{x}\mathrm{p}\left[\theta ·{\delta }_x-\psi \left(\theta \right)\right]{\overline{p}}_x,$$

(1)

where a stochastic variable, $x=\mathrm{0,1},\dots ,n$, $n$-dimensional parameter vector: $\theta =\left({\theta }_1, {\theta }_2,\dots ,{\theta }_n\right)$ and $\theta ·{\delta }_x\equiv {\displaystyle {\sum }_{i=1}^n}{\theta }_i{\delta }_{x,i}$. And ${\overline{p}}_x$ is a reference distribution, ${\overline{p}}_x>0 \mathrm{a}\mathrm{n}\mathrm{d} {\sum }_x{\overline{p}}_x=1.$

To satisfy the normalization condition, ${\sum }_x{p}_x\left(\theta \right)=1,$

$$\mathrm{e}\mathrm{x}\mathrm{p}\left[\psi \left(\theta \right)\right]={\mathrm{E}}_{\overline{p}}\left[\mathrm{e}\mathrm{x}\mathrm{p}\left[\theta ·{\delta }_x\right]\right],$$

(2)

where we write the expectation value with respect to distribution $p_x$ as, $E_{p_x}\left[·\right]\equiv {\displaystyle {\sum }_{x=0}^n}·p_x.$ Then, the Massieu potential is written as the cumulant generating function with respect to the reference distribution,

$$\psi \left(\theta \right)=\log {\mathrm{E}}_{\overline{p}}\left[\mathrm{e}\mathrm{x}\mathrm{p}\left[\theta ·{\delta }_x\right]\right]=-\log \left({\displaystyle \frac{p_0}{{\overline{p}}_0}}\right).$$

(3)

The $i$-th element of the $n$-dimensional parameter vector $\theta$ is written as,

$${\theta }_i=\log \left({\displaystyle \frac{p_i}{{\overline{p}}_i}}{\displaystyle \frac{{\overline{p}}_0}{p_0}}\right).$$

(4)

Since the Massieu potential is a convex function, the Hessian,

$${\partial }_{{\theta }_i}{\partial }_{{\theta }_j}\psi \left(\theta \right),$$

(5)

is the positive definite matrix.

2.2. Legendre Transformation

The dual parameters are defined by the Legendre transformation based on the Massieu potential,

$${\theta }_i^{*}={\partial }_{{\theta }_i}\psi \left(\theta \right).$$

(6)

They are written as the average of ${\delta }_{x,i}$ with respect to $p_x$,

$${\theta }_i^{*}={\mathrm{E}}_{p_x}\left[{\delta }_{x,i}\right]=p_i.$$

(7)

The dual potential is also defined as,

$$\phi \left({\theta }^{*}\right)=\theta ·{\theta }^{*}-\psi \left(\theta \right)=D_{KL}\left[p:\overline{p}\right],$$

(8)

where the KL divergence with respect to $p_x$,

$$D_{KL}\left[p:\overline{p}\right]=\sum _{x=0}^n{p}_x\log \left({\displaystyle \frac{p_x}{{\overline{p}}_x}}\right),$$

(9)

which reduces to the well-known negentropy for the uniform reference distribution up to a constant.

2.3. Divergences

In the information geometry, a probability distribution $p\left(\theta \right)$ in $S_n$ is represented as a point $\theta$ in the $n$ dimensional parameter space. The information distance between two probability distributions, $p\left({\theta }_A\right)$ and $p\left({\theta }_B\right)$, is given by the Bregman divergence based on the convex function, the Massieu potential $\psi \left(\theta \right)$,

$$D_{\psi }\left[{\theta }_A:{\theta }_B\right]=\psi \left({\theta }_A\right)-\psi \left({\theta }_B\right)-\left({\theta }_A-{\theta }_B\right)·{\nabla }_{{\theta }_B}\psi \left({\theta }_B\right).$$

(10)

Using the dual potential, the Bregman divergence is rewritten as follows,

$$D\left[{\theta }_A:{\theta }_B^{*}\right]=\psi \left({\theta }_A\right)+\phi \left({\theta }_B^{*}\right)-{\theta }_A·{\theta }_B^{*}.$$

(11)

We mainly use this divergence without any subscript in this article, since it has a dual symmetric form under the Legendre transformation.

The divergence is related to the KL divergence as,

$$D\left[{\theta }_A:{\theta }_B^{*}\right]=D_{KL}\left[p\left({\theta }_B\right):p\left({\theta }_A\right)\right],$$

(12)

where we used the following cross-entropy with the reference distribution,

$$-{\mathrm{E}}_{p\left({\theta }_B\right)}\left[\log \left({\displaystyle \frac{p\left({\theta }_A\right)}{\overline{p}}}\right)\right]=-{\theta }_A·{\theta }_B^{*}+\psi \left({\theta }_A\right).$$

(13)

3. Dual Probability Distribution

3.1. Dual Probability Distribution

We will consider how the parameter vector, $\theta$, fluctuates. Although parameters are usually treated as constants, we will demonstrate that they can be interpreted as the expected value of certain stochastic variables.

The dual potential, the KL divergence with respect to $p_x$, is considered as the logarithm of the multinomial distribution in $N$ independent trials,

$$-{\displaystyle \frac{1}{N}}\log P_{MD}\left(Np|\overline{p}\right)=-{\displaystyle \frac{1}{N}}\log \left[{\displaystyle \frac{N!}{N_0!N_1!\dots N_n!}}{\overline{p}}_0^{N_0}{\overline{p}}_1^{N_1}\dots {\overline{p}}_n^{N_n}\right]=D_{KL}\left[p:\overline{p}\right]+O\left({\displaystyle \frac{\log N}{N}}\right),$$

(14)

where $p_x=N_x/N$. The derivation is based on the Stirling formula for large $N$ [2]. This relation is known as Sanov’s Lemma [6] in the large deviation theory [12].

Using the integral form of the Gamma function,

$$N!=\Gamma \left(N+1\right)={\int }_{-\infty }^{\infty }{e}^{Ny}{e}^{y-e^y}dy,$$

(15)

we can introduce dual stochastic variables, $y_x \left(x=0, 1,\dots ,n\right)$.

We can rewrite the dual potential as the cumulant generating function with respect to the dual stochastic variables up to a constant and a negligible function,

$$\phi \left({\theta }^{*}\right)-{∆\phi }_N\left({\theta }^{*}\right)={\displaystyle \frac{1}{N}}\log {\mathrm{E}}_{\overline{P}\left(y\right)}\left[e^{N{\theta }^{*}·\tilde{y}}\right]+c_N,$$

(16)

where ${\tilde{y}}_i=y_i-y_0 \left(i=1,\dots ,n\right)$, the dual reference distribution,

$$\overline{P}\left(y\right)={\displaystyle \frac{N^N}{N!}}{\overline{p}}_0^N{e}^{{Ny}_0}{e}^{{\sum }_x\{y_x+\log (N{\overline{p}}_x)-e^{y_x+\log (N{\overline{p}}_x)}\}},$$

(17)

and the constant $c_N$ and a convex function ${∆\phi }_N\left({\theta }^{*}\right)$ defined as follows,

$$c_N\equiv -\log {\overline{p}}_0-{\displaystyle \frac{n+1}{2N}}\log \left(2\pi N\right)+1-\log N+{\displaystyle \frac{1}{N}}\log \left(N!\right),$$

(18)

$$\begin{gathered} {∆\phi }_N\left({\theta }^{*}\right)\equiv -{\displaystyle \frac{1}{N}}\sum _{x=0}^n\left[\log \left({(Np}_x)!\right)-\left\{{Np}_x\log \left({Np}_x\right)-{Np}_x\right\}-{\displaystyle \frac{1}{2}}\log \left(2\pi N\right)\right] \\ =-{\displaystyle \frac{1}{N}}\sum _{x=0}^n\left\{{\displaystyle \frac{1}{2}}\log p_x+{\lambda }_{N_x}\right\}=O\left({\displaystyle \frac{1}{N}}\right). \end{gathered}$$

(19)

where ${\displaystyle \frac{1}{12N+1}}<{\lambda }_N<{\displaystyle \frac{1}{12N}}$ is known as Robbins’ bound [13]. We note that ${∆\phi }_N\left({\theta }^{*}\right)$ is a convex function for $N$ large so that $\phi \left({\theta }^{*}\right)-{∆\phi }_N\left({\theta }^{*}\right)$ is interpreted as a modified dual potential. These derivations are shown in Appendix A.

Since the modified dual potential is the cumulant generating function up to a constant, the inverse Legendre transformation gives the following modified parameter vector as averages of the dual stochastic variables,

$$\theta -∆\theta ={\nabla }_{{\theta }^{*}}\left(\phi \left({\theta }^{*}\right)-{∆\phi }_N\left({\theta }^{*}\right)\right)={\mathrm{E}}_{P\left(y|{\theta }^{*}\right)}\left[\tilde{y}\right],$$

(20)

where the dual probability distribution,

$$P\left(y|{\theta }^{*}\right)=e^{N\{{\theta }^{*}·\tilde{y}-\phi \left({\theta }^{*}\right)+{∆\phi }_N\left({\theta }^{*}\right)+c_N\}}\overline{P}\left(y\right).$$

(21)

The modified Massieu potential is rewritten as the following scaled KL divergence between $P\left(y|{\theta }^{*}\right) \mathrm{a}\mathrm{n}\mathrm{d} \overline{P}\left(y\right)$ up to a constant,

$$\begin{gathered} \psi \left(\theta \right)-∆{\psi }_N\left(\theta \right)={\mathrm{E}}_{P\left(y|{\theta }^{*}\right)}\left[{\theta }^{*}·\tilde{y}-\phi \left({\theta }^{*}\right)+{∆\phi }_N\left({\theta }^{*}\right)+c_N\right]-c_N \\ ={\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }^{*}\right):\overline{P}\left(y\right)\right]-c_N. \end{gathered}$$

(22)

The Massieu potential can be considered as the information/negentropy of the dual stochastic variables.

The dual stochastic variables represent fluctuations in the distribution of the data sample. The order of the deviation is $O(1/\sqrt{N})$. From the Central Limit Theorem, the distribution becomes a normal distribution:

$$P\left(y\right)\cong \mathcal{N}\left(\overline{y},\Sigma \right),$$

(23)

where ${\overline{y}}_x=\log p_x-\log {\overline{p}}_x+1$, and ${\Sigma }_{x,x’}={\delta }_{x,x^{\prime }}/\left(N{p}_x\right).$

Although fluctuation is negligible for large $N$, it is important for finite $N.$ For example, the inverse of temperature is the typical parameter in the canonical distribution. Its fluctuation is negligible for a large reservoir in the thermodynamical limit. But the temperature can fluctuate in a finite system.

3.2. Divergence

The KL divergence between $P\left(y|{\theta }_A^{*}\right)$ and $P\left(y|{\theta }_B^{*}\right)$ is rewritten by the modified potentials as,

$${\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_A^{*}\right):P\left(y|{\theta }_B^{*}\right)\right]=\psi \left({\theta }_A\right)-∆{\psi }_N\left({\theta }_A\right)+\phi \left({\theta }_B^{*}\right)-{∆\phi }_N\left({\theta }_B^{*}\right)-\left({\theta }_A-∆{\theta }_A\right)·{\theta }_B^{*},$$

(24)

where we used (22) for the modified Massieu potential and the following dual cross-entropy derived from (20) and (21),

$$-{\displaystyle \frac{1}{N}}{E}_{P\left(y|{\theta }_A^{*}\right)}\left[\log {\displaystyle \frac{P\left(y|{\theta }_B^{*}\right)}{\overline{P}\left(y\right)}}\right]=-({\theta }_A-∆{\theta }_A)·{\theta }_B^{*}+\phi \left({\theta }_B^{*}\right)-{∆\phi }_N\left({\theta }_B^{*}\right)-c_N.$$

(25)

The KL divergence between the dual probability distributions, $P\left(y|{\theta }_A^{*}\right)$ and $P\left(y|{\theta }_B^{*}\right)$, is given as the following modified divergence between ${\theta }_B$ and ${\theta }_A^{*}$ as,

$${\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_A^{*}\right):P\left(y|{\theta }_B^{*}\right)\right]=D\left[{\theta }_A:{\theta }_B^{*}\right]-{∆D}_N\left[{\theta }_A:{\theta }_B^{*}\right],$$

(26)

where ${∆D}_N\left[{\theta }_A:{\theta }_B^{*}\right]$ is a Bregman divergence while ${∆\phi }_N\left({\theta }^{\mathrm{*}}\right)$ is a convex function with respect to ${\theta }^{\mathrm{*}}$,

$${∆D}_N\left[{\theta }_A:{\theta }_B^{*}\right]\equiv ∆{\psi }_N\left({\theta }_A\right)+{∆\phi }_N\left({\theta }_B^{*}\right)-∆{\theta }_A·{\theta }_B^{*}=O\left({\displaystyle \frac{1}{N}}\right).$$

(27)

The KL divergence between $P\left(y|{\theta }_A^{*}\right)$ and $P\left(y|{\theta }_B^{*}\right)$ asymptotically becomes the dual KL divergence between $p\left({\theta }_B\right)$ and $p\left({\theta }_A\right)$ in the limit of large $N$,

$$\underset{N\to \infty }{\lim }{D}_{KL}\left[P\left(y|{\theta }_A^{*}\right):P\left(y|{\theta }_B^{*}\right)\right]=D_{KL}\left[p\left({\theta }_B\right):p\left({\theta }_A\right)\right].$$

(28)

3.3. Pythagorean Theorem

The Pythagorean theorem for the KL divergences between the dual probability distributions is given as follows,

$$\begin{array}{c}{\displaystyle \frac{1}{N}}{D}_{KL}[P\left(y|{\theta }_A^{*}\right):P\left(y|{\theta }_C^{*}\right)]-{\displaystyle \frac{1}{N}}{D}_{KL}[P\left(y|{\theta }_A^{*}\right):P\left(y|{\theta }_B^{*}\right)]-{\displaystyle \frac{1}{N}}{D}_{KL}[P\left(y|{\theta }_B^{*}\right):P\left(y|{\theta }_C^{*}\right)]\\ =D\left[{\theta }_A:{\theta }_c^{*}\right]-{∆D}_N\left[{\theta }_A:{\theta }_c^{*}\right]-D\left[{\theta }_A:{\theta }_B^{*}\right]+{∆D}_N\left[{\theta }_A:{\theta }_B^{*}\right]-D\left[{\theta }_B:{\theta }_C^{*}\right]+{∆D}_N\left[{\theta }_B:{\theta }_C^{*}\right]\\ =\left({\theta }_A-∆{\theta }_A-{\theta }_B+∆{\theta }_B\right)·\left({\theta }_B^{*}-{\theta }_C^{*}\right)=0,\end{array}$$

(29)

where the parameter geodesic is modified.

4. Maximum Entropy Principle and the Dual Principle

4.1. Maximum Entropy Principle

We introduce a canonical parameter following the one-parameter geodesic,

$${\theta }_{can}\left(\beta \right)-∆{\theta }_{can}\left(\beta \right)=-\beta E,$$

(30)

that is corresponding to the canonical distribution in physics,

$$p\left({\theta }_{can}\left(\beta \right)-∆{\theta }_{can}\left(\beta \right)\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left[-\beta E_x+\mathcal{F}\left(\beta \right)\right]{\overline{p}}_x.$$

(31)

The parameter $\beta$ is the inverse temperature included in the Boltzmann constant, $E_x$ is the $n+1$-dimensional energy function, $E_x=E·{\delta }_x\equiv {\displaystyle {\sum }_{i=1}^n}{E}_i{\delta }_{x,i}$ and $E=\left(E_1,\dots E_n\right)$ where we choose $E_0=0$ for which the free potential is given as the negative Massieu potential,

$$\mathcal{F}\left(\beta \right)\equiv \beta F\left(\beta \right)=-\psi \left({\theta }_{can}\left(\beta \right)-∆{\theta }_{can}\left(\beta \right)\right).$$

(32)

Note that $F\left(\beta \right)$ is the free energy and $\mathcal{F}\left(\beta \right)$ is the dimensionless potential.

The orthogonal condition for the Pythagorean theorem in (29) to hold can be rewritten as for ${\theta }_A-∆{\theta }_A={\theta }_{can}\left(\alpha \right)-∆{\theta }_{can}\left(\alpha \right)$, ${\theta }_B-∆{\theta }_B={\theta }_{can}\left(\beta \right)-∆{\theta }_{can}\left(\beta \right), {\theta }_B^{*}={\theta }_{can}^{*}\left(\beta \right)=p\left({\theta }_{can}\left(\beta \right)\right) \mathrm{and} {\theta }_C^{*}={\theta }^{*}=p\left(\theta \right)$,

$$\left(\alpha -\beta \right)E·\left({\theta }_{can}^{*}\left(\beta \right)-{\theta }^{*}\right)=\left(\alpha -\beta \right)\left(E·p\left({\theta }_{can}\left(\beta \right)\right)-E·p\left(\theta \right)\right)=0.$$

(33)

This implies that the canonical geodesic is perpendicular to the iso-energy surface, as is shown in Figure 1. Under this orthogonal condition, the Pythagorean theorem holds as,

$$\begin{gathered} {\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_{can}^{*}\left(\alpha \right)\right):P\left(y|{\theta }^{*}\right)\right] \\ ={\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_{can}^{*}\left(\alpha \right)\right):P\left(y|{\theta }_{can}^{*}\left(\beta \right)\right)\right]+{\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_{can}^{*}\left(\beta \right)\right):P\left(y|{\theta }^{*}\right)\right]. \end{gathered}$$

(34)

The maximum entropy principle is derived under the iso-energy condition. From (34), the following inequality is derived,

$${\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_{can}^{*}\left(\alpha \right)\right):P\left(y|{\theta }^{*}\right)\right]\ge {\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }_{can}^{*}\left(\alpha \right)\right):P\left(y|{\theta }_{can}^{*}\left(\beta \right)\right)\right].$$

(35)

By using (20) and (30),

$$\alpha E·{\theta }^{*}-\phi \left({\theta }^{*}\right)+{∆\phi }_N\left({\theta }^{*}\right)\le \alpha E·{\theta }_{can}^{*}\left(\beta \right)-\phi \left({\theta }_{can}^{*}\left(\beta \right)\right)+{∆\phi }_N\left({\theta }_{can}^{*}\left(\beta \right)\right).$$

(36)

From the iso-energy condition, we rigorously obtained the following inequality,

$$-\phi \left({\theta }^{*}\right)+{∆\phi }_N\left({\theta }^{*}\right)\le -\phi \left({\theta }_{can}^{*}\left(\beta \right)\right)+{∆\phi }_N\left({\theta }_{can}^{*}\left(\beta \right)\right).$$

(37)

The maximum entropy principle is derived in the limit of large $N$,

$$-\phi \left({\theta }^{*}\right)\le -\phi \left({\theta }_{can}^{*}\left(\beta \right)\right).$$

(38)

Figure 1. The canonical geodesic, ${\theta }_{can}\left(\alpha \right)-∆{\theta }_{can}\left(\alpha \right)-{\theta }_{can}\left(\beta \right)+∆{\theta }_{can}\left(\beta \right)\propto E$ is perpendicular to iso-energy surface, $E·{\theta }_{can}^{*}\left(\beta \right)=E·{\theta }^{*}$.

We note that the dual potential is the negative entropy for the uniform reference distribution up to a constant.

4.2. Minimum Massieu-Potential Principle

We introduce a dual canonical parameter following the one-parameter dual-geodesic,

$${\vartheta }_{can}^{*}\left(\kappa \right)·\tilde{y}=\kappa {\vartheta }_{can}^{*}\left(\lambda \right)·\tilde{y}/\lambda =-\kappa \mathcal{E}\left(\tilde{y}\right),$$

(39)

that is corresponding to the dual canonical distribution,

$$P\left(y|{\vartheta }_{can}^{*}\left(\kappa \right)\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left[-\kappa \mathcal{E}\left(\tilde{y}\right)-\phi \left({\vartheta }_{can}^{*}\left(\kappa \right)\right)+∆\phi \left({\vartheta }_{can}^{*}\left(\kappa \right)\right)+c_N\right]\overline{P}\left(y\right).$$

(40)

Two dual canonical parameters are connected through a scale transformation in (39), ${\vartheta }_{can,i}^{*}\left(\kappa \right)=\kappa {\vartheta }_{can,i}^{*}\left(\lambda \right)/\lambda \left(i=1,\dots ,n\right)$. However, ${\vartheta }_{can,x}^{*}\left(\kappa \right)=$ $p_{can,x}\left({\vartheta }_{can}\left(\kappa \right)\right)$ in $S_n$, the normalization of $p_{can,x}\left({\vartheta }_{can}\left(\kappa \right)\right)$ is the same as $p_{can,x}\left({\vartheta }_{can}\left(\lambda \right)\right)$ due to the following relation between their respective zero-components,

$${\vartheta }_{can,0}^{*}\left(\kappa \right)=1-{\displaystyle \frac{\kappa }{\lambda }}\left(1-{\vartheta }_{can,0}^{*}\left(\lambda \right)\right)>0.$$

(41)

The orthogonal condition in the Pythagorean theorem can be rewritten as for ${\theta }_A-∆{\theta }_A=\theta -∆\theta =E_{P\left(y|{\theta }^{*}\right)}\left[\tilde{y}\right]$, ${\theta }_B-∆{\theta }_B={\vartheta }_{can}\left(\lambda \right)-∆{\vartheta }_{can}\left(\lambda \right)=E_{P\left(y|{\vartheta }_{can}^{*}\left(\lambda \right)\right)}\left[\tilde{y}\right] ,$ ${\theta }_B^{*}={\vartheta }_{can}^{*}\left(\lambda \right),$ and ${\theta }_C^{*}={\vartheta }_{can}^{*}\left(\kappa \right)=\kappa {\vartheta }_{can}^{*}\left(\lambda \right)/\lambda$,

$$\begin{gathered} \left(\theta -∆\theta -{\vartheta }_{can}\left(\lambda \right)+∆{\vartheta }_{can}\left(\lambda \right)\right)·{\vartheta }_{can}^{*}\left(\lambda \right)\left(1-\kappa /\lambda \right) \\ =\left(E_{P\left(y|{\vartheta }_{can}^{*}\left(\lambda \right)\right)}\left[\mathcal{E}\left(\tilde{y}\right)\right]-E_{P\left(y|{\theta }^{*}\right)}\left[\mathcal{E}\left(\tilde{y}\right)\right]\right)\left(\lambda -\kappa \right)=0. \end{gathered}$$

(42)

This implies that the canonical dual geodesic is perpendicular to the iso-dual-energy surface, as is shown in Figure 2. Under this orthogonal condition, the Pythagorean theorem holds as,

$$\begin{gathered} {\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }^{*}\right):P\left(y|{\vartheta }_{can}^{*}\left(\kappa \right)\right)\right] \\ ={\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }^{*}\right):P\left(y|{\vartheta }_{can}^{*}\left(\lambda \right)\right)\right]+{\displaystyle \frac{1}{N}}{D}_{KL}[P\left(y|{\vartheta }_{can}^{*}\left(\lambda \right)\right):P\left(y|{\vartheta }_{can}^{*}\left(\kappa \right)\right)]. \end{gathered}$$

(43)

The minimum Massieu-potential principle is derived under the iso-dual-energy condition. From the Pythagorean theorem (43), the following inequality is derived,

$${\displaystyle \frac{1}{N}}{D}_{KL}\left[P\left(y|{\theta }^{*}\right):P\left(y|{\vartheta }_{can}^{*}\left(\kappa \right)\right)\right]\ge {\displaystyle \frac{1}{N}}{D}_{KL}[P\left(y|{\vartheta }_{can}^{*}\left(\lambda \right)\right):P\left(y|{\vartheta }_{can}^{*}\left(\kappa \right)\right)].$$

(44)

After canceling $\phi \left({\vartheta }_{can}^{*}\left(\kappa \right)\right)-{∆\phi }_N\left({\vartheta }_{can}^{*}\left(\kappa \right)\right)$ from both sides, the inequality is rewritten as,

$$\begin{gathered} \psi \left(\theta \right)-∆{\psi }_N\left(\theta \right)-\left(\theta -∆\theta \right)·{\vartheta }_{can}^{*}\left(\kappa \right) \\ \ge \psi \left({\vartheta }_{can}\left(\lambda \right)\right)-∆{\psi }_N\left({\vartheta }_{can}\left(\lambda \right)\right)-\left({\vartheta }_{can}\left(\lambda \right)-∆{\vartheta }_{can}\left(\lambda \right)\right)·{\vartheta }_{can}^{*}\left(\kappa \right), \end{gathered}$$

(45)

where we used (24). From the iso-dual-energy condition in (42), we rigorously obtained the following inequality,

$$\psi \left(\theta \right)-∆{\psi }_N\left(\theta \right)\ge \psi \left({\vartheta }_{can}\left(\lambda \right)\right)-∆{\psi }_N\left({\vartheta }_{can}\left(\lambda \right)\right).$$

(46)

The minimum Massieu-potential principle is derived in the limit of large $N$,

$$\psi \left(\theta \right)\ge \psi \left({\vartheta }_{can}\left(\lambda \right)\right).$$

(47)

Figure 2. The dual canonical geodesic, $({\vartheta }_{can}^{*}\left(\kappa \right)-{\vartheta }_{can}^{*}\left(\lambda \right))·\tilde{y}\propto \mathcal{E}\left(\tilde{y}\right)$ is perpendicular to the iso-dual-energy surface.

5. Application to a Simple Chaotic Map System

We will reconsider the time evolution of a simple chaotic map system from the information-geometric viewpoint.

We consider the following chaotic map of the unit interval into itself, satisfying the following equation:

$$z_{t+1 }=f\left(z_{t }\right)=\left\{\begin{array}{ll}\left(1-a\right){\displaystyle \frac{z_t}{a}}+a& \mathrm{i}\mathrm{f} z_t\in \left[0,a\right)\\ {\displaystyle \frac{z_t-a}{1-a}}& \mathrm{i}\mathrm{f} z_t\in \left[a,1\right)\end{array},\right.$$

(48)

where $0<a<1$, as is illustrated in Figure 3. The map is a kind of modified Bernoulli map. Similar maps were discussed in the analysis of chaos in dynamical systems [14,15].

Figure 3. Purple lines: the piecewise linear function $f\left(z\right)$, defined as $\left(1-a\right){\displaystyle \frac{z}{a}}+a \mathrm{i}\mathrm{f} z\in \left[0,a\right)$ and ${\displaystyle \frac{z-a}{1-a}} \mathrm{i}\mathrm{f} z\in \left[a,1\right)$. Green lines: auxiliary lines.

The time evolution of a probability density is given as,

$${\rho }_{t+1 }\left(z\right)=\mathcal{L}{\rho }_t\left(z\right),$$

(49)

where the Frobenius—Perron operator is written as,

$$\mathcal{L}\rho \left(z\right)=\left\{\begin{array}{ll}\left(1-a\right)\rho \left(\left(1-a\right)z+a\right)& \mathrm{i}\mathrm{f} z\in \left[0,a\right)\\ {\displaystyle \frac{a}{1-a}}\rho \left({\displaystyle \frac{a}{1-a}}\left(z-a\right)\right)+\left(1-a\right)\rho \left(\left(1-a\right)z+a\right)\hfill & \mathrm{i}\mathrm{f} z\in \left[a,1\right)\end{array}\right..$$

(50)

The invariant density is uniform on each of the two intervals,

$${\rho }_{inv}\left(z\right)=\left\{\begin{array}{ll}{\displaystyle \frac{1}{1+a}}& \mathrm{i}\mathrm{f} z\in \left[0,a\right) \\ {\displaystyle \frac{1}{1-a^2}}& \mathrm{i}\mathrm{f} z\in \left[a,1\right)\end{array}\right..$$

(51)

To introduce $p_x$ $(x=\mathrm{0,1},\dots ,n)$ in the chaotic system, we define $n+1$ intervals as follows,

$$I_x=\left\{\begin{array}{ll}\left[0,a\right)& \mathrm{i}\mathrm{f} x=0\\ \left[a+{\displaystyle \frac{1-a}{n}}(i-1),a+{\displaystyle \frac{1-a}{n}}i\right)\hfill & \mathrm{i}\mathrm{f} x=i, i=\mathrm{1,2},\dots ,n\end{array}\right.,$$

(52)

where the second interval $\left[a,1\right)$ was divided into $n$ equal parts, for simplicity.

We define,

$$p_{t,x}=Prob\left(z_t\in I_x\right).$$

(53)

Then,

$$p_{inv,0}={\displaystyle \frac{a}{1+a}}, p_{inv,i}={\displaystyle \frac{1}{1+a}}{\displaystyle \frac{1}{n}}, i=\mathrm{1,2},\dots ,n.$$

(54)

We also define the reference distribution as,

$$\overline{p}=p_{inv}.$$

(55)

We define the dual canonical distribution as a uniform distribution on each of the two intervals $\left[0,a\right)$ and $\left[a,1\right)$. The dual canonical distribution evolves along the dual canonical geodesic in Figure 2 and converges to the invariant distribution [13].

When the initial distribution is chosen as the dual canonical distribution, the absolute value of the Massieu potential exponentially converges to zero, which is that of the invariant distribution. As shown by the purple line in Figure 4, the smooth convergence is numerically confirmed. The Massieu potential is written as

$${\psi }_t=-\log \left({\displaystyle \frac{p_{t,0}}{p_{inv,0}}}\right)=-\log \left({\displaystyle \frac{\left(1+a\right)N_{t,0}}{aN}}\right),$$

(56)

where $N$ is the total number of particles and $N_{t,0}$ is the number of particles in the interval $\left[0,a\right)$ at time $t$. It is easy to estimate the Massieu potential by just counting the number of particles, $N_{t,0}$.

Figure 4. Time evolutions of the absolute value of the Massieu potential at the parameter, $a=0.3$. The purple line: the initial state on the dual canonical geodesic: the total number of particles, $N=1.3\times {10}^7$, the initial number of particles in the interval $\left[0,a\right)$, $N_{t=\mathrm{0,0}}=4\times {10}^6$. The green line: the initial state on the iso-dual-energy surface: the total number of particles, $N=1.3\times {10}^7$, the initial number of particles in the interval $\left[0,a\right)$, $N_{t=\mathrm{0,0}}=3\times {10}^6$, the initial particles are uniformly distributed in the interval $\left[0,a\right)$, the initial number of particles in the interval $\left[a,1\right)$, ${\sum }_{i=1}^n{N}_{t=0,i}=1.0\times {10}^7$, the initial particles are distributed exponentially as $\exp [-4 z]$ in the interval $\left[a,1\right)$.

Next, we consider the time evolution of the following non-uniform initial density,

$${\rho }_{t=0}\left(z\right)=\left\{\begin{array}{ll}{e}^{-{\psi }_{t=0}}{\rho }_{inv}\left(z\right)& z<a\\ e^{{\theta }_{t=0}\left(z\right)-{\psi }_{t=0}}{\rho }_{inv}\left(z\right)& z>a\end{array}\right.,$$

(57)

where we choose,

$${\theta }_{t=0}\left(z\right)=-\beta \left(z-{\displaystyle \frac{1+a}{2}}\right),$$

(58)

so that the initial state lies on the iso-dual-energy surface,

$${\theta }_{t=0}·{\vartheta }_{inv}^{*}={\displaystyle \frac{1}{1-a^2}}{\int }_a^1{\theta }_{t=0}\left(z\right)dz=0,$$

(59)

since ${\theta }_{t=0}\left(z\right)$ is an anti-symmetric function within the interval $\left[a,1\right)$. We replaced the dot product with the integration.

The initial Massieu potential, ${\psi }_{t=0}$, is determined by the normalization condition,

$$e^{{\psi }_{t=0}}=p_{inv,0}+\left(1-p_{inv,0}\right){\int }_a^1{e}^{{\theta }_{t=0}\left(z\right)}dz.$$

(60)

After integrating the right-hand side, we obtain,

$${\psi }_{t=0}=\log \left[1+{\displaystyle \frac{1}{1+a}}\left\{{\displaystyle \frac{2}{\beta \left(1-a\right)}}\sinh \left\{{\displaystyle \frac{\beta }{2}}\left(1-a\right)\right\}-1\right\}\right]={\displaystyle \frac{1}{6}}{\displaystyle \frac{1}{1+a}}{\left\{{\displaystyle \frac{\beta }{2}}\left(1-a\right)\right\}}^2-\dots .$$

(61)

Then,

$$p_{t=\mathrm{0,0}}={\displaystyle \frac{1}{1+\frac{2}{a\beta (1-a)}\sinh \left\{\frac{\beta }{2}\left(1-a\right)\right\}}}.$$

(62)

In the numerical simulation, we prepared a total of $N$ particles at $t=0$. A total of $p_{t=\mathrm{0,0}}N$ particles are uniformly distributed in the interval $[0,a)$ while the remaining particles are distributed exponentially as $\exp [-\beta z]$ in the interval $[a,1)$. In this case, the following behavior can be expected: if the system moves on the iso-dual-energy surface orthogonal to the dual canonical geodesic line, the Massieu potential smoothly decreases to its minimum value of zero.

As shown by the green line in Figure 4, this non-uniform initial distribution causes a large change in the particle number in the interval $[0,a)$ at $t=1$, resulting in a significant jump in the absolute value of the Massieu potential. This implies that the state has deviated significantly from the iso-dual-energy surface. Although the state subsequently converges to the invariant distribution, its decay profile differs from the one along the dual canonical geodesic, suggesting a different underlying mechanism of decay.

6. Conclusions and Remarks

In this paper, we introduce the dual probability distribution of the $n$-dimensional simplex $S_n$. According to Sanov’s Lemma for the multinomial distribution, the dual potential can be interpreted as an exponent of the sampling distribution generated by the reference distribution. The integral representation of this exponent introduces new stochastic variables that follow the dual probability distribution.

The dual probability distribution yields asymptotically dual-symmetric correspondences. The dual potential can be expressed as a cumulant generating function with respect to the dual reference distribution. The parameter vector is obtained by averaging the dual potential using the inverse Legendre transformation. The Massieu potential is equal to the KL divergence between the dual probability distribution and the dual reference distribution.

Information-theoretic principles, such as the maximum-entropy and minimum Massieu-potential principles, can be derived asymptotically from the dual-probability-distribution perspective. For the minimum Massieu potential principle, we introduced a dual canonical distribution explicitly, whose exponent is written using one dual parameter and a dual energy function. The minimum Massieu potential principle is derived asymptotically from the Pythagorean theorem, based on the orthogonality between the dual canonical geodesic and the iso-dual-energy surface.

We expect new insights to play an important role in data analysis, given that dual probability distributions naturally arise in data sampling. Specifically, we intend to apply our argument to the analysis of time series for simple chaotic systems [10], the probability distribution of which evolves according to a gradient flow equation [11,12,13]. In this paper, we present the results of applying our discussion to a simple chaotic map system. We simulate the time evolution for two initial states: One is on the dual canonical geodesic, and the other is on the iso-dual-energy surface. The time evolution of the former initial state is comprehensible because it follows the gradient flow equation. However, the latter is difficult to understand because the dual-energy conservation does not hold.

Finally, this paper provides comments on future research. Our arguments can be easily applied to the maximum work formulation (the dual formulation) using KL divergences scaled by the canonical parameter (the dual-canonical parameter) [16]. The Pythagorean theorem for the scaled divergences gives orthogonality between the canonical-parameter geodesic and the isentropic surface (the dual canonical-parameter geodesic and the iso-Massieu-potential surface). We expect that the information-geometric perspective gives us a clear understanding of the time evolution of chaotic systems in an isentropic process (iso-Massieu potential process).