# An Introduction to the Non-Equilibrium Steady States of Maximum Entropy Spike Trains

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Preliminary Considerations

#### 2.1. Binning and Spike Trains

#### 2.2. Elementary Properties of Markov Chains

- (i)
- An initial probability distribution, encoded by a vector $\mu :=({\mu}_{i}:i\in \mathbb{S})$.
- (ii)
- A collection of $\mathbb{S}$-indexed stochastic matrices $\{{P}_{t}:={\left({p}_{t}(i,j)\right)}_{i,j\in \mathbb{S}}:t\in \mathbb{N}\}$.

#### 2.3. Homogeneity, Ergodicity and Stationarity

- (a)
- There exists a unique stationary distribution $\pi $ for P that satisfies that ${\pi}_{i}>0$ for every $i\in \mathbb{S}$.
- (b)
- For every $j\in \mathbb{S}$,$$\begin{array}{c}\hfill \underset{m\to +\infty}{lim}{p}_{ij}^{\left(m\right)}={\pi}_{j}\phantom{\rule{3.33333pt}{0ex}}.\end{array}$$Equivalently, for every distribution $\nu $, ${lim}_{t\to \infty}{\mathbb{P}}_{\nu}({X}_{t}=j)={\pi}_{j}\phantom{\rule{3.33333pt}{0ex}}.$ This property guarantees the uniqueness of the maximum entropy Markov chain.

#### 2.4. The Reversed Markov Chain

#### 2.5. Reversibility and Detailed Balance

#### 2.6. Law of Large Numbers for Ergodic Markov Chains

## 3. Observables of Markov Chains and Their Properties

#### 3.1. Observables and Their Empirical Averages

#### 3.2. Moments and Cumulants

#### 3.3. Observables and Ergodicity

#### 3.4. Central Limit Theorem for Observables

**Theorem**

**1**

**(Central**

**limit**

**theorem**

**for**

**ergodic**

**Markov**

**chains).**

#### 3.5. Large Deviations of Average Values of Observables

**Theorem**

**2**

**(Gärtner-Ellis**

**theorem).**

## 4. Building Maximum Entropy Temporal Models

#### 4.1. The Entropy Rate of a Temporal Model

#### 4.1.1. Basic Definitions

**Definition**

**1**

**(entropy**

**rate).**

#### 4.1.2. The Entropy Rate of I.I.D. and Markov Models

#### 4.2. Entropy Rate Maximization under Constraints

#### 4.3. Solving the Optimization Problem

#### 4.3.1. Lagrange Multipliers and the Variational Principle

#### 4.3.2. Transfer Matrix Method

**Theorem**

**3**

**(Perron-Frobenius**

**theorem).**

- There is a positive maximal eigenvalue ρ > 0 such that all other eigenvalues satisfy $\mid {\rho}^{\prime}\mid <\rho $. Moreover ρ is simple;
- There are positive left- and right-eigenvectors $u=({u}_{1},\cdots ,{u}_{k}),v=({v}_{1},\cdots ,{v}_{k})$ s.t. $uA\phantom{\rule{3.33333pt}{0ex}}=\phantom{\rule{3.33333pt}{0ex}}\rho \phantom{\rule{3.33333pt}{0ex}}u,Av\phantom{\rule{3.33333pt}{0ex}}=\phantom{\rule{3.33333pt}{0ex}}\rho v.$

#### 4.3.3. Finite Range Gibbs Measures

#### 4.4. Example

## 5. Statistical Properties of Markov Maximum Entropy Measures

#### 5.1. Cumulants from Free Energy

#### 5.2. Fluctuation-Dissipation Relations

#### 5.3. Resonances and Decay of Correlations

#### 5.4. Large Deviations for Average Values of Observables in MEMC

#### 5.5. Information Entropy Production

#### 5.6. Gallavotti-Cohen Fluctuation Theorem

#### 5.7. Linear Response

## 6. Discussion and Future Work

## Author Contributions

## Funding

## Conflicts of Interest

## Abbreviations

MEP | Maximum entropy principle |

MEMC | Maximum entropy Markov chain |

SCGF | Scaled cumulant generating function |

CLT | Central limit theorem |

LLN | Law of large numbers |

LDP | Large deviation principle |

IEP | Information entropy production |

KSE | Kolmogorov-Sinai entropy |

NESS | Non-equilibrium steady states |

Symbol list | |

$\mathbb{S}$ | ${\{0,1\}}^{N}$ the state space of spike patterns of N neuron |

$\mathsf{\Omega}$ | The set of infinite sequences of spike patterns |

${x}_{n}^{k}$ | Spiking state of neuron k at time n |

${\mathit{x}}_{n}$ | Spike pattern at time n |

${\mathit{x}}_{{t}_{1},{t}_{2}}$ | Spike block from time ${t}_{1}$ to ${t}_{2}$ |

$\nu \left(f\right)$ | Expectation of the observable f w.r.t. the probability measure $\nu $ |

${A}_{T}\left(f\right)$ | Empirical Average value of the observable f considering T spike patterns |

${\mathbb{S}}^{R}$ | Space of spike blocks of N neurons and length R |

$\mathcal{S}\left[\mu \right]$ | Entropy of the probability measure $\mu $ |

$\mathcal{H}$ | Energy function |

$\mathcal{F}\left[\mathcal{H}\right]$ | Free energy |

## References

- Rieke, F.; Warland, D.; de Ruyter van Steveninck, R.; Bialek, W. Spikes, Exploring the Neural Code; M.I.T. Press: Cambridge, MA, USA, 1996. [Google Scholar]
- Bialek, W. Biophysics: Searching for Principles; Princeton University Press: Princeton, NJ, USA, 2012. [Google Scholar]
- Schneidman, E.; Berry, M.J.; Segev, R.; Bialek, W. Weak pairwise correlations imply strongly correlated network states in a neural population. Nature
**2006**, 440, 1007–1012. [Google Scholar] [CrossRef] [PubMed][Green Version] - Ganmor, E.; Segev, R.; Schneidman, E. Sparse low-order interaction network underlies a highly correlated and learnable neural population code. Proc Natl. Acad. Sci. USA
**2011**, 108, 9679–9684. [Google Scholar] [CrossRef] [PubMed][Green Version] - Tkačik, G.; Marre, O.; Amodei, D.; Schneidman, E.; Bialek, W.; Berry, M.J. Searching for collective behavior in a large network of sensory neurons. PLoS Comput. Biol.
**2014**, 10, e1003408. [Google Scholar] [CrossRef] [PubMed] - Palsso, B. Systems Biology: Properties of Reconstructed Networks; Cambridge University Press: Cambridge, UK, 2006. [Google Scholar]
- Tang, A.; Jackson, D.; Hobbs, J.; Chen, W.; Smith, J.; Patel, H.; Prieto, A.; Petrusca, D.; Grivich, M.; Sher, A.; et al. A maximum entropy model applied to spatial and temporal correlations from cortical networks in vitro. J. Neurosci.
**2008**, 28, 505–518. [Google Scholar] [CrossRef] [PubMed] - Marre, O.; El Boustani, S.; Frégnac, Y.; Destexhe, A. Prediction of spatiotemporal patterns of neural activity from pairwise correlations. Phys. Rev. Lett.
**2009**, 102, 138101. [Google Scholar] [CrossRef] - Vasquez, J.; Palacios, A.; Marre, O.; Berry, M., II; Cessac, B. Gibbs distribution analysis of temporal correlation structure on multicell spike trains from retina ganglion cells. J. Physiol. Paris
**2012**, 106, 120–127. [Google Scholar] [CrossRef] [PubMed] - Mora, T.; Deny, S.; Marre, O. Dynamical criticality in the collective activity of a population of retinal neurons. Phys. Rev. Lett.
**2015**, 114, 078105. [Google Scholar] [CrossRef] - Cofré, R.; Cessac, B. Exact computation of the maximum entropy potential of spiking neural networks models. Phys. Rev. E
**2014**, 89, 052117. [Google Scholar] [CrossRef] - Cofré, R.; Maldonado, C. Information entropy production of maximum entropy Markov chains from spike trains. Entropy
**2018**, 20, 34. [Google Scholar] [CrossRef] - Schulman, L.S. Time’s Arrows and Quantum Measurement; Cambridge University Press: Cambridge, UK, 1997. [Google Scholar]
- Jiang, D.Q.; Qian, M.; Qian, M.P. Mathematical Theory of Non-Equilibrium Steady States; Springer: Berlin/Heidelberg, Germany, 2004. [Google Scholar]
- Schrödinger, E. What Is Life? The Physical Aspect of the Living Cell; Cambridge University Press: Cambridge, UK, 1944. [Google Scholar]
- Prigogine, I. Nonequilibrium Statistical Mechanics; Monographs in Statistical Physics; Interscience publishers, John Wiley & Sons: Hoboken, NJ, USA, 1962. [Google Scholar]
- Deem, M. Mathematical adventures in biology. Phys. Today
**2007**, 60, 42–47. [Google Scholar] [CrossRef] - Filyukov, A.; Karpov, V. Description of steady transport processes by the method of the most probable path of evolution. Inzhenerno-Fizicheskii Zhurnal
**1967**, 13, 624–630. [Google Scholar] [CrossRef] - Filyukov, A.; Karpov, V. Method of the most probable path of evolution in the theory of stationary irreversible processes. Inzhenerno-Fizicheskii Zhurnal
**1967**, 13, 798–804. [Google Scholar] [CrossRef] - Favretti, M. The maximum entropy rate description of a thermodynamic system in a stationary non-equilibrium state. Entropy
**2009**, 4, 675–687. [Google Scholar] [CrossRef] - Monthus, C. Non-equilibrium steady states: maximization of the Shannon entropy associated with the distribution of dynamical trajectories in the presence of constraints. J. Stat. Mech. Theor. Exp.
**2011**, 3, P03008. [Google Scholar] [CrossRef] - Shi, P.; Qian, H. Frontiers in Computational and Systems Biology; Feng, J., Fu, W., Sun, F., Eds.; Springer: London, UK, 2010; chapter Irreversible Stochastic Processes, Coupled Diffusions and Systems Biochemistry; pp. 175–201. [Google Scholar]
- Galves, A.; Löcherbach, E. Infinite systems of interacting chains with memory of variable length-A stochastic model for biological neural nets. J. Stat. Phys.
**2013**, 151, 896–921. [Google Scholar] [CrossRef] - Cofré, R.; Cessac, B. Dynamics and spike trains statistics in conductance-based Integrate-and-Fire neural networks with chemical and electric synapses. Chaos Solitons Fractals
**2013**, 50, 13–31. [Google Scholar] [CrossRef] - Halmos, P.R. Measure Theory; Graduate Texts in Mathematics; Springer: New York, NY, USA, 1974. [Google Scholar]
- Levin, D.; Peres, Y. Markov Chains and Mixing Times, 2nd ed.; American Mathematical Society: Providence, RI, USA, 2017. [Google Scholar]
- Gerstner, W.; Kistler, W. Spiking Neuron Models; Cambridge University Press: Cambridge, UK, 2002. [Google Scholar]
- Jones, G.L. On the Markov chain central limit theorem. Probab. Surv.
**2004**, 1, 299–320. [Google Scholar] [CrossRef] - Ellis, R. Entropy, Large Deviations and Statistical Mechanics; Springer: Berlin, Germany, 1985. [Google Scholar]
- Touchette, H. The large deviation approach to statistical mechanics. Phys. Rep.
**2009**, 478, 1–69. [Google Scholar] [CrossRef][Green Version] - Dembo, A.; Zeitouni, O. Large deviations techniques and applications. In Stochastic Modelling and Applied Probability; Springer: Berlin, Germany, 2010; Volume 38. [Google Scholar]
- Marre, O.; Amodei, D.; Deshmukh, N.; Sadeghi, K.; Soo, F.; Holy, T.; Berry, M., II. Mapping a complete neural population in the Retina. J. Neurosci.
**2012**, 43, 14859–14873. [Google Scholar] [CrossRef] - Tkačik, G.; Mora, T.; Marre, O.; Amodei, D.; Berry, M., II; Bialek, W. Thermodynamics for a network of neurons: Signatures of criticality. Proc Natl. Acad. Sci. USA
**2015**, 112, 11508–11513. [Google Scholar] [CrossRef] - Granot-Atedgi, E.; Tkačik, G.; Segev, R.; Schneidman, E. Stimulus-dependent maximum entropy models of neural population codes. PLoS Comput. Biol.
**2013**, 9, e1002922. [Google Scholar] [CrossRef] - Shimazaki, H.; Amari, S.; Brown, E.N.; Grün, S. State-space analysis of time-varying higher-order spike correlation for multiple neural spike train data. PLoS Comput. Biol.
**2012**, 8, e1002385. [Google Scholar] [CrossRef] - Dewar, R. Information theory explanation of the fluctuation theorem, maximum entropy production and self-organized criticality in non-equilibrium stationary states. J. Phys. A Math. Gen.
**2003**, 36, 631. [Google Scholar] [CrossRef] - Dewar, R. Maximum entropy production and the fluctuation theorem. J. Phys. A Math. Gen.
**2005**, 38, L371. [Google Scholar] [CrossRef] - Martyushev, L.; Seleznev, V. Maximum entropy production principle in physics, chemistry and biology. Phys. Rep.
**2006**, 426, 1–45. [Google Scholar] [CrossRef] - Jaynes, E. The minimum entropy production principle. Ann. Rev. Phys. Chem.
**1980**, 31, 579–601. [Google Scholar] [CrossRef] - Pressé, S.; Ghosh, K.; Lee, J.; Dill, K.A. Principles of maximum entropy and maximum caliber in statistical physics. Rev. Mod. Phys.
**2013**, 85, 1115. [Google Scholar] [CrossRef] - Ruelle, D. Thermodynamic Formalism; Addison-Wesley: Reading, MA, USA, 1978. [Google Scholar]
- Jaynes, E. Information theory and statistical mechanics. Phys. Rev.
**1957**, 106, 620. [Google Scholar] [CrossRef] - Tkačik, G.; Marre, O.; Mora, T.; Amodei, D.; Berry, M., II; Bialek, W. The simplest maximum entropy model for collective behavior in a neural network. J. Stat. Mech.
**2013**, 2013, P03011. [Google Scholar] [CrossRef] - Seneta, E. Non-Negative Matrices and Markov Chains; Springer: New York, NY, USA, 2006. [Google Scholar]
- Walters, P. Ruelle’s operator theorem and g-measures. Trans. Am. Math. Soc.
**1975**, 214, 375–387. [Google Scholar] - Bowen, R. Equilibrium States and the Ergodic Theory of Anosov Diffeomorphisms. In Lecture Notes in Mathematics, revised ed.; Springer: Berlin, Germany, 2008; Volume 470. [Google Scholar]
- Parry, W.; Pollicott, M. Zeta functions and the periodic orbit structure of hyperbolic dynamics. Astérisque, Société mathématique de France
**1990**, 187–188. Available online: http://www.numdam.org/issue/AST_1990__187-188__1_0.pdf (accessed on 11 September 2019). - Chazottes, J. Fluctuations of observables in dynamical systems: From limit theorems to concentration inequalities. In Nonlinear Dynamics New Directions; González-Aguilar, H., Ugalde, E., Eds.; Springer: Cham, Switzerland, 2015; pp. 47–85. [Google Scholar]
- Gaspard, P. Chaos, Scattering and Statistical Mechanics; Non-Linear Science series; Cambridge University Press: Cambridge, UK, 1998; Volume 9. [Google Scholar]
- Bettolo, U.M.; Puglisi, A.; Rondoni, L.; Vulpiani, A. Fluctuation–dissipation: Response theory in statistical physics. Phys. Rep.
**2008**, 461, 111–195. [Google Scholar] - Gaspard, P. Random paths and current fluctuations in nonequilibrium statistical mechanics. J. Math. Phys.
**2014**, 55, 075208. [Google Scholar] [CrossRef][Green Version] - Cofré, R.; Maldonado, C.; Rosas, F. Large deviations properties of maximum entropy Markov shains from spike trains. Entropy
**2018**, 20, 573. [Google Scholar] [CrossRef] - Touchette, H. A Basic Introduction to Large Deviations: Theory, Applications, Simulations. arXiv
**2012**, arXiv:1106.4146v3. [Google Scholar] - Ellis, R.S. The theory of large deviations and applications to statistical mechanics. In Long-Range Interacting Systems; Oxford University Press: Oxford, UK, 2010. [Google Scholar]
- Nicolis, G.; Nicolis, C. Foundations of Complex Systems: Emergence, Information and Prediction; World Scientific: Singapore, 2012. [Google Scholar]
- Maes, C. The fluctuation theorem as a Gibbs property. J. Stat. Phys.
**1999**, 95, 367–392. [Google Scholar] [CrossRef] - Nasser, H.; Cessac, B. Parameter estimation for spatio-temporal maximum entropy distributions: Application to neural spike trains. Entropy
**2014**, 16, 2244–2277. [Google Scholar] [CrossRef]

**Figure 1.**Illustration of a spike train, a spiking state and spike pattern. The time bin size $\Delta {t}_{b}$ determine the binary patterns.

**Figure 2.**Plot of the auto-correlation of the observable ${x}_{0}^{2}{x}_{1}^{1}$ with respect to the MEMC consistent with constraints $\langle {x}_{0}^{1}{x}_{1}^{2}\rangle =0.1$ and $\langle {x}_{0}^{2}{x}_{1}^{1}\rangle =0.3$. The plot show the sum of Equation (18) from $r=1$ up to the number in the abscissa. Note the fast convergence towards ${\chi}_{22}$.

**Figure 3.**Auto-correlations of the observable ${x}_{0}^{2}{x}_{1}^{1}$ for the MEMC with the same parameters as in Figure 1. Modulations in the decay of correlations are due to the complex eigenvalues of the MEMC.

**Figure 4.**Rate functions of observables ${x}_{0}^{1}{x}_{1}^{2}$ in red, and ${x}_{0}^{2}{x}_{1}^{1}$ in blue for the MEMC consistent with constraints $\langle {x}_{0}^{1}{x}_{1}^{2}\rangle =0.1$ and $\langle {x}_{0}^{2}{x}_{1}^{1}\rangle =0.3$. The minimum value of both functions coincide with their expected values with respect to the MEMC. Around the minimum Gaussian fluctuations are expected (9). Far from the expected values are the large deviations.

**Figure 5.**IEP for the MEMC of the example (Section 4.4) for different values of parameters ${h}_{1},{h}_{2}$. Observe that $IEP(P,\pi )=0$ when ${h}_{1}={h}_{2}$ and that increases as the parameters become more different (more asymmetry in P).

**Figure 6.**Gallavotti-Cohen symmetry property for the SCGF and rate function of the IEP (Equation (25)). Left: SCGF of the IEP of the MEMC with the same parameters considered in the previous examples. Right: Rate function of the observable W, the minimum is attained at the expected value of IEP.

**Figure 7.**Linear response for the MEMC of the example (Section 4.4) for different values of perturbations $\delta {h}_{1}$ and $\delta {h}_{2}$. The colors represent $\parallel {\mathbb{E}}_{\tilde{\mu}}\left[{f}_{k}\right]-{\mathbb{E}}_{\mu}\left[{f}_{k}\right]\parallel $ computed using two methods. The “forward” method consists in computing ${\mathbb{E}}_{\tilde{\mu}}\left[{f}_{k}\right]$ from $\tilde{\mu}$ and ${\mathbb{E}}_{\mu}\left[{f}_{k}\right]$ from $\mu $. The figure in the middle is obtained by computing $\parallel {\mathbb{E}}_{\tilde{\mu}}\left[{f}_{k}\right]-{\mathbb{E}}_{\mu}\left[{f}_{k}\right]\parallel $ from $\chi $ using Equation (28). (

**Right**) The difference between both methods illustrated in a scatter plot in logarithmic scale.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Cofré, R.; Videla, L.; Rosas, F. An Introduction to the Non-Equilibrium Steady States of Maximum Entropy Spike Trains. *Entropy* **2019**, *21*, 884.
https://doi.org/10.3390/e21090884

**AMA Style**

Cofré R, Videla L, Rosas F. An Introduction to the Non-Equilibrium Steady States of Maximum Entropy Spike Trains. *Entropy*. 2019; 21(9):884.
https://doi.org/10.3390/e21090884

**Chicago/Turabian Style**

Cofré, Rodrigo, Leonardo Videla, and Fernando Rosas. 2019. "An Introduction to the Non-Equilibrium Steady States of Maximum Entropy Spike Trains" *Entropy* 21, no. 9: 884.
https://doi.org/10.3390/e21090884