# The Liang-Kleeman Information Flow: Theory and Applications

^{1}

^{2}

## Abstract

**:**

## 1. Introduction

## 2. Mathematical Formalism

#### 2.1. Theoretical Framework

#### 2.2. Toward a Rigorous Formalism—A Heuristic Argument

#### 2.3. Mathematical Formalism

## 3. Discrete Systems

#### 3.1. Frobenius-Perron Operator

**Figure 1.**Illustration of the Frobenius-Perron operator $\mathcal{P}$, which takes $\rho \left(\mathit{x}\right)$ to $\mathcal{P}\rho \left(\mathit{x}\right)$ as Φ takes $\mathit{x}$ to $\Phi \mathit{x}$.

#### 3.2. Information Flow

#### 3.3. Properties

**Theorem 3.1**

**Theorem 3.2**

## 4. Continuous Systems

- Discretize the continuous system in time on $[t,t+\Delta t]$, and construct a mapping Φ to take $\mathit{x}\left(t\right)$ to $\mathit{x}(t+\Delta t)$;
- Freeze ${x}_{2}$ in Φ throughout $[t,t+\Delta t]$ to obtain a modified mapping ${\Phi}_{\setminus \phantom{\rule{-3.00003pt}{0ex}}2}$;
- Compute the marginal entropy change $\Delta {H}_{1}$ as Φ steers the system from t to $t+\Delta t$;
- Derive the marginal entropy change $\Delta {H}_{1\setminus \phantom{\rule{-3.00003pt}{0ex}}2}$ as ${\Phi}_{\setminus \phantom{\rule{-3.00003pt}{0ex}}2}$ steers the modified system from t to $t+\Delta t$;
- Take the limit$${T}_{2\to 1}=\underset{\Delta t\to 0}{lim}\frac{\Delta {H}_{1}-\Delta {H}_{1\setminus \phantom{\rule{-3.00003pt}{0ex}}2}}{\Delta t}$$

#### 4.1. Discretization of the Continuous System

#### 4.2. Information Flow

**Proposition 4.1**

**Theorem 4.1**

#### 4.3. Properties

**Theorem 4.2**

**Theorem 4.3 (Causality)**

## 5. Stochastic Systems

**Proposition 5.1**

**Theorem 5.1**

**Theorem 5.2**

**Theorem 5.3**

## 6. Applications

#### 6.1. Baker Transformation

#### 6.2. H$\stackrel{\xb4}{\mathrm{e}}$non Map

**Figure 3.**A trajectory of the canonical H$\stackrel{\xb4}{\mathrm{e}}$non map ($a=1.4$, $b=0.3$) starting at $({x}_{1},{x}_{2})=(1,0)$.

#### 6.3. Truncated Burgers–Hopf System

- Initialize the joint density of $({x}_{1},{x}_{2},{x}_{3},{x}_{4})$ with some distribution ${\rho}_{0}$; make random draws according to ${\rho}_{0}$ to form an ensemble. The ensemble should be large enough to resolve adequately the sample space.
- Discretize the sample space into “bins.”
- Do ensemble prediction for the system (73)–(74).
- At each step, estimate the probability density function ρ by counting the bins.
- Plug the estimated ρ back to Equation (39) to compute the rates of information flow at that step.

**Figure 4.**The invariant attractor of the truncated Burgers–Hopf system (73)–(76). Shown here is the trajectory segment for $2\le t\le 20$ starting at $(40,40,40,40)$. (a) and (b) are the 3-dimensional projections onto the subspaces ${x}_{1}$-${x}_{2}$-${x}_{3}$ and ${x}_{2}$-${x}_{3}$-${x}_{4}$, respectively.

**Figure 5.**Information flows within the 4D truncated Burgers-Hopf system. The series prior to $t=2$ are not shown because some trajectories have not entered the attractor by that time.

#### 6.4. Langevin Equation

**Figure 6.**A solution of Equation (78), the model examined in [54], with ${a}_{21}=0$ and initial conditions as shown in the text: (

**a**) $\mathbf{\mu}$; (

**b**) Σ; and (

**c**) a sample path starting from (1,2).

**Figure 7.**The computed rates of information flow for the system (77): (a) ${T}_{2\to 1}$, (b) ${T}_{1\to 2}$.

## 7. Summary

## Acknowledgments

## References

- Baptista, M.S.; Garcia, S.P.; Dana, S.K.; Kurths, J. Transmission of information and synchronization in a pair of coupled chaotic circuits: An experimental overview. Eur. Phys. J.-Spec. Top.
**2008**, 165, 119–128. [Google Scholar] [CrossRef] - Baptista, M.D.S.; Kakmeni, F.M.; Grebogi, C. Combined effect of chemical and electrical synapses in Hindmarsh-Rose neural networks on synchronization and the rate of information. Phys. Rev. E
**2010**, 82, 036203. [Google Scholar] [CrossRef] [PubMed] - Bear, M.F.; Connors, B.W.; Paradiso, M.A. Neuroscience: Exploring the Brain, 3rd ed.; Lippincott Williams & Wilkins: Baltimore, MD, USA, 2007; p. 857. [Google Scholar]
- Vakorin, V.A.; MiAiA, B.; Krakovska, O.; McIntosh, A.R. Empirical and theoretical aspects of generation and transfer of information in a neuromagnetic source network. Front. Syst. Neurosci.
**2011**, 5, 96. [Google Scholar] [CrossRef] [PubMed] - Ay, N.; Polani, D. Information flows in causal networks. Advs. Complex Syst.
**2008**, 11. [Google Scholar] [CrossRef] - Peruani, F.; Tabourier, L. Directedness of information flow in mobile phone communication networks. PLoS One
**2011**, 6, e28860. [Google Scholar] [CrossRef] [PubMed] - Sommerlade, L.; Amtage, F.; Lapp, O.; Hellwig, B.; Licking, C.H.; Timmer, J.; Schelter, B. On the estimation of the direction of information flow in networks of dynamical systems. J. Neurosci. Methods
**2011**, 196, 182–189. [Google Scholar] [CrossRef] [PubMed] - Donner, R.; Barbosa, S.; Kurths, J.; Marwan, N. Understanding the earth as a complex system-recent advances in data analysis and modelling in earth sciences. Eur. Phys. J.
**2009**, 174, 1–9. [Google Scholar] [CrossRef] - Kleeman, R. Information flow in ensemble weather prediction. J. Atmos. Sci.
**2007**, 64, 1005–1016. [Google Scholar] [CrossRef] - Materassi, M.; Ciraolo, L.; Consolini, G.; Smith, N. Predictive space weather: An information theory approach. Adv. Space Res.
**2011**, 47, 877–885. [Google Scholar] [CrossRef] - Tribbia, J.J. Waves, Information and Local Predictability. In Proceedings of the Workshop on Mathematical Issues and Challenges in Data Assimilation for Geophysical Systems: Interdisciplinary Perspectives, IPAM, UCLA, 22–25 February 2005.
- Chen, C.R.; Lung, P.P.; Tay, N.S.P. Information flow between the stock and option markets: Where do informed traders trade? Rev. Financ. Econ.
**2005**, 14, 1–23. [Google Scholar] [CrossRef] - Lee, S.S. Jumps and information flow in financial markets. Rev. Financ. Stud.
**2012**, 25, 439–479. [Google Scholar] [CrossRef] - Sommerlade, L.; Eichler, M.; Jachan, M.; Henschel, K.; Timmer, J.; Schelter, B. Estimating causal dependencies in networks of nonlinear stochastic dynamical systems. Phys. Rev. E
**2009**, 80, 051128. [Google Scholar] [CrossRef] [PubMed] - Zhao, K.; Karsai, M.; Bianconi, G. Entropy of dynamical social networks. PLoS One
**2011**. [Google Scholar] [CrossRef] [PubMed] - Cane, M.A. The evolution of El Niño, past and future. Earth Planet. Sci. Lett.
**2004**, 164, 1–10. [Google Scholar] [CrossRef] - Jin, F.-F. An equatorial ocean recharge paradigm for ENSO. Part I: conceptual model. J. Atmos. Sci.
**1997**, 54, 811–829. [Google Scholar] [CrossRef] - Philander, S.G. El Niño, La Niña, and the Southern Oscillation; Academic Press: San Diego, CA, USA, 1990. [Google Scholar]
- Ghil, M.; Chekroun, M.D.; Simonnet, E. Climate dynamics and fluid mechanics: Natural variability and related uncertainties. Physica D
**2008**, 237, 2111–2126. [Google Scholar] [CrossRef] - Mu, M.; Xu, H.; Duan, W. A kind of initial errors related to “spring predictability barrier” for El Niño events in Zebiak-Cane model. Geophys. Res. Lett.
**2007**, 34, L03709. [Google Scholar] [CrossRef] - Zebiak, S.E.; Cane, M.A. A model El Niño-Southern Oscillation. Mon. Wea. Rev.
**1987**, 115, 2262–2278. [Google Scholar] [CrossRef] - Chen, D.; Cane, M.A. El Niño prediction and predictability. J. Comput. Phys.
**2008**, 227, 3625–3640. [Google Scholar] [CrossRef] - Mayhew, S.; Sarin, A.; Shastri, K. The allocation of informed trading across related markets: An analysis of the impact of changes in equity-option margin requirements. J. Financ.
**1995**, 50, 1635–1654. [Google Scholar] [CrossRef] - Goldenfield, N.; Woese, C. Life is physics: Evolution as a collective phenomenon far from equilibrium. Ann. Rev. Condens. Matt. Phys.
**2011**, 2, 375–399. [Google Scholar] [CrossRef] - K¨ppers, B. Information and the Origin of Life; MIT Press: Cambridge, UK, 1990. [Google Scholar]
- Murray, J.D. Mathematical Biology; Springer-Verlag: Berlin, Germany, 2000. [Google Scholar]
- Allahverdyan, A.E.; Janzing, D.; Mahler, G. Thermodynamic efficiency of information and heat flow. J. Stat. Mech.
**2009**, PO9011. [Google Scholar] [CrossRef] - Crutchfield, J.P.; Shalizi, C.R. Thermodynamic depth of causal states: Objective complexity via minimal representation. Phys. Rev. E
**1999**, 59, 275–283. [Google Scholar] [CrossRef] - Davies, P.C.W. The Physics of Downward Causation. In The Re-emergence of Emergence; Clayton, P., Davies, P.C.W., Eds.; Oxford University Press: Oxford, UK, 2006; pp. 35–52. [Google Scholar]
- Ellis, G.F.R. Top-down causation and emergence: Some comments on mechanisms. J. R. Soc. Interface
**2012**, 2, 126–140. [Google Scholar] [CrossRef] [PubMed] - Okasha, S. Emergence, hierarchy and top-down causation in evolutionary biology. J. R. Soc. Interface
**2012**, 2, 49–54. [Google Scholar] [CrossRef] [PubMed] - Walker, S.I.; Cisneros, L.; Davies, P.C.W. Evolutionary transitions and top-down causation. arXiv:1207.4808v1 [nlin.AO], 2012. [Google Scholar]
- Wu, B.; Zhou, D.; Fu, F.; Luo, Q.; Wang, L.; Traulsen, A. Evolution of cooperation on stochastic dynamical networks. PLoS One
**2010**, 5, e11187. [Google Scholar] [CrossRef] [PubMed] - Pope, S. Turbulent Flows, 8th ed.; Cambridge University Press: Cambridge, UK, 2011. [Google Scholar]
- Faes, L.; Nollo, G.; Erla, S.; Papadelis, C.; Braun, C.; Porta, A. Detecting Nonlinear Causal Interactions between Dynamical Systems by Non-uniform Embedding of Multiple Time Series. In Proceedings of the Engineering in Medicine and Biology Society, Buenos Aires, Argentina, 31 August–4 September 2010; pp. 102–105.
- Kantz, H.; Shreiber, T. Nonlinear Time Series Analysis; Cambridge University Press: Cambridge, UK, 2004. [Google Scholar]
- Schindler-Hlavackova, K.; Palus, M.; Vejmelka, M.; Bhattacharya, J. Causality detection based on information-theoretic approach in time series analysis. Phys. Rep.
**2007**, 441, 1–46. [Google Scholar] [CrossRef] - Granger, C. Investigating causal relations by econometric models and cross-spectral methods. Econometrica
**1969**, 37, 424–438. [Google Scholar] [CrossRef] - McWilliams, J.C. The emergence of isolated, coherent vortices in turbulence flows. J. Fluid Mech.
**1984**, 146, 21–43. [Google Scholar] [CrossRef] - Salmon, R. Lectures on Geophysical Fluid Dynamics; Oxford University Press: Oxford, UK, 1998; p. 378. [Google Scholar]
- Bar-Yam, Y. Dynamics of Complex Systems; Addison-Welsley Press: Reading, MA, USA, 1997; p. 864. [Google Scholar]
- Crutchfield, J.P. The calculi of emergence: computation, dynamics, and induction induction. “Special issue on the Proceedings of the Oji International Seminar: Complex Systems-From Complex Dynamics to Artifical Reality”. Physica D
**1994**, 75, 11–54. [Google Scholar] [CrossRef] - Goldstein, J. Emergence as a construct: History and issues. Emerg. Complex. Org.
**1999**, 1, 49–72. [Google Scholar] [CrossRef] - Corning, P.A. The re-emergence of emergence: A venerable concept in search of a theory. Complexity
**2002**, 7, 18–30. [Google Scholar] [CrossRef] - Vastano, J.A.; Swinney, H.L. Information transport in sptiotemporal systems. Phys. Rev. Lett.
**1988**, 60, 1773–1776. [Google Scholar] [CrossRef] [PubMed] - Kaiser, A.; Schreiber, T. Information transfer in continuous processes. Physica D
**2002**, 166, 43–62. [Google Scholar] [CrossRef] - Schreiber, T. Measuring information transfer. Phys. Rev. Lett.
**2000**, 85, 461. [Google Scholar] [CrossRef] [PubMed] - Liang, X.S.; Kleeman, R. A rigorous formalism of information transfer between dynamical system components. I. Discrete mapping. Physica D
**2007**, 231, 1–9. [Google Scholar] [CrossRef] - Liang, X.S.; Kleeman, R. Information transfer between dynamical system components. Phys. Rev. Lett.
**2005**, 95, 244101. [Google Scholar] [CrossRef] [PubMed] - Lasota, A.; Mackey, M.C. Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics; Springer: New York, NY, USA, 1994. [Google Scholar]
- Liang, X.S.; Kleeman, R. A rigorous formalism of information transfer between dynamical system components. II. Continuous flow. Physica D
**2007**, 227, 173–182. [Google Scholar] [CrossRef] - Liang, X.S. Uncertainty generation in deterministic fluid flows: Theory and applications with an atmospheric stability model. Dyn. Atmos. Oceans
**2011**, 52, 51–79. [Google Scholar] [CrossRef] - Majda, A.J.; Harlim, J. Information flow between subspaces of complex dynamical systems. Proc. Natl. Acad. Sci. USA
**2007**, 104, 9558–9563. [Google Scholar] [CrossRef] - Liang, X.S. Information flow within stochastic dynamical systems. Phys. Rev. E
**2008**, 78, 031113. [Google Scholar] [CrossRef] [PubMed] - Majda, A.J.; Timofeyev, I. Remarkable statistical behavior for truncated Burgers-Hopf dynamics. Proc. Natl. Acad. Sci. USA
**2000**, 97, 12413–12417. [Google Scholar] [CrossRef] [PubMed] - Gardiner, C.W. Handbook of Stochastic Methods for Physics, Chemistry, and the Natural Sciences; Springer-Verlag: Berlin/Heidelberg, Germany, 1985. [Google Scholar]
- Liang, X.S.; Anderson, D.G.M.A. Multiscale window transform. SIAM J. Multiscale Model. Simul.
**2007**, 6, 437–467. [Google Scholar] [CrossRef] - Liang, X.S.; Robinson, A.R. Multiscale processes and nonlinear dynamics of the circulation and upwelling events off Monterey Bay. J. Phys. Oceanogr.
**2009**, 39, 290–313. [Google Scholar] [CrossRef] - Kleeman, R. Measuring dynamical prediction utility using relative entropy. J. Atmos. Sci.
**2002**, 59, 2057–2072. [Google Scholar] [CrossRef] - Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley: New York, NY, USA, 1991. [Google Scholar]
- Ao, P. Emerging of stochastic dynamical equalities and steady state thermodynamics from Darwinian dynamics. Commun. Theor. Phys.
**2008**, 49, 1073–1090. [Google Scholar] [CrossRef] [PubMed] - Tang, Y.; Deng, Z.; Zhou, X.; Cheng, Y.; Chen, D. Interdecadal variation of ENSO predictability in multiple models. J. Clim.
**2008**, 21, 4811–4832. [Google Scholar] [CrossRef]

© 2013 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

## Share and Cite

**MDPI and ACS Style**

Liang, X.S. The Liang-Kleeman Information Flow: Theory and Applications. *Entropy* **2013**, *15*, 327-360.
https://doi.org/10.3390/e15010327

**AMA Style**

Liang XS. The Liang-Kleeman Information Flow: Theory and Applications. *Entropy*. 2013; 15(1):327-360.
https://doi.org/10.3390/e15010327

**Chicago/Turabian Style**

Liang, X. San. 2013. "The Liang-Kleeman Information Flow: Theory and Applications" *Entropy* 15, no. 1: 327-360.
https://doi.org/10.3390/e15010327