# Bundled Causal History Interaction

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methodology

#### 2.1. Interactive Information Flow from Two Bundled Variables

#### 2.2. Two-Stage Dimensionality Reduction

**Stage 1: From infinite to finite cardinality—a probabilistic graphical model approach.**Figure 2 illustrates the use of a DAG for time-series representation. The DAG, $\mathcal{G}=({\overrightarrow{X}}_{:t},E)$, includes a set of directed edges E and a set of nodes ${\overrightarrow{X}}_{:t}$ connected by edges in E. Every state in ${\overrightarrow{X}}_{:t}$ is represented by a node in $\mathcal{G}$, and a directed edge in E connecting from an earlier node ${X}_{t-{\tau}_{X}}$ to a recent node ${Y}_{t-{\tau}_{Y}}$ ($0\le {\tau}_{Y}<{\tau}_{X}$), ${X}_{t-{\tau}_{X}}\to {Y}_{t-{\tau}_{Y}}$, refers to the direct influence from ${X}_{t-{\tau}_{X}}$ to ${Y}_{t-{\tau}_{Y}}$. An illustration of using the DAG for time-series to depict the temporal multivariate dynamics is shown in Figure 2a through a system consisting of seven components, $\{{X}_{t}^{({m}_{1})},{X}_{t}^{({m}_{2})},{X}_{t}^{({n}_{1})},{X}_{t}^{({n}_{2})},{X}_{t}^{({r}_{1})},{X}_{t}^{({r}_{2})},{X}_{t}^{({r}_{3})}\}$. We consider ${X}_{t}^{({r}_{1})}$ as the target, ${\overrightarrow{X}}_{:t}^{m}=\{{X}_{:t}^{({m}_{1})},{X}_{:t}^{({m}_{2})}\}$ as the first bundled causal history, and ${\overrightarrow{X}}_{:t}^{n}=\{{X}_{:t}^{({n}_{1})},{X}_{:t}^{({n}_{2})}\}$ as the second bundled causal history. The nodes involved in the bundled causal histories are highlighted in blue for immediate bundled causal history ${\overrightarrow{X}}_{t-{\tau}_{\mathbb{C}}:t}^{mn}$ up to the partitioning time lag ${\tau}_{\mathbb{C}}$, and in orange for the remaining distant bundled causal history ${\overrightarrow{X}}_{:t-{\tau}_{\mathbb{C}}}^{mn}$. The historical states outsize of those two bundled sets in the system, ${\overrightarrow{X}}_{:t}^{\mathrm{rest}}=\{{X}_{t}^{({r}_{1})},{X}_{t}^{({r}_{2})},{X}_{t}^{({r}_{3})}\}$, are denoted as gray nodes.

**Stage 2: From high to low cardinality—MIWTR approach.**The cardinality of Equation (5) can be potentially high in a strongly interacting multivariate system, leading to higher uncertainty in the estimation of information measures. Here, we adopt a recently-proposed Momentary Information Weighted Transitive Reduction approach [12] to further reduce the dimensionality of Equation (5) by simplifying the DAG. The basic idea of MIWTR is to first exclude any “redundant” edges connecting a node in ${\overrightarrow{W}}_{{\tau}_{\mathbb{C}}}$ with node in immediate history ${\overrightarrow{X}}_{t-{\tau}_{\mathbb{C}}:t}^{mn}$ by using weighted transitive reduction, and then remove any node in ${\overrightarrow{W}}_{{\tau}_{\mathbb{C}}}$ which are now not directly linked to the nodes in ${\overrightarrow{X}}_{t-{\tau}_{\mathbb{C}}:t}^{mn}$, thereby resulting in reduced cardinality of ${\overrightarrow{W}}_{{\tau}_{\mathbb{C}}}$. Here, the edge weight, representing the information flowing through the edge, is measured by momentary information transfer [8] which quantifies the shared dependency between two linked nodes conditioned on their parents. The “redundancy” of a directed edge linking two nodes by using WTR, say ${X}_{t-{\tau}_{X}}$ to ${Y}_{t-{\tau}_{Y}}$, is assessed according to the existence of an indirect path connecting ${X}_{t-{\tau}_{X}}$ and ${Y}_{t-{\tau}_{Y}}$ as well as the weights of the edges involved. That is, a directed edge, ${X}_{t-{\tau}_{X}}\to {Y}_{t-{\tau}_{Y}}$, is considered “redundant” and thus removed, if and only if there exists a path indirectly linking ${X}_{t-{\tau}_{X}}$ and ${Y}_{t-{\tau}_{Y}}$ and the minimum weight of all the edges in this indirect pathway is larger than that of ${X}_{t-{\tau}_{X}}\to {Y}_{t-{\tau}_{Y}}$. In other words, the existence of an indirect pathway, whose capacity of conveying information from ${X}_{t-{\tau}_{X}}$ to ${Y}_{t-{\tau}_{Y}}$ is stronger than the direct channel between the two nodes, makes the direct edge ${X}_{t-{\tau}_{X}}\to {Y}_{t-{\tau}_{Y}}$ obsolete. More details of MIWTR can be found in [12].

## 3. Application: Bundled Causal Interaction in Stream Chemistry Dynamics

^{+}, Al

^{3+}, Ca

^{2+}} as the first bundled set, the anions {Cl

^{−}, SO4

^{2−}} as the second bundled set, and {pH, $lnQ$ (the logarithm of flow rate)} as the remaining variables. Based on the observed data shown in Figure 3a, we constructed the DAG for time-series by using Tigramite algorithm [8,18,19,20]. Generally, the algorithm first builds up preliminary links between nodes by using mutual information-based independence test, and then removes any spurious links by using CMI-based independence test by conditioning on the parents of the connected two nodes. The resulting DAG is shown in Figure 3b, with the estimation methodology for the graph detailed in [10].

## 4. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Tononi, G.; Edelman, G.M. Consciousness and Complexity. Science
**1998**, 282, 1846–1851. [Google Scholar] [CrossRef] [PubMed] - Kirchner, J.W.; Neal, C. Universal fractal scaling in stream chemistry and its implications for solute transport and water quality trend detection. Proc. Natl. Acad. Sci. USA
**2013**, 110, 12213–12218. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Granger, C.W.J. Investigating Causal Relations by Econometric Models and Cross-spectral Methods. Econometrica
**1969**, 37, 424–438. [Google Scholar] [CrossRef] - Pearl, J. Causal diagrams for empirical research. Biometrika
**1995**, 82, 669–688. [Google Scholar] [CrossRef] - Sugihara, G.; May, R.; Ye, H.; Hsieh, C.H.; Deyle, E.; Fogarty, M.; Munch, S. Detecting Causality in Complex Ecosystems. Science
**2012**, 338, 496–500. [Google Scholar] [CrossRef] - Imbens, G.W.; Rubin, D.B. Causal Inference in Statistics, Social, and Biomedical Sciences; Cambridge University Press: Cambridge, UK, 2015. [Google Scholar] [CrossRef]
- Schreiber, T. Measuring Information Transfer. Phys. Rev. Lett.
**2000**, 85, 461–464. [Google Scholar] [CrossRef] [Green Version] - Runge, J.; Heitzig, J.; Petoukhov, V.; Kurths, J. Escaping the Curse of Dimensionality in Estimating Multivariate Transfer Entropy. Phys. Rev. Lett.
**2012**, 108, 258701. [Google Scholar] [CrossRef] - Jiang, P.; Kumar, P. Interactions of information transfer along separable causal paths. Phys. Rev. E
**2018**, 97, 042310. [Google Scholar] [CrossRef] [Green Version] - Jiang, P.; Kumar, P. Information transfer from causal history in complex system dynamics. Phys. Rev. E
**2019**, 99, 012306. [Google Scholar] [CrossRef] [Green Version] - Williams, P.L.; Beer, R.D. Nonnegative Decomposition of Multivariate Information. arXiv
**2010**, arXiv:1004.2515. [Google Scholar] - Jiang, P.; Kumar, P. Using Information Flow for Whole System Understanding From Component Dynamics. Water Resour. Res.
**2019**, 55, 8305–8329. [Google Scholar] [CrossRef] - Shannon, C.E.; Weaver, W. A Mathematical Theory of Communication; University of Illinois Press: Urbana, IL, USA, 1949. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley Series in Telecommunications and Signal Processing; Wiley-Interscience: Hoboken, NJ, USA, 2006. [Google Scholar]
- Eichler, M. Graphical modelling of multivariate time series. Probab. Theory Relat. Fields
**2012**, 153, 233–268. [Google Scholar] [CrossRef] [Green Version] - Bosnacki, D.; Lightenberg, W.; Odenbrett, M.; Wijs, A.; Hilbers, P. Parallel algorithms for transitive reduction of weighted graphs. Math. Maced.
**2010**, 8, 95–106. [Google Scholar] - Lauritzen, S.L.; Dawid, A.P.; Larsen, B.N.; Leimer, H. Independence properties of directed markov fields. Networks
**1990**, 20, 491–505. [Google Scholar] [CrossRef] - Runge, J. Quantifying information transfer and mediation along causal pathways in complex systems. Phys. Rev. E
**2015**, 92, 062829. [Google Scholar] [CrossRef] [Green Version] - Runge, J.; Petoukhov, V.; Donges, J.F.; Hlinka, J.; Jajcay, N.; Vejmelka, M.; Hartman, D.; Marwan, N.; Paluš, M.; Kurths, J. Identifying causal gateways and mediators in complex spatio-temporal systems. Nat. Commun.
**2015**, 6, 8502. [Google Scholar] [CrossRef] [Green Version] - Runge, J.; Sejdinovic, D.; Flaxman, S. Detecting causal associations in large nonlinear time series datasets. arXiv
**2017**, arXiv:1702.07007. [Google Scholar] [CrossRef] [Green Version] - Neal, C.; Reynolds, B.; Kirchner, J.W.; Rowland, P.; Norris, D.; Sleep, D.; Lawlor, A.; Woods, C.; Thacker, S.; Guyatt, H.; et al. High-frequency precipitation and stream water quality time series from Plynlimon, Wales: An openly accessible data resource spanning the periodic table. Hydrol. Process.
**2013**, 27, 2531–2539. [Google Scholar] [CrossRef] [Green Version] - Kraskov, A.; Stögbauer, H.; Grassberger, P. Estimating mutual information. Phys. Rev. E
**2004**, 69, 066138. [Google Scholar] [CrossRef] [Green Version] - Khan, S.; Bandyopadhyay, S.; Ganguly, A.R.; Saigal, S.; Erickson, D.J.; Protopopescu, V.; Ostrouchov, G. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Phys. Rev. E
**2007**, 76, 026209. [Google Scholar] [CrossRef] [Green Version] - Walters-Williams, J.; Li, Y. Estimation of Mutual Information: A Survey. In Rough Sets and Knowledge Technology; Wen, P., Li, Y., Polkowski, L., Yao, Y., Tsumoto, S., Wang, G., Eds.; Springer: Berlin/Heidelberg, Germany, 2009; pp. 389–396. [Google Scholar]
- Goodwell, A.E.; Kumar, P. Temporal information partitioning: Characterizing synergy, uniqueness, and redundancy in interacting environmental variables. Water Resour. Res.
**2017**, 53, 5920–5942. [Google Scholar] [CrossRef]

**Figure 1.**(color online) Illustration of pairwise interaction (

**a**) and different types of multivariate interactions (

**b**–

**d**): (

**a**) the influence from a source variable to a target variable; and (

**b**–

**d**) the influences on the target variable from two individual variables, a group of variables, and two groups of variables, respectively.

**Figure 2.**(color online) Illustration of bundled causal history analysis framework by using a system consisting of seven components: (

**a**) the influence from immediate (${\overrightarrow{X}}_{t-{\tau}_{\mathbb{C}}:t}^{mn}$) and distant (${\overrightarrow{X}}_{:t-{\tau}_{\mathbb{C}}}^{mn}$) bundled causal histories to the target (${X}_{t}^{\mathrm{tar}}$) in Equation (3); and (

**b**) the different components in Equation (5) based on using the Markov property for DAG as well as the Order-1 approximation ${\overrightarrow{F}}_{1}$ for ${\overrightarrow{F}}_{}$.

**Figure 3.**(color online) Illustration of using Momentary Information Weighted Transitive Reduction (MIWTR) to reduce the dimensionality of ${\overrightarrow{W}}_{{\tau}_{\mathbb{C}}}$ for the present state of pH influenced by the selected cation and anion groups, with ${\tau}_{\mathbb{C}}=6$. (

**a**) The used stream chemistry time-series data recorded in the Upper Hafren catchment in the United Kingdom [21]. (

**b**) The estimated directed acyclic graph (DAG) using Tigramite algorithm [8,18,19,20], with the original identified ${\overrightarrow{W}}_{{\tau}_{\mathbb{C}}}$ (orange nodes) and $\overrightarrow{V}$ (blue nodes) in Equation (5) (the edges removed by MIWTR are highlighted in magenta). (

**c**) The reduced ${\overrightarrow{W}}_{{\tau}_{\mathbb{C}}}$ (orange nodes) using MIWTR.

**Figure 4.**(color online) Plots of partial information decomposition of the immediate (${\mathcal{J}}_{mn}$) and distant (${\mathcal{D}}_{mn}$) bundled causal histories based on the stream solute time-series data and the estimated directed acyclic graph for time-series in Figure 2 under the Order-0 (

**left**) and Order-1 (

**right**) of approximation for ${\overrightarrow{F}}_{}$ by using k-nearest-neighbor estimators with $k\in \{5,6,7,8,9,10,15\}$.

**Figure 5.**(color online) Plots of partial information decomposition of the immediate (${\mathcal{J}}_{mn}$) and distant (${\mathcal{D}}_{mn}$) bundled causal histories based on the stream solute time-series data and the estimated directed acyclic graph for time-series in Figure 3b with the Order-0 (

**top**) and Order-1 (

**bottom**) of approximation for ${\overrightarrow{F}}_{}$. The ${\mathcal{J}}_{mn}$ and ${\mathcal{D}}_{mn}$ (Equation (4a,b)) components are separated by a black dotted line.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jiang, P.; Kumar, P.
Bundled Causal History Interaction. *Entropy* **2020**, *22*, 360.
https://doi.org/10.3390/e22030360

**AMA Style**

Jiang P, Kumar P.
Bundled Causal History Interaction. *Entropy*. 2020; 22(3):360.
https://doi.org/10.3390/e22030360

**Chicago/Turabian Style**

Jiang, Peishi, and Praveen Kumar.
2020. "Bundled Causal History Interaction" *Entropy* 22, no. 3: 360.
https://doi.org/10.3390/e22030360