The Causal Interaction between Complex Subsystems

Abstract

Information flow provides a natural measure for the causal interaction between dynamical events. This study extends our previous rigorous formalism of componentwise information flow to the bulk information flow between two complex subsystems of a large-dimensional parental system. Analytical formulas have been obtained in a closed form. Under a Gaussian assumption, their maximum likelihood estimators have also been obtained. These formulas have been validated using different subsystems with preset relations, and they yield causalities just as expected. On the contrary, the commonly used proxies for the characterization of subsystems, such as averages and principal components, generally do not work correctly. This study can help diagnose the emergence of patterns in complex systems and is expected to have applications in many real world problems in different disciplines such as climate science, fluid dynamics, neuroscience, financial economics, etc.

1. Introduction

When investigating the properties of a complex system, it is often necessary to study the interaction between one subsystem and another subsystem, which themselves also form complex systems, usually with a large number of components involved. In climate science, for example, there is much interest in understanding how one sector of the system collaborates with another sector to cause climate change (see [1] and the references therein); in neuroscience, it is important to investigate the effective connectivity from one brain region to another, each with millions of neurons involved (e.g., [2,3]), and the interaction between structures (e.g., [4,5,6]; see more references in a recent review [7]). This naturally raises a question: How can we study the interaction between two subsystems in a large parental system?

An immediate answer coming to mind might be to study the componentwise interactions by assessing the causalities between the respective components using, for instance, the classical causal inference approaches (e.g., [8,9,10]). This is generally infeasible if the dimensionality is large. For two subsystems, each with, say, 1000 components, they end up with 1 million causal relations, making it impossible to analyze, albeit with all the details. In this case, the details are not a benefit; they need to be re-analyzed for a big, interpretable picture of the phenomena. On the other hand, in many situations, this is not necessary; one needs only a “bulk” description of the subsystems and their interactions. Such examples are seen from the Reynolds equations for turbulence (e.g., [11]) and the thermodynamic description of molecular motions (e.g., [12]). In some fields (e.g., climate science, neuroscience, geography, etc.), a common practice is simply to take respective averages and form the mean properties, and to study the interactions between the proxies, i.e., the mean properties. A more sophisticated approach is to extract the respective principal components (PCs) (e.g., [13,14,15]), based on which the interactions are analyzed henceforth. These approaches, as we will be examining in this study, however, may not work satisfactorily; their validities need to be carefully checked before being put into application.

During the past 16 years, it has been gradually realized that causality in terms of information flow (IF) is a real physical notion that can be rigorously derived from first principles (see [16]). When two processes interact, IF provides not only the direction but also the strength of the interaction. Thus far, the formalism of the IF between two components has been well established (see [16,17,18,19,20], among others). It has been shown promising to extend the formalism to subspaces with many components involved. A pioneering effort is [21], where the authors show that the heuristic argument in [17] equally applies to that between subsystems in the case with only one-way causality. A recent study on the role of individual nodes in a complex network [22] may be viewed as another effort. (Causality analyses between subspaces with the classical approaches are rare; a few examples are [23,24], etc.) However, a rigorous formalism for more generic problems (e.g., with mutual causality involved) is yet to be implemented. This makes the objective of this study, i.e., to study the interactions between two complex subsystems within a large parental system by investigating the “bulk” information flow between them.

The rest of the paper is organized as follows. In Section 2, we first present the setting for the problem and then derive the IF formulas. Maximum likelihood estimators of these formulas are given in Section 3, which is followed by a validation. Finally, Section 5 summarizes the study.

2. Information Flow between Two Subspaces of a Complex System

Consider an n-dimensional dynamical system

$$\begin{array}{ccc}& & A:\left\{\begin{array}{c}\frac{d{x}_1}{dt}=F_1(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{1,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \\ ⋮⋮\hfill \\ \frac{d{x}_r}{dt}=F_r(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{r,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \end{array}\right.\hfill \end{array}$$

(1)

$$\begin{array}{ccc}& & B:\left\{\begin{array}{c}\frac{d{x}_{r+1}}{dt}=F_{r+1}(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{r+1,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \\ ⋮⋮\hfill \\ \frac{d{x}_s}{dt}=F_s(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{s,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \end{array}\right.\hfill \end{array}$$

(2)

$$\begin{array}{ccc}& & \left\{\begin{array}{c}\frac{d{x}_{s+1}}{dt}=F_{s+1}(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{s+1,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \\ ⋮⋮\hfill \\ \frac{d{x}_n}{dt}=F_n(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{nk}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k.\hfill \end{array}\right.\hfill \end{array}$$

(3)

where $\mathbf{x}\in {ℝ}^n$ denotes the vector of state variable $(x_1,x_2,\dots ,x_n)$, $\mathbf{F}=(F_1,\dots ,F_n)$ are differentiable functions of $\mathbf{x}$ and time t, $\mathbf{w}$ is a vector of m independent standard Wiener processes, and $\mathbf{B}=\left(b_{ij}\right)$ is an $n\times m$ matrix of stochastic perturbation amplitudes. Here we follow the convention in physics not to distinguish a random variable from its deterministic counterpart. From the components $(x_1,\dots ,x_n)$, we separate out two sets, $(x_1,\dots ,x_r)$ and $(x_{r+1},\dots ,x_s)$, and denote them as ${\mathbf{x}}_{1\dots r}$ and ${\mathbf{x}}_{r+1,\dots ,s}$, respectively. The remaining components $(x_{s+1},\dots ,x_n)$ are denoted as ${\mathbf{x}}_{s+1,\dots ,n}$. The subsystems formed by them are henceforth referred to as A and B, and the following is a derivation of the information flow between them. Note that, for convenience, here A and B are put adjacent to each other; if not, the equations can always be rearranged to make them so.

Associated with Equations (1)–(3) there is a Fokker–Planck equation governing the evolution of the joint probability density function (pdf) $\rho$ of $\mathbf{x}$:

$$\begin{array}{c}\hfill \frac{\partial \rho }{\partial t}+\frac{\partial \rho F_1}{\partial x_1}+\frac{\partial \rho F_2}{\partial x_2}+\dots +\frac{\partial \rho F_n}{\partial x_n}=\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^n\frac{{\partial }^2{g}_{ij}\rho }{\partial x_i\partial x_j},\end{array}$$

(4)

where $g_{ij}={\sum }_{k=1}^m{b}_{ik}{b}_{jk}$, $i,j=1,\dots ,n$. Without much loss of generality, $\rho$ is assumed to be compactly supported on ${ℝ}^n$. The joint pdfs of ${\mathbf{x}}_{1\dots r}$ and ${\mathbf{x}}_{r+1,\dots ,s}$ are, respectively,

$$\begin{array}{ccc}& & {\rho }_{1\dots r}={\int }_{{ℝ}^{n-r}}\rho \left(\mathbf{x}\right)d{x}_{r+1}\dots d{x}_n\equiv {\int }_{{ℝ}^{n-r}}\rho \left(\mathbf{x}\right)d{\mathbf{x}}_{r+1,\dots ,n},\hfill \\ & & {\rho }_{r+1,\dots ,s}={\int }_{{ℝ}^{n-s+r}}\rho \left(\mathbf{x}\right)d{x}_1\dots d{x}_{r}d{x}_{s+1}\dots d{x}_n\equiv {\int }_{{ℝ}^{n-s+r}}\rho \left(\mathbf{x}\right)d{\mathbf{x}}_{1,\dots ,r,s+1,\dots ,n}.\hfill \end{array}$$

With respect to them, the joint entropies are then

$$\begin{array}{ccc}& & H_A=-{\int }_{{ℝ}^r}{\rho }_{1\dots r}log{\rho }_{1\dots r}d{\mathbf{x}}_{1\dots r},\hfill \end{array}$$

(5)

$$\begin{array}{ccc}& & H_B=-{\int }_{{ℝ}^{s-r}}{\rho }_{r+1,\dots ,s}log{\rho }_{r+1,\dots ,s}d{\mathbf{x}}_{r+1,\dots ,s}.\hfill \end{array}$$

(6)

To derive the evolution of ${\rho }_{1\dots r}$, integrate out $(x_{r+1},\dots ,x_n)$ in Equation (4). This yields, by using the assumption of compactness for $\rho$,

$$\begin{array}{ccc}\hfill & & \frac{\partial {\rho }_{1\dots r}}{\partial t}+\sum _{i=1}^r\frac{\partial }{\partial x_i}{\int }_{{ℝ}^{n-r}}\rho F_{i}d{\mathbf{x}}_{r+1,\dots ,n}=\frac{1}{2}\sum _{i=1}^r\sum _{j=1}^r{\int }_{{ℝ}^{n-r}}\frac{{\partial }^2{g}_{ij}\rho }{\partial x_i\partial x_j}d{\mathbf{x}}_{r+1,\dots ,n}.\hfill \end{array}$$

(7)

Similarly,

$$\begin{array}{ccc}& & \frac{\partial {\rho }_{r+1,\dots ,s}}{\partial t}+\sum _{i=r+1}^s\frac{\partial }{\partial x_i}{\int }_{{ℝ}^{n-s+r}}\rho F_{i}d{\mathbf{x}}_{1,\dots ,r,s+1,\dots ,n}=\frac{1}{2}\sum _{i=r+1}^s\sum _{j=r+1}^s{\int }_{{ℝ}^{n-s+r}}\frac{{\partial }^2{g}_{ij}\rho }{\partial x_i\partial x_j}d{\mathbf{x}}_{1,\dots ,r,s+1,\dots ,n}.\hfill \end{array}$$

(8)

Multiplication of Equation (7) by $-(1+log{\rho }_{1\dots r})$, followed by an integration with respect to ${\mathbf{x}}_{1\dots r}$ over ${ℝ}^r$, yields

$$\begin{array}{ccc}& & \frac{d{H}_A}{dt}-\sum _{i=1}^r{\int }_{{ℝ}^r}\left[(1+log{\rho }_{1\dots r})·\frac{\partial }{\partial x_i}{\int }_{{ℝ}^{n-r}}\rho F_{i}d{\mathbf{x}}_{r+1,\dots ,n}\right]d{\mathbf{x}}_{1\dots r}\hfill \\ & & =-\frac{1}{2}{\int }_{{ℝ}^r}\left[(1+log{\rho }_{1\dots r})·\sum _{i=1}^r\sum _{j=1}^r{\int }_{{ℝ}^{n-r}}\frac{{\partial }^2{g}_{ij}\rho }{\partial x_i\partial x_j}d{\mathbf{x}}_{r+1,\dots ,n}\right]d{\mathbf{x}}_{1\dots r}.\hfill \end{array}$$

Note that in the second term of the left hand side, the part within the summation is, by integration by parts,

$$\begin{array}{ccc}& & {\int }_{{ℝ}^r}(log{\rho }_{1\dots r})·\frac{\partial }{\partial x_i}\left({\int }_{{ℝ}^{n-r}}\rho F_{i}d{\mathbf{x}}_{r+1,\dots ,n}\right)d{\mathbf{x}}_{1\dots r}\hfill \\ & & =-{\int }_{{ℝ}^r}{\int }_{{ℝ}^{n-r}}\rho F_i\frac{\partial log{\rho }_{1\dots r}}{\partial x_i}d{\mathbf{x}}_{r+1,\dots ,n}d{\mathbf{x}}_{1\dots r}\hfill \\ & & =-{\int }_{{ℝ}^n}\rho F_i\frac{\partial log{\rho }_{1\dots r}}{\partial x_i}d\mathbf{x}\hfill \\ & & =-E\left[F_i\frac{\partial log{\rho }_{1\dots r}}{\partial x_i}\right].\hfill \end{array}$$

In the derivation, the compactness assumption has been used (variables vanish at the boundaries). By the same approach, the right hand side becomes

$$\begin{array}{ccc}& & -\frac{1}{2}{\int }_{{ℝ}^r}\left[log{\rho }_{1\dots r}·\sum _{i=1}^r\sum _{j=1}^r{\int }_{{ℝ}^{n-r}}\frac{{\partial }^2{g}_{ij}\rho }{\partial x_i\partial x_j}d{\mathbf{x}}_{r+1,\dots ,n}\right]d{\mathbf{x}}_{1\dots r}\hfill \\ & & =-\frac{1}{2}\sum _{i=1}^r\sum _{j=1}^r{\int }_{{ℝ}^n}\left[log{\rho }_{1\dots r}·\frac{{\partial }^2{g}_{ij}\rho }{\partial x_i\partial x_j}\right]d\mathbf{x}\hfill \\ & & =-\frac{1}{2}\sum _{i=1}^r\sum _{j=1}^r{\int }_{{ℝ}^n}\left[g_{ij}\rho \frac{{\partial }^{2}log{\rho }_{1\dots r}}{\partial x_i\partial x_j}\right]d\mathbf{x}\hfill \\ & & =-\frac{1}{2}\sum _{i=1}^r\sum _{j=1}^{r}E\left[g_{ij}\frac{{\partial }^{2}log{\rho }_{1\dots r}}{\partial x_i\partial x_j}\right].\hfill \end{array}$$

Hence,

$$\begin{array}{c}\hfill \frac{d{H}_A}{dt}=-\sum _{i=1}^{r}E\left[F_i\frac{\partial log{\rho }_{1\dots r}}{\partial x_i}\right]-\frac{1}{2}\sum _{i=1}^r\sum _{j=1}^{r}E\left[g_{ij}\frac{{\partial }^{2}log{\rho }_{1\dots r}}{\partial x_i\partial x_j}\right].\end{array}$$

(9)

Likewise, we have

$$\begin{array}{c}\hfill \frac{d{H}_B}{dt}=-\sum _{i=r+1}^{s}E\left[F_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}\right]-\frac{1}{2}\sum _{i=r+1}^s\sum _{j=r+1}^{s}E\left[g_{ij}\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}\right].\end{array}$$

(10)

Now consider the impact of the subsystem A on its peer B, written $T_{A\to B}$. Following Liang (2016) [16], this is associated with the evolution of the joint entropy of the latter:

$$\begin{array}{c}\hfill \frac{d{H}_B}{dt}=\frac{d{H}_{B\setminus A}}{dt}+T_{A\to B},\end{array}$$

(11)

where $H_{B\setminus A}$ signifies the entropy evolution with the influence of A excluded, which is found by instantaneously freezing $(x_1,\dots ,x_r)\equiv {\mathbf{x}}_{1\dots r}$ as parameters. To do this, examine, on an infinitesimal interval $[t,t+\Delta t]$, a system modified from the original Equations (1)–(3) by removing the r equations for $x_1$, $x_2$, ..., $x_r$ from the equation set

$$\begin{array}{ccc}\hfill & & \frac{d{x}_{r+1}}{dt}=F_{r+1}(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{r+1,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \end{array}$$

(12)

$$\begin{array}{c}⋮⋮\hfill \end{array}$$

$$\begin{array}{ccc}& & \frac{d{x}_s}{dt}=F_s(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{s,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \end{array}$$

(13)

$$\begin{array}{ccc}& & \frac{d{x}_{s+1}}{dt}=F_{s+1}(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{s+1,k}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k\hfill \end{array}$$

(14)

$$\begin{array}{c}⋮⋮\hfill \end{array}$$

$$\begin{array}{ccc}& & \frac{d{x}_n}{dt}=F_n(x_1,x_2,\dots ,x_n;t)+\sum _{k=1}^m{b}_{nk}(x_1,x_2,\dots ,x_n;t){\dot{w}}_k.\hfill \end{array}$$

(15)

Notice that the $F_i$s and $b_{ik}$s still have dependence on $(x_1,\dots ,x_r)\equiv {\mathbf{x}}_{1\dots r}$, which, however, appear in the modified system as parameters. By [16], we can construct a mapping $\Phi :{ℝ}^{n-r}\to {ℝ}^{n-r}$, ${\mathbf{x}}_{\setminus A}\left(t\right)\mapsto {\mathbf{x}}_{\setminus A}(t+\Delta t)$, where ${\mathbf{x}}_{\setminus A}$ means $\mathbf{x}$ but with ${\mathbf{x}}_{1\dots r}$ appearing as parameters, and study the Frobenius–Perron operator (see, for example, [25]) of the modified system. An alternative approach is given by Liang in [18], which we henceforth follow. Observe that on the interval $[t,t+\Delta t]$, corresponding to the modified dynamical system, there is also a Fokker–Planck equation

$$\begin{array}{ccc}& & \frac{\partial {\rho }_{\setminus A}}{\partial t}+\sum _{i=r+1}^n\frac{\partial F_i{\rho }_{\setminus A}}{\partial x_i}=\frac{1}{2}\sum _{i=r+1}^n\sum _{j=r+1}^n\frac{{\partial }^2{g}_{ij}{\rho }_{\setminus A}}{\partial x_i\partial x_j},\hfill \\ & & {\rho }_{\setminus A}={\rho }_{r+1,\dots ,n}\mathrm{at}\mathrm{time}\mathrm{t}.\hfill \end{array}$$

Here $g_{ij}={\sum }_{k=1}^m{b}_{ik}{b}_{jk}$, ${\rho }_{\setminus A}$ means the joint pdf of $(x_{r+1},\dots ,x_n)$ with ${\mathbf{x}}_{1\dots r}$ frozen as parameters. Note the difference between ${\rho }_{\setminus A}$ and ${\rho }_{r+1,\dots ,n}$; the former has ${\mathbf{x}}_{1\dots r}$ as parameters, while the latter has no dependence on ${\mathbf{x}}_{1\dots r}$. However, they are equal at time t.

Integration of the above Fokker–Planck equation with respect to $d{\mathbf{x}}_{s+1,\dots ,n}$ gives the evolution of the pdf of subsystem B with A frozen as parameters, written ${\rho }_{B,\setminus A}$:

$$\begin{array}{ccc}\hfill & & \frac{\partial {\rho }_{B,\setminus A}}{\partial t}+\sum _{i=r+1}^s{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{\setminus A}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}=\frac{1}{2}\sum _{i=r+1}^s\sum _{j=r+1}^s{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{\setminus A}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n},\hfill \end{array}$$

(16)

$$\begin{array}{ccc}& & {\rho }_{B,\setminus A}={\rho }_{r+1,\dots ,s}\mathrm{at}\mathrm{time}\mathrm{t}.\hfill \end{array}$$

(17)

Divide Equation (16) by ${\rho }_{B,\setminus A}$ and simplify the notation ${\mathbf{x}}_{r+1,\dots ,s}$ by ${\mathbf{x}}_B$ to obtain

$$\begin{array}{c}\hfill \frac{\partial log{\rho }_{B,\setminus A}}{\partial t}+\sum _{i=r+1}^s\frac{1}{{\rho }_{B,\setminus A}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{\setminus A}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}=\frac{1}{2{\rho }_{B,\setminus A}}\sum _{i=r+1}^s\sum _{j=r+1}^s{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{\setminus A}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}.\end{array}$$

Discretizing and noticing that ${\rho }_{B,\setminus A}\left(t\right)={\rho }_{r+1,\dots ,s}\left(t\right)$, we have (in the following, unless otherwise indicated, the variables without arguments explicitly specified are assumed to be at time step t)

$$\begin{array}{ccc}& & log{\rho }_{B,\setminus A}({\mathbf{x}}_B;t+\Delta t)\hfill \\ & & =log{\rho }_{r+1,\dots ,s}({\mathbf{x}}_B;t)-\Delta t·\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\hfill \\ & & +\frac{\Delta t}{2}\sum _{i=r+1}^s\sum _{j=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}+o\left(\Delta t\right).\hfill \end{array}$$

To arrive at $d{H}_{B,\setminus A}/dt$, we need to find $log{\rho }_{B,\setminus A}({\mathbf{x}}_B(t+\Delta t);t+\Delta t)$. Using the Euler–Bernstein approximation,

$$\begin{array}{c}\hfill {\mathbf{x}}_B(t+\Delta t)={\mathbf{x}}_B\left(t\right)+{\mathbf{F}}_B\Delta t+{\mathbf{B}}_B\Delta \mathbf{w}\end{array}$$

(18)

where, just as the notation ${\mathbf{x}}_B$,

$$\begin{array}{ccc}& & {\mathbf{F}}_B={(F_{r+1},\dots ,F_s)}^T,\hfill \\ & & {\mathbf{B}}_B=\left[\begin{array}{ccc}{b}_{r+1,1}& \cdots & b_{r+1,m}\\ ⋮& \ddots & ⋮\\ b_{s1}& \cdots & b_{sm}\end{array}\right]\hfill \\ & & \Delta \mathbf{w}={(\Delta w_1,\dots ,\Delta w_m)}^T\hfill \end{array}$$

and $\Delta w_k\sim N(0,\Delta t)$, we have

$$\begin{array}{ccc}& & log({\rho }_{B,\setminus A}({\mathbf{x}}_B(t+\Delta t);t+\Delta t)\hfill \\ & & =log{\rho }_{r+1,\dots ,s}({\mathbf{x}}_B\left(t\right)+{\mathbf{F}}_B\Delta t+{\mathbf{B}}_B\Delta \mathbf{w};t)\hfill \\ & & -\Delta t·\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}+\frac{\Delta t}{2}\sum _{r+1}^s\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}+o\left(\Delta t\right).\hfill \\ & & =log{\rho }_{r+1,\dots ,s}\left({\mathbf{x}}_B\left(t\right)\right)+\sum _{i=r+1}^s\left[\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}(F_i\Delta t+\sum _{k=1}^m{b}_{ik}\Delta w_k)\right]\hfill \\ & & +\frac{1}{2}·\sum _{i=r+1}^s\sum _{j=r+1}^s\left[\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}(F_i\Delta t+\sum _{k=1}^m{b}_{ik}\Delta w_k)·(F_j\Delta t+\sum _{l=1}^m{b}_{jl}\Delta w_l)\right]\hfill \\ & & -\Delta t·\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}+\frac{\Delta t}{2}\sum _{r+1}^s\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}+o\left(\Delta t\right).\hfill \end{array}$$

Take mathematical expectation on both sides. The left hand side is $-H_{B,\setminus A}(t+\Delta t)$. By Corollary III.I of [16], and noting $E\Delta w_k=0$, $E\Delta w_k^2=\Delta t$ and the fact that $\Delta \mathbf{w}$ are independent of ${\mathbf{x}}_B$, we have

$$\begin{array}{ccc}& & -H_{B,\setminus A}(t+\Delta t)=-H_B\left(t\right)+\Delta t·E\sum _{i=r+1}^s{F}_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}\hfill \\ & & +\frac{\Delta t}{2}·E\sum _{i=r+1}^s\sum _{j=r+1}^s\sum _{k=1}^m\sum _{l=1}^m{b}_{ik}{b}_{jl}{\delta }_{kl}\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}\hfill \\ & & -\Delta t·E\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\hfill \\ & & +\frac{\Delta t}{2}E\sum _{r+1}^s\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}+o\left(\Delta t\right)\hfill \\ & & =-H_B\left(t\right)+\Delta t·E\sum _{i=r+1}^s{F}_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}+\frac{\Delta t}{2}·E\sum _{i=r+1}^s\sum _{j=r+1}^s{g}_{ij}\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}\hfill \\ & & -\Delta t·E\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\hfill \\ & & +\frac{\Delta t}{2}E\sum _{r+1}^s\sum _{r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}+o\left(\Delta t\right).\hfill \end{array}$$

Thus,

$$\begin{array}{ccc}& & \frac{d{H}_{B,\setminus A}}{dt}=\underset{\Delta t\to 0}{lim}\frac{H_{B,\setminus A}-H_B\left(t\right)}{\Delta t}\hfill \\ & & =-E\sum _{i=r+1}^n\left(F_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}-\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right)\hfill \\ & & -\frac{1}{2}E\sum _{i=r+1}^s\sum _{j=r+1}^s\left(g_{ij}\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}+\frac{1}{{\rho }_{r+1,\dots ,s}}\frac{{\partial }^2}{\partial x_i\partial x_j}{\int }_{{ℝ}^{n-s}}{g}_{ij}{\rho }_{r+1,\dots ,n}d{\mathbf{x}}_{s+1,\dots ,n}\right).\hfill \end{array}$$

Hence, the information flow from ${\mathbf{x}}_{1\dots r}$ to ${\mathbf{x}}_{r+1,\dots ,s}$ is

$$\begin{array}{ccc}& & T_{A\to B}=\frac{d{H}_B}{dt}-\frac{d{H}_{B,\setminus A}}{dt}\hfill \\ & & =-E\sum _{i=r+1}^s\left(F_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}\right)-\frac{1}{2}E\sum _{i=r+1}^s\sum _{j=r+1}^s\left(g_{ij}\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}\right)\hfill \\ & & +E\sum _{i=r+1}^s\left(F_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}-\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right)\hfill \\ & & +\frac{1}{2}E\sum _{i=r+1}^s\sum _{j=r+1}^s\left(g_{ij}\frac{{\partial }^{2}log{\rho }_{r+1,\dots ,s}}{\partial x_i\partial x_j}+\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right)\hfill \\ & & =-E\left[\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & & +\frac{1}{2}E\left[\sum _{i=r+1}^s\sum _{j=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right].\hfill \end{array}$$

Likewise, we can obtain the information flow from subsystem B to subsystem A. These are summarized in the following theorem.

Theorem 1.

For the dynamical system Equations (1)–(3), if the probability density function (pdf) of $\mathbf{x}$ is compactly supported, then the information flow from ${\mathbf{x}}_{1\dots r}$ to ${\mathbf{x}}_{r+1,\dots ,s}$ and that from ${\mathbf{x}}_{r+1,\dots ,s}$ to ${\mathbf{x}}_{1\dots r}$ are (in nats per unit time), respectively,

$$\begin{array}{ccc}\hfill & & T_{A\to B}=-E\left[\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & & +\frac{1}{2}E\left[\sum _{i=r+1}^s\sum _{j=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right],\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill & & T_{B\to A}=-E\left[\sum _{i=1}^r\frac{1}{{\rho }_{1\dots r}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & & +\frac{1}{2}E\left[\sum _{i=1}^r\sum _{j=1}^r\frac{1}{{\rho }_{1\dots r}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right],\hfill \end{array}$$

(20)

where $g_{ij}={\sum }_{k=1}^m{b}_{ik}{b}_{jk}$, and E signifies mathematical expectation.

When $r=1$, $s=n=2$, (20) reduces to

$$\begin{array}{c}\hfill T_{B\to A}=-E\left[\frac{1}{{\rho }_1}\frac{\partial F_1{\rho }_1}{\partial x_1}\right]+\frac{1}{2}E\left[\frac{1}{{\rho }_1}\frac{{\partial }^2{g}_{11}{\rho }_1}{\partial x_1^2}\right]\end{array}$$

which is precisely the same as the Equation (15) in [18]; the same holds for Equation (19). These equations are hence verified.

The following theorem forms the basis for causal inference.

Theorem 2.

If the evolution of subsystem A (resp. B) does not depend on ${\mathbf{x}}_{r+1,\dots ,s}$ (resp. ${\mathbf{x}}_{1\dots r}$), then $T_{B\to A}=0$ (resp. $T_{A\to B}=0$).

Proof. 

We only check the formula for $T_{B\to A}$. In (20), the deterministic part

$$\begin{array}{ccc}& & -E\left[\sum _{i=1}^r\frac{1}{{\rho }_{1\dots r}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & & =-\sum _{i=1}^r{\int }_{{ℝ}^r}{\int }_{{ℝ}^{s-r}}\left[\left(\frac{{\rho }_{1,\dots ,s}}{{\rho }_{1\dots r}}\right){\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]d{\mathbf{x}}_{1\dots r}d{\mathbf{x}}_{r+1,\dots ,s}.\hfill \end{array}$$

Now, $F_i$ is independent of ${\mathbf{x}}_{r+1,\dots ,s}$, and note that ${\rho }_{1,\dots ,r,s+1,\dots ,n}$ is also so. Thus, we may integrate ${\rho }_{1,\dots ,s}$ within the parentheses directly with respect to $d{\mathbf{x}}_{r+1,\dots ,s}$, yielding

$$\begin{array}{c}\hfill \frac{{\int }_{{ℝ}^{s-r}}{\rho }_{1,\dots ,s}d{\mathbf{x}}_{r+1,\dots ,s}}{{\rho }_{1\dots r}}=\frac{{\rho }_{1\dots r}}{{\rho }_{1\dots r}}=1.\end{array}$$

By the compactness of $\rho$, the whole deterministic part hence vanishes. Likewise, it can be proved that the stochastic part also vanishes.

This theorem allows us to identify the causality with information flow. Ideally, if $T_{B\to A}=0$, then B is not causal to A, and vice versa; the same holds for $T_{A\to B}$. □

3. Information Flow between Linear Subsystems and Its Estimation

Linear systems provide the simplest framework which is usually taken as the first step toward a more generic setting. Simple as it may be, it has been demonstrated in practice that linear results often provide a good approximation of an otherwise much more complicated problem. It is hence of interest to examine this special case.

Let

$$\begin{array}{c}\hfill F_i=f_i+\sum _{j=1}^n{a}_{ij}{x}_j,\end{array}$$

(21)

where $f_i$ and $a_{ij}$ are constants. Additionally, suppose that $b_{ij}$ are constants—that is to say, the noises are additive. Then, $g_{ij}$ are also constants. Thus, in Equation (20),

$$\begin{array}{ccc}& & E\left(\frac{1}{{\rho }_{1\dots r}}{\int }_{n-s}\frac{{\partial }^2{g}_{ij}{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right)\hfill \\ & & =g_{ij}{\int }_{{ℝ}^s}\rho \left({\mathbf{x}}_{1\dots s}\right)\frac{1}{{\rho }_{1\dots r}}\frac{{\partial }^2{\int }_{}{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}d{\mathbf{x}}_{1\dots s}\hfill \\ & & =g_{ij}{\int }_{{ℝ}^r}{\int }_{{ℝ}^{s-r}}\frac{{\rho }_{1\dots s}}{{\rho }_{1\dots r}}\frac{{\partial }^2{\rho }_{1\dots r}}{\partial x_i\partial x_j}d{\mathbf{x}}_{r+1,\dots ,s}d{\mathbf{x}}_{1\dots r}\hfill \\ & & =g_{ij}{\int }_{{ℝ}^r}1·\frac{{\partial }^2{\rho }_{1\dots r}}{\partial x_i\partial x_j}d{\mathbf{x}}_{1\dots r}\hfill \\ & & =0.\hfill \end{array}$$

The same holds in Equation (19). Thus, the stochastic parts in both Equations (19) and (20) vanish.

Since a linear system initialized with a Gaussian process will always be Gaussian, we may write the joint pdf of $\mathbf{x}$ as

$$\begin{array}{c}\hfill \rho (x_1,\dots ,x_n)=\frac{1}{\sqrt{{\left(2\pi \right)}^{n}det\Sigma }}{e}^{-\frac{1}{2}{(\mathbf{x}-\mu )}^T{\Sigma }^{-1}(\mathbf{x}-\mu )},\end{array}$$

(22)

where $\Sigma ={\left({\sigma }_{ij}\right)}_{n\times n}$ is the population covariance matrix of $\mathbf{x}$. By the property of the Gaussian process, it is easy to show

$$\begin{array}{c}\hfill {\rho }_{r+1,\dots ,s}(x_{r+1},\dots ,x_s)=\frac{1}{\sqrt{{\left(2\pi \right)}^{s-r}det{\Sigma }_B}}{e}^{-\frac{1}{2}{({\mathbf{x}}_B-{\mu }_B)}^T{\Sigma }_B^{-1}({\mathbf{x}}_B-{\mu }_B)},\end{array}$$

(23)

where ${\mathbf{x}}_B=(x_{r+1},\dots ,x_s)$, ${\mu }_B=({\mu }_{r+1},\dots ,{\mu }_s)$ is the vector of the means of ${\mathbf{x}}_B$, and ${\Sigma }_B$ the covariance matrix of ${\mathbf{x}}_B$. For easy correspondence, we will augment ${\mathbf{x}}_B$, ${\mu }_B$, and ${\Sigma }_B$, so that their entries have the same indices as their counterparts in $\mathbf{x}$, $\mu$ and $\Sigma$. Separate $F_i$ into two parts:

$$\begin{array}{ccc}\hfill F_i& =& \left[f_i+\sum _{j=1}^r{a}_{ij}{x}_j+\sum _{j=s+1}^n{a}_{ij}{x}_j\right]+\left[\sum _{j=r+1}^s{a}_{ij}{x}_j\right]\hfill \\ & \equiv & F_i^{\prime }+F_i^{\prime },\hfill \end{array}$$

where $F_i^{\prime }$ and $F_i^{\prime }$ correspond to the respective parts in the two square brackets. Thus, $F_i^{\prime }$ has nothing to do with the subspace B. By Theorem 2, this part does not contribute to the causality from A to B, so we only need to consider $F_i^{″}$ in evaluating $T_{A\to B}$; that is to say,

$$\begin{array}{ccc}\hfill T_{A\to B}& =& -E\left[\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}\frac{\partial }{\partial x_i}{\int }_{{ℝ}^{n-s}}{F}_i{\rho }_{r+1,\dots ,n}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & =& -E\left[\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}\frac{\partial }{\partial x_i}{\int }_{{ℝ}^{n-s}}{F}_i^{″}{\rho }_{r+1,\dots ,n}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & =& -E\left[\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}\frac{\partial F_i^{″}{\rho }_{r+1,\dots ,s}}{\partial x_i}\right]\hfill \\ & =& -\sum _{i=r+1}^s\left[E\left(F_i^{″}\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}\right)+E\left(\frac{\partial F_i^{″}}{\partial x_i}\right)\right].\hfill \end{array}$$

The second term in the bracket is $a_{ii}$. The first term is

$$\begin{array}{ccc}& & F_i^{″}\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}=\left[\sum _{j=r+1}^s{a}_{ij}{x}_j\right]·\frac{\partial }{\partial x_i}\left[-\frac{1}{2}{({\mathbf{x}}_B-{\mu }_B)}^T{\Sigma }_B^{-1}({\mathbf{x}}_B-{\mu }_B)\right]\hfill \\ & & =\left(\sum _{j=r+1}^s{a}_{ij}{x}_j\right){\dot{\sum }}_{j=r+1}^s\left(-\frac{{\sigma }_{ij}^{\prime }+{\sigma }_{ji}^{\prime }}{2}\right)·(x_j-{\mu }_j).\hfill \end{array}$$

Here, ${\sigma }_{ij}^{\prime }$ is the $(i,j)\mathrm{th}$ entry of the matrix

$$\begin{array}{c}\hfill \left[\begin{array}{ccc}\mathbf{I}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\Sigma }_B^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{I}\end{array}\right].\end{array}$$

Since, here, only $1\le i,j\le s$ are in question, this is equal to the $(i,j)\mathrm{th}$ entry of the matrix

$$\begin{array}{c}\hfill \left[\begin{array}{cc}{\mathbf{I}}_{r\times r}& {\mathbf{0}}_{r\times (s-r)}\\ {\mathbf{0}}_{(s-r)\times r}& {\Sigma }_B^{-1}\end{array}\right]\end{array}$$

As ${\Sigma }_B$ is symmetric, so is ${\Sigma }_B^{-1}$, and hence $({\sigma }_{ij}^{\prime }+{\sigma }_{ji}^{\prime })/2={\sigma }_{ij}^{\prime }$. Thus,

$$\begin{array}{ccc}& & -E{F}_i\frac{\partial log{\rho }_{r+1,\dots ,s}}{\partial x_i}=-E\sum _{j=1}^s{a}_{ij}{x}_j·\sum _{j=r+1}^s(-{\sigma }_{ij}^{\prime })·(x_j-{\mu }_j)\hfill \\ & & =E\sum _{k=1}^s{a}_{ik}(x_k-{\mu }_k)·\sum _{j=r+1}^s{\sigma }_{ij}^{\prime }(x_j-{\mu }_j)\hfill \\ & & =\sum _{k=1}^s\sum _{j=r+1}^s{a}_{ik}{\sigma }_{ij}^{\prime }E(x_k-{\mu }_k)(x_j-{\mu }_j)\hfill \\ & & =\sum _{k=1}^s\sum _{j=r+1}^s{a}_{ik}{\sigma }_{ij}^{\prime }{\sigma }_{k}j.\hfill \end{array}$$

Substituting back, we obtain a very simplified result for $T_{A\to B}$. Likewise, $T_{B\to A}$ can also be obtained, as shown in the following.

Theorem 3.

In Equations (1)–(3), suppose $b_{ij}$ are constants, and

$$\begin{array}{ccc}& & F_i=f_i+\sum _{j=1}^n{a}_{ij}{x}_j,\hfill \end{array}$$

(24)

where $f_i$ and $a_{ij}$ are also constants. Furthermore, suppose that initially $\mathbf{x}$ has a Gaussian distribution; then,

$$\begin{array}{ccc}\hfill T_{A\to B}& =& \sum _{i=r+1}^s\left[\sum _{j=r+1}^s{\sigma }_{ij}^{\prime }\left(\sum _{k=1}^s{a}_{ik}{\sigma }_{kj}\right)-a_{ii}\right],\hfill \end{array}$$

(25)

where ${\sigma }_{ij}^{\prime }$ is the $(i,j)\mathrm{th}$ entry of $\left[\begin{array}{cc}{\mathbf{I}}_{r\times r}& \mathbf{0}\\ \mathbf{0}& {\Sigma }_B^{-1}\end{array}\right]$, and

$$\begin{array}{ccc}\hfill T_{B\to A}& =& \sum _{i=1}^r\left[\sum _{j=1}^r{\sigma }_{ij}^{″}\left(\sum _{k=1}^s{a}_{ik}{\sigma }_{kj}\right)-a_{ii}\right],\hfill \end{array}$$

(26)

where ${\sigma }_{ij}^{\prime \prime }$ is the $(i,j)\mathrm{th}$ entry of $\left[\begin{array}{cc}{\Sigma }_A^{-1}& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{(s-r)\times (s-r)}\end{array}\right]$.

Given a system such as (1)–(3), we can evaluate in a precise sense the information flows among the components. Now, suppose that instead of the dynamical system, what we have are just n time series with K steps, $K\gg n$, $\left\{x_1\left(k\right)\right\},\left\{x_2\left(k\right)\right\},\dots ,\left\{x_n\left(k\right)\right\}$. We can estimate the system from the series and then apply the information flow formula to fulfill the task. Assume a linear model as shown above, and assume $m=1$. following Liang (2014) [19], the maximum likelihood estimator (mle) of $a_{ij}$ is equal to the least-square solution of the following over-determined problem:

$$\begin{array}{c}\hfill \left(\begin{array}{ccccc}1& x_1\left(1\right)& x_2\left(1\right)& \dots & x_n\left(1\right)\\ 1& x_1\left(2\right)& x_2\left(2\right)& \dots & x_n\left(2\right)\\ 1& x_1\left(3\right)& x_2\left(3\right)& \dots & x_n\left(3\right)\\ ⋮& ⋮& ⋮\ddots & ⋮\\ 1& x_1\left(K\right)& x_2\left(K\right)& \dots & x_n\left(K\right)\end{array}\right)\left(\begin{array}{c}{f}_i\\ a_{i1}\\ a_{i2}\\ ⋮\\ a_{in}\end{array}\right)=\left(\begin{array}{c}{\dot{x}}_i\left(1\right)\\ {\dot{x}}_i\left(2\right)\\ {\dot{x}}_i\left(3\right)\\ ⋮\\ {\dot{x}}_i\left(K\right)\end{array}\right)\end{array}$$

where ${\dot{x}}_i\left(k\right)=(x_i(k+1)-x_i\left(k\right))/\Delta t$ ($\Delta t$ is the time stepsize), for $i=1,2,\dots ,n$, $k=1,\dots ,K$. Use an overbar to denote the time mean over the K steps. The above equation is

$$\begin{array}{c}\hfill \left(\begin{array}{ccccc}1& {\overline{x}}_1& {\overline{x}}_2& \dots & {\overline{x}}_n\\ 0& x_1\left(2\right)-{\overline{x}}_1& x_2\left(2\right)-{\overline{x}}_2& \dots & x_n\left(2\right)-{\overline{x}}_n\\ 0& x_1\left(3\right)-{\overline{x}}_1& x_2\left(3\right)-{\overline{x}}_2& \dots & x_n\left(3\right)-{\overline{x}}_n\\ ⋮& ⋮& ⋮\ddots & ⋮\\ 0& x_1\left(K\right)-{\overline{x}}_1& x_2\left(K\right)-{\overline{x}}_2& \dots & x_n\left(K\right)-{\overline{x}}_n\end{array}\right)\left(\begin{array}{c}{f}_i\\ a_{i1}\\ a_{i2}\\ ⋮\\ a_{in}\end{array}\right)=\left(\begin{array}{c}{\overline{\dot{x}}}_i\\ {\dot{x}}_i\left(2\right)-{\overline{\dot{x}}}_i\\ {\dot{x}}_i\left(3\right)-{\overline{\dot{x}}}_i\\ ⋮\\ {\dot{x}}_i\left(K\right)-{\overline{\dot{x}}}_i\end{array}\right)\end{array}$$

Denote by $\mathbf{R}$ the matrix

$$\left(\begin{array}{cccc}{x}_1\left(2\right)-{\overline{x}}_1& x_2\left(2\right)-{\overline{x}}_2& \dots & x_n\left(2\right)-{\overline{x}}_n\\ ⋮& ⋮& ⋮\ddots & ⋮\\ x_1\left(K\right)-{\overline{x}}_1& x_2\left(K\right)-{\overline{x}}_2& \dots & x_n\left(K\right)-{\overline{x}}_n\end{array}\right),$$

$\mathbf{q}$ the vector ${(x_i\left(2\right)-{\overline{\dot{x}}}_i,\dots ,x_i\left(K\right)-{\overline{\dot{x}}}_i)}^T$, and ${\mathbf{a}}_i$ the row vector ${(a_{i1},\dots ,a_{in})}^T$. Then, $\mathbf{R}{\mathbf{a}}_i=\mathbf{q}$. The least square solution of ${\mathbf{a}}_i$, ${\widehat{\mathbf{a}}}_i$, solves

$$\begin{array}{c}\hfill {\mathbf{R}}^T\mathbf{R}{\widehat{\mathbf{a}}}_i={\mathbf{R}}^T\mathbf{q}.\end{array}$$

Note that ${\mathbf{R}}^T\mathbf{R}$ is $K\mathbf{C}$, where $\mathbf{C}=\left(c_{ij}\right)$ is the sample covariance matrix. Thus,

$$\begin{array}{c}\hfill \left(\begin{array}{c}{\widehat{a}}_{i1}\\ {\widehat{a}}_{i2}\\ ⋮\\ {\widehat{a}}_{in}\end{array}\right)={\mathbf{C}}^{-1}\left(\begin{array}{c}{c}_{1,di}\\ c_{2,di}\\ ⋮\\ c_{n,di}\end{array}\right)\end{array}$$

(27)

where $c_{j,di}$ is the sample covariance between the series $\left\{x_j\left(k\right)\right\}$ and $\{(x_i(k+1)-x_i\left(k\right))/\Delta t\}$.

Thus, finally, the mle of $T_{A\to B}$ is

$$\begin{array}{c}\hfill {\widehat{T}}_{A\to B}=\sum _{i=r+1}^s\left[\sum _{j=r+1}^s{c}_{ij}^{\prime }\left(\sum _{k=1}^s{\widehat{a}}_{ik}{c}_{kj}\right)-{\widehat{a}}_{ii}\right],\end{array}$$

(28)

where $c_{ij}^{\prime }$ is the $(i,j)\mathrm{th}$ entry of ${\tilde{\mathbf{C}}}^{-1}$, and

$$\begin{array}{c}\hfill \tilde{\mathbf{C}}=\left(\begin{array}{cc}{\mathbf{I}}_{r\times r}& {\mathbf{0}}_{r\times (s-r)}\\ {\mathbf{0}}_{(s-r)\times r}& \left[\begin{array}{ccc}{c}_{r+1,r+1}& \dots & c_{r+1,s}\\ ⋮& ⋮& ⋮\\ c_{s,r+1}& \dots & c_{s,s}\end{array}\right]\end{array}\right).\end{array}$$

(29)

Likewise,

$$\begin{array}{ccc}\hfill {\widehat{T}}_{B\to A}& =& \sum _{i=1}^r\left[\sum _{j=1}^r{c}_{ij}^{\prime \prime }\left(\sum _{k=1}^s{\widehat{a}}_{ik}{c}_{kj}\right)-{\widehat{a}}_{ii}\right]\hfill \end{array}$$

(30)

Here,

$$\begin{array}{c}\hfill \widetilde{\tilde{\mathbf{C}}}=\left(\begin{array}{cc}\left[\begin{array}{ccc}{c}_{11}& \dots & c_{1r}\\ ⋮& ⋮& ⋮\\ c_{r1}& \dots & c_{rr}\end{array}\right]& {\mathbf{0}}_{r\times (s-r)}\\ {\mathbf{0}}_{(s-r)\times r}& {\mathbf{I}}_{(s-r)\times (s-r)}\end{array}\right),\end{array}$$

(31)

and $c_{ij}^{″}$ is the $(i,j)th$ entry of ${\widetilde{\tilde{\mathbf{C}}}}^{{}_{-}1}$.

When $n=2$ and $r=1$, and hence, $s=1$, $\widetilde{\tilde{\mathbf{C}}}=\left[\begin{array}{cc}{c}_{11}& 0\\ 0& 1\end{array}\right]$, so $c_{11}^{″}=c_{11}^{-1}$. Equation (30) thus becomes

$$\begin{array}{ccc}\hfill {\widehat{T}}_{B\to A}& =& c_{11}^{″}\sum _{k=1}^2{\widehat{a}}_{1k}{c}_{k1}-{\widehat{a}}_{11}\hfill \\ & =& \frac{1}{c_{11}}({\widehat{a}}_{11}{c}_{11}+{\widehat{a}}_{12}{c}_{12})-{\widehat{a}}_{11}\hfill \\ & =& \frac{c_{11}{c}_{12}{c}_{2,d1}-c_{12}^2{c}_{1,d1}}{c_{11}^2-c_{11}{c}_{12}^2},\hfill \end{array}$$

recovering the well-known Equation (10) in [19].

4. Validation

4.1. One-Way Causal Relation

To see if the above formalism works, consider the vector autoregressive (VAR) process:

$$\begin{array}{ccc}& & X:\left\{\begin{array}{c}{x}_1(n+1)=-0.5x_1\left(n\right)+0.5x_2\left(n\right)+0.2x_3\left(n\right)+e_{x1}(n+1),\hfill \\ x_2(n+1)=0x_1\left(n\right)-0.2x_2\left(n\right)-0.6x_3\left(n\right)+e_{x2}(n+1),\hfill \\ x_3(n+1)=-0.2x_1\left(n\right)+0.4x_2\left(n\right)-0.2x_3\left(n\right)+{\epsilon }_3{y}_3\left(n\right)+e_{x3}(n+1),\hfill \end{array}\right.\hfill \end{array}$$

(32)

$$\begin{array}{ccc}& & Y:\left\{\begin{array}{c}{y}_1(n+1)=-0.2y_1\left(n\right)-0.5y_2\left(n\right)+0y_3\left(n\right)-{\epsilon }_1{x}_1\left(n\right)+e_{y1}(n+1),\hfill \\ y_2(n+1)=0.5y_1\left(n\right)-0.6y_2\left(n\right)+0.4y_3\left(n\right)+e_{y2}(n+1),\hfill \\ y_3(n+1)=-0.1y_1\left(n\right)-0.4y_2\left(n\right)-0.5y_3\left(n\right)+e_{y3}(n+1)\hfill \end{array}\right.,\hfill \end{array}$$

(33)

where $e_{xi},e_{yi}\sim N(0,1)$, $i=1,2,3$, are independent. As schematized in Figure 1, $(x_1,x_2,x_3)$ and $(y_1,y_2,y_3)$ form two subsystems, written as X and Y, respectively. They are coupled only through the first and third components; more specifically, $x_1$ drives $y_1$, and Y feeds back to X through coupling $y_3$ with $x_3$. The strength of the coupling is determined by the parameters ${\epsilon }_1$ and ${\epsilon }_3$. In this subsection, ${\epsilon }_3=0$, so the causality is one-way, i.e., from X to Y without feedback.

Initialized with random numbers, we iterate the process for 20,000 steps and discard the first 10,000 steps to form six time series with a length of 10,000 steps. Using the algorithm by Liang (e.g., [16,18,19,20]), the information flows between $x_1$ and $y_1$ can be rather accurately obtained. As shown in Figure 2a, the information flow/causality from X to Y increases with ${\epsilon }_1$, and there is no causality the other way around, just as expected. Since no other coupling exists, one can imagine that the bulk information flows must also bear a similar trend. Using Equations (28) and (30), the estimators are indeed similar to that, as shown in Figure 2b. This demonstrates the success of the above formalism.

Since practically averages and principal components (PCs) have been widely used to measure complex subsystem variations, we also compute the information flows between $\overline{x}=\frac{1}{3}(x_1+x_2+x_3)$ and $\overline{y}=\frac{1}{3}(y_1+y_2+y_3)$, and that between the first PCs of $(x_1,x_2,x_3)$ and $(y_1,y_2,y_3)$. The results are plotted in Figure 2c,d, respectively. As can be seen, the principal component analysis (PCA) method works just fine in this case. By comparison, the averaging method yields an incorrect result.

The incorrect inference based on averaging is within expectation. In a network with complex causal relations, for example, with a causality from $y_2$ to $y_1$, the averaging of $y_1$ with $y_2$ is equivalent to mixing $y_1$ with its future state, which is related to the contemporary state of $x_1$, and hence will yield a spurious causality to $x_1$. The PCA here functions satisfactorily, perhaps because in selecting the most coherent structure, it discards most of the influences from other (implicit) time steps. However, the relative success of PCA may not be robust, as evidenced in the following mutually causal case.

4.2. Mutually Causal Relation

If both the coupling parameters, ${\epsilon }_1$ and ${\epsilon }_3$, are turned on, the resulting causal relation has a distribution on the ${\epsilon }_1-{\epsilon }_3$ plane. Figure 3 shows the componentwise information flows $T_{x_1\to y_1}$ (bottom) and $T_{y_3\to x_3}$ (top) on the plane. The other two flows, i.e., their counterparts $T_{y_1\to x_1}$ and $T_{x_3\to y_3}$, are by computation essentially zero. As argued in the preceding subsection, the bulk information flows should follow the general pattern, albeit perhaps in a more coarse and/or mild pattern, since it is a property on the whole. This is indeed true. Shown in Figure 4 are the bulk information flows between X and Y computed using Equations (28) and (30).

Again, as usual, we try the averages and first PCs as proxies for estimating the causal interaction between X and Y. Figure 5 shows the distributions of the information flows between $\overline{x}$ and $\overline{y}$. The resulting patterns are totally different from what Figure 3 displays; obviously, these patterns are incorrect.

One may expect that the PCA method should yield more reasonable causal patterns. We have computed the first PCs for $(x_1,x_2,x_3)$ and $(y_1,y_2,y_3)$, respectively, and estimated the information flows using the algorithm by Liang [20]. The resulting distributions, however, are no better than those with the averaged series (Figure 6). That is to say, this seemingly more sophisticated approach does not yield the right interaction between the complex subsystems, either.

5. Summary

Information flow provides a natural measure of the causal interaction between dynamical events. In this study, the information flows between two complex subsystems of a large dimensional system are studied, and analytical formulas have been obtained in a closed form. For easy reference, the major results are summarized hereafter.

For an n-dimensional

$$\begin{array}{c}\hfill \frac{d\mathbf{x}}{dt}=\mathbf{F}(\mathbf{x},t)+\mathbf{B}(\mathbf{x},t)\dot{\mathbf{w}},\end{array}$$

if the probability density function (pdf) of $\mathbf{x}$ is compactly supported, then the information flows from subsystem A, which are made of ${\mathbf{x}}_{1\dots r}$, to subsystem B, made of ${\mathbf{x}}_{r+1,\dots ,s}$ ($1\le r<s\le n$), and that from B to A are, respectively (in nats per unit time),

$$\begin{array}{ccc}& & T_{A\to B}=-E\left[\sum _{i=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{r+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & & +\frac{1}{2}E\left[\sum _{i=r+1}^s\sum _{j=r+1}^s\frac{1}{{\rho }_{r+1,\dots ,s}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{r+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right],\hfill \end{array}$$

$$\begin{array}{ccc}& & T_{B\to A}=-E\left[\sum _{i=1}^r\frac{1}{{\rho }_{1\dots r}}{\int }_{{ℝ}^{n-s}}\frac{\partial F_i{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i}d{\mathbf{x}}_{s+1,\dots ,n}\right]\hfill \\ & & +\frac{1}{2}E\left[\sum _{i=1}^r\sum _{j=1}^r\frac{1}{{\rho }_{1\dots r}}{\int }_{{ℝ}^{n-s}}\frac{{\partial }^2{g}_{ij}{\rho }_{1,\dots ,r,s+1,\dots ,n}}{\partial x_i\partial x_j}d{\mathbf{x}}_{s+1,\dots ,n}\right],\hfill \end{array}$$

where $g_{ij}={\sum }_{k=1}^m{b}_{ik}{b}_{jk}$, and E signifies mathematical expectation. Given n stationary time series, $T_{A\to B}$ and $T_{B\to A}$ can be estimated. The maximum likelihood estimators under a Gaussian assumption are referred to in Equations (28) and (30).

We have constructed a VAR process to validate the formalism. The system has a dimension of 6, with two subsystems respectively denoted by X and Y, each with a dimension of 3. X drives Y via the coupling at one component, and Y feeds back to X via another. The detailed, componentwise causal relation can be easily found using our previous algorithms such as that in [20]. It is expected that the bulk information flow should in general also follow a similar trend, though the structure could be in a more coarse and mild fashion, as now displayed is an overall property. The above formalism does yield such a result. On the contrary, the commonly used proxies for subsystems, such as averages and principal components (PCs), generally do not work. Particularly, the averaged series yield the wrong results in the two cases considered in this study; the PC series do not work either for the mutually causal case, though they result in a satisfactory characterization for the case with a one-way causality.

The result of this study is applicable in many real world problems. As explained in the Introduction, it will be of particular use in the related fields of climate science, neuroscience, financial economics, fluid mechanics, etc. For example, it helps clarify the role of greenhouse gas emissions in bridging the climate system and the socioeconomic system (see a review in [26]). Likewise, the interaction between the earth system and public health [27] can also be studied. In short, it is expected to play a role in the frontier field of complexity, namely, multiplex networks or networks of networks (see the references in [28,29,30]). We are therefore working on these applications.