On Geometry of Information Flow for Causal Inference

Abstract

Causal inference is perhaps one of the most fundamental concepts in science, beginning originally from the works of some of the ancient philosophers, through today, but also weaved strongly in current work from statisticians, machine learning experts, and scientists from many other fields. This paper takes the perspective of information flow, which includes the Nobel prize winning work on Granger-causality, and the recently highly popular transfer entropy, these being probabilistic in nature. Our main contribution will be to develop analysis tools that will allow a geometric interpretation of information flow as a causal inference indicated by positive transfer entropy. We will describe the effective dimensionality of an underlying manifold as projected into the outcome space that summarizes information flow. Therefore, contrasting the probabilistic and geometric perspectives, we will introduce a new measure of causal inference based on the fractal correlation dimension conditionally applied to competing explanations of future forecasts, which we will write $Geo{C}_{y\to x}$. This avoids some of the boundedness issues that we show exist for the transfer entropy, $T_{y\to x}$. We will highlight our discussions with data developed from synthetic models of successively more complex nature: these include the Hénon map example, and finally a real physiological example relating breathing and heart rate function.

1. Introduction

Causation Inference is perhaps one of the most fundamental concepts in science, underlying questions such as “what are the causes of changes in observed variables”. Identifying, indeed even defining causal variables of a given observed variable is not an easy task, and these questions date back to the Greeks [1,2]. This includes important contributions from more recent luminaries such as Russel [3], and from philosophy, mathematics, probability, information theory, and computer science. We have written that [4], “a basic question when defining the concept of information flow is to contrast versions of reality for a dynamical system. Either a subcomponent is closed or alternatively there is an outside influence due to another component”. Claude Granger’s Nobel prize [5] winning work leading to Granger Causality (see also Wiener [6]) formulates causal inference as a concept of quality of forecasts. That is, we ask, does system X provide sufficient information regarding forecasts of future states of system X or are there improved forecasts with observations from system Y? We declare that X is not closed, as it is receiving influence (or information) from system Y, when data from Y improve forecasts of X. Such a reduction of uncertainty perspective of causal inference is not identical to the interventionists’ concept of allowing perturbations and experiments to decide what changes indicate influences. This data oriented philosophy of causal inference is especially appropriate when (1) the system is a dynamical system of some form producing data streams in time, and (2) a score of influence may be needed. In particular, contrasting forecasts is the defining concept underlying Granger Causality (G-causality), and it is closely related to the concept of information flow as defined by transfer entropy [7,8], which can be proved as a nonlinear version of Granger’s otherwise linear (ARMA) test [9]. In this spirit, we find methods such as Convergent Cross-Mapping method (CCM) [10], and causation entropy (CSE) [11] to disambiguate direct versus indirect influences [11,12,13,14,15,16,17,18]. On the other hand, closely related to information flow are concepts of counter factuals: “what would happen if...” [19] that are foundational questions for another school leading to the highly successful Pearl “Do-Calculus” built on a specialized variation of Bayesian analysis [20]. These are especially relevant for nondynamical questions (inputs and outputs occur once across populations), such as a typical question of the sort, “why did I get fat” may be premised on inferring probabilities of removing influences of saturated fats and chocolates. However, with concepts of counter-factual analysis in mind, one may argue that Granger is less descriptive of causation inference, but rather more descriptive of information flow. In fact, there is a link between the two notions for so-called “settable" systems under a conditional form of exogeneity [21,22].

This paper focuses on the information flow perspective, which is causation as it relates to G-causality. The role of this paper is to highlight connections between the probabilistic aspects of information flow, such as Granger causality and transfer entropy, to a less often discussed geometric picture that may underlie the information flow. To this purpose, here we develop both analysis and data driven concepts to serve in bridging what have otherwise been separate philosophies. Figure 1 illustrates the two nodes that we tackle here: causal inference and geometry. In the diagram, the equations that are most central in serving to bridge the main concepts are highlighted, and the main role of this paper then could be described as building these bridges.

Figure 1. Summary of the paper and relationship of causation and geometry.

When data are derived from a stochastic or deterministic dynamical system, one should also be able to understand the connections between variables in geometric terms. The traditional narrative of information flow is in terms of comparing stochastic processes in probabilistic terms. However, the role of this paper is to offer a unifying description for interpreting geometric formulations of causation together with traditional statistical or information theoretic interpretations. Thus, we will try to provide a bridge between concepts of causality as information flow to the underlying geometry since geometry is perhaps a natural place to describe a dynamical system.

Our work herein comes in two parts. First, we analyze connections between information flow by transfer entropy to geometric quantities that describe the orientation of underlying functions of a corresponding dynamical system. In the course of this analysis, we have needed to develop a new “asymmetric transfer operator” (asymmetric Frobenius–Perron operator) evolving ensemble densities of initial conditions between spaces whose dimensionalities do not match. With this, we proceed to give a new exact formula for transfer entropy, and from there we are able to relate this Kullback–Leibler divergence based measure directly to other more geometrically relevant divergences, specifically total variation divergence and Hellinger divergence, by Pinsker’s inequality. This leads to a succinct upper bound of the transfer entropy by quantities related to a more geometric description of the underlying dynamical system. In the second part of this work, we present numerical interpretations of transfer entropy $T{E}_{y\to x}$ in the setting of a succession of simple dynamical systems, with specifically designed underlying densities, and eventually we include a heart rate versus breathing rate data set. Then, we present a new measure in the spirit of G-causality that is more directly motivated by geometry. This measure, $Geo{C}_{y\to x}$, is developed in terms of the classical fractal dimension concept of correlation dimension.

In summary, the main theme of this work is to provide connections between probabilistic interpretations and geometric interpretations of causal inference. The main connections and corresponding sections of this paper are summarized as a dichotomy: Geometry and Causation (information flow structure) as described in Figure 1. Our contribution in this paper is as follows:

• In traditional methods, causality is estimated by probabilistic terms. In this study, we present analytical and data driven approach to identify causality by geometric methods, and thus also a unifying perspective.
• We show that a derivative (if it exists) of the underlining function of the time series has a close relationship to the transfer entropy (Section 2.3).
• We provide a new tool called $geoC$ to identify the causality by geometric terms (Section 3).
• Correlation dimension can be used as a measurement for dynamics of a dynamical system. We will show that this measurement can be used to identify the causality (Section 3).

Part I: Analysis of Connections between Probabilistic Methods and Geometric Interpretations

2. The Problem Setup

For now, we assume that $x,y$ are real valued scalars, but the multi-variate scenario will be discussed subsequently. We use a shorthand notation, $x:=x_n$, $x^{\prime }:=x_{n+1}$ for any particular time n, where the prime (${}^{\prime }$) notation denotes “next iterate”. Likewise, let $z=(x,y)$ denote the composite variable, and its future composite state, $z^{\prime }$. Consider the simplest of cases, where there are two coupled dynamical systems written as discrete time maps,

$$\begin{array}{ccc}\hfill x^{\prime }& =& f_1(x,y),\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill y^{\prime }& =& f_2(x,y).\hfill \end{array}$$

(2)

The definition of transfer entropy [7,8,23], measuring the influence of coupling from variables y onto the future of the variables x, denoted by $x^{\prime }$ is given by:

$$T_{y\to x}=D_{KL}(p\left(x^{\prime }\right|x)\left|\right|p\left(x^{\prime }\right|x,y)).$$

(3)

This hinges on the contrast between two alternative versions of the possible origins of $x^{\prime }$ and is premised on deciding one of the following two cases: Either

$$x^{\prime }=f_1\left(x\right),\mathrm{or}{x}^{\prime }=f_1(x,y)$$

(4)

is descriptive of the actual function $f_1$. The definition of $T_{y\to x}$ is defined to decide this question by comparing the deviation from a proposed Markov property,

$$p\left(x^{\prime }\right|x)\stackrel{?}{=}p\left(x^{\prime }\right|x,y).$$

(5)

The Kullback–Leibler divergence used here contrasts these two possible explanations of the process generating $x^{\prime }$. Since $D_{KL}$ may be written in terms of mutual information, the units are as any entropy, bits per time step. Notice that we have overloaded the notation writing $p\left(x^{\prime }\right|x)$ and $p\left(x^{\prime }\right|x,y)$. Our practice will be to rely on the arguments to distinguish functions as otherwise different (likewise distinguishing cases of $f_1\left(x\right)$ versus $f_1(x,y)$.

Consider that the coupling structure between variables may be characterized by the directed graph illustrated in Figure 2.

Figure 2. A directed graph presentation of the coupling stucture questions corresponding to Equations (1) and (2).

In one time step, without loss of generality, we may decide on Equation (4), the role of y on $x^{\prime }$, based on $T_{y\to x}>0$, exclusively in terms of the details of the argument structure of $f_1$. This is separate from the reverse question of $f_2$ as to whether $T_{x\to y}>0$. In geometric terms, assuming $f_1\in C^1\left({\Omega }_1\right)$, it is clear that, unless the partial derivative $\frac{\partial f_1}{\partial y}$ is zero everywhere, then the y argument in $f_1(x,y)$ is relevant. This is not a necessary condition for $T_{y\to x}>0$, which is a probabilistic statement, and almost everywhere is sufficient.

2.1. In Geometric Terms

Consider a manifold of points $(x,y,x^{\prime })\in X\times Y\times X^{\prime }$ as the graph over ${\Omega }_1$, which we label ${\mathcal{M}}_2$. In the following, we assume $f_1\in C^1\left({\Omega }_1\right),{\Omega }_1\subset X\times Y$. Our primary assertion here is that the geometric aspects of the set $(x,y,x^{\prime })$ projected into $(x,x^{\prime })$ distinguishes the information flow structure. Refer to Figure 3 for notation. Let the level set for a given fixed y be defined,

$$L_y:=\{(x,x^{\prime }):x^{\prime }=f(x,y),y=constant\}\in {\Omega }_2=X\times X^{\prime }$$

(6)

Figure 3. Ω₂ = X × X′ manifold and L_y level set for (a) x′ = f₁(x) = −0.005x² + 100, (b) x′ = f₁(x, y) = −0.005x² + 0.01y² + 50. The dimension of the projected set of (x, x′) depends on the causality as just described. Compare to Figure 4 and Equation (27).

When these level sets are distinct, then the question of the relevance of y to the outcome of $x^{\prime }$ is clear:

• If $\frac{\partial f_1}{\partial y}=0$ for all $(x,y)\in {\Omega }_1$, then $L_y=L_{\tilde{y}}$ for all $y,\tilde{y}$.

Notice that, if the y argument is not relevant as described above, then $x^{\prime }=f_1\left(x\right)$ better describes the associations, but if we nonetheless insist to write $x^{\prime }=f_1(x,y)$, then $\frac{\partial f_1}{\partial y}=0$ for all $(x,y)\in {\Omega }_1$. The converse is interesting to state explicitly,

• If $L_y\ne L_{\tilde{y}}$ for some $y,\tilde{y}$, then $\frac{\partial f_1}{\partial y}\ne 0$ for some $(x,y)\in {\Omega }_1$, and then $x^{\prime }=f_1\left(x\right)$ is not a sufficient description of what should really be written $x^{\prime }=f_1(x,y)$. We have assumed $f_1\in C^1\left({\Omega }_1\right)$ throughout.

2.2. In Probabilistic Terms

Considering the evolution of x as a stochastic process [8,24], we may write a probability density function in terms of all those variables that may be relevant, $p(x,y,x^{\prime })$. Contrasting the role of the various input variables requires us to develop a new singular transfer operator between domains that do not necessarily have the same number of variables. Notice that the definition of transfer entropy (Equation (3)) seems to rely on the absolute continuity of the joint probability density $p(x,y,x^{\prime })$. However, that joint distribution of $p(x,y,f(x,y\left)\right)$ is generally not absolutely continuous, noticing its support is $\{(x,y,f(x,y)):(x,y)\in {\Omega }_x\times {\Omega }_y\subseteq {\mathbb{R}}^2\}$, a measure 0 subset of ${\mathbb{R}}^3$. Therefore, the expression $h\left(f\right(X,Y\left)\right|X,Y)$ is not well defined as a differential entropy and hence there is a problem with transfer entropy. We expand upon this important detail in the upcoming subsection. To guarantee existence, we interpret these quantities by convolution to smooth the problem. Adding an “artificial noise” with standard deviation parameter $ϵ$ allows definition of the conditional entropy at the singular limit $ϵ$ approaches to zero, and likewise the transfer entropy follows.

The probability density function of the sum of two continuous random variables ($U,Z$) can be obtained by convolution, $P_{U+Z}=P_U*P_Z$. Random noise (Z with mean $\mathbb{E}\left(Z\right)=0$ and variance $\mathbb{V}\left(Z\right)=C{ϵ}^2$) added to the original observable variables regularizes, and we are interested in the singular limit, $ϵ\to 0$. We assume that Z is independent of $X,Y$. In experimental data from practical problems, we argue that some noise, perhaps even if small, is always present. Additionally, noise is assumed to be uniform or normally distributed in practical applications. Therefore, for simplicity of the discussion, we mostly focused on those two distributions. With this concept, Transfer Entropy can now be calculated by using $h\left(X^{\prime }\right|X,Y)$ and $h\left(X^{\prime }\right|X)$ when

$$\begin{array}{c}\hfill X^{\prime }=f(X,Y)+Z,\end{array}$$

(7)

where now we assume that $X,Y,Z\in \mathbb{R}$ are independent random variables and we assume that $f:{\Omega }_x\times {\Omega }_y\to \mathbb{R}$ is a component-wise monotonic (we will consider the monotonically increasing case for consistent explanations, but one can use monotonically decreasing functions in similar manner) continuous function of $X,Y$ and ${\Omega }_x,{\Omega }_y\subseteq \mathbb{R}$.

Relative Entropy for a Function of Random Variables

Calculation of transfer entropy depends on the conditional probability. Hence, we will first focus on conditional probability. Since for any particular values $x,y$ the function value $f(x,y)$ is fixed, we conclude that $X^{\prime }|x,y$ is just a linear function of Z. We see that

$$\begin{array}{c}\hfill p_{X^{\prime }|X,Y}\left(x^{\prime }\right|x,y)=Pr(Z=x^{\prime }-f(x,y))=p_Z(x^{\prime }-f(x,y)),\end{array}$$

(8)

where $p_Z$ is the probability density function of Z.

Note that the random variable $X^{\prime }|x$ is a function of $(Y,Z)$. To write $U+Z$, let $U=f(x,Y)$. Therefore, convolution of densities of U and Z gives the density function for $p\left(x^{\prime }\right|x)$ (See Section 4.1 for examples). Notice that a given value of the random variable, say $X=\alpha$, is a parameter in U. Therefore, we will denote $U=f(Y;\alpha )$. We will first focus on the probability density function of U, $p_U\left(u\right)$, using the Frobenius–Perron operator,

$$\begin{array}{c}\hfill p_U\left(u\right)=\sum _{y:u=f(y;\alpha )}\frac{p_Y\left(f(y;\alpha )\right)}{|f^{\prime }\left(f(y;\alpha )\right)|}.\end{array}$$

(9)

In the multivariate setting, the formula is extended similarly interpreting the derivative as the Jacobian matrix, and the absolute value is interpreted as the absolute value of the determinant. Denote $\mathbf{Y}=(Y_1,Y_2,\cdots ,Y_n),\mathbf{g}(\mathbf{Y};\alpha )=(g_1,g_2,\cdots ,g_n)$ and $U=f(\alpha ,\mathbf{Y}):=g_1(\mathbf{Y};\alpha )$; and the vector $\mathbf{V}=(V_1,V_2,\cdots ,V_{n-1})\in {\mathbb{R}}^{n-1}$ such that $V_i=g_{i+1}\left(\mathbf{Y}\right):=Y_{i+1}$ for $i=1,2,\cdots ,n-1$. Then, the absolute value of the determinate of the Jacobian matrix is given by: $|J_g\left(\mathbf{y}\right)|=|\frac{\partial g_1(\mathbf{y};\alpha )}{\partial y_1}|$. As an aside, note that J is lower triangular with diagonal entries $d_{ii}=1$ for $i>1$. The probability density function of U is given by

$$\begin{array}{c}\hfill p_U\left(u\right)={\int }_S{p}_{\mathbf{Y}}\left(g^{-1}(u,\mathbf{v};\alpha )\right)|\frac{\partial g_1}{\partial y_1}\left(g^{-1}(u,\mathbf{v};\alpha )\right){|}^{-1}d\mathbf{v},\end{array}$$

(10)

where S is the support set of the random variable $\mathbf{V}$.

Since the random variable $X^{\prime }|x$ can be written as a sum of U and Z, we find the probability density function by convolution as follows:

$$\begin{array}{c}\hfill p_{X^{\prime }|x}\left(x^{\prime }\right|x)=\int p_U\left(u\right)p_Z(x^{\prime }-u)du.\end{array}$$

(11)

Now, the conditional differential entropy $h\left(Z\right|X,Y)$ is in terms of these probability densities. It is useful that translation does not change the differential entropy, $h_{ϵ}(f(X,Y)+Z|X,Y)=h\left(Z\right|X,Y)$. In addition, Z is independent from $X,Y$, $h\left(Z\right|X,Y)=h(Z)$. Now, we define

$$\begin{array}{c}\hfill h\left(f(X,Y)\right|X,Y):=\underset{ϵ\to 0^{+}}{lim}{h}_{ϵ}(f(X,Y)+Z|X,Y)\end{array}$$

(12)

if this limit exists.

We consider two scenarios: (1) Z is a uniform random variable or (2) Z is a Gaussian random variable. If it is uniform in the interval $[-ϵ/2,ϵ/2],$ then the differential entropy is $h\left(Z\right)=ln\left(ϵ\right)$. If specifically, Z is Gaussian with zero mean and $ϵ$ standard deviation, then $h\left(Z\right)=\frac{1}{2}ln\left(2\pi e{ϵ}^2\right)$. Therefore, $h_{ϵ}(f(X,Y)+Z|X,Y)\to -\infty$ as $ϵ\to 0^{+}$ in both cases. Therefore, $h\left(f\right(X,Y\left)\right|X,Y\left)\right)$ is not finite in this definition (Equation (12)) as well. Thus, instead of calculating $X^{\prime }=f(X,Y)$, we need to use a noisy version of data $X^{\prime }=f(X,Y)+Z$. For that case,

$$\begin{array}{c}\hfill h\left(X^{\prime }\right|X,Y)=h\left(Z\right)=\left\{\begin{array}{cc}ln\left(ϵ\right);\hfill & Z\sim U(-ϵ/2,ϵ/2)\hfill \\ \frac{1}{2}ln\left(2\pi e{ϵ}^2\right);\hfill & Z\sim \mathcal{N}(0,{ϵ}^2)\hfill \end{array}\right.,\end{array}$$

(13)

where $U(-ϵ/2,ϵ/2)$ is the uniform distribution in the interval $[-ϵ/2,ϵ/2],$ and $\mathcal{N}(0,{ϵ}^2)$ is a Gaussian distribution with zero mean and $ϵ$ standard deviation.

Now, we focus on $h\left(X^{\prime }\right|X)$. If $X^{\prime }$ is just a function of X, then we can similarly show that: if $X^{\prime }=f\left(X\right),$ then

$$\begin{array}{c}\hfill h(f\left(X\right)+Z|X)=h\left(Z\right)=\left\{\begin{array}{cc}ln\left(ϵ\right);\hfill & Z\sim U(-ϵ/2,ϵ/2)\hfill \\ \frac{1}{2}ln\left(2\pi e{ϵ}^2\right);\hfill & Z\sim \mathcal{N}(0,{ϵ}^2).\hfill \end{array}\right.\end{array}$$

(14)

In addition, notice that, if $X^{\prime }=f(X,Y)$, then $h\left(X^{\prime }\right|X)$ will exist, and most of the cases will be finite. However, when we calculate $T_{y\to x}$, we need to use the noisy version to avoid the issues in calculating $h\left(X^{\prime }\right|X,Y)$. We will now consider the interesting case $X^{\prime }=f(X,Y)+Z$ and calculate $h\left(X^{\prime }\right|X)$. We require $p_{X^{\prime }|X}$ and Equation (11) can be used to calculate this probability. Let us denote $I:=\int p_U\left(u\right)p_Z(x^{\prime }-u)du;$ then,

$$\begin{array}{cc}\hfill h_{ϵ}\left(X^{\prime }\right|X)& =\int \int I{p}_X\left(x\right)ln\left(I\right)d{x}^{\prime }dx\hfill \\ \hfill & =\int p_X\left(x\right)\int Iln\left(I\right)d{x}^{\prime }dx\hfill \\ \hfill & ={\mathbb{E}}_X\left(Q\right),\hfill \end{array}$$

(15)

where $Q=\int Iln\left(I\right)d{x}^{\prime }$. Notice that, if Q does not depend on $x,$ then $h\left(X^{\prime }\right|X)=Q\int p_{X}dx=Q$ because $\int p_{X}dx=1$(since $p_x$ is a probability density function). Therefore, we can calculate $h_{ϵ}\left(X^{\prime }\right|X)$ by four steps. First, we calculate the density function for $U=f(x,Y)$ (by using Equation (9) or (10)). Then, we calculate $I=p_{X^{\prime }|X}$ by using Equation (11). Next, we calculate the value of $Q,$ and finally we calculate the value of $h_{ϵ}\left(X^{\prime }\right|X)$.

Thus, the transfer entropy from y to x follows in terms of comparing conditional entropies,

$$\begin{array}{c}\hfill T_{y\to x}=h\left(X^{\prime }\right|X)-h\left(X^{\prime }\right|X,Y).\end{array}$$

(16)

This quantity is not well defined when $X^{\prime }=f(X,Y)$, and therefore we considered the $X^{\prime }=f(X,Y)+Z$ case. This interpretation of transfer entropy depends on the parameter $ϵ$, as we define

$$\begin{array}{c}\hfill T_{y\to x}:=\underset{ϵ\to 0^{+}}{lim}{T}_{y\to x}\left(ϵ\right)=\underset{ϵ\to 0^{+}}{lim}{h}_{ϵ}\left(X^{\prime }\right|X)-h_{ϵ}\left(X^{\prime }\right|X,Y)\end{array}$$

(17)

if this limit exists.

Note that

$$\begin{array}{c}\hfill T_{y\to x}=\left\{\begin{array}{cc}\underset{ϵ\to 0^{+}}{lim}h\left(Z\right)-h\left(Z\right)=0;\hfill & X^{\prime }=f\left(X\right)\hfill \\ \infty ;\hfill & X^{\prime }=f(X,Y)\ne f\left(X\right).\hfill \end{array}\right.\end{array}$$

(18)

Thus, we see that a finite quantity is ensured by the noise term. We can easily find an upper bound for the transfer entropy when $X^{\prime }=f(X,Y)+Z$ is a random variable with finite support (with all the other assumptions mentioned earlier) and suppose $Z\sim U(-ϵ/2,ϵ/2)$. First, notice that the uniform distribution maximizes entropy amongst all distributions of continuous random variables with finite support. If f is component-wise monotonically increasing continuous function, then the support of $X^{\prime }|x$ is $[f(x,y_{min})-ϵ/2,f(x,y_{min})+ϵ/2]$ for all $x\in {\Omega }_x$. Here, $y_{min}$ and $y_{max}$ are minimum and maximum values of Y. Then, it follows that

$$\begin{array}{c}\hfill h_{ϵ}\left(X^{\prime }\right|X)\le ln\left(\right|f(x_{max},y_{max})-f(x_{max},y_{min})+ϵ\left|\right),\end{array}$$

(19)

where $x_{max}$ is the maximum x value. We see that an interesting upper bound for transfer entropy follows:

$$\begin{array}{c}\hfill T_{y\to x}\left(ϵ\right)\le ln\left(|\frac{f(x_{max},y_{max})-f(x_{max},y_{min})}{ϵ}+1|\right).\end{array}$$

(20)

2.3. Relating Transfer Entropy to a Geometric Bound

Noting that transfer entropy and other variations of the G-causality concept are expressed in terms of conditional probabilities, we recall that

$$\rho \left(x^{\prime }\right|x,y)\rho (x,y)=\rho (x,y,x^{\prime }).$$

(21)

Again, we continue to overload the notation on the functions $\rho$, the details of the arguments distinguishing to which of these functions we refer.

Now, consider the change of random variable formulas that map between probability density functions by smooth transformations. In the case that $x^{\prime }=f_1\left(x\right)$ (in the special case that $f_1$ is one-one), then

$$\rho \left(x^{\prime }\right)=\frac{\rho \left(x\right)}{\left|\frac{d{f}_1}{dx}\left(x\right)\right|}=\frac{\rho \left(f_1^{-1}\left(x^{\prime }\right)\right)}{|\frac{d{f}_1}{dx}\left(f_1^{-1}\left(x^{\prime }\right)\right)|}.$$

(22)

In the more general case, not assuming one-one-ness, we get the usual Frobenius–Perron operator,

$$\rho \left(x^{\prime }\right)=\sum _{x:x^{\prime }=f_1\left(x\right)}\rho (x,x^{\prime })=\sum _{x:x^{\prime }=f_1\left(x\right)}\frac{\rho \left(x\right)}{\left|\frac{d{f}_1}{dx}\left(x\right)\right|},$$

(23)

in terms of a summation over all pre-images of $x^{\prime }$. Notice also that the middle form is written as a marginalization across x of all those x that lead to $x^{\prime }$. This Frobenius–Perron operator, as usual, maps densities of ensembles of initial conditions under the action of the map $f_1$.

Comparing to the expression

$$\rho (x,x^{\prime })=\rho \left(x^{\prime }\right|x)\rho \left(x\right),$$

(24)

we assert the interpretation that

$$\rho \left(x^{\prime }\right|x):=\frac{1}{|\frac{d{f}_1}{dx}\left(x\right)|}\delta (x^{\prime }-f_1\left(x\right)),$$

(25)

where $\delta$ is the Dirac delta function. In the language of Bayesian uncertainty propagation, $p\left(x^{\prime }\right|x)$ describes the likelihood function, if interpreting the future state $x^{\prime }$ as data, and the past state x as parameters, in a standard Bayes description, $p\left(\mathrm{data}\right|\mathrm{parameter})\times p(\mathrm{parameter})$. As usual for any likelihood function, while it is a probability distribution over the data argument, it may not necessarily be so with respect to the parameter argument.

Now, consider the case where $x^{\prime }$ is indeed nontrivially a function with respect to not just x, but also with respect to y. Then, we require the following asymmetric space transfer operator, which we name here an asymmetric Frobenius–Perron operator for smooth transformations between spaces of dissimilar dimensionality:

Theorem 1

(Asymmetric Space Transfer Operator). If $x^{\prime }=f_1(x,y)$, for $f_1:{\Omega }_1\to \mathsf{{\rm Y}}$, given bounded open domain $(x,y)\in {\Omega }_1\subset {\mathbb{R}}^{2d}$, and range $x^{\prime }\in \mathsf{{\rm Y}}\subset {\mathbb{R}}^d$, and $f_1\in C^1\left({\Omega }_1\right)$, and the Jacobian matrices, $\frac{\partial f_1}{\partial x}(x,y)$, and $\frac{\partial f_1}{\partial y}(x,y)$ are not both rank deficient at the same time, then taking the initial density $\rho (x,y)\in L^1\left({\Omega }_1\right)$, the following serves as a transfer operator mapping asymmetrically defined densities $P:L^1\left({\Omega }_1\right)\to L^1\left(\mathsf{{\rm Y}}\right)$

$$\rho \left(x^{\prime }\right)=\sum _{(x,y):x^{\prime }=f_1(x,y)}\rho (x,y,x^{\prime })=\sum _{(x,y):x^{\prime }=f_1(x,y)}\frac{\rho (x,y)}{\left|\frac{\partial f_1}{\partial x}(x,y)\right|+\left|\frac{\partial f_1}{\partial y}(x,y)\right|}.$$

(26)

The proof of this is in Appendix A. Note also that, by similar argumentation, one can formulate the asymmetric Frobenius–Perron type operator between sets of dissimilar dimensionality in an integral form.

Corollary 1

(Asymmetric Transfer Operator, Kernel Integral Form). Under the same hypothesis as Theorem 1, we may alternatively write the integral kernel form of the expression,

$$\begin{array}{cccc}\hfill P:L^2\left({\mathbb{R}}^2\right)& \to \hfill & L^2\left(\mathbb{R}\right)\hfill & \\ \hfill \rho (x,y)& & \mapsto & {\rho }^{\prime }\left(x^{\prime }\right)=P\left[\rho \right](x,y)]\hfill \\ & & =& \hfill \\ & & =& {\int }_{L_{x^{\prime }}}\rho (x,y,x^{\prime })dxdy={\int }_{L_{x^{\prime }}}\rho \left(x^{\prime }\right|x,y)\rho (x,y)dxdy\hfill \end{array}$$

(27)

$$\begin{array}{cc}=& {\int }_{L_{x^{\prime }}}\frac{1}{|\frac{\partial f_1}{\partial x}(x,y)|+|\frac{\partial f_1}{\partial y}(x,y)|}\rho (x,y)dxdy.\hfill \end{array}$$

(28)

This is in terms of a line integration along the level set, $L_{x^{\prime }}$. See Figure 4:

$$L_{x^{\prime }}=\{(x,y)\in {\Omega }_1:f(x,y)=x^{\prime }a chosen constant.\}$$

(29)

Figure 4. The asymmetric transfer operator, Equation (27), is written in terms of intefration over the level set, $L_{x^{\prime }}$ of $x^{\prime }=f_1(x,y)$ associated with a fixed value $x^{\prime }$, Equation (29).

In Figure 4, we have shown a typical scenario where a level set is a curve (or it may well be a union of disjoint curves), whereas, in a typical FP-operator between sets of the same dimensionality, generally the integration is between pre-images that are usually either singletons, or unions of such points, ${\rho }^{\prime }\left(x^{\prime }\right)=\int \delta (s-f\left(x\right))\rho \left(s\right)ds={\sum }_{x:f\left(x\right)=x^{\prime }}\frac{\rho \left(x\right)}{\left|Df\right(x\left)\right|}$.

Contrasting standard and the asymmetric forms of transfer operators as described above, in the next section, we will compute and bound estimates for the transfer entropy. However, it should already be apparent that, if $\frac{\partial f_1}{\partial y}=0$ in probability with respect to $\rho (x,y)$, then $T_{y\to x}=0$.

Comparison to other statistical divergences reveals geometric relevance: Information flow is quite naturally defined by the KL-divergence, in that it comes in the units of entropy, e.g., bits per second. However, the well-known Pinsker’s inequality [25] allows us to more easily relate the transfer entropy to a quantity that has a geometric relevance using the total variation, even if this is only by an inequality estimate.

Recall that Pinsker’s inequality [25] relates random variables with probability distributions p and q over the same support to the total variation and the KL-divergence as follows:

$$0\le \frac{1}{2}TV(P,Q)\le \sqrt{D_{KL}\left(P\right|\left|Q\right)},$$

(30)

written as probability measures P, Q. The total variation distance between probability measures is a maximal absolute difference of possible events,

$$TV(P,Q)=\underset{A}{sup}|P\left(A\right)-Q\left(A\right)|,$$

(31)

but it is well known to be related to 1/2 of the L1-distance in the case of a common dominating measure, $p\left(x\right)d\mu =dP$, $q\left(x\right)d\mu =dQ$. In this work, we only need absolute continuity with respect to Lebesgue measure, $p\left(x\right)=dP\left(x\right)$, $q\left(x\right)=dQ\left(x\right)$; then,

$$TV(P,Q)=\frac{1}{2}\int |p\left(x\right)-q\left(x\right)|dx=\frac{1}{2}{\parallel p-q\parallel }_{L^1},$$

(32)

here with respect to Lebesgue measure. In addition, we write $D_{KL}\left(P\right|\left|Q\right)=\int p\left(x\right)log\frac{p\left(x\right)}{q\left(x\right)}dx$; therefore,

$$\frac{1}{2}{\parallel p-q\parallel }_{L^1}^2\le \int p\left(x\right)log\frac{p\left(x\right)}{q\left(x\right)}dx.$$

(33)

Thus, with the Pinsker inequality, we can bound the transfer entropy from below by inserting the definition Equation (3) into the above:

$$0\le \frac{1}{2}{\parallel p\left(x^{\prime }\right|x,y)-p\left(x^{\prime }\right|x)\parallel }_{L^1}^2\le T_{y\to x}.$$

(34)

The assumption that the two distributions correspond to a common dominating measure requires that we interpret $p\left(x^{\prime }\right|x)$ as a distribution averaged across the same $\rho (x,y)$ as $p\left(x^{\prime }\right|x,y)$. (Recall by definition [26] that $\lambda$ is a common dominating measure of P and Q if $p\left(x\right)=dP/d\lambda$ and $q\left(x\right)=dQ/d\lambda$ describe corresponding densities). For the sake of simplification, we interpret transfer entropy relative to a uniform initial density, $\rho (x,y)$, for both entropies of Equation (16). With this assumption, we interpret

$$0\le \frac{1}{2}{\parallel \frac{1}{|\frac{\partial f_1}{\partial x}(x,y)|+|\frac{\partial f_1}{\partial y}(x,y)|}-\frac{1}{\left|\frac{d{f}_1}{dx}\left(x\right)\right|}\parallel }_{L^1({\Omega }_1,\rho (x,y))}^2\le T_{y\to x}.$$

(35)

In the special case that there is very little information flow, we would expect that $|\frac{\partial f_1}{\partial y}|<b<<1$, and $b<<|\frac{\partial f_1}{\partial x}|$, almost every $x,y$; then, a power series expansion in small b gives

$$\frac{1}{2}{\parallel \frac{1}{\left|\frac{\partial f_1}{\partial x}(x,y)\right|+\left|\frac{\partial f_1}{\partial y}(x,y)\right|}-\frac{1}{\left|\frac{d{f}_1}{dx}\left(x\right)\right|}\parallel }_{L^1({\Omega }_1,\rho (x,y))}^2\approx \frac{Vol\left({\Omega }_1\right)}{2}\frac{<|\frac{\partial f_1}{\partial y}|{>}^2}{<|\frac{\partial f_1}{\partial x}|{>}^4},$$

(36)

which serves approximately as the TV-lower bound for transfer entropy where have used the notation $<·>$ to denote an average across the domain. Notice that, therefore, $\delta (p\left(x^{\prime }\right|x,y),p\left(x^{\prime }\right|x))\downarrow$ as $|\frac{\partial f_1}{\partial y}|\downarrow$. While Pinsker’s inequality cannot guarantee that $T_{y\to x}\downarrow$, since TV is only an upper bound, it is clearly suggestive. In summary, comparing inequality Equation (35) to the approximation (36) suggests that, for $|\frac{\partial f_1}{\partial y}|<<b<<|\frac{\partial f_1}{\partial x}|$, for $b>0$, for a.e. $x,y$, then $T_{y\to x}\downarrow$ as $b\downarrow$.

Now, we change to a more computational direction of this story of interpreting information flow in geometric terms. With the strong connection described in the following section, we bring to the problem of information flow between geometric concepts to information flow concepts, such as entropy, it is natural to turn to studying the dimensionality of the outcome spaces, as we will now develop.

Part II: Numerics and Examples of Geometric Interpretations

Now, we will explore numerical estimation aspects of transfer entropy for causation inference in relationship to geometry as described theoretically in the previous section, and we will compare this numerical approach to geometric aspects.

3. Geometry of Information Flow

As theory suggests, see the sections above, there is a strong relationship between the information flow (causality as measured by transfer entropy) and the geometry, encoded for example in the estimates leading to Equation (36). The effective dimensionality of the underlying manifold as projected into the outcome space is a key factor to identify the causal inference between chosen variables. Indeed, any question of causality is in fact observer dependent. To this point, suppose $x^{\prime }$ only depends on $x,y$ and $x^{\prime }=f(x,y)$, where $f\in C^1\left({\Omega }_1\right)$. We noticed that (Section 2) $T_{y\to x}=0\iff \frac{\partial f}{\partial y}=0,\forall (x,y)\in {\Omega }_1$. Now, notice that $\frac{\partial f}{\partial y}=0,\forall (x,y)\in {\Omega }_1\iff x^{\prime }=f(x,y)=f\left(x\right).$ Therefore, in the case that ${\Omega }_1$ is two-dimensional, then $(x,x^{\prime })$ would be a one-dimensional, manifold if and only if $\frac{\partial f}{\partial y}=0,\forall (x,y)\in {\Omega }_1$. See Figure 3. With these assumptions,

$$T_{y\to x}=0\iff (x,x^{\prime })\mathrm{data}\mathrm{lie}\mathrm{on}\mathrm{a}1D\mathrm{manifold}.$$

Likewise, for more general dimensionality of the initial ${\Omega }_1$, the story of the information flow between variables is in part a story of how the image manifold is projected. Therefore, our discussion will focus on estimating the dimensionality in order to identify the nature of the underlying manifold. Then, we will focus on identifying causality by estimating the dimension of the manifold, or even more generally of the resulting set if it is not a manifold but perhaps even a fractal. Finally, this naturally leads us to introduce a new geometric measure for characterizing the causation, which we will identify as $Ge{o}_{y\to x}$.

3.1. Relating the Information Flow as Geometric Orientation of Data

For a given time series $x:=x_n\in {\mathbb{R}}^{d_1},y:=y_n\in {\mathbb{R}}^{d_2}$, consider the $x^{\prime }:=x_{n+1}$ and contrast the dimensionalities of $(x,y,x^{\prime })$ versus $(x,x^{\prime })$, in order to identify that $x^{\prime }=f\left(x\right)$ or $x^{\prime }=f(x,y)$. Thus, in mimicking the premise of Granger causality, or likewise of Transfer entropy, contrasting these two versions of the explanations of $x^{\prime }$, in terms of either $(x,y)$ or x, we decide the causal inference, but this time, by using only the geometric interpretation. First, we recall how fractal dimensionality evolves under transformations, [27].

Theorem 2

([27]). Let A be a bounded Borel subset of ${\mathbb{R}}^{d_1}$. Consider the function $F:A\to {\mathbb{R}}^{d_1}\times {\mathbb{R}}^{d_1}$ such that $F\left(x\right)=(x,x^{\prime })$ for some $x^{\prime }\in {\mathbb{R}}^{d_1}$. The correlation dimension $D_2\left(F\left(A\right)\right)\le d_1$, if and only if there exists a function $f:A\to {\mathbb{R}}^{d_1}$ such that $x^{\prime }=f\left(x\right)$ with $f\in C^1\left(A\right)$.

The idea of the arguments in the complete proof found in Sauer et. al., [27], are as follows. Let A be bounded Borel subset of ${\mathbb{R}}^{d_1}$ and $f:A\to {\mathbb{R}}^{d_1}$ with $f\in C^1\left(A\right)$. Then, $D_2\left(f\left(A\right)\right)=D_2\left(A\right)$, where $D_2$ is the correlation dimension [28]. Note that $D_2\left(A\right)\le d_1$. Therefore, $D_2\left(F\left(A\right)\right)=D_2\left(A\right)\le d_1$, with $F:A\to {\mathbb{R}}^{d_1}\times {\mathbb{R}}^{d_1}$ if and only if $F\left(x\right)=(x,f(x\left)\right)$.

Now, we can describe this dimensional statement in terms of our information flow causality discussion, to develop an alternative measure of inference between variables. Let $(x,x^{\prime })\in {\Omega }_2\subset {\mathbb{R}}^{2d_1}$ and $(x,y,x^{\prime })\in {\Omega }_3\subset {\mathbb{R}}^{2d_1+d_2}$. We assert that there is a causal inference from y to x, if $dim\left({\Omega }_2\right)>d_1$ and $d_1<dim\left({\Omega }_3\right)\le d_1+d_2$, (Theorem 1). In this paper, we focus on time series $x_n\in \mathbb{R}$ which might also depend on time series $y_n\in \mathbb{R}$, and we will consider the geometric causation from y to x, for $(x,y)\in A\times B={\Omega }_1\subset {\mathbb{R}}^2$. We will denote geometric causation by $Geo{C}_{y\to x}$ and assume that $A,B$ are Borel subsets of $\mathbb{R}$. Correlation dimension is used to estimate the dimensionality. First, we identify the causality using the dimensionality of on $(x,x^{\prime })$ and $(x,y,x^{\prime })$. Say, for example, that $(x,x^{\prime })\in {\Omega }_2\subset {\mathbb{R}}^2$ and $(x,y,x^{\prime })\in {\Omega }_3\subset {\mathbb{R}}^3$; then, clearly we would enumerate a correlation dimension causal inference from y to x, if $dim\left({\Omega }_2\right)>1$ and $1<dim\left({\Omega }_3\right)\le 2$ (Theorem 1).

3.2. Measure Causality by Correlation Dimension

As we have been discussing, the information flow of a dynamical system can be described geometrically by studying the sets (perhaps they are manifolds) $X\times X^{\prime }$ and $X\times Y\times X^{\prime }$. As we noticed in the last section, comparing the dimension of these sets can be interpreted as descriptive of information flow. Whether dimensionality be estimated from data or by a convenient fractal measure such as the correlation dimension ($D_2(.)$), there is an interpretation of information flow when contrasting $X\times X^{\prime }$ versus $X\times Y\times X^{\prime }$, in a spirit reminiscent of what is done with transfer entropy. However, these details are geometrically more to the point.

Here, we define $Geo{C}_{y\to x}$ (geometric information flow) by $GeoC(.|.)$ as conditional correlation dimension.

Definition 1

(Conditional Correlation Dimensional Geometric Information Flow). Let $\mathcal{M}$ be the manifold of data set $(X_1,X_2,\cdots ,X_n,X^{\prime })$ and let ${\Omega }_1$ be the data set $(X_1,X_2,\cdots ,X_n)$. Suppose that the $\mathcal{M},{\Omega }_1$ are bounded Borel sets. The quantity

$$\begin{array}{c}\hfill GeoC\left(X^{\prime }\right|X_1,\cdots ,X_n):=D_2\left(\mathcal{M}\right)-D_2\left({\Omega }_1\right)\end{array}$$

(37)

is defined as “Conditional Correlation Dimensional Geometric Information Flow". Here, $D_2(.)$ is the usual correlation dimension of the given set, [29,30,31].

Definition 2

(Correlation Dimensional Geometric Information Flow). Let $x:=x_n,y=y_n\in \mathbb{R}$ be two time series. The correlation dimensional geometric information flow from y to x as measured by the correlation dimension and denoted by $Geo{C}_{y\to x}$ is given by

$$\begin{array}{c}\hfill Geo{C}_{y\to x}:=GeoC\left(X^{\prime }\right|X)-GeoC\left(X^{\prime }\right|X,Y).\end{array}$$

(38)

A key observation is to notice that, if $X^{\prime }$ is a function of $(X_1,X_2,\cdots ,X_n)$, then $D_2\left(\mathcal{M}\right)=D_2\left({\Omega }_1\right)$; otherwise, $D_2\left(\mathcal{M}\right)>D_2\left({\Omega }_1\right)$ (Theorem 1). If X is not influenced by y, then $GeoC\left(X^{\prime }\right|X)=0,$$GeoC\left(X^{\prime }\right|X,Y)=0$ and therefore $Geo{C}_{y\to x}=0$. In addition, notice that $Geo{C}_{y\to x}\le D_2\left(X\right)$, where $X=\left\{x_n\right|n=1,2,\cdots \}$. For example, if $x_n\in \mathbb{R}$, then $Geo{C}_{y\to x}\le 1$. Since we assume that influence of any time series $z_n\ne x_n,y_n$ to $x_n$ is relatively small, we can conclude that $Geo{C}_{y\to x}\ge 0$, and, if $x^{\prime }=f(x,y)$, then $GeoC\left(X^{\prime }\right|X,Y)=0$. Additionally, the dimension ($GeoC\left(X^{\prime }\right|X)$) in the $(X,X^{\prime })$ data scores how much additional (other than X) information is needed to describe the $X^{\prime }$ variable. Similarly, the dimension $GeoC\left(X^{\prime }\right|X,Y)$ in the $(X,Y,X^{\prime })$ data describes how much additional (other than $X,Y$) information is needed to define $X^{\prime }$. However, when the number of data points $N\to \infty$, the value $Geo{C}_{y\to x}$ is not negative (equal to the dimension of X data). Thus, theoretically, $GeoC$ identifies a causality in the geometric sense we have been describing.

4. Results and Discussion

Now, we present specific examples to contrast the transfer entropy with our proposed geometric measure to further highlight the role of geometry in such questions. Table 1 provides a summary of our numerical results. We use synthetic examples with known underlining dynamics to understand the accuracy of our model. Calculating transfer entropy has theoretical and numerical issues for those chosen examples while our geometric approach accurately identifies the causation. We use the correlation dimension of the data because data might be fractals. Using a Hénon map example, we demonstrate that fractal data will not affect our calculations. Furthermore, we use a real-world application that has a positive transfer entropy to explain our data-driven geometric method. Details of these examples can be found in the following subsections.

Table 1. Summary of the results. Here, we experiment our new approach by synthetics and real world application data.

Data	Transfer Entropy (Section 4.1)	Geometric Approach
Synthetic: f(x,y)=$aX+bY+C$, $a,b,c\in \mathbb{R}$	Theoretical issues can be noticed. Numerical estimation have boundedness issues when $b<<1$.	Successfully identify the causation for all the cases ($100\%$).
Synthetic: f(x,y)=$a{g}_1\left(X\right)+b{g}_2\left(Y\right)+C$, $a,b,c\in \mathbb{R}$	Theoretical issues can be noticed. Numerical estimation have boundedness issues when $b<<1$.	Successfully identify the causation for all the cases ($100\%$).
Hénon map: use data set invariant under the map.	special case of $a{X}^2+bY+C$ with $a=-1.4,b=c=1$. Estimated transfer entropy is positive.	Successfully identify the causation.
Application: heart rate vs. breathing rate	Positive transfer entropy.	Identify positive causation. It also provides more details about the data.

4.1. Transfer Entropy

In this section, we will focus on analytical results and numerical estimators for conditional entropy and transfer entropy for specific examples (see Figure 5 and Figure 6). As we discussed in previous sections starting with Section 2.2, computing the transfer entropy for $X^{\prime }=f(X,Y)$ has technical difficulties due to the singularity of the quantity $h\left(X^{\prime }\right|X,Y)$. First, we will consider the calculation of $h\left(X^{\prime }\right|X)$ for $X^{\prime }=f(X,Y)$, and then we will discuss the calculation for noisy data. In the following examples, we assumed that $X,Y$ are random variables such that $X,Y\stackrel{iid}{\sim }U\left([1,2]\right)$. A summary of the calculations for a few examples are listed in Table 2.

Figure 5. Conditional entropy h(X′|X). Note that these numerical estimates for the conditional entropy by the KSG method [32], converge (as N → ∞) to the analytic solutions (see Table 2).

Figure 6. Numerical results and analytical results for transfer entropy T_y→x(ε; b) to the problem X′ = X + bY + ε. Transfer entropy vs. ε shows in (a) for fixed b value. (b) and (c) show the behavior of the transfer entropy for b values with fixed e values. Notice that convergence of numerical solution is slow when epsilon is small.

Table 2. Conditional entropy $h\left(X^{\prime }\right|X)$ for $X^{\prime }=f(X,Y)$, for specific parametric examples listed, under the assumption that $X,Y\stackrel{iid}{\sim }U\left([1,2]\right)$.

$\mathit{f}(\mathit{X},\mathit{Y})$	$\mathit{h}\left({\mathit{X}}^{\prime }\right|\mathit{X})$
$g\left(X\right)+bY$	$ln\left(b\right)$
$g\left(X\right)+b{Y}^2$	$ln\left(8b\right)-5/2$
$g\left(X\right)+bln\left(Y\right)$	$ln\left(\frac{be}{4}\right)$

We will discuss the transfer entropy with noisy data because making $h\left(X^{\prime }\right|X,Y)$ well defined requires absolute continuity of the probability density function $p(x,y,x^{\prime })$. Consider, for example, the problem form $X^{\prime }=g\left(X\right)+bY+C$, where $X,Y$ are uniformly distributed independent random variables over the interval $[1,2]$ (the same analysis can be extend to any finite interval) with b being a constant, and g a function of random variable X. We will also consider C to be a random variable, which is distributed uniformly on $[-ϵ/2,ϵ/2]$. Note that it follows that $h\left(X^{\prime }\right|X,Y)=lnϵ$. To calculate the $h\left(X^{\prime }\right|X)$, we need to find the conditional probability $p\left(X^{\prime }\right|x)$ and observe that $X^{\prime }|x=U+C$, where $U=g\left(x\right)+bY$. Therefore,

$$\begin{array}{c}\hfill p_U\left(u\right)=\left\{\begin{array}{cc}\frac{1}{b}\hfill & ;g_1\left(x\right)+b\le X^{\prime }\le g_1\left(x\right)+2b\hfill \\ 0\hfill & ;otherwise.\hfill \end{array}\right.\end{array}$$

(39)

and

$$\begin{array}{c}\hfill p_{X^{\prime }|X}\left(X^{\prime }\right|x)=\left\{\begin{array}{cc}\frac{x^{\prime }+ϵ/2-g\left(x\right)}{bϵ}\hfill & ;g\left(x\right)-ϵ/2\le X^{\prime }\le g\left(x\right)+ϵ/2\hfill \\ \frac{1}{b}\hfill & ;g\left(x\right)+ϵ/2\le X^{\prime }\le b+g\left(x\right)-ϵ/2\hfill \\ \frac{-x^{\prime }+ϵ/2+g\left(x\right)+b}{bϵ}\hfill & ;b+g\left(x\right)-ϵ/2\le X^{\prime }\le b+g\left(x\right)+ϵ/2\hfill \\ 0\hfill & ;otherwise\hfill \end{array}\right..\end{array}$$

(40)

By the definition of transfer entropy, we can show that

$$\begin{array}{c}\hfill h\left(X^{\prime }\right|X)=lnb+\frac{ϵ}{2b}\end{array}$$

(41)

and hence transfer entropy of this data are given by

$$\begin{array}{c}\hfill T_{y\to x}(ϵ;b)=\left\{\begin{array}{cc}ln\frac{b}{ϵ}+\frac{ϵ}{2b};\hfill & b\ne 0\hfill \\ 0;\hfill & b=0.\hfill \end{array}\right.\end{array}$$

(42)

Therefore, when $b=0$, the transfer entropy $T_{y\to x}=lnϵ-lnϵ=0$. In addition, notice that $T_{y\to x}(ϵ;b)\to \infty$ as $ϵ\to 0$. Therefore, convergence of the numerical estimates is slow when $ϵ>0$ is small (see Figure 6).

4.2. Geometric Information Flow

Now, we focus on quantifying the geometric information flow by comparing dimensionalities of the outcomes’ spaces. We will contrast this to the transfer entropy computations for a few examples of the form $X^{\prime }=g\left(X\right)+bY+C$.

To illustrate the idea of geometric information flow, let us first consider a simple example, $x^{\prime }=ax+by+c$. If $b=0$, we have $x^{\prime }=f\left(x\right)$ and, when $b\ne 0$, we have the $x^{\prime }=f(x,y)$ case. Therefore, dimensionality of the data set $(x^{\prime },x)$ will change with parameter b (see Figure 7). When the number of data points $N\to \infty$ and $b\ne 0$, then $Geo{C}_{y\to x}\to 1$. Generally, this measure of causality depends on the value of b, but also the initial density of initial conditions.

Figure 7. Manifold of the data (x′, x) with x′ = by and y is uniformly distributed in the interval [0, 1]. Notice that, when (a) b = 0, we have a 1D manifold, (b) b ≠ 0 we have 2D manifold, in the (x′, x) plane.

In this example, we contrast theoretical solutions with the numerically estimated solutions (Figure 8). Theoretically, we expect $T_{y\to x}=\left\{\begin{array}{cc}0\hfill & ;b=0\hfill \\ \infty \hfill & ;b\ne 0\hfill \end{array}\right.$ as $N\to \infty$. In addition, the transfer entropy for noisy data can be calculated by Equation (42).

Figure 8. Geometric information flow vs. Transfer entropy for X′ = bY data.

4.3. Synthetic Data: $X^{\prime }=aX+bY$ with $a\ne 0$

The role of the initial density of points in the domain plays an important role in how the specific information flow values are computed depending on the measure used. To illustrate this point, consider the example of a unit square, ${[0,1]}^2$, that is uniformly sampled, and mapped by

$$X^{\prime }=aX+bY, \mathrm{with} a\ne 0.$$

(43)

This fits our basic premise that $(x,y,x^{\prime })$ data embeds in a 2D manifold, by ansatz of Equations (1) and (43), assuming for this example that each of $x,y$ and $x^{\prime }$ are scalar. As the number of data point grows, $N\to \infty$, we can see that $Geo{C}_{y\to x}=\left\{\begin{array}{cc}0\hfill & ;b=0\hfill \\ 1\hfill & ;b\ne 0\hfill \end{array}\right.$ because $(X,X^{\prime })$ data are on 2D manifold iff $b\ne 0$ (numerical estimation can be seen in Figure 9b). On the other hand, the conditional entropy $h\left(X^{\prime }\right|X,Y)$ is not defined, becoming unbounded when defined by noisy data. Thus, it follows that transfer entropy shares this same property. In other words, boundedness of transfer entropy depends highly on the $X^{\prime }|X,Y$ conditional data structure, while, instead, our geometric information flow measure highly depends on $X^{\prime }|X$ conditional data structure. Figure 9c demonstrates this observation with estimated transfer entropy and analytically computed values for noisy data. The slow convergence can be observed, Equation (42), Figure 6.

Figure 9. (a) shows the geometric information flow and (b) represents the Transfer entropy for x′ = x + by data. The figures show the changes with parameter b. We can notice that the transfer entropy has similar behavior to the geometric information flow of the data.

4.4. Synthetic Data: Nonlinear Cases

Now, consider the Hénon map,

$$\begin{array}{cc}\hfill x^{\prime }& =1-1.4x^2+y\hfill \\ \hfill y^{\prime }& =x\hfill \end{array}$$

(44)

as a special case of a general quadratic relationship, $x^{\prime }=ax+b{y}^2+c$, for discussing how $x^{\prime }$ may depend on $(x,y)\in {\Omega }_1$. Again, we do not worry here if $y^{\prime }$ may or may not depend on x and or y when deciding dependencies for $x^{\prime }$. We will discuss two cases, depending on how the $(x,y)\in {\Omega }_1$ data are distributed. For the first case, assume $(x,y)$ is uniformly distributed in the square, ${[-1.5,1.5]}^2$. The second and dynamically more realistic case will assume that $(x,y)$ lies on the invariant set (the strange attractor) of the Hénon map. The geometric information flow is shown for both cases in Figure 10. We numerically estimate the transfer entropy for both cases, which gives $T_{y\to x}=2.4116$ and $0.7942$, respectively. (However, recall that the first case for transfer entropy might not be finite analytically, and there is slow numerical estimation—see Table 3).

Figure 10. Consider the Hénon map, Equation (44), within the domain [−1.5, 1.5]² and the invariant set of Hénon map. (a) the uniform distribution case (green) as well as the natural invariant measure of the attractor (blue) are shown regarding the (x, y, x′) data for both cases; (b) when (x, y) ∈ [−1.5, 1.5]², notice that GeoC_y→x = 0.9, and (c) if (x, y) is in an invariant set of Hénon map, then GeoC_y→x = 0.2712.

Table 3. Hénon Map Results. Contrasting geometric information flow versus transfer entropy in two different cases, 1st relative to uniform distribution of initial conditions (reset each time) and 2nd relative to the natural invariant measure (more realistic).

Domain	$\mathit{GeoC}$	${\mathit{T}}_{\mathit{y}\to \mathit{x}}$
${[-1.5,1.5]}^2$	0.90	2.4116
Invariant Set	0.2712	0.7942

4.5. Application Data

Now, moving beyond bench-marking with synthetic data, we will contrast the two measures of information flow in a real world experimental data set. Consider heart rate ($x_n$) vs. breathing rate ($y_n$) data (Figure 11) as published in [33,34], consisting of 5000 samples. Correlation dimension of the data X is $D_2\left(X\right)=1.00$, and $D_2(X,X^{\prime })=1.8319>D_2\left(X\right)$. Therefore, $X^{\prime }=X_{n+1}$ depends not only on x, but also on an extra variable (Theorem 2). In addition, correlation dimension of the data $(X,Y)$ and $(X,Y,X^{\prime })$ is computed $D_2(X,Y)=1.9801$ and $D_2(X,Y,X^{\prime })=2.7693>D_2(X,Y)$, respectively. We conclude that $X^{\prime }$ depends on extra variable(s) other that $(x,y)$ (Theorem 2) and the correlation dimension geometric information flow, $Geo{C}_{y\to x}=0.0427$, is computed by Equations (38) and (37). Therefore, this suggests the conclusion that there is a causal inference from breathing rate to heart rate. Since breathing rate and heart rate share the same units, the quantity measured by geometric information flow can be described without normalizing. Transfer entropy as estimated by the KSG method [32] with parameter $k=30$ is $T_{y\to x}=0.0485$, interestingly relatively close to the $GeoC$ value. In summary, both measures for causality ($GeoC,T$) are either zero or positive together. It follows that there exists a causal inference (see Table 4).

Figure 11. Result for heart rate(x_n) (a,c) vs. breathing rate(y_n) data (b,d). The top row is the scatter plot of the data, and the second row represents the dimension of the data.

Table 4. Heart rate vs. breathing rate data—contrasting geometric information flow versus transfer entropy in breath rate to heart rate.

${\mathit{GeoC}}_{\mathit{y}\to \mathit{x}}$	${\mathit{T}}_{\mathit{y}\to \mathit{x}}$
0.0427	0.0485

5. Conclusions

We have developed here a geometric interpretation of information flow as a causal inference as usually measured by a positive transfer entropy, $T_{y\to x}$. Our interpretation relates the dimensionality of an underlying manifold as projected into the outcome space and summarizes the information flow. Furthermore, the analysis behind our interpretation involves standard Pinsker’s inequality that estimates entropy in terms of total variation, and, through this method, we can interpret the production of information flow in terms of details of the derivatives describing relative orientation of the manifolds describing inputs and outputs (under certain simple assumptions).

A geometric description of causality allows for new and efficient computational methods for causality inference. Furthermore, this geometric perspective provides a different view of the problem and facilitates the richer understanding that complements the probabilistic descriptions. Causal inference is weaved strongly throughout many fields and the use of transfer entropy has been a popular black box tool for this endeavor. Our method can be used to reveal more details of the underling geometry of the data-set and provide a clear view of the causal inference. In addition, one can use the hybrid method of this geometric aspect and existing other methods in their applications.

We provided a theoretical explanation (part I: Mathematical proof of the geometric view of the problem) and numerical evidence (part 2: A data-driven approach for mathematical framework) of a geometric view for the causal inference. Our experiments are based on synthetic (toy problems) and practical data. In the case of synthetic data, the underlining dynamics of the data and the actual solution to the problem are known. For each of these toy problems, we consider a lot of cases by setting a few parameters. Our newly designed geometric approach can successfully capture these cases. One major problem may be if data describes a chaotic attractor. We prove theoretically (Theorem 2) and experimentally (by Hénon map example: in this toy problem, we also know actual causality) that correlation dimension serves to overcome this issue. Furthermore, we present a practical example based on heart rate vs. breathing rate variability, which was already shown to have positive transfer entropy, and here we relate this to show positive geometric causality.

Furthermore, we have pointed out that transfer entropy has analytic convergence issues when future data ($X^{\prime }$) are exactly a function of current input data ($X,Y$) versus more generally $(X,Y,X^{\prime })$. Therefore, referring to how the geometry of the data can be used to identify the causation of the time series data, we develop a new causality measurement based on a fractal measurement comparing inputs and outputs. Specifically, the correlation dimension is a useful and efficient way to define what we call correlation dimensional geometric information flow, $Geo{C}_{y\to x}$. The $Geo{C}_{y\to x}$ offers a strongly geometric interpretable result as a global picture of the information flow. We demonstrate the natural benefits of $Geo{C}_{y\to x}$ versus $T_{y\to x}$, in several synthetic examples where we can specifically control the geometric details, and then with a physiological example using heart and breathing data.