Causal Inference in Time Series in Terms of Rényi Transfer Entropy

Abstract

Uncovering causal interdependencies from observational data is one of the great challenges of a nonlinear time series analysis. In this paper, we discuss this topic with the help of an information-theoretic concept known as Rényi’s information measure. In particular, we tackle the directional information flow between bivariate time series in terms of Rényi’s transfer entropy. We show that by choosing Rényi’s parameter $\alpha$, we can appropriately control information that is transferred only between selected parts of the underlying distributions. This, in turn, is a particularly potent tool for quantifying causal interdependencies in time series, where the knowledge of “black swan” events, such as spikes or sudden jumps, are of key importance. In this connection, we first prove that for Gaussian variables, Granger causality and Rényi transfer entropy are entirely equivalent. Moreover, we also partially extend these results to heavy-tailed $\alpha$-Gaussian variables. These results allow establishing a connection between autoregressive and Rényi entropy-based information-theoretic approaches to data-driven causal inference. To aid our intuition, we employed the Leonenko et al. entropy estimator and analyzed Rényi’s information flow between bivariate time series generated from two unidirectionally coupled Rössler systems. Notably, we find that Rényi’s transfer entropy not only allows us to detect a threshold of synchronization but it also provides non-trivial insight into the structure of a transient regime that exists between the region of chaotic correlations and synchronization threshold. In addition, from Rényi’s transfer entropy, we could reliably infer the direction of coupling and, hence, causality, only for coupling strengths smaller than the onset value of the transient regime, i.e., when two Rössler systems are coupled but have not yet entered synchronization.

1. Introduction

The time evolution of a complex system is usually recorded in the form of a time series. Time series analysis is a traditional field of mathematical statistics; however, the development of nonlinear dynamical systems and the theory of deterministic chaos have opened up new vistas in the analysis of nonlinear time series [1,2]. The discovery of the synchronization of chaotic systems [3] has changed the study of interactions and cooperative behavior of complex systems and brought new approaches to studying the relations between nonlinear time series [4]. During the process of synchronization, two systems can either mutually interact or only one can influence the other. In order to distinguish these two ways, and to find which system is the driver (“master”) and which is the response (“slave”) system, a number of approaches from the dynamical system theory have been proposed [5,6,7,8]. The aforementioned problem of synchronization can be seen as part of a broader framework known as causality or causal relations between systems, processes, or phenomena. The mathematical formulation of causality, in terms of predictability, was first proposed by Wiener [9] and formulated for the time series by Granger [10]. In particular, Granger introduced what is now known as Granger causality, which is a statistical concept of causality that is based on the evaluation of predictability in bivariate autoregressive models.

Extracting causal interdependencies from observational data is one of the key tasks in a nonlinear time series analysis. Apart from the linear Granger causality and various nonlinear extensions thereof [11,12,13], existing methods for this purpose include state-space-based approaches, such as conditional probabilities of recurrence [14,15,16], or information-theoretic quantities, such as conditional mutual information [17,18] and transfer entropies [2,19,20,21]. In particular, the latter information-theoretic quantities represent powerful instruments in quantifying causality between time-evolving systems. This is because ensuing information-theoretic functionals (typically based on Shannon entropy) quantify—in a non-parametric and explicitly non-symmetric way—the flow of information between two (or more) time series. In particular, transfer entropies (TEs) have recently received considerable attention. The catalyst was the infusion of new (numerical and conceptional) ideas. For instance, the performances of the Shannon entropy-based conditional entropies and conditional mutual entropies have been, in recent years, extensively tested using numerically-generated time series [17,22]. Sophisticated algorithms have been developed to uncover direct causal relations in multivariate time series [23,24,25]. In parallel, increasing attention has been devoted to the development of reliable estimators of entropic functionals to detect causality from nonlinear time series [26]. At the same time, it has been recognized that information-theoretic approaches play important roles in dealing with complex dynamical systems that are multiscale and/or non-Gaussian [21,27,28,29]. The latter class includes complex systems with heavy-tailed probability distributions epitomized, e.g., in financial and climatological time series [30,31].

In this paper, we extend the popular Shannon entropy-based TE (STE), which represents a prominent tool for assessing directed information flow between joint processes, and quantifies information transfer in terms of Rényi’s TE (RTE). RTE was introduced by one of us (PJ) in reference [21] in the context of a bivariate financial time series. The original idea was to use the RTE in order to exploit the theoretical formulation that could identify and quantify peculiar features in multiscale bivariate processes (e.g., multiscale patterns, generalized fractal dimensions, or multifractal cross-correlations) that are often seen in finance. In contrast to [21], where the focus was mostly on qualitative aspects of Rényian information flow between selected stock-market time series, in the present work, we wish to be more quantitative by analyzing coupled time series that are numerically generated from known dynamics. Specifically, we demonstrate how the RTE method performs in the detection of the coupling direction and onset of synchronization between two Rössler oscillators [32] that are unidirectionally coupled in the first variable x. The Rössler system (RS) is a paradigmatic and well-studied low-dimensional chaotic dynamical system. When coupled, RSs allow for synchronization as well as a subtle phenomenon known as “phase synchronization”, i.e., when the amplitudes of both systems are not correlated while the phases are approximately equal. In this respect, the synthetic bivariate time series (generated from coupled RSs) serves as an excellent test-bed, allowing to numerically analyze, e.g., drive–response relationships or identify the ensuing onset (or threshold) of synchronization. In doing so, we identify factors and influences that can lead to either decreases in the RTE sensitivity or false detections and propose some ways to cope with them. The aforementioned issues have not been explicitly studied in the framework of the RTE; this work presents the first attempt in this direction.

To set the stage, we shall first, in Section 2, provide the information-theoretic background on Rényi entropy (RE), which will be needed in the main body of the text. For self-consistency of our exposition, we briefly review Shannon’s transfer entropy of Schreiber and motivate and derive the core quantity of this work—the Rényi transfer entropy. The issue of causality (and its connection to RTE) is examined in Section 3. In particular, we prove that the Granger causality is entirely equivalent to the RTE for Gaussian processes and show how the Granger causality and the RTE are related in the case of heavy-tailed (namely $\alpha$-Gaussian) processes. Section 4 is dedicated to derived information-theoretic concepts, such as the balance of transfer entropy and effective transfer entropy that will be employed in our analysis. The proposed framework is then illustrated on two unidirectionally coupled Rössler systems as a paradigmatic example. To cultivate our intuition about the latter RSs, we discuss in Section 5 the inner workings of such RSs in terms of simple numerical experiments. The ensuing numerical analysis is presented in Section 6, where we discuss how the RTE can be used to detect causality and the onset of synchronization in the two coupled RSs. We also demonstrate how the RTE provides non-trivial insight into the structure of a transient regime that exists between the regions of chaotic correlations and the onset of synchronization. Finally, Section 7 summarizes our theoretical and numerical findings and discusses possible extensions of the present work. For the reader’s convenience, we relegate some technical issues concerning the RE estimator employed and the statistical significance of results presented to Appendix A and Appendix B.

2. Rényi Entropy

Information theory approaches based on Shannon entropy currently belong in the portfolio of techniques and tools that are indispensable in addressing causality issues in complex dynamical systems. At the same time, Shannon’s information theory is limited in its scope. In fact, since Shannon’s seminal papers [33], it has been known that Shannon’s information measure (or entropy) represents mere idealized information, appearing only in situations when the buffer memory (or storage capacity) of a transmitting channel is infinite. In particular, Shannon’s source coding theorem (or noiseless coding theorem), which establishes the limits to possible data compression and, thus, provides operational meaning to the Shannon entropy, assumes that the cost of a codeword is a linear function of its length (so the optimal code has a minimal cost out of all codes). However, the linear costs of codewords are not always desirable. For instance, when the storage capacity is finite one would aim to penalize excessively lengthy codewords with a price that is, e.g., exponential rather than the linear function of the length.

For these reasons, information theorists have devised various remedies to deal with such cases. This usually consists of substituting Shannon’s information measure with information measures of other types. Consequently, numerous generalizations of Shannon’s entropy have started to proliferate in the information-theory literature, ranging from additive entropies [34,35] to a rich class of non-additive entropies [36,37,38,39,40], to more exotic types of entropies [41]. The one-parametric class of information measures, known as Rényi entropies, introduced by Hungarian mathematician and information theorist Alfred Rényi in the early 1960s [42,43], is particularly prominent among such generalizations. Applications of RE in information theory, namely its generalization to coding theorems, were carried over by Campbel [44], Csiszár [45,46], Aczél [47], and others. In a physical setting, RE was popularized in the context of chaotic dynamical systems by Kadanoff et al. [48] and in connection with multifractals by Mandelbrot [49]. RE is also indispensable in the quantum information theory where it quantifies multipartite entanglement [50].

In its essence, REs constitute a one-parametric family of information measures labeled by parameter $\alpha$, fulfilling the additivity with respect to the composition of statistically independent systems. The special case with $\alpha =1$ corresponds to ordinary Shannon’s entropy. REs belong to a broader class of so-called Uffink entropic functionals [51,52], i.e., the most general class of solutions that satisfy Shorem–Johnson axioms for the maximum entropy principle in the statistical estimation theory. Moreover, it might be shown that Rényi entropies belong to the class of the so-called mixing homomorphic functions [53] and that they are analytic for $\alpha \in {\mathbb{C}}_{I\cup IV}$, cf. [34].

2.1. Definition

RE is defined as an exponentially weighted mean of the Hartley information measure $-logp$ (i.e., elementary measure of information) [54]. In fact, it was shown by Rényi that, except for a linearly-weighted average (which leads to Shannon entropy), exponential weighting is the only possible averaging that is both compatible with the Kolmogorov–Nagumo average prescription and leads to entropies that are additive, with respect to independent systems [42,43]. RE, associated with a system described with a probability distribution $\mathcal{P}$, reads

$$H_{\alpha }\left[\mathcal{P}\right]=\frac{1}{1-\alpha }{log}_2\sum _{i=1}^n{p}_i^{\alpha }.$$

(1)

RE has the following properties [34,43]:

• RE is symmetric, i.e., $H_{\alpha }\left[\{p_1,\cdots ,p_n\}\right]=H_{\alpha }\left[\{p_{\pi \left(1\right)},\cdots ,p_{\pi \left(n\right)}\}\right]$;
• RE is non-negative, i.e., $H_{\alpha }\ge 0$;
• ${lim}_{\alpha \to 1}{H}_{\alpha }=H_1$, where $H_1=H$ is the Shannon entropy;
• $H_0={log}_{2}n$ is the Hartley entropy and $H_2=-{log}_2{\sum }_{i=1}^n{p}_i^2$ is the Collision entropy;
• $0\le H_{\alpha }\left[\mathcal{P}\right]\le {log}_{2}n$;
• $H_{\alpha }$ is a positive, decreasing the function of $\alpha \ge 0$.

Let us mention that $H_{\alpha }\left[\mathcal{P}\right]$ with different $\alpha$s complement each other. This is because for each specific $\alpha$, the ensuing $H_{\alpha }\left[\mathcal{P}\right]$ carries extra information that is not present in any other $H_{\beta }\left[\mathcal{P}\right]$ with $\beta \ne \alpha$. In information theory, this fact is known as the reconstruction theorem, namely, the underlying distribution $\mathcal{P}$ can be uniquely reconstructed only if all $H_{\alpha }\left[\mathcal{P}\right]$ are known, [21,34,55]. In chaotic dynamical systems, the reconstruction theorem goes under the name complementary generalized dimensions [56] (cf. also next subsection).

2.2. Multifractals, Chaotic Systems, and Rényi Entropy

Another appealing property of the Rényi entropy is its close connection to multifractals, i.e., the mathematical paradigm that is often encountered in complex dynamical systems with examples ranging from turbulence and strange attractors to meteorology and finance, see, e.g., [57]. The aforementioned connection is established through the so-called generalized dimensions, which are defined as [2,48]

$$D_{\alpha }=-\underset{\delta \to 0}{lim}\frac{H_{\alpha }\left(\delta \right)}{log\delta }$$

(2)

where $\delta$ is a size of a $\delta -$mesh covering of a configuration space of a system. Generalized dimensions $D_{\alpha }$ are conjugate to the multifractal spectrum $f\left(\beta \right)$ through the Legendre transform [48]

$$(\alpha -1)D_{\alpha }=\alpha \beta -f\left(\beta \right).$$

(3)

The function $f\left(\beta \right)$ is called the multifractal spectrum because $\beta$ plays the role of the scaling exponent in the local probability distribution, e.g., distribution with support on the i-th hypercube of a mesh size $\delta$ scale, as $p_i\left(\delta \right)\sim {\delta }^{{\beta }_i}$. The key assumption in the multifractal analysis is that in the small $\delta -$ limit, the local probability distribution depends smoothly on $\beta$. It can be argued that $f\left(\beta \right)$ corresponds to the (box-counting) fractal dimension of the portion of the configuration space where local probability distributions have the scaling exponent $\beta$, cf., e.g., reference [34]. In this way, the multifractal can be viewed as an ensemble of intertwined (uni)fractals, each with its own fractal dimension $f\left(\beta \right)$.

The multifractal paradigm is particularly pertinent in the theory of chaotic systems. For instance, chaotic dynamics and strange attractors, in particular, are uniquely characterized by the infinite sequences of generalized dimensions $D_{\alpha }$, cf. reference [56]. In particular, the generalized dimensions can help to recognize (in a quantitative way) the main geometric features of chaotic systems. For instance, they may help to distinguish chaotic behavior from noisy behavior, determine the number of variables that are needed to model the dynamics of the system or classify systems into universality classes. On the other hand, dynamical features of chaotic systems are often analyzed through such quantifiers as Lyapunov exponent, which is a measure of the divergence of nearby trajectories, or ensuing Kolmogorov-Sinai entropy rate (KSE), which quantifies the change of entropy as the system evolves and is given by the sum of all positive Lyapunov exponents. The connection between KSE and the time evolution of the information-theoretic or statistical entropy is quite delicate, see, e.g., the discussion in reference [58], though the upshot is clear, in order to describe the dynamics of a (complex) system, the temporal change or the difference in entropy is more relevant than the entropy itself. Consequently, while RE (alongside with $D_{\alpha }$) is a suitable quantifier of geometric properties of chaotic systems, its temporal differences or temporal rates are useful for the description of the dynamics of such systems. Rényi’s transfer entropy follows the latter route.

2.3. Shannon Transfer Entropy

In order to understand the concept of Rényi transfer entropy, we recall first its Shannon’s counterpart.

Let $X={\left\{x_i\right\}}_{i=1}^N$ be a discrete random variable with ensuing probability distribution ${\mathcal{P}}_X$, then the Shannon entropy of this process is

$$H\left(X\right)\equiv H\left({\mathcal{P}}_X\right)=-\sum _{x\in \mathcal{X}}p\left(x\right){log}_{2}p\left(x\right).$$

(4)

Let $Y={\left\{y_i\right\}}_{i=1}^N$ be another random variable, then mutual information between X and Y is

$$\begin{array}{ccc}\hfill I(X:Y)& =& \sum _{x\in X,y\in Y}p(x,y){log}_2\frac{p(x,y)}{p\left(x\right)p\left(y\right)}\hfill \\ & =& H\left(X\right)-H\left(X\right|Y)=H(Y)-H(Y\left|X\right),\hfill \end{array}$$

(5)

where quantity $H\left(X\right|Y)$ is the conditional entropy, defined as

$$H\left(X\right|Y)=-\sum _{x\in X,y\in Y}p(x,y){log}_{2}p\left(x\right|y).$$

(6)

Mutual information quantifies an average reduction in uncertainty (i.e., gain in information) about X resulting from the observation of Y, or vice versa. Since $I(X:Y)=I(Y:X)$, it cannot be used as a measure of directional information flow. Note also that the amount of information contained in X about itself is just the Shannon entropy, i.e., $I(X:X)=H\left(X\right)$.

The mutual information between two processes X and Y conditioned on the third process Z is called conditional mutual information and is defined as

$$I(X:Y|Z)=H(X\left|Z\right)-H\left(X\right|Y,Z)=I(X:(Y,Z))-I(X:Y).$$

(7)

Let us now consider two time sequences (e.g., two stock market time series) described by stochastic (possibly vector-type) random variables $X_t$ and $Y_t$. Let us assume further that the time steps (e.g., data ticks) are discrete with the time step $\tau$ and with $t_n=t_0+n\tau$ where $t_0$ is some reference time. For practical purposes, it is also useful to assume that $X_t$ and $Y_t$ represent discrete-time stochastic Markov processes of order k and l, respectively.

We wish to know what information will be gained on $X_{t_{n+1}}$ by observing $Y_t$ up to time $t_n$. To this end, we introduce the joint process $X_{t_n},X_{t_{n-1}},\dots ,X_{t_{n-k+1}}$, which we denote as $X_n^{\left(k\right)}$, and similarly, we define the joint process $Y_n^{\left(l\right)}\equiv Y_{t_n},Y_{t_{n-1}},\dots ,Y_{t_{n-l+1}}$. By replacing X in (7) by $X_{t_{n+1}}$, Y by $Y_n^{\left(l\right)}$, and Z by $X_n^{\left(k\right)}$, we obtain the desired conditional mutual information

$$\begin{array}{ccc}\hfill I(X_{t_{n+1}}:Y_n^{\left(l\right)}|X_n^{\left(k\right)})& =& H\left(X_{t_{n+1}}\right|X_n^{\left(k\right)})-H\left(X_{t_{n+1}}\right|Y_n^{\left(l\right)},X_n^{\left(k\right)})\hfill \\ & =& \sum _{x_n^{\left(k\right)}\in X_{n+1}^{\left(k\right)},y_n^{\left(l\right)}\in Y_n^{\left(l\right)}}p(x_{n+1},x_n^{\left(k\right)},y_n^{\left(l\right)}){log}_2\left(\frac{p\left(x_{n+1}\right|x_n^{\left(k\right)},y_n^{\left(l\right)})}{p\left(x_{n+1}\right|x_n^{\left(k\right)})}\right).\hfill \end{array}$$

(8)

The conditional mutual information (8) is also known as Shannon transfer entropy from $Y_t$ to $X_t$ (or simply from Y to X) and as a measure of the directed (time asymmetric) information transfer between joint processes, it was introduced by Schreiber in reference [19]. The latter is typically denoted as

$$T_{Y\to X}(k,l)\equiv I(X_{t_{n+1}}:Y_n^{\left(l\right)}|X_n^{\left(k\right)}).$$

(9)

As already mentioned, for independent processes, TE is equal to zero. For a non-zero case transfer, entropy measures the deviation from the independence of the two processes. An important property of the transfer entropy is that it is directional, i.e., in general, $T_{Y\to X}\ne T_{X\to Y}$.

2.4. Rényi Transfer Entropy

In the same manner as in (7), we can introduce the Rényi transfer entropy of order α from Y to X (see also reference [21]) as

$$\begin{array}{ccc}\hfill T_{\alpha ,Y\to X}^R(k,l)& =& H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)})-H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)},Y_n^{\left(l\right)})\hfill \\ & =& I_{\alpha }(X_{t_{n+1}}:Y_n^{\left(l\right)}|X_n^{\left(k\right)}),\hfill \end{array}$$

(10)

where $H_{\alpha }\left(X\right|Y)$ is the conditional entropy of order α and $I_{\alpha }(X:Y)$ is the mutual information of order α. These can be explicitly written as [21,43]

$$\begin{array}{ccc}\hfill H_{\alpha }\left(X\right|Y)& =& \frac{1}{1-\alpha }{log}_2\frac{{\sum }_{x\in X,y\in Y}{p}^{\alpha }(x,y)}{{\sum }_{y\in Y}{p}^{\alpha }\left(y\right)},\hfill \\ \hfill I_{\alpha }(X:Y)& =& \frac{1}{1-\alpha }{log}_2\frac{{\sum }_{x\in X,y\in Y}{p}^{\alpha }\left(x\right)p^{\alpha }\left(y\right)}{{\sum }_{x\in X,y\in Y}{p}^{\alpha }(x,y)}.\hfill \end{array}$$

(11)

It can be checked (via L’Hospital’s rule) that Rényi’s transfer $\alpha$-entropy reduces to Shannon TE in the $\alpha \to 1$ limit, i.e.,

$$\begin{array}{c}\hfill \underset{\alpha \to 1}{lim}{T}_{\alpha ,Y\to X}^R=T_{Y\to X}.\end{array}$$

(12)

From (10), we see that $T_{\alpha ,Y\to X}^R(k,l)$ may be intuitively interpreted as the degree of ignorance (or uncertainty) about $X_{t_{n+1}}$ resolved by the past states $Y_n^{\left(l\right)}$ and $X_n^{\left(k\right)}$, over and above the degree of ignorance about $X_{t_{n+1}}$ already resolved by its own past state alone. Here, the ignorance is quantified by the Rényi information measure (i.e., RE) of order $\alpha$.

Rényi TE can also be negative (unlike the Shannon TE). This means that the uncertainty of the process $X_t$ becomes bigger knowing the past of $Y_t$, i.e., $H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)})\le H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)},Y_n^{\left(l\right)})$. If $X_t$ and $Y_t$ are independent, then $T_{\alpha ,Y\to X}^R=T_{\alpha ,X\to Y}^R=0$. However, in contrast to Shannon’s case, the fact that $T_{\alpha ,Y\to X}^R=0$ does necessarily imply the independence of the two underlying stochastic processes. Nonetheless, in Section 3, we prove that in case of Gaussian (Wiener) processes, 0-valued RTE is a clear signature of independence.

Due to the reconstruction theorem mentioned in Section 2.1, RTE $T_{\alpha ,Y\to X}^R$ conveys for each $\alpha$ a different type of directional information from Y to X. The essence of this statement can be understood qualitatively by introducing the so-called escort distribution.

2.5. Escort Distribution

Because of the nonlinear way in which probability distributions enter in the definition of RE, cf. Equation (1), the RTE represents a useful measure of transmitted information that quantifies the dominant information flow between certain parts of underlying distributions. In fact, for $0<\alpha <1$, the corresponding information flow accentuates marginal events, while for $\alpha >1$, more probable (close-to-average) events are emphasized [21]. In this respect, one can zoom or amplify different parts of probability density functions involved by merely choosing appropriate values of $\alpha$. This is particularly useful in studies of time sequences, where marginal events are of crucial importance, for instance, in financial time series.

In order to better understand the aforementioned “zooming” property of RTE, we rewrite (10) in the form

$$T_{\alpha ,Y\to X}^R(k,l)=\frac{1}{1-\alpha }{log}_2\left(\frac{\sum \frac{p^{\alpha }\left(x_n^{\left(k\right)}\right)}{\sum p^{\alpha }\left(x_n^{\left(k\right)}\right)}{p}^{\alpha }\left(x_{n+1}\right|x_n^{\left(k\right)})}{\sum \frac{p^{\alpha }(x_n^{\left(k\right)},y_n^{\left(l\right)})}{\sum p^{\alpha }(x_n^{\left(k\right)},y_n^{\left(l\right)})}{p}^{\alpha }\left(x_{n+1}\right|x_n^{\left(k\right)},y_n^{\left(l\right)})}\right).$$

(13)

This particular representation shows how the underlying distribution changes (or deforms) with the change of parameter $\alpha$. The numerator and denominator inside the log-function contain the so-called escort (or zooming) distributions ${\rho }_{\alpha }$

$${\rho }_{\alpha }\left(x\right)\equiv \frac{p^{\alpha }\left(x\right)}{{\sum }_{x\in X}{p}^{\alpha }\left(x\right)},$$

(14)

which emphasize less probable events for $0<\alpha <1$ and more probable events when $\alpha >1$, see Figure 1.

Note also that ${\rho }_{\alpha }(x_n^{\left(k\right)},y_n^{\left(l\right)})$ is not the joint probability distribution of $X_n^{\left(k\right)}$ and $Y_n^{\left(l\right)}$ as it does not satisfy the Kolmogorov–de Finetti relation for conditional probabilities [59].

In connection with (13), we may note that for $0<\alpha <1$ the multiplicative factor is positive, and so the RTE is negative if, by learning $Y_n^{\left(l\right)}$, the rare events are (on average) more emphasized than in the case when only $X_n^{\left(k\right)}$ alone is known. Analogically, for $\alpha >1$ the RTE can be negative when—by learning $Y_n^{\left(l\right)}$—the more probable events are (on average) more accentuated in comparison with the situation when $Y_n^{\left(l\right)}$ is not known. It should be stressed that the analogous situation does not hold for Shannon’s TE. This is because in the limit $\alpha \to 1$ we regain expression (8), which is nothing but relative entropy, and as such, it is always non-negative due to Gibbs inequality. At the same time, Shannon’s TE is, by its very definition, also mutual information. While RTE is also defined to be a mutual information, it is not relative entropy (in the RE case, those two concepts do not coincide). It can be shown (basically via Jensen’s inequality) [34] that the relative entropy based on RE is also non-negative but this is not true for ensuing mutual information, which serves as a conceptual basis for the definition of RTE.

3. Rényi Transfer Entropy and Causality

As already seen, Rényi TE (analogously to Shannon TE) is a directional measure of information transfer. Let us now comment on the connection of the RTE with the causality concept.

3.1. Granger Causality—Gaussian Variables

The first general definition of causality, which could be quantified and measured computationally was given by Wiener in 1956, namely “…For two simultaneously measured signals, if we can predict the first signal better by using the past information from the second one than by using the information without it, then we call the second signal causal to the first one…” [9].

The introduction of the concept of causality into the experimental practice, namely into analyses of data observed in consecutive time instants (i.e., time series), is due to the Nobel prize winner (economy, 2003) C.W.J. Granger. The so-called Granger causality is defined so that the process $Y_t$ Granger causes another process $X_t$ if, in an appropriate statistical sense, $Y_t$ assists in predicting the future of $X_t$ beyond the degree to which $X_t$ already predicts its own future.

The standard test of the Granger causality was developed by Granger himself [10] and it is based on a linear regression model, namely

$$\begin{array}{c}\hfill X_t=a_{0t}+\sum _{\ell =1}^k{a}_{1\ell }{X}_{t-\ell }+\sum _{\ell =1}^l{a}_{2\ell }{Y}_{t-\ell }+e_t,\end{array}$$

(15)

where $a_0,a_{1\ell },a_{2\ell }$ are (constant) regression coefficients, l and k represent the maximum number of lagged observations included in the model (i.e., memory indices), t is a discrete time with the time step $\tau$ (ℓ is also quantified in units of $\tau$) and $e_t$ is the uncorrelated random variable (residual) with zero mean and variance ${\sigma }^2$. The null hypothesis that $Y_t$ does not cause $X_t$ (in the sense of Granger) is not rejected if and only if $a_{2\ell }=0$ for $\ell =1,\dots ,l$. In the latter case, we will call the ensuing regression model the reduced regression model.

It is not difficult to show that for Gaussian variables, the RTE and Granger causality are entirely equivalent. To see this, we use the standard measure of the Granger causality, which is defined as [60]

$$\begin{array}{c}\hfill {\mathcal{F}}_{Y\to X}^{(k,l)}={log}_2\frac{|\sum (e_t^{\prime }\left)\right|}{|\sum (e_t\left)\right|},\end{array}$$

(16)

where $\sum (\dots )$ is the covariance matrix, $|\dots |$ denotes the matrix determinant, and $e_t$, $e_t^{\prime }$ are residuals in the full and reduced regression model, respectively. We chose the logarithm to the base 2, rather than e for technical convenience. We now prove the following theorem:

Theorem 1.

If the joint process $X_t$, $Y_t$ is Gaussian, then there is an exact equivalence between the Granger causality and RTE, namely

$$\begin{array}{c}\hfill {\mathcal{F}}_{Y\to X}^{(k,l)}=2T_{\alpha ,Y\to X}^R(k,l).\end{array}$$

(17)

This can be proved in the following way (for an analogous proof for Shannon’s TE, see [61]). We first define the partial covariance as

$$\begin{array}{c}\hfill \sum \left(\mathbf{X}\right|\mathbf{Y})=\mathbf{\Sigma }\left(\mathbf{X}\right)-\mathbf{\Sigma }(\mathbf{X},\mathbf{Y})\mathbf{\Sigma }{\left(\mathbf{Y}\right)}^{-\mathbf{1}}\mathbf{\Sigma }{(\mathbf{X},\mathbf{Y})}^{\top },\end{array}$$

(18)

where ${\sum \left(\mathbf{X}\right)}_{ij}=\mathrm{cov}(X_i,X_j)$ and ${\sum (\mathbf{X},\mathbf{Y})}_{ij}=\mathrm{cov}(X_i,Y_j)$ with $\mathbf{X}$ and $\mathbf{Y}$ being random vector (or multivariate) variables. Let $\mathbf{X}$ and $\mathbf{Y}$ be jointly distributed random vectors in the linear regression model

$$\begin{array}{c}\hfill \mathbf{X}=\mathbf{a}+\mathbf{Y}\mathbb{A}+\mathbf{e}.\end{array}$$

(19)

Here, $\mathbf{a}$ is a constant vector, $\mathbb{A}$ contains regression coefficients, and e is a residual random vector with zero mean. In the subsequent, we will identify both $\mathbf{X}$ and $\mathbf{Y}$ with stochastic vectors (see text after Equation (28)). In such a case, one can always choose a specified number of time lags, so that system (19) (or better (23) and, consequently, (22)) is uniquely solvable, as neither vector $\mathbf{a}$ nor matrix $\mathbb{A}$ are time-dependent.

We now apply the least square method to the mean square error

$$\begin{array}{c}\hfill {ℰ}^2\equiv \sum _i\mathbb{E}\left(e_i^2\right)=\sum _i\mathbb{E}\left[{(\mathbf{X}-\mathbf{Y}\mathbb{A}-\mathbf{a})}_i^2\right],\end{array}$$

(20)

Here, $\mathbb{E}(\dots )$ denotes the average value. The ensuing least square equations

$$\begin{array}{c}\hfill \frac{\partial {ℰ}^2}{\partial {\mathbb{A}}_{ij}}=0and\frac{\partial {ℰ}^2}{\partial a_k}=0,\end{array}$$

(21)

yield

$$\begin{array}{ccc}& & a_l=\mathbb{E}\left(X_l\right)-\sum _k\mathbb{E}\left(Y_k\right){\mathbb{A}}_{kl},\hfill \end{array}$$

(22)

$$\begin{array}{ccc}& & {\mathbb{A}}_{li}=\sum _j{[\sum \left(\mathbf{X}\right)]}_{lj}^{-1}\sum {(\mathbf{Y},\mathbf{X})}_{ji}.\hfill \end{array}$$

(23)

From (19) follows that

$$\begin{array}{c}\hfill \mathbb{E}\left(X_i{X}_j\right)=\mathbb{E}\left[{(\mathbf{a}+\mathbf{Y}\mathbb{A}+\mathbf{e})}_i{(\mathbf{a}+\mathbf{Y}\mathbb{A}+\mathbf{e})}_j\right],\end{array}$$

(24)

which after employing (22) can be equivalently rewritten as

$$\begin{array}{c}\hfill \mathrm{cov}(X_i,X_j)=\sum _{l,k}\mathrm{cov}(Y_l,Y_k){\mathbb{A}}_{li}{\mathbb{A}}_{kj}+\mathrm{cov}(e_i,e_j),\end{array}$$

(25)

or equivalently

$$\begin{array}{c}\hfill \sum \left(\mathbf{X}\right)={\mathbb{A}}^{\top }\sum \left(\mathbf{Y}\right)\mathbb{A}+\sum \left(\mathbf{e}\right).\end{array}$$

(26)

If we now insert (23)–(26), we obtain

$$\begin{array}{c}\hfill \mathrm{cov}(e_i,e_j)=\mathrm{cov}(X_i,X_j)-\mathrm{cov}(X_i,Y_k){\left[\mathrm{cov}(Y_k,Y_i)\right]}^{-1}{\left[\mathrm{cov}(X_i,Y_j)\right]}^{\top },\end{array}$$

(27)

which might be equivalently written as

$$\begin{array}{c}\hfill \sum \left(\mathbf{e}\right)=\sum \left(\mathbf{X}\right|\mathbf{Y}).\end{array}$$

(28)

If we now take $\mathbf{X}=\left(X_{t_{n+1}}\right)$, $\mathbf{a}=\left(a_0\right)$, $\mathbf{Y}=(X^{\left(k\right)},Y^{\left(l\right)})$, $\mathbb{A}=\mathrm{diag}(a_{1n}^{\left(k\right)},a_{2n}^{\left(l\right)})$ for the full regression model and $\mathbf{Y}=\left(X_n^{\left(k\right)}\right)$, $\mathbb{A}=\mathrm{diag}\left(a_1^{\left(k\right)}\right)$ for the reduced regression model, we might write that

$$\begin{array}{c}\hfill {\mathcal{F}}_{Y\to X}^{(k,l)}={log}_2\frac{|\sum (e_t^{\prime }\left)\right|}{|\sum (e_t\left)\right|}={log}_2\left(\frac{|\sum (X_{t_{n+1}}\left|X_n^{\left(k\right)}\right)|}{|\sum (X_{t_{n+1}}|X_n^{\left(k\right)},Y_n^{\left(l\right)})|}\right).\end{array}$$

(29)

At this stage, we can use the fact that RE of the multivariate Gaussian variable $\mathbf{X}$ is [62]

$$\begin{array}{c}\hfill H_{\alpha }\left(\mathbf{X}\right)=\frac{1}{2}{log}_2|\sum \left(\mathbf{X}\right)|+\frac{D_{{}_{\mathbf{X}}}}{2}{log}_2\left(2\pi {\alpha }^{{\alpha }^{\prime }/\alpha }\right).\end{array}$$

(30)

Here, $D_{{}_{\mathbf{X}}}$ is the dimension of $\mathbf{X}$ and ${\alpha }^{\prime }$ is a Hölder dual variable to $\alpha$ (i.e., $1/\alpha +1/{\alpha }^{\prime }=1$). In particular, for jointly multivariate Gaussian variables $\mathbf{X}$ and $\mathbf{Y}$, we can use (11) to write

$$\begin{array}{ccc}\hfill H_{\alpha }\left(\mathbf{X}\right|\mathbf{Y})& =& \left[\frac{1}{2}{log}_2|\sum (\mathbf{X}\oplus \mathbf{Y})|+\frac{D_{{}_{\mathbf{X}}}+D_{{}_{\mathbf{Y}}}}{2}{log}_2\left(2\pi {\alpha }^{{\alpha }^{\prime }/\alpha }\right)\right]\hfill \\ & -& \left[\frac{1}{2}{log}_2|\sum \left(\mathbf{Y}\right)|+\frac{D_{{}_{\mathbf{Y}}}}{2}{log}_2\left(2\pi {\alpha }^{{\alpha }^{\prime }/\alpha }\right)\right]\hfill \\ & =& \frac{1}{2}{log}_2|\sum \left(\mathbf{X}\right|\mathbf{Y})|+\frac{D_{{}_{\mathbf{X}}}}{2}{log}_2\left(2\pi {\alpha }^{{\alpha }^{\prime }/\alpha }\right).\hfill \end{array}$$

(31)

Here, ⊕ denotes the direct sum. Employing finally the defining relation (10), we obtain

$$\begin{array}{ccc}\hfill T_{\alpha ,Y\to X}^R(k,l)& =& H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)})-H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)},Y_n^{\left(l\right)})\hfill \\ & =& \frac{1}{2}{log}_2\left(\frac{|\sum (X_{t_{n+1}}\left|X_n^{\left(k\right)}\right)|}{|\sum (X_{t_{n+1}}|X_n^{\left(k\right)},Y_n^{\left(l\right)})|}\right).\hfill \end{array}$$

(32)

This confirms the statement of Theorem 1. In addition, since the standard measure of Granger causality (16) is typically defined only for the univariate target and source variables $X_t$ and $Y_t$, we can omit $|\dots |$ in (29) and (32).

Theorem 1 deserves two comments. First, the theorem is clearly true for any $\alpha$. In fact, it is $\alpha$ independent, which means that for Gaussian processes we can employ any RTE to test the Granger causality. This naturally generalizes the classical result of Barnett et al. [61] (see also [1]) that is valid for Shannon’s TE. When TE is phrased in terms of the Shannon entropy, it is typically easier to use various multivariate autoregressive model fitting techniques (e.g., the Lewinson–Wiggins–Robinson algorithm or the least-squares linear regression approach [63]) to derive ${\mathcal{F}}_{Y\to X}^{(k,l)}$ more efficiently than by employing direct entropy/mutual information-based estimators. On the other hand, since the efficiency and robustness of RTE estimators crucially hinge on the parameter $\alpha$ employed [64] (see also our discussion in Section 4), it might be, in many cases, easier to follow the information-theoretic route to the Granger causality (provided the Gaussian framework is justified). One can even test the Gaussian assumption in the actual time series by determining the RTE for various $\alpha$ parameters and checking if the results are $\alpha$ independent.

Second, the exact equivalence between the Granger causality and RTE can be (in the Gaussian case) retraced to the fact that in Equation (30) the second additive term on the RHS is proportional to $D_{\mathbf{X}}$. It is not difficult to see (by a direct inspection) that this proportionality will be preserved in many other exponential distributions that satisfy the Markov factorization property. In these cases, the equivalence between the Granger causality and RTE statistics will also be preserved. However, for generic distributions, the additive term in (30) will no longer be a linear function of $D_{\mathbf{X}}$ and, hence, it will not be canceled. This, in turn, spoils the desired equivalence. In the following section, we will discuss one possible generalization of Theorem 1 in the context of heavy-tailed distributions.

3.2. Granger Causality—Heavy-Tailed Variables

It is not difficult to find relations analogous to (32) in a more general setting. Here, we will illustrate this point with heavy-tailed (namely $\alpha$-Gaussian) random variables, where computations can be conducted analytically.

It is well known that if variance and mean are the only statistical observables, then the conventional maximum entropy principle (MaxEnt) based on Shannon entropy yields Gaussian distribution. Similarly, if the very same MaxEnt is applied to Rényi entropy $H_{\alpha }$, one obtains the so-called $\alpha$-Gaussian distribution [34] (cf. also Figure 2)

$$\begin{array}{c}\hfill p_i=\frac{1}{{\mathcal{Z}}_{\alpha }}{\left[1-\beta (\alpha -1)x_i^2\right]}_{+}^{1/(\alpha -1)},\end{array}$$

(33)

that decays asymptotically following power law. Here, $\beta \in {\mathbb{R}}^{+}$ and ${\left[z\right]}_{+}=z$ if $z\ge 0$ and 0, otherwise, ${\mathcal{Z}}_{\alpha }$ is the normalization factor. It is more conventional to write (33) as

$$\begin{array}{c}\hfill p_i={\mathcal{Z}}_{\alpha }^{-1}{exp}_{\{2-\alpha \}}(-\beta x_i^2),\end{array}$$

(34)

where

$$\begin{array}{c}\hfill e_{\left\{\alpha \right\}}^x={\left[1+(1-\alpha )x\right]}_{+}^{1/(1-\alpha )},\end{array}$$

(35)

is the Box–Cox $\alpha$-exponential [30].

$\alpha$-Gaussian distribution (33) has finite variance (and, more generally, the covariance matrix) for $\frac{D}{2+D}<\alpha \le 1$. Let us now assume that Granger’s linear (full/reduced) regression model is described by joint processes $X_t$ and $Y_t$ that are $\alpha$-Gaussian. We now prove the following theorem:

Theorem 2.

If the joint process $X_t$, $Y_t$ is α-Gaussian with $\alpha \in \left(\frac{1+k+l}{3+k+l},1\right]$ (i.e., a finite covariance matrix region) then ${\mathcal{F}}_{Y\to X}^{(k,l)}-2T_{\alpha ,Y\to X}^R(k,l)$ is a monotonically decreasing function of α (at fixed k and l) with zero reached at a stationary point $\alpha =1$. The leading-order correction to the Granger causality is “k”-independent and has the form

$$\begin{array}{c}\hfill {\mathcal{F}}_{Y\to X}^{(k,l)}=2T_{\alpha ,Y\to X}^R(k,l)+\frac{l{(\alpha -1)}^2}{4}+\mathcal{O}({(\alpha -1)}^3).\end{array}$$

(36)

This result explicitly illustrates how certain “soft” heavy-tailed processes can be related to the concept of the Granger causality via universal types of corrections that are principally discernible in data analysis.

Theorem 2 can be proved in close analogy with our proof of Theorem 1. In fact, all steps in the proof are identical up to Equation (29). For the D-dimensional $\alpha$-Gaussian process, the scaling property (30) reads

$$\begin{array}{c}\hfill H_{\alpha }\left(\mathbf{X}\right)=\frac{1}{2}{log}_2|\sum \left(\mathbf{X}\right)|+H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D}).\end{array}$$

(37)

Here, ${\mathbf{Z}}_{\alpha }^{\mathbf{1},D}$ represents an $\alpha$-Gaussian random vector with zero mean and unit ($D\times D$) covariance matrix. Relation (37) results from the following chain of identities

$$\begin{array}{ccc}\hfill H_{\alpha }\left(\mathbf{X}\right)& =& H_{\alpha }(\sqrt{\sum \left(\mathbf{X}\right)}{\mathbf{Z}}_{\alpha }^{\mathbf{1},D})\hfill \\ & =& \frac{1}{1-\alpha }{log}_2{\int }_{{\mathbb{R}}^D}{d}^D\mathbf{y}{\left({\int }_{{\mathbb{R}}^D}{d}^D\mathbf{z}\delta \left(\mathbf{y}-\sqrt{\sum \left(\mathbf{X}\right)}\mathbf{z}\right)\mathcal{F}\left(\mathbf{z}\right)\right)}^{\alpha }\hfill \\ & =& \frac{1}{1-\alpha }{log}_2\left[{|\sum \left(\mathbf{X}\right)|}^{(1-\alpha )/2}{\int }_{{\mathbb{R}}^D}{d}^D\mathbf{y}{\mathcal{F}}^{\alpha }\left(\mathbf{y}\right)\right]\hfill \\ & =& \frac{1}{2}{log}_2|\sum \left(\mathbf{X}\right)|+H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D}),\hfill \end{array}$$

(38)

which is clearly valid for any non-singular covariance matrix. The derivation $\mathcal{F}(\dots )$ denoted the $\alpha$-Gaussian probability density function with the unit covariance matrix and zero mean. We can now use the simple fact that

$$\begin{array}{ccc}& & H_{\alpha }\left({\mathbf{Z}}_{\alpha }^{\mathbf{1},D}\right)={log}_2\left[{\left(\frac{\pi }{\mathfrak{b}(1-\alpha )}\right)}^{D/2}\frac{\Gamma \left(\frac{1}{1-\alpha }-\frac{D}{2}\right)}{\Gamma \left(\frac{1}{1-\alpha }\right)}{\left(1-\frac{D}{2\alpha }(1-\alpha )\right)}^{1/(\alpha -1)}\right]\hfill \\ & & =\frac{D}{2}{log}_2\left[2\pi \alpha \right]+{log}_2\left[\frac{\Gamma \left(\frac{1}{1-\alpha }-\frac{D}{2}\right)}{{(1-\alpha )}^{D/2}\Gamma \left(\frac{1}{1-\alpha }\right)}\right]+{log}_2\left[{\left(1-\frac{D}{2\alpha }(1-\alpha )\right)}^{\frac{D}{2}-\frac{1}{1-\alpha }}\right],\hfill \end{array}$$

(39)

(where $\mathfrak{b}={[2\alpha -D(1-\alpha )]}^{-1}$), to write

$$\begin{array}{ccc}\hfill H_{\alpha }\left(\mathbf{X}\right|\mathbf{Y})& =& \frac{1}{2}{log}_2|\sum \left(\mathbf{X}\right|\mathbf{Y})|+H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}+D_{{}_{\mathbf{Y}}}})-H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{Y}}}}).\hfill \end{array}$$

(40)

At this stage, we note that

$$\begin{array}{ccc}\hfill H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}+D_{{}_{\mathbf{Y}}}})& -& H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{Y}}}})-H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}})\hfill \\ & =& H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{Y}}}})-H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}}),\hfill \end{array}$$

(41)

which is not zero as it was in the case of the Gaussian distribution. In fact, from the foregoing discussion, it is clear that for the $\alpha$-Gaussian random variables, we can write the RTE in the form

$$\begin{array}{ccc}& & T_{\alpha ,Y\to X}^R(k,l)=H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)})-H_{\alpha }\left(X_{t_{n+1}}\right|X_n^{\left(k\right)},Y_n^{\left(l\right)})\hfill \\ & & =\frac{1}{2}{log}_2\left(\frac{\sum \left(X_{t_{n+1}}\right|X_n^{\left(k\right)})}{\sum \left(X_{t_{n+1}}\right|X_n^{\left(k\right)},Y_n^{\left(l\right)})}\right)+H_{\alpha }\left({\mathbf{Z}}_{\alpha }^{1,1}\right|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})-H_{\alpha }\left({\mathbf{Z}}_{\alpha }^{1,1}\right|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k+l})\hfill \\ & & =\frac{1}{2}{\mathcal{F}}_{Y\to X}^{(k,l)}+I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k}).\hfill \end{array}$$

(42)

Here, we have set ${\mathbf{Z}}_{\alpha }^{1,1}$ to correspond to the random variable $X_{t_{n+1}}$ with unit variance. Similarly, ${\mathbf{Z}}_{\alpha }^{\mathbf{1},k}$ and ${\mathbf{Z}}_{\alpha }^{\mathbf{1},l}$ correspond to unit covariance random variables $X_n^{\left(k\right)}$ and $Y_n^{\left(l\right)}$, respectively.

Clearly, when $Y_t$ and $X_t$ processes are independent (and, hence, not causal in the Granger sense), their joint distribution factorizes and, thus, $H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}+D_{{}_{\mathbf{Y}}}})\mapsto H_{\alpha }({\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{X}}}}\times {\mathbf{Z}}_{\alpha }^{\mathbf{1},D_{{}_{\mathbf{Y}}}})$. Additivity of the RE then ensures that $H_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})=H_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k+l})$ and, hence, $I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})$ is zero. In other words, when two processes are not Granger causal, their RTEs are zero. Actually, it is not difficult to see that this is true irrespective of a specific form of the distribution involved. However, the opposite is not true since $I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})$ might be (unlike in Shannon’s case) negative; consequently, $T_{\alpha ,Y\to X}^R(k,l)$ can be zero even if ${\mathcal{F}}_{Y\to X}^{(k,l)}$ is not. To understand this point better, we explicitly evaluate $I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})$ for our $\alpha$-Gaussian random variables. Using (39), we can write

$$\begin{array}{ccc}\hfill I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})& =& {log}_2\left[\frac{\Gamma \left(\frac{1}{1-\alpha }-\frac{1+k}{2}\right)}{\Gamma \left(\frac{1}{1-\alpha }-\frac{k}{2}\right)}\frac{\Gamma \left(\frac{1}{1-\alpha }-\frac{k+l}{2}\right))}{\Gamma \left(\frac{1}{1-\alpha }-\frac{1+k+l}{2}\right)}\right]\hfill \\ & +& {log}_2\left[\frac{{\left(\frac{\alpha }{1-\alpha }-\frac{1+k}{2}\right)}^{\frac{1+k}{2}-\frac{1}{1-\alpha }}}{{\left(\frac{\alpha }{1-\alpha }-\frac{k}{2}\right)}^{\frac{k}{2}-\frac{1}{1-\alpha }}}\frac{{\left(\frac{\alpha }{1-\alpha }-\frac{k+l}{2}\right)}^{\frac{k+l}{2}-\frac{1}{1-\alpha }}}{{\left(\frac{\alpha }{1-\alpha }-\frac{1+k+l}{2}\right)}^{\frac{1+k+l}{2}-\frac{1}{1-\alpha }}}\right].\hfill \end{array}$$

(43)

By setting $\zeta =\frac{1}{1-\alpha }-\frac{k}{2}$ and $\xi =\frac{1}{1-\alpha }-\frac{k+l}{2}$, we can rewrite (43) as

$$\begin{array}{ccc}\hfill I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})& =& {log}_2\left[\frac{\Gamma \left(\zeta -\frac{1}{2}\right)}{\Gamma \left(\zeta \right)}\frac{{(\zeta -1)}^{\zeta }}{{\left(\zeta -\frac{3}{2}\right)}^{\zeta -\frac{1}{2}}}\frac{\Gamma \left(\xi \right)}{\Gamma \left(\xi -\frac{1}{2}\right)}\frac{{\left(\xi -\frac{3}{2}\right)}^{\xi -\frac{1}{2}}}{{(\xi -1)}^{\xi }}\right]\hfill \\ & =& {log}_2\left[\frac{\Gamma \left(\zeta -\frac{3}{2}\right)}{\Gamma \left(\zeta -1\right)}\frac{{(\zeta -1)}^{\zeta -1}}{{\left(\zeta -\frac{3}{2}\right)}^{\zeta -\frac{3}{2}}}\frac{\Gamma \left(\xi -1\right)}{\Gamma \left(\xi -\frac{3}{2}\right)}\frac{{\left(\xi -\frac{3}{2}\right)}^{\xi -\frac{3}{2}}}{{(\xi -1)}^{\xi -1}}\right]\hfill \\ & \le & -\frac{1}{2}{log}_2\left[\frac{(\xi -1)}{\left(\xi -\frac{3}{2}\right)}\right]\le 0,\hfill \end{array}$$

(44)

where on the last line we use the Kečkić–Vasić inequality [65]

$$\begin{array}{c}\hfill \frac{{(x+1)}^{x+1}}{{(x+s)}^{x+s}}{e}^{s-1}\le \frac{\Gamma (x+1)}{\Gamma (x+s)}\le \frac{{(x+1)}^{x+\frac{1}{2}}}{{(x+s)}^{x+s-\frac{1}{2}}}{e}^{s-1},\end{array}$$

(45)

valid for $s\in (0,1)$. In addition, it can be numerically checked that $\frac{d{I}_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})}{d\alpha }>0$, for all $l,k$ from the definition, so the maximum of $I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})$ is attained at $\alpha =1$, see Figure 3. When $\alpha$ is close to 1, then one can employ the asymptotic relation $\Gamma [x+\gamma ]\sim \Gamma \left[x\right]x^{\gamma }$ valid for $x\gg 1$, $\gamma \in \mathbb{C}$, and rewrite (39) in the form $(D/2){log}_2\left[2\pi \alpha e^{\alpha }\right]$. In this case, (43) tends to zero and we obtain equivalence between TE and the Granger causality. This result should not be so surprising because in the limit $\alpha \to 1$, RE tends to Shannon’s entropy and the $\alpha$-Gaussian distribution tends to the Gaussian distribution.

The leading order behavior near $\alpha =1$ can be obtained directly from (43). The ensuing Taylor expansion gives

$$\begin{array}{c}\hfill I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})=-\frac{l{(\alpha -1)}^2}{8}+\mathcal{O}({(\alpha -1)}^3),\end{array}$$

(46)

so, the point $\alpha =1$ is a stationary point of $I_{\alpha }({\mathbf{Z}}_{\alpha }^{1,1}:{\mathbf{Z}}_{\alpha }^{\mathbf{1},l}|{\mathbf{Z}}_{\alpha }^{\mathbf{1},k})$. This closes the proof.

4. Estimation of Rényi Entropy

4.1. RTE and Derived Concepts

From a data analysis point of view, it is not very practical to use the full joint processes $X_n^{\left(k\right)}$ and $Y_n^{\left(l\right)}$ (cf. the defining relation (10)) because (possibly) high values of k and l negatively influence the accuracy of estimation of RTE. In the following sections, we will thus switch to a more expedient definition of RTE given by

$$\begin{array}{ccc}\hfill T_{\alpha ,Y\to X}^R(\left\{k\right\},\left\{m\right\},\left\{l\right\})& =& H_{\alpha }\left(X_{{}_n}^{\left\{m\right\},+}\right|X_n^{\left\{k\right\},-})-H_{\alpha }\left(X_{{}_n}^{\left\{m\right\},+}\right|X_n^{\left\{k\right\},-},Y_n^{\left\{l\right\},-})\hfill \\ & =& I_{\alpha }(X_{{}_n}^{\left\{m\right\},+}:Y_n^{\left\{l\right\},-}|X_n^{\left\{k\right\},-}),\hfill \end{array}$$

(47)

where $X_n^{\left\{k\right\},\Omega }$ is a subset of past ($\Omega =-$) or future ($\Omega =+$) values of $X_{t_n}$ with the number of elements equal to k, such that $\left\{k\right\}=\{{\kappa }_1,\dots ,{\kappa }_k\}$ is a set of indices and $X_n^{\left\{k\right\},\Omega }\equiv X_{t_{n\Omega {\kappa }_1}},X_{t_{n\Omega {\kappa }_2}},\dots ,X_{t_{n\Omega {\kappa }_k}}$ is a selected subsequence of $X_{t_n}$, i.e., $n_X$-dimensional vectors. The same notational convention applies to $Y_n^{\left\{l\right\},\Omega }$ as a subsequence of $Y_{t_n}$, i.e., $n_Y$-dimensional vectors. In definition (47), we added a third parameter, m—the so-called future step. Though such a parametrization is often used in the literature on Shannon’s TE, cf., e.g., reference [17], we will (in the following) only employ $m=\left\{1\right\}$ so as to conform with the definition (10). In such a case, we will often omit the middle index in $T_{\alpha ,Y\to X}^R(\left\{k\right\},\left\{1\right\},\left\{l\right\})$.

4.1.1. Balance of Transfer Entropy

In order to compare RTE that flows in the direction from $Y\to X$ with the RTE that flows in the opposite direction $X\to Y$, we define the balance of transfer entropy

$$T_{\alpha ,Y\to X}^{R,\mathrm{balance}}(\left\{k\right\},\left\{l\right\})=T_{\alpha ,Y\to X}^R(\left\{k\right\},\left\{l\right\})-T_{\alpha ,X\to Y}^R(\left\{k\right\},\left\{l\right\}).$$

(48)

4.1.2. Effective Transfer Entropy

To mitigate the finite size effects, we employ the idea of a surrogate time series. To this end, we define the effective transfer entropy

$$T_{\alpha ,Y\to X}^{R,\mathrm{effective}}(\left\{k\right\},\left\{l\right\})=T_{\alpha ,Y\to X}^R(\left\{k\right\},\left\{l\right\})-T_{\alpha ,Y^{\left(\mathrm{sur}\right)}\to X}^R(\left\{k\right\},\left\{l\right\}),$$

(49)

where $Y^{\left(\mathrm{sur}\right)}$ stands for the randomized (reordered) time series—the surrogate data sequence. Such a series has the same mean, the same variance, the same autocorrelation function and, therefore, the same power spectrum as the original sequence, but (nonlinear) phase relations are destroyed. In effect, all the potential correlations between $X_n^{\left\{k\right\}}$ and $Y_n^{\left\{l\right\}}$ are removed, which means that $T_{\alpha ,Y^{\left(\mathrm{sur}\right)}\to X}^R(\left\{k\right\},\left\{l\right\})$ should be zero. In practice, this is not the case, despite the fact that there are no obvious structures in the data. The non-zero value of $T_{\alpha ,Y^{\left(\mathrm{sur}\right)}\to X}^R(\left\{k\right\},\left\{l\right\})$ must then be a byproduct of the finite data set. Definition (49) then ensures that spurious effects caused by finite k and l are removed. In our computations, we used the Fisher–Yates algorithm [66] together with Mersenne twister random generation algorithm [67] for the randomized surrogates. For a more technical exposition, see, e.g., refs. [68,69,70].

4.1.3. Balance of Effective Transfer Entropy

Finally, we combined both previous definitions to form the balance effective transfer entropy

$$\begin{array}{ccc}\hfill T_{\alpha ,Y\to X}^{R,\mathrm{balance},\mathrm{effective}}(\left\{k\right\},\left\{l\right\})& =& T_{\alpha ,Y\to X}^{R,\mathrm{effective}}(\left\{k\right\},\left\{l\right\})-T_{\alpha ,X\to Y}^{R,\mathrm{effective}}(\left\{k\right\},\left\{l\right\})\hfill \\ & =& T_{\alpha ,Y\to X}^R(\left\{k\right\},\left\{l\right\})-T_{\alpha ,Y^{\left(\mathrm{sur}\right)}\to X}^R(\left\{k\right\},\left\{l\right\})\hfill \\ & -& T_{\alpha ,X\to Y}^R(\left\{k\right\},\left\{l\right\})+T_{\alpha ,X^{\left(\mathrm{sur}\right)}\to Y}^R(\left\{k\right\},\left\{l\right\}),\hfill \end{array}$$

(50)

to quantify the direction of flow of transfer entropy without finite size effects.

4.1.4. Choice of Parameters k and l

The choice of the parameters k and l is essential to reliably analyze the information transfer between variables in a system. So, a natural question arises as to how one should choose such parameters.

The order of k and l, both in the RTE and Shannon’s TE, but also in approximating autoregression in the Granger case, is often (in practice) set rather arbitrarily at some moderately high number. In the literature, there are theoretical criteria for optimal choices of k and l—with no unique answer. In our numerical simulations, we employed two pragmatic criteria: $\left(a\right)$ results should be stable under the increase of k and l and, additionally, $\left(b\right)$ k, and l should be equal to—or higher than—those used in the literature for the analysis of Shannon’s TE in Rössler systems, e.g., references [18,22], so that we could make a comparison with the existence results. The chosen values $\left(\right\{k\},\{l\left\}\right)\equiv \left(\right\{k\},\{1\},\{l\left\}\right)=\left(\right\{0,1\},\{1\},\{0\left\}\right)$ often well-satisfied both aforementioned conditions. In Section 6.3, it was sufficient to set $\left\{k\right\}=\left\{0\right\}$ and $\left\{l\right\}=\left\{0\right\}$, in agreement with [18]. When a need has arisen to emphasize some finer details in the behavior of the RTE (cf. Figures 6 and 10), $\left\{k\right\}$ was chosen to be $\{0,1,2,3,4\}$ or even $\{0,1,2,3,4,5,6\}$.

5. Rössler System

5.1. Equations for Master System

In order to illustrate the use of RTE, we considered two unidirectionally coupled Rössler systems (oscillators). These often serve as testbeds for various measures of synchronization, including Shannon’s TE [71,72,73]. Rössler’s system is described by three non-linearly coupled partial differential equations

$$\begin{array}{ccc}\hfill & & {\dot{x}}_1=-{\omega }_1{x}_2-x_3,\hfill \\ & & {\dot{x}}_2={\omega }_1{x}_1+a{x}_2,\hfill \\ & & {\dot{x}}_3=b+x_3(x_1-c),\hfill \end{array}$$

(51)

with four coefficients ${\omega }_1$, a, b, and c. Strictly speaking, only three coefficients are independent, as ${\omega }_1$ can be set to one by appropriately rescaling $x_2$. RS was invented in 1976 by O.E. Rössler [32] and it likely represents the most elementary geometric construction of chaos in the continuous systems. In fact, since the Poincaré–Bendixson theorem precludes the existence of (other than) steady, periodic, or quasi-periodic attractors in autonomous systems, defined in one- or two-dimensional manifolds, the minimal dimension for chaos is three [74]. The simplicity of the RS is bolstered by the fact that it only has one nonlinear (quadratic) coupling.

RS classifies as the continuous (deterministic) chaotic system, and more specifically as the chaotic attractor. The word “attractor” refers to the fact that whatever is the initial condition for the solution of the differential Equations (52), the trajectory $\mathbf{x}\left(t\right)$ ends up (after a short transient period) at the same geometrical structure (see Figure 5), which is neither a fixed point nor a limit cycle. This attractive geometrical structure is known as the Rössler attractor.

For future convenience, we will call the RS (51) as driving or master system and denote it as $\left\{X\right\}$.

5.2. Equations for the Slave System

In the following, we investigate RTE between two Rössler systems that are unidirectionally coupled in the variable $x_1$ via a small adjustable parameter $\epsilon$. The corresponding second RS—driven or slave system, is defined as

$$\begin{array}{ccc}\hfill & & {\dot{y}}_1=-{\omega }_2{y}_2-y_3+\epsilon (x_1-y_2),\hfill \\ & & {\dot{y}}_2={\omega }_2{y}_1+a{y}_2,\hfill \\ & & {\dot{y}}_3=b+y_3(y_1-c).\hfill \end{array}$$

(52)

Here, we fix the coefficients so that $a=0.15$, $b=0.2$, $c=10.0$, and frequencies ${\omega }_1=1.015$ and ${\omega }_2=0.985$, and initial conditions $(x_1\left(0\right),x_2\left(0\right),x_3\left(0\right))=(0,0,0)$ and $(y_1\left(0\right),y_2\left(0\right),y_3\left(0\right))=(0,0,1)$. This parametrization is adopted from reference [18] where Shannon’s TE between systems (51) and (52) was studied. In the following, we will denote the slave system also as $\left\{Y\right\}$.

5.3. Numerical Experiments with Coupled RSs

Before we embark on the RTE analysis, let us first take a look at the phenomenology of the coupled RSs (51) and (52) by means of simple numerical experiments. In our numerical treatment, we simulate coupled RSs by using the integration method, which is implemented in a package SciPy named solve_ivp with the LSODA option that exploits the Addams/BDF method, see, e.g., reference [75]. Projections of the $\epsilon$-dependent RSs dynamics to various planes are presented in Figure 5. For visualization purposes, we used the toolkit Matplotlib [76] that exploits toolkit NumPy [77]. The sources are part of the Pyclits project [78]. In the future, the work can be rebased. The resulting data set analyzed consisted of 100,000 data points. To gain insight into the transient region, we chose shorter time lags in the data set generated from RS with $0.1\le \epsilon \le 0.15$, namely, we reduced the time steps from $0.01$ to $0.001$. In parallel, we display in Figure 4 the behaviors of the corresponding Lyapunov exponents, as adapted from [22], which help to elucidate our discussion.

Projections

Instead of a conventional stereoscopic plotting, we found it more convenient (and illuminating) to focus on various plane projections of the coupled RSs. First, we noticed that, in Figure 5, the projections of RSs on the $x_2$-$x_1$, $x_3$-$x_2$, and $x_1$-$x_3$ planes do not depend on the coupling between systems (i.e., they are $\epsilon$-independent), as expected, because the slave system (52) does not influence dynamics of the master system (51), which is autonomous (irrespective of $\epsilon$). However, it is clear that signatures of the interaction between non-symmetrically coupled RSs (51) and (52) will show up in projections on the $x_i$-$y_j$ and $y_i$-$y_j$ planes.

Secondly, when the RSs are not coupled (i.e., when $\epsilon =0$), we have two autonomous RSs—in fact, two strange attractors that differ only by values of their frequency coefficients and initial values. The autonomies of the respective RSs are clearly seen in projections on the $x_i$-$x_j$ and $y_i$-$y_j$ planes (cf. Figure 5). A different density of trajectories (in a given time window t = 100,000) can be ascribed to the frequency mismatch. Projections on the $x_1$-$y_1$ and $x_2$-$y_2$ planes show how the ensuing chaotic and (component-wise) uncorrelated trajectories fill their support regions. In particular, we can observe that on the background of densely packed chaotic trajectories, clear vertical stripes of dominantly-visited regions appear in the slave system. Vertical stripes are clearly visible because limit cycles in the autonomous slave system are far more localized than in the master system. The projection on the $x_3$-$y_3$ plane indicates that (most of the time) the master system orbits venture to the $x_3$ direction, the slave system orbits are in the vicinity of the $y_1$-$y_2$ plane, and vice versa.

By continuously increasing the coupling strength $\epsilon$ from the zero value, we can observe that, already, a small interaction significantly changes the evolution of the slave system. For instance, in Figure 5, we see that when $\epsilon =0.01$, then the diffusive term $\epsilon (x_1-y_2)$ significantly disperses the limit cycles in the slave system. This is reflected not only in all projections on the $y_i$-$y_j$ planes but also in projections on the $x_1$-$y_1$ and $x_2$-$y_2$ planes. In the latter two cases, the diffusion causes that horizontal stripes to completely disappear. Finally, the projection on the $x_3$-$y_3$ plane does not change significantly from the $\epsilon =0$ case.

When we further increase $\epsilon$, we see that the behavior of the slave system starts to qualitatively depart from that of the master system. For $\epsilon$, around $0.1$, the slave system orbit diffuses to the region around the origin that is basically not visited (apart from an initial transient orbit) by the master system orbit (cf. projections on the $y_i$-$y_j$ planes). In addition, projections on the $x_1$-$y_1$ and $x_2$-$y_2$ planes disclose that the ensuing support areas are not filled anymore. In fact, we can see a development of a slant stripe structure. On the other hand, the projection on the $y_3$-$x_3$ plane reveals that the slave system orbits stop visiting regions further from $y_3=0$. A yet higher $\epsilon$ (around $0.14$) orbit of the system $\left\{Y\right\}$ first converges to a single limit cycle before it makes (again) a transition into a chaotic regime. Finally, we can observe that at $\epsilon \sim 0.14$, the slave system rarely deviates far from $y_3=0$ and spends most of its time in the close vicinity of the $y_1$-$y_2$ plane—its evolution is “flattened”.

Moreover, at $\epsilon \sim 0.14$, we can also notice that projections on the $y_1$-$x_1$ and $y_2$-$x_2$ planes underwent a change in topology (in fact, this happened already at around $\epsilon \sim 0.12$). The onset of this “topological phase transition” is closely correlated with the behavior of the largest Lyapunov exponent (LE) of the slave system. In fact, coupled RSs altogether have six Lyapunov exponents. The $\epsilon =0$ one has two autonomous RSs each with three LEs—one positive, one zero, and one negative (signature $+0-$ is a typical hallmark of a strange attractor in three dimensions). While at $\epsilon =0$, the signature of LEs is $++00--$, increasing $\epsilon$ all three LEs associated with $\left\{Y\right\}$ decreasing (initially) monotonically, cf. Figure 4. After a transient negativity and a return to zero (red curve in Figure 4), the originally positive LE of the slave system monotonically decreases and the negative for $\epsilon \gtrsim 0.15$. In particular, we see that the critical value $\epsilon \sim 0.12$ at which the “topological phase transition” occurs coincides with the value at which the largest LE of the system $\left\{Y\right\}$ crosses zero.

What is particularly noteworthy is an abrupt (non-analytic) change in the behavior of LEs at the value $\epsilon \sim 0.145$. At this value, the LE changes direction and starts to increase with increasing $\epsilon$. The increase stops at $\epsilon \sim 0.15$ when the yellow-colored LE in Figure 4 reaches (approximately) value zero, after which it monotonically decreases. Such a decrease also starts for the second red-colored LE, but at a slightly different value of $\epsilon$.

For stronger interactions with $0.15\lesssim \epsilon \lesssim 0.2$, we see (cf. Figure 5 with $\epsilon =0.16$) that the slave system starts to approach the structure of the master system strange attractor (cf. $x_i$-$x_j$ and $y_i$-$y_j$ projections). From the tilt and thinning of projections on the $x_1$-$y_1$ and $x_2$-$y_2$ planes, one may deduce that the amplitude synchronizations in the $x_1$ and $y_1$ (as well as $x_2$ and $y_2$) directions increase. Projection on the $x_3$-$y_3$ plane shows that amplitudes in the $x_3$ and $y_3$ directions are also synchronized (being roughly a half-cycle behind each other).

Finally, for very strong interactions, e.g., for $\epsilon \sim 0.5$, the synchronization is almost complete: the system $\left\{Y\right\}$ basically fully emulates the master system’s behavior with both systems now being structurally identical (cf. $x_i$-$x_j$ and $y_i$-$y_j$ projections). Full synchronization is nicely seen in projections on the $x_1$-$y_1$ and $x_2$-$y_2$ planes. Note that the amplitudes in the $x_3$ and $y_3$ directions start to synchronize.

6. Numerical Analysis of RTE for Coupled RSs

In the previous section, we learned some essentials about the coupled RS (51) and (52). In order to demonstrate the inner workings of the RTE and to gain further insight into how the two RSs approach synchronization, we compute here the RTE for various salient situations, such as the RTE between the $x_1$- and $y_1$-component, between the $x_1$- and $y_3$-component, or RTE between the full master and slave system. In our numerical analysis, we employed the RE estimator introduced by Leonenko et al. [26]. Some fundamentals associated with this estimator are relegated to Appendix A.

6.1. Effective RTE between x₁ and y₁ Directions

In order to understand the dynamics of the two coupled nonlinear dynamical systems (51) and (52) on their routes to synchronization, we first analyzed the effective RTE between the $x_1$ and $y_1$ components. Corresponding plots for different coupling strengths $\epsilon$ and different orders $\alpha$ are depicted in Figure 6. We can observe first that the effective RTE from $x_1$ to $y_1$ gradually increases with the increasing coupling strength until $\epsilon \sim 0.12$. The regime between $\epsilon \sim 0.12$ and $\epsilon \sim 0.15$, as seen from Figure 5, corresponds to a transient synchronization behavior, which stabilizes only after $\epsilon \sim 0.15$. This can also be seen from the behavior of the LEs at Figure 4. It should also be noted that the behavior of effective RTEs in the transient regime is apparently almost identical for all $\alpha$ in both $T_{\alpha ,x_1\to y_1}^{R,\mathrm{effective}}(\{0,1\},\left\{1\right\},\left\{0\right\})$ and $T_{\alpha ,y_1\to x_1}^{R,\mathrm{effective}}(\{0,1\},\left\{1\right\},\left\{0\right\})$. This would, in turn, indicate that the information transfer is the same across all sectors of the underlying probability distributions. Upon closer inspection though, such a highly correlated behavior will disappear when more historic data on $\left\{X\right\}$ and $\left\{Y\right\}$ are included (cf. $T_{\alpha ,x_1\to y_1}^{R,\mathrm{effective}}(\{0,1,2,3,4,5,6\},\left\{1\right\},\left\{0\right\})$ and $T_{\alpha ,y_1\to x_1}^{R,\mathrm{effective}}(\{0,1,2,3,4,5,6\},\left\{1\right\},\left\{0\right\})$ in Figure 6).

The same conclusion can be reached when the effective RTEs for the full six-dimensional systems are considered, cf. Figure 7.

Nevertheless, from Figure 6, it can clearly be inferred that—in the transient region—strong correlations do exist, albeit not for all $\alpha$s. In particular, one starts with the correlated flow for $\alpha \gtrsim 1.2$, which becomes stronger as $\epsilon$ increases. On the other hand, as $\epsilon$ approaches $0.15$, the information flow decreases for $\alpha \lesssim 1$. This can be seen clearly in both Figure 6 and Figure 7. At $\epsilon =0.15$, the information flow abruptly increases for all $\alpha$s. This is similar to a first order phase transition in statistical physics. In this respect, our “topological phase transition” would be more similar to a second order phase transition due to a smooth change in the entropic flow across the critical point $\epsilon =0.12$. This scenario is also supported by Figure 8, where the actual behavior of the RS between the two critical points for four selected values of $\epsilon$’s is depicted. Note, in particular, how the increase in the RTE for $\alpha \gtrsim 1.2$ (as well as the decrease of RTE for $\alpha \lesssim 1$) are reflected in the contractions (measure concentrations) of the regions with denser orbit populations in the slave system. This, in turn, reinforces the picture that RTEs with higher $\alpha$s describe the transfer of information between more central parts of underlying distributions, which, in this case, relate to higher occupation densities of the $\left\{Y\right\}$ system orbit. From Figure 8, we can also note that, at the critical point $\epsilon =0.15$, the contracted orbit regions abruptly expand and the slave system starts its way toward full synchronization with the master system. This is again compatible with the fact that the RTE abruptly increases for all $\alpha$s at this point—i.e., all parts of underlying distributions participate in this transition and, consequently, the occupation density of the $\left\{Y\right\}$ system orbit spreads. In this respect, point $\epsilon =0.15$ represents the threshold to full synchronization while point $\epsilon =0.12$ denotes the threshold to transient behavior prior to full synchronization. The latter can be identified with a phase synchronization threshold, which should be at (or very close to) this point [22].

After the critical point $\epsilon \sim 0.15$, both RSs enter full synchronization. In fact, the full synchronization starts when the information flow from all sectors of underlying distributions (i.e., for all $\alpha$s) starts to be (almost) $\epsilon$-independent and when $T_{\alpha ,X\to Y}^{R,\mathrm{balance},\mathrm{effective}}$ approach zero—so there is a one-to-one relation between the states of the systems, and the time series of the $\left\{X\right\}$ system can be predicted from the time series $\left\{Y\right\}$ system, and vice versa. Indeed, from Figure 6 (cf. also Figure 7 and Figure 9), we see that all $T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$ proceed in a slow increase toward their asymptotic values in the fully-synchronized state.

6.2. Effective RTE between x₃ and y₃ Directions

As already seen from Figure 5 and Figure 8, projections in the $x_3$-$y_3$ plane are particularly distinct. In Figure 10, we see the ensuing effective RTE between $x_3$ and $y_3$ directions.

What is particularly noticeable is a sudden increase in entropy transfer from the master to slave system at $\epsilon =0.12$ (i.e., at the threshold to transient behavior) for $\alpha <1$. No comparable increase is observed from slave to master. This, might be explained as an influx of information needed to organize the chaotically correlated regime that exists prior the (correlated) transient regime (cf. $x_i$–$y_i$ projections in Figure 5 and Figure 8). It should also be noticed that ordinary Shannonian TE ($\alpha =1$) is completely blind to such an information transfer.

As for the transient region, we can observe that the effective RTE has qualitatively very similar behavior to the effective RTE between $x_1$ and $y_1$, namely a distinct decrease in the information transfer for $\alpha <1$ and an increase for $\alpha >1$. This again reveals a measure concentration. In this case, the orbit occupation density concentrates around the $y_1$-$y_2$ plane of the slave systems, cf. projections depicted in Figure 8. The situation abruptly changes at the synchronization threshold $\epsilon =0.15$ after which the effective RTE approaches for each $\alpha$ a fixed asymptotic value that turns out to be the same for both $T_{\alpha ,x_3\to y_3}^{R,\mathrm{effective}}$ and $T_{\alpha ,y_3\to x_3}^{R,\mathrm{effective}}$.

6.3. Effective RTE for the Full System

In general, for a reliable inference, it is desirable that the conditioning variable in the definition or RTE (10) contains all relevant information about future values of the system or processes generating this variable in the uncoupled case. So, it should be a full three-dimensional vector X or Y in the case of RS. To this end, we display in Figure 7 the effective RTE for the full six-dimensional RS with information transfers in both $X\to Y$ and $Y\to X$ directions. Corresponding plots are depicted for different coupling strengths $\epsilon$, different order $\alpha$s, and different memories.

In particular, we can see that the information flow in the transient region starts after a brief decrease at around $\epsilon \sim 0.12$ and sharply increases (in both directions) for $\alpha \gtrsim 1.2$. This implies that there is an increase in the correlating activity in between regions with higher occupation densities in both REs. The behavior depicted in Figure 8 can help us to better understand this situation. In particular, we see that in the transient region the $\left\{Y\right\}$ system reshapes its orbit occupation density so that the ensuing measure concentrates more around its peak while its tail parts are thinner. In fact, Figure 8 also shows that this measure concentration increases until almost $\epsilon \sim 0.15$. The measure concentration behavior is reflected by the decrease of the RTE for $\alpha \lesssim 1$, i.e., decreasing information transfers between tail parts. This situation is even more pronounced when more memory is included in the effective RTEs, cf. both right pictures in Figure 7.

At the synchronization threshold $\epsilon =0.15$, the information flow abruptly changes for all $\alpha$s, with a particularly strong increase for $\alpha \lesssim 1$. This indicates that the orbit occupation density of the $\left\{Y\right\}$ system abruptly reshapes by lowering the measure concentrated around its peak and broadening it in tails, so that the tail parts may also enter the full synchronization regime.

Let us finally comment on the issue of bidirectional information flown for single-component RTEs. By envisioning the discretized versions of RSs, (51) and (52), one can see that RTE from the slave to the master system (e.g., between the $x_3$ and $y_3$ direction) cannot easily be zero. This is because $H_{\alpha }(X_{3,t_{n+1}}|X_{3,n}^{\left(k\right)},Y_{3,n}^{\left(l\right)})$ in the relation (10) is not simply $H_{\alpha }(X_{3,t_{n+1}}|X_{3,n}^{\left(k\right)})$. Note that due to the nonlinear nature of the coupled RSs, $y_3\left(t_n\right)$ depends both on $y_1\left(t_n\right)$ and $y_1\left(t_{n-1}\right)$ (via the third equation in (52)), while $y_1\left(t_n\right)$ depends on $x_1\left(t_n\right)$ and $x_1\left(t_{n-1}\right)$ (via the first equation in (52)); finally, $x_1\left(t_n\right)$ depends on $x_3\left(t_n\right)$ and $x_3\left(t_{n-1}\right)$ and also $x_2\left(t_n\right)$ and $x_2\left(t_{n-1}\right)$ (via the first equation in (51)); hence, $y_3\left(t_n\right)$ depends not only on $x_3\left(t_n\right)$, $x_3\left(t_{n-1}\right)$, $x_3\left(t_{n-2}\right)$ and $x_3\left(t_{n-3}\right)$ but also on historical values of $x_2$. In this way, $H_{\alpha }(X_{3,t_{n+1}}|X_{3,n}^{\left(k\right)},Y_{3,n}^{\left(l\right)}\left(\mathbf{X}\right))$ is not simply $H_{\alpha }(X_{3,t_{n+1}}|X_{3,n}^{\left(k\right)})$, as other components beyond $X_{3,n}$ are also needed. Consequently, when single-component RTEs for RS are computed, we inevitably find a non-zero information transfer from the slave to the master system. The latter is not so much a problem of k and l but rather the fact that we did not account for all relevant components (we simply missed some information).

It is true that for a reliable inference, in general, it would be desirable to obtain a zero value in the uncoupled direction $Y\to X$. This should be attained by proper conditioning—the conditioning variable should contain full information about future values of the system or processes generating this variable in the uncoupled case. So, it should be a three-dimensional vector $\mathbf{X}$ or $\mathbf{Y}$ for RS. Here, we computed effective RTE for the full six-dimensional system (vectors $\mathbf{X}$ and $\mathbf{Y}$). From Figure 7, we can see that $T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$ in the uncoupled direction stays at the zero value (particularly for larger values of $\alpha$) up to close to the synchronization threshold ($\epsilon =0.12$), while $T_{\alpha ,X\to Y}^{R,\mathrm{effective}}$ is distinctly positive there. So, RTE is a good causal measure only if the conditioning has a sufficient dimension (in our case, 3); otherwise, it can be viewed only as a measure of dependence.

6.4. Balance of Effective RTE

In order to quantify the difference between coupled ($X\to Y$) and uncoupled ($Y\to X$) information flow directions, we depict in Figure 9 the balance of effective RTEs between $T_{\alpha ,X\to Y}^{R,\mathrm{effective}}$ and $T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$ for two different situations. Let us first concentrate on the balance of effective RTE $T_{\alpha ,x_1\to y_1}^{R,\mathrm{balance},\mathrm{effective}}(\{0,1\},\left\{1\right\},\left\{0\right\})$. There, we can clearly see that before the synchronization threshold (“topological phase transition”), i.e., for $\epsilon \lesssim 0.12$, we have $T_{\alpha ,x_1\to y_1}^{R,\mathrm{effective}}>T_{\alpha ,y_1\to x_1}^{R,\mathrm{effective}}$, which indicates the correct direction of coupling. The fact that for $\alpha >1.6$ and $\epsilon \lesssim 0.04$ one has $T_{\alpha ,x_1\to y_1}^{R,\mathrm{balance},\mathrm{effective}}(\{0,1\},\left\{1\right\},\left\{0\right\})<0$ can be attributed to smaller reliability of the estimator in this region, cf. Figure 11 for estimation of ensuing the standard deviations. We can also observe that the synchronization threshold $T_{\alpha ,x_1\to y_1}^{R,\mathrm{balance},\mathrm{effective}}(\{0,1\},\left\{1\right\},\left\{0\right\})$ changes sign and slowly return back to positive values in the fully synchronized regime. Similar behavior was reported in [22] for Shannon’s TE. Moreover, in this transient region, the effective RTEs have the same values irrespective of $\alpha$, or, in other words, information transfer is the same across all sectors of the underlying probability distributions. This is akin to the behavior, which, in statistical physics, is typically associated with phase transitions—except for the fact that now we have a critical line rather than a critical point. However, as we already mentioned in the previous two paragraphs, this degeneracy is only spurious and will be removed by considering either the effective RTE for the full (six-dimensional) RS or longer memory.

After $\epsilon \sim 0.15$, the approach to full synchronization proceeds at slightly different rates for different $\alpha$s. This can equivalently be restated as saying that different parts of the underlying distributions enter synchronization differently. The dependence of the balance of effective RTE for the full (six-dimensional) system is shown on the right in Figure 9. Here, the behavior is less reliable for larger values of $\alpha$ ($\alpha \gtrsim 1.2$) and for smaller $\alpha$s ($\alpha \lesssim 0.8$), cf. Figure 11. In the region of reliable $\alpha$s, the behavior is qualitatively similar to that of $T_{\alpha ,x_1\to y_1}^{R,\mathrm{balance},\mathrm{effective}}(\{0,1\},\left\{1\right\},\left\{0\right\})$. On the other hand, apart from the region of a transient synchronization, we clearly have $T_{\alpha ,X\to Y}^{R,\mathrm{effective}}>T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$, which implies the correct direction of coupling. The approach to full synchronization is also easily recognized—the RTEs saturate to constant values (i.e., information transfer is $\epsilon$-independent) and both $T_{\alpha ,X\to Y}^{R,\mathrm{effective}}$ and $T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$ start to approach each other. In this respect, RTEs with lower $\alpha$s enter the synchronization regime slower than RTEs with larger $\alpha$s. In other words, events described by the tail parts of the distributions $p(x_{n+1}|x_n^{\left(k\right)})$ and $p(x_{n+1}|x_n^{\left(k\right)},y_n^{\left(l\right)})$ (corresponding to $\alpha <1$) will fully synchronize at higher values of $\epsilon$ than corresponding events described by central parts ($\alpha >1$).

In passing, we might notice that since both $T_{\alpha ,X\to Y}^{R,\mathrm{effective}}$ and $T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$ approach each other in the fully synchronized state, both the $\left\{X\right\}$ and $\left\{Y\right\}$ systems have to have the same underlying distributions (due to the reconstruction theorem for REs [21,34]) and, hence, they are indistinguishable, as one would expect.

7. Discussion and Conclusions

7.1. Theoretical Results

How one discerns ‘cause’ from ‘effect’ is the main question in many scientific areas. The seminal contribution of Wiener and Granger led to the so-called Granger causality principle and time series analysis method for inference of causality from experimental data. The traditional Granger causality method is based on linear autoregressive processes. However, nonlinear complex systems cannot be well-described by linear autoregressive models and require appropriate generalizations of the Granger causality method. One successful generalization stems from information theory, using a form of conditional mutual information, also known as transfer entropy. Shannon entropy-based TE has become a standard tool used for inferring causality from time series in all areas of science (including finance, climatology, neuroscience, etc.).

In this paper, instead of the Shannon entropy, we employed yet another information quantity, namely Rényi entropy. The ensuing RTE has the principal advantage that it is based on a bona fide information measure. In this way, one has a clear quantifier of the conveyed directional information (measured in bits or nats). Consequently, statements, such as: “the conveyed directional information from a tail part of the distribution is comparable with information from central part of distribution” or “information transfer is small/large (or good/bad)” are meaningful. RE is a measurable quantity; in principle, it can be measured directly (similar to Clausius entropy or Shannon entropy) without invoking the concept of the underlying distribution. This is because RE has an operational meaning given by various coding theorems. In practice, this is how RE is measured, e.g., in quantum optics (or more generally quantum information theory) [50,55]. In a conventional time series, one does not proceed this way because coding theorems (such as the Campbell coding theorem [44]) are difficult to implement for a large number of data.

As a proof of principle, we tested the concept of RTE on two unidirectionally coupled Rössler systems. The idea was to illustrate how the RTE can deal with such issues as synchronization and, more generally, causality in systems that are complex enough and yet amenable to a numerical analysis. Coupled RS is one of a handful of (simple) coupled chaotic systems that have been studied in the literature by means of Shannon’s TE. This point is particularly important because we needed a gauge to which we could compare our results (and to which our results should reduce for $\alpha =1$). Despite the earlier applications of the RTE in bivariate (mostly financial) time series, many questions remained unanswered about how to properly qualify and quantify the results obtained. Here, we went ‘some way’ toward this goal.

First, we showed that the concept of the Granger causality is exactly equivalent to the RTE for Gaussian processes, which may, in turn, be used as a test of Gaussianity. This is because RTEs are in the Gaussian framework all the same, and, hence, the results should be $\alpha$-independent. On the other hand, since the efficiency and robustness of RTE estimators crucially hinge on the parameter $\alpha$ employed, it might be (in many cases) easier to follow the information-theoretic route to Granger causality (provided the Gaussian framework is justified).

Second, we demonstrated that the equivalence between the Granger causality and RTE can also be established for certain heavy-tailed processes—for instance, for soft $\alpha$-Gaussian processes. In particular, in this latter case, one could clearly see the connection between Granger causality, Rényi’s parameter $\alpha$, and the heavy-tail power.

7.2. Numerical Analysis of RTE for Rössler Systems

In order to estimate the RTE, we employed the ℓ-nearest-neighbor entropy estimator of Leonenko et al. [26]. The latter is not only suitable for RE evaluation but it can also be easily numerically implemented to RTEs so that these can be computed almost in real time, which is relevant, e.g., in finance, regarding various risk-aversion decisions. Spurious effects caused by the finite size of the data set were taken into account by working with effective RTEs.

In order to gain further insight into the practical applicability and efficiency of the RTE, we tested it on two unidirectionally coupled Rössler systems—the master and slave system. To have a clear idea about what to expect, we first looked at the phenomenology of the coupled RSs by means of simple numerical simulations (presented in Figure 5). This was also accompanied by comparisons with Lyapunov exponents computed in references [18,22] and reproduced in Figure 4. In particular, we could clearly observe how the RSs synchronized with the increasing value of coupling strength. In this connection, we also identified critical values of coupling strengths at which thresholds to transient behavior (or the “topological phase transition”) and the threshold to full synchronization occurred.

More specifically, we were particularly interested in the transient region between chaotic correlation regimes and full synchronization, which had not as yet been discussed in the literature. To gain a better understanding of this region, we employed in the range $\epsilon \in [0.1,0.15]$ a higher frequency sampling, namely $0.001$, in contrast to the standard $0.01$ one used for other $\epsilon$s. The threshold to transient behavior was identified at the scale $\epsilon =0.12$ where the positive LE crossed to negative values and where the projection on the $x_1$-$y_1$ and $x_2$-$y_2$ planes underwent topology changes (cf. Figure 5). From the point of view of RTEs, this threshold behavior was reflected in peaking the information flow in various directions. The increase in the effective RTE between $x_1$ and $y_1$ (in both directions) for $\alpha >1$ was pronounced, in particular, which reflected the increase in orbit occupation density around the peak in the $y_1$-$y_2$ plane in the slave system. Even more marked was the high peak in information flow from $x_3$ to $y_3$ for $\alpha <1$ (see Figure 10), which described an influx of information needed to “organize” chaotic correlations that existed between the $x_3$ and $y_3$ directions prior to $\epsilon \lesssim 0.12$. Furthermore, the RTE was especially instrumental in understanding the measure concentration phenomenon in the transient regime. Finally, after a sharp “first-order-type” transition at the threshold of synchronization, the effective RTEs slowly approached their asymptotic values (distinct for each $\alpha$) in the synchronized state. In addition, in the synchronized state, both $T_{\alpha ,X\to Y}^{R,\mathrm{effective}}$ and $T_{\alpha ,Y\to X}^{R,\mathrm{effective}}$ approached each other, which reveals that both $\left\{X\right\}$ and $\left\{Y\right\}$ systems have the same underlying distributions and, hence, they are indistinguishable.

As for the causality issue, we observed that the RTE is a good causal measure only if the conditioning has a sufficient dimension (in our case 3); otherwise, it is merely a measure of dependence. By employing effective RTE for the full system, we could reliably infer the coupling direction but only until $\epsilon \lesssim 0.12$, i.e., until the threshold to transient behavior. After this value, the RSs started to synchronize, first partially (in the transient regime) and then fully $\epsilon =0.15$. In fact, the full synchronization started when the information flows from all sectors of underlying distributions (i.e., for all $\alpha$s) began to be (almost) $\epsilon$ independent and when $T_{\alpha ,X\to Y}^{R,\mathrm{balance},\mathrm{effective}}$ approached zero— so there was a one-to-one relation between the states of the systems and the time series of the $\left\{X\right\}$ system could be predicted from the time series $\left\{Y\right\}$ system, and vice versa; hence, one could not make any statement about the coupling direction.

We should also reemphasize that the standard deviation of the RTE importantly depends on $\alpha$, cf. Equation (Figure 11). For instance, the balance effective RTE for the full system is around the transient region quite reliably described by $0.8\lesssim \alpha \lesssim 1.25$, though the minimal noise value is not attained at $\alpha =1$ (Shannon transfer entropy) but at $\alpha =1.16$. Clearly, the $\alpha$-dependence of fluctuations is generally dynamics-dependent, and in many interesting real-world processes, it is simply more reliable to utilize non-Shannonian TEs.

7.3. Conclusions

In this paper, we discussed the Rényi transfer entropy and its role in the inference of causal relations between two systems, i.e., in the identification of the driving and driven systems from the experimental time series. On the theoretical side, our focus was on understanding the connection between RTE and Granger causality. In particular, we proved that the Granger causality is entirely equivalent to the RTE for Gaussian processes. This generalizes the classic result of Barnett et al. [61] that is valid for Shannon’s TE. Furthermore, we have also shown how the Granger causality and the RTE are related in the case of heavy-tailed (namely $\alpha$-Gaussian) processes. These results allow one to bridge the gap between autoregressive and Rényi entropy-based information-theoretic approaches.

On the experimental side, we illustrated the inner workings of the RTE by analyzing RTE between the synthetic time series generated from two unidirectionally coupled Rössler systems that are known to undergo synchronization. The route to synchronization was scrutinized by considering the effective RTE (and other derived concepts) between various master–slave components as well as between the full master and slave systems. We observed that with the effective RTE one could clearly identify a transient synchronization region (in the coupling strength), i.e., the regime between chaotic (master–slave) correlations and the synchronization threshold. In the transient region, the effective RTE allowed inferring the measure concentration for the orbit occupation density. It is noteworthy to mention that the latter cannot be deduced from Shannon’s TE alone.

We also saw that the direction of coupling and, hence, causality, could be reliably inferred only for coupling strengths $\epsilon <0.12$ (the onset of the transient regime), i.e., when two RSs were coupled, but not yet fully. This is in agreement with earlier observations, cf., e.g., reference [22]. As soon as the RSs were synchronized, they produced identical time series; hence, there is no way to infer the correct causality relation solely from the measured data.

We conclude with a general observation—a clear conceptual advantage of information-theoretic measures in general, and RTE in particular, as compared to the standard Granger causality, are sensitive to nonlinear signal properties, as they do not rely on linear regression models. On the other hand, a clear limitation of RTEs, in comparison to the Granger causality, is that they are—by their very formulation—restricted to bivariate situations (though multivariate generalization is possible, it substantially increases dimensionality in the estimation problem, which might be hard to solve with a limited amount of available data). In addition, the RTEs often require substantially more data than regression methods.