Approximate Continuous Time Measures of Information Movement in Complex Extended Networks ^†

Abstract

The differential entropy of a continuous waveform is defined over the period spanned by the recording. The time-dependent information dynamics of the observed process are not accessible to this measure. We review here the construction and validation of approximate time-dependent measures of information dynamics.

1. Introduction

The object of this investigation is to review and extend methods for approximating time-dependent information dynamics and then proceed to their use in the dynamical characterization of information movement in networks. The investigation is constructed in four steps.

• Local entropy rate and specific entropy rate: approximate continuous time measures of information generation
• Specific transfer entropy: an approximate continuous time measure of information movement
• Time-dependent measures of information movement in networks
• Hierarchical transition chronometries in information networks

2. Local Entropy Rate and Specific Entropy Rate

Consider a time series $\left\{x_1,\cdots x_N\right\}.$ Time $t_i$ is the time when $x_i$ was observed. Local entropy rate, ${H^L\left(t_i\right)=H}^L\left(x_i\right|x_{i-p}^{i-1})$, is a time-dependent, locally determined conditional entropy where the entropy of an observed variable is conditioned on its past. It quantifies the surprise on observing $x_t$ given the preceding values $x_{t-p},\cdots x_{t-1}$. ${H^L\left(t_i\right)=H}^L\left(x_i\right|x_{i-p}^{i-1})$, can be interpreted as the rate of information generation at time t [1,2] (It is a single number that is a function of p + 1 inputs.

$${H^L\left(t_i\right)=H}^L\left(x_i|x_{i-p}^{i-1}\right)=-\mathrm{l}\mathrm{o}\mathrm{g}{f}_{-i}\left(x_i\right|x_{i-p}^{i-1})$$

The essential step in the analysis is to recognize the relationship between a conditional density function, the joint distribution, and the marginal distribution. Specifically,

$${H^L\left(t_i\right)=H}^L\left(x_i|x_{i-p}^{i-1}\right)=-\mathrm{l}\mathrm{o}\mathrm{g}{f}_{-i}\left(x_i|x_{i-p}^{i-1}\right)=-\mathrm{l}\mathrm{o}\mathrm{g}\left\{\frac{f_{-i}(x_i,x_{i-p}^{i-1})}{f_{-i}\left(x_{i-p}^{i-1}\right)}\right\}$$

The subscript −i is used to indicate that the value $x_i$ is not used when evaluating the local entropy rate at time $t_i$. Density functions can be approximated by kernel density estimation [3,4]) or by k-th nearest neighbor estimation [5] Autoregressive order p can be determined by the minimization of a negative log predictive likelihood.

The local entropy rate is exquisitely sensitive to noise. This is consistent with its definition. Noise at time $t_i$ reflects information generation at time $t_i$ and is reflected in the value of the local entropy rate. Noise is usually not, however, particularly interesting. This difficulty is addressed with a specific entropy rate [6] Like the local entropy rate, the specific entropy rate is a time-dependent, locally determined measure. In this case, the specific entropy rate gives a quantitative estimate of uncertainty at time $t_i$. Specific entropy rate differs from local entropy rate in being an average of local entropy rate over a defined set. Therefore, it is intermediate to the local entropy rate, where no averaging occurs, and the total entropy rate, which is the average of all available values of the local entropy rate.

A summary is given by Gilpin et al. [7] “Local and specific entropy rates quantify complementary properties of a dynamical system. If we have a time series representing an observable, the local entropy rate (LER) represents the statistical surprise of seeing an already observed local future given specific past. It also can be thought of as the rate of information generation at a given time point. The specific entropy rate (SER) represents the statistical uncertainty in an as yet unobserved future given a specific past. The LER, in isolation, is a retrospective measure, and the SER is a prospective measure, and each yields distinct information about the behavior of the dynamical system.” The notation ${H^S\left(t_i\right)=H}^S\left(x_{i-p}^{i-1}\right)$ is used to emphasize the fact that specific entropy rate is determined by values in the past. The present, $x_i$, is not an argument of the specific entropy rate.

An operational requirement is the specification of the procedure used to calculate a specific entropy rate from a subset of local entropy rate values. An unnecessarily simplistic response would be a moving average. As an alternative, note that the set $(x_{i-p},\cdots x_{i-1})$ is a point in an ${\mathbb{R}}^p$ dimensional space. Averaging local entropy rate values from points close to $(x_{i-p},\cdots x_{i-1})$ exploits the possibility, suggested by the Takens embedding theorem [8] that these points are subject to a dynamical process similar to that in operation at point $\left(x_{i-p},\cdots x_{i-1}\right)$. Ideally, one examines the same underlying dynamics at each point in the average, and noise will be reduced in the averaging process. We refer to this as dynamical adjacency in contrast with the temporal adjacency of a moving average. The estimator for the specific entropy rate is as follows:

$${\widehat{H}}^S\left(t_i\right)=H^s\left(x_{i-p}^{i-1}\right)=\frac{1}{k^{\ast }}\sum _{j\in N\left(x_{i-p}^{i-1}\right)}{\widehat{H}}^L\left(x_j\right|x_{j-p}^{j-1})$$

$N\left(x_{i-p}^{i-1}\right)$ is the set of the indices of the $k^{\ast }$ points closest to $(x_{i-p},\cdots x_{i-1})\in {\mathbb{R}}^p$. An appropriate value of $k^{\ast }$ can be determined by a split-half minimization argument.

3. Specific Transfer Entropy

In the simplest case, consider two sources producing simultaneously measured time series. There is a long history of procedures that can be used to quantify the correlation between time series. These correlation measures do not, however, quantify information movement between the two sources. In the “Treatise Concerning Human Knowledge,” Bishop Berkeley identified prediction rather than correlation as the essential criterion for causation [9] This is consistent with Wiener’s operationalization of causation. To paraphrase Wiener [10] if measuring variable X improves the prediction of variable Y, then Y is, in this very limited operational sense of causation, causally dependent on X. Where we recognize that the relationship is not necessarily unidirectional. Granger [11] introduced a measure of causal relations based on marginal predictability that has been used extensively in econometrics. It is recognized, however, that Granger causality can fail to detect nonlinear relationships. This motivated the introduction of alternative measures, most notably transfer entropy [12] As an example, Amblard and Michel [13] give an example system that has zero Granger causality in a system that detects information movement with transfer entropy.

Y is said to transfer information to X if the uncertainty in X alone exceeds the uncertainty in X given Y. Schreiber defined transfer entropy $T_{Y\to X}$ as the difference between the marginal conditional entropy and the joint conditional entropy. The corresponding local transfer entropy is defined as

$${\widehat{T}}_{Y\to X}^L\left(t_i\right)={\widehat{H}}^C\left(x_i|x_{i-p}^{i-1}\right)-{\widehat{H}}^C\left(x_i\right|x_{i-q}^{i-1},y_{i-r}^{i-1})$$

Conditional entropies can be re-expressed

$${\widehat{T}}_{Y\to X}^L\left(t_i\right)=-\log {\widehat{f}}_{-i}\left(x_i|x_{i-p}^{i-1}\right)+\log {\widehat{f}}_{-i}\left(x_i\right|x_{i-q}^{i-1},y_{i-r}^{i-1})$$

Again, recalling that a conditional density can be expressed as the ratio of a joint density and a marginal density, the estimator of $T_{Y\to X}^L\left(t_i\right)$ is

$${\widehat{T}}_{Y\to X}^L\left(t_i\right)=-\mathrm{l}\mathrm{o}\mathrm{g}\left\{\frac{{\widehat{f}}_{-i}\left(x_i,x_{i-p}^{i-1}\right)}{{\widehat{f}}_{-i}\left(x_{i-p}^{i-1}\right)}\right\}+\mathrm{l}\mathrm{o}\mathrm{g}\left\{\frac{{\widehat{f}}_{-i}\left(x_i,x_{i-q}^{i-1},y_{i-r}^{i-1}\right)}{{\widehat{f}}_{-i}\left(x_{i-q}^{i-1},y_{i-r}^{i-1}\right)}\right\}$$

The relationship between local transfer entropy and specific transfer entropy follows the logic established by local entropy rate and specific entropy rate. Specific transfer entropy is a local, dynamically informed average of local transfer entropy [14] (Darmon and Rapp, 2017).

$${\widehat{T}}_{Y\to X}^S\left(t_i\right)=\frac{1}{k^{\ast }}\sum _{j\in N^{\ast }}{\widehat{T}}_{Y\to X}^L(t_i)$$

The Figure 1 below shows the results of a simulation using a coupled Hénon map [14] Y is independent of X, but X includes a dependence on both Y and X. The top graph shows Y as a function of time, and the second shows X as a function of time. The third graph shows ${\widehat{T}}_{Y\to X}^L(t_i)$ and correctly indicates a nonzero transfer of information from Y to X. The fourth graph shows ${\widehat{T}}_{X\to Y}^L$, and because of simulated noise, it indicates a spurious nonzero transfer of information from X to Y, though the amplitude is less than that observed for ${\widehat{T}}_{Y\to X}^L$. The fifth and sixth graphs show specific transfer entropy. The fifth graph shows the expected nonzero value for ${\widehat{T}}_{Y\to X}^S$. In contrast to ${\widehat{T}}_{X\to Y}^L$, ${\widehat{T}}_{X\to Y}^S$ is effectively zero.

4. Time-Dependent Measures of Information Movement in Networks

Networks consist of nodes and edges. Each node is not necessarily connected to all other nodes. When applied to the analysis of multichannel EEGs and event-related potentials, which is our primary interest area, the electrodes are nodes and edges are quantified by functional connectivity or information transfer. These between node relationships are summarized in the adjacency matrix.

In the simplest case, the relationship between two nodes is quantified by the correlation between the signals measured at the corresponding electrodes. A value of zero is entered in the matrix if the correlation is less than a threshold value, and a value of one is entered if the value is greater than the threshold. The resulting adjacency matrix is binary, symmetric, and static. If the correlation itself is entered, the adjacency matrix is real, symmetric, and static. Alternatively, transfer entropy can be used. As defined, transfer entropy, in contrast with specific transfer entropy, is not time-dependent, but the information movement from Node A to Node B is typically not the same as the information movement from Node B to Node A. Therefore, in this case the adjacency matrix is real, asymmetric, and static. If specific transfer entropy is used, the time-dependent behavior of information movement in a complex network is summarized in a matrix that is real, asymmetric, and time-dependent.

In principle, any measure that is applicable to a real, asymmetric, static adjacency matrix can be used iteratively at each time point to produce a time-dependent measure of matrix dynamics which we denote by $\mathsf{\Lambda }\left(t\right)$. In many applications, particularly in the analysis of biological systems, the detection of transitions in network behavior is particularly interesting. This can now be implemented by an examination of the transition behavior of $\mathsf{\Lambda }\left(t\right)$. This is addressed in the fourth step of the project.

5. Hierarchical Transition Chronometries in Information Networks

There is a large well developed mathematical technology for identifying transitions in a time series, but care must be exercised when using these methods. As previously argued, “the typically stated challenge, ‘find transition times in the time series’ is unacceptably naïve. Transitions are timescale dependent. A stock market change that is significant to a day trader is not necessarily important to a pension fund manager. The challenge to find a time series’ transitions must be revised: ‘find the transitions in time series X that are important on a timescale of Y’ [15].

One possible approach to meeting this requirement uses the quantification of recurrence diagrams. Recurrence diagrams were introduced by [16] with the specific objective of identifying subtle time-dependent transitions in dynamical systems. Recurrence diagrams provide a graphical representation of geometrical relationships between points in an embedded set. Quadrant scans [17] quantify this visually presented structure. A change in the system’s dynamics generating the time series produces a local maximum in the quadrant scan. A change in the embedding dimension changes the time scale of the transitions identified in the recurrence diagram, and there is a corresponding change in the time scale of the quadrant scan. The utility of this method in identifying hierarchical transition chronometries has yet to be explored.

6. Conclusions

We have presented a four-step analysis trajectory that begins with the measurement of a multichannel time series, proceeds with the construction of a time-dependent network, and concludes with the identification of transitions in the behavior of that network. Applications in the investigation of central nervous system psychopathology are now underway.