Symbolic time series analysis is a useful tool for modeling and characterization of nonlinear dynamical systems. It provides a rigorous way of looking at “real” dynamics with finite precision [8,9]. Briefly, it is a way of coarse-graining or simplifying the description.

The basic idea is quite simple. One divides the phase space into a finite number of classes and labels each class with a symbol (e.g., a letter from some alphabet). Instead of representing the trajectories by sequences of (typically real-valued) numbers (iterations of a discrete map or sampled points along the trajectories of a continuous flow), one watches the alteration of symbols. Of course, in doing so, one loses a considerable amount of detailed information, but some of the invariant, robust properties of the dynamics are typically kept, such as periodicity, symmetry or the chaotic nature of an orbit [8].

In the framework of symbolic dynamics, time series are transformed into a series of symbols by using an appropriate partition, which results in relatively few symbols. After symbolization, the next step is the construction of “symbol sequences” (“words” in the language of symbolic dynamics) from the symbol series by collecting groups of symbols together in temporal order.

To be more precise, the simplest possible coarse-graining of a time series is given by choosing a single threshold (usually the mean value or median of the data considered [10]), and assigning the symbols “1” and “0” to the signal, depending on whether it is above or below the threshold (binary partition) (The generalization to obtaining a classification using more than one threshold is trivial. Moreover, in addition to this kind of “static” encoding, there are other possibilities for obtaining a “dynamic” symbolization of observed time series data, such as using order patterns (cf. Section 2.1.4) or even mixed strategies [11]). Thus, we generate a symbolic time series from a two-letter ( λ = 2 ) alphabet (0,1), e.g., 0110100110010110 … . We can read this symbolic sequence in terms of distinct consecutive blocks of a length of n. For example, using n = 2 , one obtains 01 / 10 / 10 / 01 / 10 / 01 / 01 / 10 / … . This reading procedure is called “lumping”.

The number of all possible kinds of words is λ n = 2 2 = 4 , namely 00, 01, 10, 11. The required probabilities for the estimation of an entropy, p 00 , p 01 , p 10 , p 11 , are the fractions of the different blocks (words), 00, 01, 10, 11, in the symbolic time series, namely, 0, 4/8, 4/8 and 0, correspondingly for the specific example. Based on these probabilities, we can estimate, for example, the probabilistic entropy measure, H S , introduced by Shannon [12]: (1) H S = - ∑ j = 1 λ n p j log 2 p j where p j are the probabilities associated with the microscopic configurations. The binary logarithm is taken here for convenience in the case of a binary symbolization (giving rise to an interpretation of the Shannon entropy as the negative mean amount of information, measured in bits), but may be replaced by another basis (e.g., using log λ ) without loss of generality.

In the case of a real-world time series, “experimental” conditions are often rather different (e.g., in terms of the system’s energy content), which can lead to ambiguities when quantitatively comparing entropy estimates, e.g., obtained from different persons in clinical studies. In order to correct for this problem, the renormalized entropy [5,13] has been introduced as the difference between the Shannon entropies obtained for the time series under study and some reference data.

A generalization of the Shannon entropy is provided by the spectrum of Rényi entropies [14]: (2) H q = 1 1 - q log 2 ∑ j = 1 λ n p j q with q ∈ R , which give different weights to rare and frequent symbols. In this spirit, Rényi entropies allow for the studying of scaling properties associated with a non-uniform probability distribution of symbols or words as arising, e.g., in the case of multi-fractal processes. Notably, the Shannon entropy, H S , is a special case of H q for q → 1 . In a similar way, for q → 0 , H q is the Hartley entropy (or Hartley information), one of the most classical information-theoretic measures of uncertainty [15]. Note that for the case of a uniform probability distribution of n-blocks, all Rényi entropies coincide with each other and, hence, the Shannon and Hartley entropies.

In general, one can statistically analyze a symbolic time series of N symbols, { A i } , i = 1 , 2 , . . . , N , in terms of the frequency distribution of blocks of a length of n n < N . For an alphabet consisting of λ letters, considering a word of a length of n, there are λ n possible combinations of the symbols that may constitute a word, here, λ n = 2 n . The probability of the occurrence, p j ( n ) , of the j-th combination of symbols ( j = 1 , 2 , . . . , 2 n ) (i.e., the j-th n-word) can be approximated by the number of occurrences of this n-word in the considered symbolic sequence divided by the total number of n-words.

Based on these probabilities, one can estimate the probabilistic entropy or other related characteristics. Various tools of information theory and entropy concepts can be used to identify statistical patterns in the symbolic sequences, onto which the dynamics of the original system under analysis has been projected. For detection of an anomaly, it suffices that a detectable change in the pattern represents a deviation of the system from nominal behavior [16]. Recently published work has reported novel methods for detection of anomalies in complex dynamical systems, which rely on symbolic time series analysis, e.g., [17,18]. Entropies depending on the word-frequency distribution in symbolic sequences are of special interest, extending Shannon’s classical definition of the entropy and providing a link between dynamical systems and information theory. These entropies take a large/small value if there are many/few kinds of patterns, i.e., they decrease while the organization of patterns is increasing. In this way, these entropies can measure the complexity of a signal.

Extending Shannon’s classical definition of the entropy of a single state [12] to the entropy of a succession of states [19,20], one reaches the definition of the n-block entropy, H ( n ) , which, for a two-letter ( λ = 2 ) alphabet and word length n, is given by: (3) H n = - ∑ j = 1 2 n p j ( n ) log 2 p j ( n ) H ( n ) is a measure of uncertainty and gives the average amount of information necessary to predict a subsequence of length n. Consequently, H ( n ) / n can be interpreted as the mean uncertainty per symbol and should converge for n → ∞ to some stationary value if the observed dynamics is deterministic. Moreover, from a practical perspective, one is often interested in quantifying the mean information gain when increasing the word length, measured by the conditional (or differential) block entropies: (4) h ( n ) = H ( n + 1 ) - H ( n ) f o r n ≥ 1 ; h ( 0 ) = H ( 1 ) For stationary and ergodic processes, the limit of h ( n ) for n → ∞ provides an estimator of the Kolmogorov-Sinai entropy or source entropy h of the dynamical system under study [21,22] (In a similar spirit, H ( n ) / n provides another estimate of the source entropy for n → ∞ .). For a broad class of systems, h ( n ) converges exponentially fast, with the characteristic scaling parameter being related to the third-order Rényi entropy, H 3 . In general, exploiting the corresponding convergence properties gives rise to a widely applicable probabilistic complexity measure, the effective measure complexity [23]: (5) E M C = ∑ n = 0 ∞ [ h ( n ) - h ]

A more extensive discussion and classification of complexity measures based on symbolic dynamics concepts has been given elsewhere [24].

It has been established that physical systems characterized by long-range interactions or long-term memory or being of a multi-fractal nature are best described by a generalized statistical-mechanical formalism proposed by Tsallis [1,25,26]. More precisely, inspired by multi-fractal concepts, Tsallis introduced an entropic expression characterized by an index, q, which leads to non-extensive statistics [25,26]: (6) S q = k 1 q - 1 1 - ∑ i = 1 W p i q where p i are probabilities associated with the microscopic configurations (i.e., the empirical frequency distribution of symbols), W is their total number, q is a real number and k is a constant, i.e., the Boltzmann’s constant from statistical thermodynamics. Notably, there is a striking conceptual similarity between Tsallis’ entropy definition and the notion of Rényi entropies.

The entropic index, q, describes the deviation of Tsallis entropy from the standard Boltzmann-Gibbs entropy. Indeed, using p i ( q - 1 ) = e ( q - 1 ) ln ( p i ) ∼ 1 + ( q - 1 ) ln ( p i ) in the limit q → 1 , we recover the usual Boltzmann-Gibbs entropy: (7) S 1 = - k ∑ i = 1 W p i ln ( p i ) as the thermodynamic analog of the information-theoretic Shannon entropy.

For q ≠ 1 , the entropic index, q, characterizes the degree of non-extensivity reflected in the following pseudo-additivity rule: (8) S q A + B k = S q A k + S q B k + q - 1 S q A k S q B k where A and B are two subsystems. If these subsystems have special probability correlations, extensivity does not hold for q = 1 ( S 1 ≠ S 1 ( A ) + S 1 ( B ) ), but may occur for S q , with a particular value of the index, q ≠ 1 . Such systems are called non-extensive [1,25]. The cases q > 1 and q < 1 correspond to sub-additivity or super-additivity, respectively. As in the case of Rényi entropies, we may think of q as a bias parameter: q < 1 privileges rare events, while q > 1 highlights prominent events [27].

We note that the parameter, q, itself is not a measure of the complexity of the system, but measures the degree of the non-extensivity of the system. In turn, the time variations of the Tsallis entropy, S q , for some q quantify the dynamic changes of the complexity of the system. Specifically, lower S q values characterize the portions of the signal with lower complexity.

In terms of symbolic dynamics, the Tsallis entropy for a two-letter ( λ = 2 ) alphabet and word length n reads [18,28,29]: (9) S q n = k 1 q - 1 1 - ∑ j = 1 2 n p j ( n ) q Broad symbol-sequence frequency distributions produce high entropy values, indicating a low degree of organization. Conversely, when certain sequences exhibit high frequencies, low values of S q ( n ) are produced, indicating a high degree of organization.

In the symbolic dynamics framework introduced above, the transformation from a continuous to a discrete-valued time series (necessary for estimating the Shannon entropy of a discrete distribution or related characteristics) has been realized by coarse-graining the range of the respective observable. This procedure has the advantage of being algorithmically simple. However, unless block lengths n ≥ 2 are used, the resulting entropies follow directly from the specific discretization and, thus, exclusively reflect statistical, but not dynamical, characteristics. In turn, dynamics comes into play when considering blocks of symbols.

An alternative way of introducing a symbolization in an explicitly dynamic way is making use of order patterns and related concepts from ordinal time series analysis [30]. In the simplest case (i.e., a pattern of order two), one considers a binary discretization of the underlying time series of a continuous random variable according to the sign of the difference between two subsequent values, e.g., using the symbols “0” if s i + 1 - s i < 0 and “1” if s i + 1 - s i > 0 (disregarding the “unlikely” case s i + 1 = s i ). Notably, this kind of symbolization corresponds to the classical symbolic dynamics approach applied to the time series of increments (i.e., a first-order difference filter applied to the original data) with the increment value of zero discriminating the two classes of the resulting binary encoding. Based on this discretization, one can again define all entropies described above.

Besides using binary encodings based on order patterns of order two, the above framework can be extended to patterns of a higher order. In this case, sequences of observations are transformed into ordinal sequences (e.g., the case s i < s i + 1 < s i + 2 would correspond to the sequence ( 1 , 2 , 3 ) ). Obviously, for a pattern of order q, there are q ! possible permutations of ( 1 , ⋯ , q ) and, hence, at most q ! different patterns (Depending on the type of observed dynamics, there might be forbidden patterns. We do not further discuss the implications of this fact here.). Using the frequency distribution of the possible order patterns for calculating a Shannon-type entropy leads to the notion of permutation entropy [31,32,33].

Besides its direct application as a proxy for dynamical disorder based on time series data, Rosso et al. [34,35] suggested utilizing permutation entropy in combination with some associated measure of complexity for defining the so-called complexity-entropy causality plane. Here, the entropic quantity is the (Shannon-type) permutation entropy normalized by its maximum value, log 2 q ! : (10) H S * [ P π ] = - 1 log 2 q ! ∑ i = 1 q ! p ( π i ) log 2 p ( π i ) where π i denote the different possible permutations of ( 1 , ⋯ , q ) . As one possible complexity measure associated with the permutation entropy, the Jensen-Shannon complexity, C J S [ P π ] , is defined as [36]: (11) C J S [ P π ] = Q J [ P π , P e ] H S * [ P π ] with the Jensen-Shannon divergence: (12) Q J [ P π , P e ] = Q 0 log 2 q ! H S * [ ( P π + P e ) / 2 ] - H S * [ P π ] 2 - 1 2 between the observed distribution, P π , and the uniform distribution P e = ( 1 / q ! , ⋯ , 1 / q ! ) , where Q 0 is a normalization constant, such that 0 ≤ Q J ≤ 1 .

Approximate entropy ( A p E n ) has been introduced by Pincus [37] as a measure for characterizing the regularity in relatively short and potentially noisy data. Specifically, A p E n quantifies the degree of irregularity or randomness within a time series (of length N) and has already been widely applied to (non-stationary) biological systems for dynamically monitoring the system’s “health” status (see [38,39] and references therein). The conceptual idea is rooted in the work of Grassberger and Procaccia [40] and makes use of distances between sequences of successive observations. More specifically, A p E n examines time series for detecting the presence of similar epochs; more similar and more frequent epochs lead to lower values of A p E n .

From a qualitative point of view, given N points, A p E n is approximately equal to the negative logarithm of the conditional probability that two sequences that are similar for m points remain similar, that is, within a tolerance, r, at the next point. Smaller A p E n values indicate a greater chance that a set of data will be followed by similar data (regularity). Thus, smaller values indicate greater regularity. Conversely, a larger value of A p E n points to a lower chance of similar data being repeated (irregularity). Hence, larger values convey more disorder, randomness and system complexity. Consequently, a low/high value of A p E n reflects a high/low degree of regularity. Notably, A p E n detects changes in underlying episodic behavior not reflected in peak occurrences or amplitudes [41].

In the following, we give a brief description of the calculation of A p E n . A more comprehensive discussion can be found in [37,38,39].

For a time series, { s k } k with s k = s t k , t k = k T , k = 1 , 2 , . . . , N , and T being the sampling period, we can define N - m + 1 vectors, each one consisting of m consecutive samples of this time series as: (13) X i m = s i , s i + 1 , . . . , s i + m - 1 , i = 1 , 2 , . . . , N - m + 1 The main idea is to consider a window of length m running through the time series and forming the corresponding vectors, X i m . The similarity between the formed vectors is used as a measure of the degree of organization of the time series. A quantitative measure of this similarity, C i m r , is given by the average number of vectors, X j m , within a distance, r, from X i m . Here, X j m is considered to be within a distance r from X i m if d i j m ≤ r , where d i j m is the maximum absolute difference of the corresponding scalar components of X i m and X j m (i.e., the two vectors have a distance smaller than r according to their supremum norm). By calculating C i m r for each i ≤ N - m + 1 and then taking the mean value of the corresponding natural logarithms: (14) ϕ m r = N - m + 1 - 1 ∑ i = 1 N - m + 1 ln C i m r the A p E n is defined as: (15) A p E n m , r = lim N → ∞ [ ϕ m r - ϕ m + 1 r ] which, for a finite time series, can be estimated by the statistic: (16) A p E n m , r , N = ϕ m r - ϕ m + 1 r By tuning r, one can reach a reasonable degree of similarity for “most” vectors, X i m , and, hence, a reliable statistics, even for relatively short time series.

In summary, the presence of repetitive patterns of fluctuation in a time series renders it more predictable than a time series in which such patterns are absent. A time series containing many repetitive patterns has a relatively small A p E n ; a less predictable (i.e., more complex) process has a higher A p E n .

Sample entropy ( S a m p E n ) was proposed by Richman and Moorman [42] as an alternative that would provide an improvement of the intrinsic bias of A p E n [43]. Specifically, when calculating the similarity within a time series using the vectors defined by Equation (13), A p E n does not exclude self-matches, since the employed measure of similarity, C i m r , which is proportional to the number of vectors within a distance, r, from X i m , does not exclude the X i m itself from this count. S a m p E n is considered to be an evolution of A p E n that displays relative consistency and less dependence on data length (see, e.g., [44] and the references therein).

First of all, we consider only the first N - m vectors of a length of m of Equation (13), ensuring that, for 1 ≤ i ≤ N - m , X i m and X i m + 1 are defined. Then, we define B i m r as N - m - 1 - 1 times the number of vectors, X j m , within a distance, r, from X i m , where j = 1 , 2 , . . . , N - m , but also j ≠ i , excluding in this way the self-matches, and set: (17) B m r = N - m - 1 ∑ i = 1 N - m B i m r Correspondingly, we define A i m r as N - m - 1 - 1 times the number of vectors, X j m + 1 , within a distance, r, from X i m + 1 , where j = 1 , 2 , . . . , N - m with j ≠ i , and: (18) A m r = N - m - 1 ∑ i = 1 N - m A i m r The S a m p E n m , r is then defined as: (19) S a m p E n m , r = lim N → ∞ - ln A m r / B m r which, for finite time series, can be estimated by the statistic: (20) S a m p E n m , r , N = - ln A m r B m r

Fuzzy entropy ( F u z z y E n ) [43,44], like its ancestors, A p E n and S a m p l e E n [44], is a “regularity statistic” that quantifies the (un)predictability of fluctuations in a time series. For the calculation of F u z z y E n , the similarity between vectors is defined based on fuzzy membership functions and the vectors’ shapes. The gradual and continuous boundaries of the fuzzy membership functions lead to a series of advantages, like the continuity, as well as the validity of F u z z y E n at small values, higher accuracy, stronger relative consistency and even less dependence on the data length. F u z z y E n can be considered as an upgraded alternative of S a m p E n (and A p E n ) for the evaluation of complexity, especially for short time series contaminated by noise.

Like S a m p E n , F u z z y E n excludes self-matches. However, it follows a slightly different definition for the employed first N - m vectors of a length of m, by removing a baseline, s ¯ i : (21) s ¯ i = m - 1 ∑ j = 0 m - 1 s i + j i.e., for the F u z z y E n calculations, we use the first N - m of the vectors: (22) X i m = s i , s i + 1 , . . . , s i + m - 1 - s ¯ i , i = 1 , 2 , . . . , N - m + 1 Then, the similarity degree, D i j m , between each pair of vectors, X j m and X i m , being within a distance, r, from each other, is defined by a fuzzy membership function: (23) D i j m = μ d i j m , r where d i j m is, as in the case of A p E n and S a m p E n , the supremum norm difference between X i m and X j m . For each vector, X i m , we calculate the average similarity degrees with respect to all other vectors, X j m , j = 1 , 2 , . . . , N - m + 1 , and j ≠ i (i.e., excluding itself): (24) ϕ i m r = N - m - 1 - 1 ∑ i = 1 , j ≠ i N - m D i j m Then, we evaluate: (25) φ m r = N - m - 1 ∑ i = 1 N - m ϕ i m r and: (26) φ m + 1 r = N - m - 1 ∑ i = 1 N - m ϕ i m + 1 r The F u z z y E n m , r is then defined as: (27) F u z z y E n m , r = lim N → ∞ ln φ m r - ln φ m + 1 r which, for finite time series, can be estimated by the statistic: (28) F u z z y E n m , r , N = ln φ m r - ln φ m + 1 r We note that by making use of vectors of different lengths, A p E n , S a m p E n and F u z z y E n are based on a similar rationale as the conditional block entropies (i.e., changes in the information content as the sequence length increases).

While Shannon entropy is a measure of the uncertainty about outcomes of one process, mutual information (MI) is a measure of the reduction thereof if another process is known (There are several other, conceptually-related symmetric measures for the strength of interdependencies between two variables, e.g., the maximal information coefficient [45]. For brevity, we do not further discuss these alternative measures here.). The Shannon-type MI can be expressed as: (29) I ( X ; Y ) = H S ( Y ) - H S ( Y | X ) = H S ( X ) - H S ( X | Y ) (30) = H S ( X ) + H S ( Y ) - H S ( X , Y ) i.e., as the difference between the uncertainty in Y and the remaining uncertainty if X is already known (and vice versa). MI is symmetric in its arguments, non-negative and zero, if and only if X and Y are independent. The lagged cross-MI for two time series is given by: (31) I X Y ( τ ) ≡ I ( X t - τ ; Y t ) For τ > 0 , one measures the information in the past of X that is contained in Y, and vice versa for τ < 0 . In analogy, the auto-MI is defined as I ( Y t - τ ; Y t ) for τ > 0 (for τ = 0 , this gives the classical Shannon entropy, H S ( Y ) ).

Just like the entropy of a time series can be estimated based on symbols or patterns, MI can be estimated by simply plugging in the corresponding marginal and joint entropies into Equation (30). In contrast to pattern-based estimators, for binning estimators (i.e., estimators using frequency distributions based on a bivariate symbolization with λ ≫ 2 symbols for each time series, but typically considering individual symbols, i.e., n = 1 ), an adaptive strategy is preferable in which the bins are chosen, such that the marginal distributions are uniform [46,47,48]. For continuous time series, a useful class of estimators is based on nearest-neighbor statistics (see Section 2.4.1). For example, making use of the entropy estimators developed in [49], Kraskov et al. introduced a nearest-neighbor estimator of MI [47].

In the limiting case of a bivariate Gaussian distribution, the MI is simply given by: (32) I Gauss ( X ; Y ) = - 1 2 ln 1 - ρ ( X ; Y ) 2 where ρ ( X ; Y ) is the linear Pearson correlation coefficient.

There are several straightforward generalizations of the mutual information concept. On the one hand, one can characterize nonlinear bivariate interrelationships between two variables by a spectrum of generalized mutual information functions obtained by replacing the Shannon entropy, H S , by Rényi entropies, H q , of arbitrary order q [50] (Note that this replacement does not conserve the non-negativity property of the classical MI function [50].). On the other hand, the idea behind mutual information can be directly extended to sets of K > 2 variables, X 1 , ⋯ , X K , leading to the so-called (generalized) redundancies [51,52,53]: (33) I q ( X 1 ; ⋯ ; X K ) = ∑ k = 1 K H q ( X k ) - H q ( X 1 , ⋯ , X K )

An important generalization of MI is the conditional mutual information (CMI): (34) I ( X ; Y | Z ) = H S ( Y | Z ) - H S ( Y | X , Z ) = H S ( X | Z ) - H S ( X | Y , Z ) (35) = H S ( X , Z ) + H S ( Y , Z ) - H S ( Z ) - H S ( X , Y , Z ) which measures the mutual information between X and Y that is not contained in a third variable, Z, which can also be multivariate. CMI shares the properties of MI and is zero, if and only if X and Y are mutually independent conditionally on Z. Moreover, it is possible to use any of the existing estimators of entropic characteristics (including binning, order patterns or k-nearest neighbors) for defining CMI and generalized versions thereof. Like in Equation (32), the CMI for multivariate Gaussian processes can be expressed in terms of the partial correlation, where the correlation, ρ ( X ; Y ) , is replaced by ρ ( X ; Y | Z ) .

Of particular interest is the use of information-theoretic measures to quantify causal relations between variables based on time series data. A first step is to assess the directionality of information transfer between two time series. While lagged MI can be used to quantify whether information in Y has already been present in the past of X, this information could have existed in the common past of both processes and, therefore, is not a sign of a transfer of unique information in X to Y. To overcome this limitation, transfer entropy (TE) [54] measures the information in the past of X that is shared in the present of Y and not already contained in the past of Y: (36) I X → Y TE ≡ I ( X t - ; Y t | Y t - ) with the upper script “ - ” denoting the (infinite) past vector, e.g., X t - = ( X t - 1 , X t - 2 , … ) . It should be noted that the notion of causality employed here is in the predictive sense of Granger causality, which was introduced for linear processes in [55] and later generalized to nonlinear frameworks [56,57]. TE and Granger causality can actually be shown to be equivalent for Gaussian variables [58].

The definition of TE leads to the problem that infinite-dimensional densities have to be estimated, which is commonly called the “curse of dimensionality”. In the usual naive estimation of TE, the infinite vectors are simply truncated at some τ max , which has to be chosen at least as large as the supposed maximal coupling delay between X and Y. This can lead to very large estimation dimensions, since the delay is not known a priori. As shown in [59,60], a high estimation dimension severely affects the reliability of the inference of directionality. In [59], the problem of high dimensionality is overcome by utilizing the concept of graphical models.

While the TE discussed before is a bivariate measure of directionality, a more comprehensive causality assessment requires the inclusion of multiple variables to be able to exclude common drivers and indirect causalities. In the graphical model approach [61,62,63], the conditional independence properties of a multivariate process are visualized in a graph; in our case, a time series graph. As depicted in Figure 1a, each node in that graph represents a subprocess of a multivariate discrete-time process, X , at a certain time, t. Nodes X t - τ and Y t are connected by a directed link “ X t - τ → Y t ”, if and only if τ > 0 and: (37) I ( X t - τ ; Y t | X t - ∖ { X t - τ } ) > 0 i.e., if they are not independent conditionally on the past of the whole process, which implies a lag-specific causality with respect to X . If Y ≠ X , we say that the link “ X t - τ → Y t ” represents a coupling at lag τ, while for Y = X , it represents an auto-dependency at lag τ. Nodes X t and Y t are connected by an undirected contemporaneous link “ X t - Y t ” (visualized by a line, cf. in Figure 1a) [63], if and only if: (38) I ( X t ; Y t | X t + 1 - ∖ { X t , Y t } ) > 0 where also the contemporaneous present X t ∖ { X t , Y t } is included in the condition. Note that stationarity implies that “ X t - τ → Y t ” whenever “ X t ′ - τ → Y t ′ ” for any t ′ . In the resulting graph, the parents and neighbors of a node Y t are defined as: (39) P Y t ≡ { Z t - τ : Z ∈ X , τ > 0 , Z t - τ → Y t } (40) N Y t ≡ { X t : X ∈ X , X t - Y t } In [59], an algorithm for the estimation of time series graphs by iteratively inferring the parents is introduced. The supplementary material of [59] also provides a suitable shuffle test and a numerical study on the detection and false positive rates of the algorithm.

Time series graphs encode the existence of a lag-specific causality, but they are not meant to assess the causal strength in a meaningful way. This topic is discussed in [64], where the concept of momentary information transfer (MIT) [65] is utilized for attributing a well-interpretable coupling strength solely to the inferred links of the time series graph. MIT between X at some lagged time, t - τ , in the past and Y at time t is the CMI that measures the part of the source entropy of Y that is shared with the source entropy of X: (41) I X → Y MIT ( τ ) ≡ I ( X t - τ ; Y t | P Y t ∖ { X t - τ } , P X t - τ ) The attribute momentary is used, because MIT measures the information of the “moment” t - τ in X that is transferred to Y t . Similar to the definition of contemporaneous links in Equation (38), we can also define a contemporaneous MIT: (42) I X - Y MIT ≡ I ( X t ; Y t | P Y t , P X t , N X t ∖ { Y t } , N Y t ∖ { X t } ) MIT is well interpretable, because it excludes past information, not only from Y, but also from X, i.e., information transmitted by serial correlations. Often X and Y are entangled by multiple links, and only the exclusion of both pasts truly yields a coupling measure that depends solely on the mutual coupling strength. This is demonstrated analytically and numerically in [64].

In their most common formulation, many of the aforementioned characteristics are explicitly based on a discretization of the system under study (either via coarse-graining and subsequent symbolization or using order patterns or related approaches), giving rise to an information-theoretic interpretation of entropies and related complexity measures. Consequently, these measures are often considered as providing one major field of nonlinear time series analysis methods, which is opposed to another class of well-developed theoretical frameworks making use of the notion of the phase space of a dynamical system. In the latter case, one commonly takes sets of variables (or time-lagged replications of one variable) into account, which are assumed to span a metric space in which the system’s behavior can be fully described by means of distances between sets of “state vectors” sampled from the observed trajectory.

Reconsidering the entropic measures described above in more detail, one recognizes that there are also multiple quantifiers that make explicit use of distances (i.e., approximate, sample and fuzzy entropy), which implies that the latter methods can also be considered as phase space-based approaches if one identifies subsequent observations as individual “coordinates” of some abstract space (i.e., a “naive” embedding space of variable dimension). In fact, as already stated above, the mentioned class of entropy characteristics is based on ideas formulated by Grassberger and Procaccia about 30 years ago, which are among the pioneering works utilizing the concept of phase space for nonlinear time series analysis.

In a similar spirit, state-of-the-art estimators of mutual information and causality concepts based on this framework make use of the phase space concept by replacing the traditional histogram-based symbolizations or order patterns by nearest-neighbor relationships between sampled state vectors. Specifically, k-nearest neighbor estimators of entropies [49], MI [47] and CMI [66] have been found practically useful and have desirable numerical properties not shared by other estimators. Here, we just briefly illustrate the basic idea of k-nearest neighbor estimates for CMI.

As the first step, the number of neighbors, k, to be considered around each sampled state vector in the joint space of ( X , Y , Z ) is chosen as a free parameter. For each sample corresponding to observations at time t, the supremum norm distance, ε t , to the k-th nearest neighbor defines a cube of a length of 2 ε t around the corresponding state vector. Next, the numbers of points in the respective subspaces with a distance less than ϵ t are counted, yielding k z , t , k x z , t and k y z , t . Then, the CMI is estimated as: (43) I ^ ( X ; Y | Z ) = ψ ( k ) + 1 T ∑ t = 1 T ψ ( k z , t ) - ψ ( k x z , t ) - ψ ( k y z , t ) where ψ is the Digamma function. Smaller k result in smaller cubes, and since (as assumed in the estimator’s derivation) the density is approximately constant in these cubes, the estimator has a low bias. Conversely, for large k, the bias is stronger, but the variance gets smaller. Note, however, that for independent processes, the bias is approximately zero, i.e., I ^ ( X ; Y | Z ) ≈ 0 ; large k are therefore better suited for independence tests, as needed in the algorithm to infer the time series graph.

Besides the above-mentioned examples, entropic quantities frequently occur in various phase space-based methods of nonlinear time series analysis. One prominent example is the Grassberger-Procaccia algorithm for estimating the correlation dimension of chaotic attractors [67,68]. As in the calculation of A p E n , S a m p E n and F u z z y E n , let us consider the m-dimensional delay vectors, X i m , where the embedding dimension, m, and the mutual temporal offset between the components of X i m have been optimized, so as to yield coordinates, which are as independent as possible and represent the “true” dimensionality of the recorded system appropriately. Computing the pairwise distance of the resulting delay vectors, we may distinguish between mutually “close” and “distant” vectors by comparing the distance with some predefined global threshold value, r, which is summarized in the recurrence matrix [69]: (44) R i j ( r ) = Θ ( r - ∥ X i m - X j m ∥ ) where Θ ( · ) is the Heaviside “function”. The fraction of mutually close pairs of vectors: (45) C m ( r ) = 2 N ′ ( N ′ - 1 ) ∑ i < j R i j ( r ) (with N ′ = N - m + 1 ) is called the correlation sum or recurrence rate of the observed trajectory. For a dissipative chaotic system with attractor dimension < m , the latter obeys a characteristic scaling behavior as: (46) C m ( r ) ∼ r D 2 exp ( - m H 2 Δ t ) with Δ t denoting the sampling interval of the time series data and D 2 the attractor’s correlation dimension. By varying r and m, it is thus possible to estimate the second-order Rényi entropy, H 2 , together with the correlation dimension from the correlation sum.

Alternatively, other properties of the recurrence matrix may be exploited for estimating different entropic quantities. Among others, the length distribution, p ( l ) , of “diagonal lines” formed by non-zero entries in R provides a versatile tool for estimating H 2 from time series data [70]. A similar approach can be used for estimating the generalized MI of second order. The Shannon entropy, E N T R , of the discrete distribution, p ( l ) , has been used as a heuristic complexity measure in the framework of the so-called recurrence quantification analysis [69]. However, it has been shown that this definition does not allow for properly distinguishing between different “degrees” of chaotic dynamics in the desired way. In turn, the Shannon entropy associated with the length distribution, p ( l ¯ ) , of diagonal lines in R formed exclusively by zero entries provides a measure of dynamical disorder that increases with the complexity of chaotic dynamics as measured by the system’s largest Lyapunov exponent [71]. Even more, the latter quantity shows an excellent agreement with the classical Shannon entropy estimate based on symbolic dynamics, but does not share some of the disadvantages, due to the coarse-graining, which arise in the latter case.

It is worth noting that, recently, there have been the first successful attempts to transfer the idea of “recurrences” in phase space to symbolic data [11,72,73]. A special case is order pattern recurrence plots [74,75], which have been successfully applied in the analysis of neurophysiological time series.

For the sake of completeness, we briefly mention that similar to the characterization of dynamical complexity with entropic characteristics, also the complementary bi- and multi-variate problem of detecting causal interrelationships can be complementarily tackled by information-theoretic and phase space methods. Distance-based characteristics allowing one to reliably detect directional couplings between different variables have been developed and applied by various authors, e.g., [76,77,78,79,80,81,82]. A specific class of approaches is based on the recurrence matrix, making use of conditional recurrence probabilities [83,84] or asymmetries in coupled network statistics [85]. Since these methods are not within the focus of this review, we refer to the aforementioned references for further details.

The quantitative characterization of dynamical complexity in today’s Earth climate has been the subject of many studies. Notably, entropic characteristics and associated measures of complexity are particularly well suited for this purpose. For example, Paluš and Novotná [167] demonstrated the use of redundancies (i.e., multivariate generalizations of MI), in combination with surrogate data preserving the linear correlation properties of the original time series, for testing for the nonlinearity of different climatic variables (surface air pressure and temperature, geopotential height data), a necessary prerequisite for the emergence of chaotic dynamics.

On the global scale, von Bloh et al. [168] studied surface air temperature observations, as well as the corresponding outcomes of a global circulation model. In order to reveal the spatial characteristics associated with the nonlinear dynamics of the climate system, they considered the spatial patterns of the second-order Rényi entropy, H 2 , estimated from recurrence plots (A similar analysis has been recently presented by Paluš et al. [169] in terms of a Gaussian process entropy rate computed for globally distributed surface air temperature anomalies). Their results suggested that state-of-the-art climate models (at the time of this study) have not been able to trace the fundamental spatial signatures of nonlinearity in observed climate dynamics, such as marked latitudinal, as well as land-ocean entropy gradients. The latter finding can be considered a well-established fact, which has been described for other nonlinear characteristics, such as the Hurst exponent describing the degree of long-range dependence in temperature fluctuations (see, e.g., [170,171,172,173]).

One reason for the observed complexity of temperature fluctuations is that the climate exhibits relevant variability on a multitude of different time-scales, from diurnal to seasonal and inter-annual scales. The longer time-scales are particularly highlighted by nonlinear large-scale oscillatory (dipole) patterns, such as the El Niño/Southern Oscillation (ENSO). Investigating the spatial profile and temporal variability of such patterns and characterizing their degree of dynamical disorder can provide a vast amount of complementary information not provided by classical linear analysis techniques. Among other entropic characteristics, Tsallis’ non-extensive entropy has been found particularly useful for this purpose, once more making use of the fact that the climate system is typically far from equilibrium.

Ausloos and Petroni [174] studied the normalized variability of the southern oscillation index (SOI) on different time-scales. Their results indicated a qualitative change in the non-extensivity as the considered scales vary, leading to a classical Boltzmann-Gibbs scaling ( q → 1 ) at larger scales. Beyond exclusive consideration of S q , in the non-extensive statistical mechanics framework, characteristic scaling exponents can also be found for other quantities, which is formalized by Tsallis’ q-triplet [175], which has been obtained for daily ENSO fluctuations [176], but also other geoscientific variables, such as the depth of the stratospheric ozone layer [177], various meteorological variables, earthquake inter-occurrence times, magnetospheric dynamics, etc. [178].

In addition to non-extensive entropy, also other information-theoretic entropy concepts have great potentials for discriminating between climatological records regarding their respective dynamical complexity. As a recent example, we mention here the consideration of the complexity-entropy causality plane for the characterization of daily streamflow time series [36], vegetation dynamics based on remote sensing data [179] or the degree distribution of a complex network obtained from correlations among surface air temperature records in the ENSO region in the equatorial Pacific [180].

In order to obtain reliable estimates of the dynamical complexity associated with the variability of large-scale climate modes, such as ENSO, on long time-scales, instrumental records may provide insufficient time coverage. Instead, it is feasible to utilize paleoclimate proxy data for obtaining information on nonlinear variability during the last few centuries, millennia or even the deeper past. In such situations, entropic characteristics can again provide interesting insights into qualitative changes in historical climate variability.

Saco et al. [181] studied the histogram-based Shannon entropy, as well as permutation entropy for a proxy record obtained from a sedimentary sequence from Laguna Pallcacocha, Ecuador. Their study revealed a distinct time interval with markedly reduced permutation entropy corresponding to a known period of large-scale rapid climate change (RCC) between 9,000 and 8,000 years before present (BP). The latter period includes the globally observed 8.2 kyr (kiloyear) event, which is believed to have originated from a strong meltwater pulse into the North Atlantic [182]. In terms of dynamical entropy, the obtained results indicate a considerable degree of regularity in the variability of environmental conditions. We hypothesize that this regularity indeed reflects the marked variability profile during the 8.2 kyr event, which exceeds the typical magnitude of “background fluctuations”. Notably, in the case of other, weaker RCC episodes during the last 8,000 years [182], there is no such clear decrease in permutation entropy. Instead, the corresponding entropy values are found to display a marked low-frequency variability, where even some of the temporary maxima of entropy coincide with known RCC episodes.

The special characteristics of the time interval between 9,000 and 8,000 years BP are supported by complementary analyses of Ferri et al. [183] using Tsallis’ q-triplet [175], who found that the three characteristic scaling exponents from the triplet change their relative ordering during this time interval. The aforementioned results once more underline the relevance of entropic characteristics (especially in the non-extensive framework) as indicators of approaching critical transitions, which is additionally supported by the results of another study on some Antarctic ice core records [184].

The previous results highlight the great potentials of nonlinear methods of time series analysis for the investigation of paleoclimate variability, which are not restricted to statistical mechanics and information-theoretic approaches, but have also been demonstrated for phase space-based characteristics, such as different quantifiers based on recurrence plots [85,185,186,187,188,189]. As an important cautionary note, we need to mention the presence of severe methodological problems, due to the non-uniform sampling and time-scale uncertainty of paleoclimate proxy data [190,191,192], which often do not permit a straightforward use of existing (linear and nonlinear) methods of time series analysis, but call for the development of specifically tailored estimators. Notable exceptions are tree rings, annually laminated (varved) sediments and other layer-forming geological archives (e.g., some ice cores, corals, etc.), where a reliable age-depth model can eventually be inferred based on layer counting.

Another climate-related recent field of application of information-theoretic concepts is the investigation of cross-scale processes in biogeochemistry and ecohydrology, i.e., the investigation of interdependencies between atmospheric dynamics and hydrology, on the one hand, and the biosphere (especially vegetation), on the other hand. In general, the biosphere is considered an important part of the Earth system, which exhibits mutual interactions with other components on, or even between, various spatial and temporal scales. The identification of such interdependencies, along with the associated relevant scales, is a challenging task for which bi- and multi-variate characteristics, based on information theory, appear particularly well suited.

Realizing the latter fact, the application of methods, such as mutual information or transfer entropy, has recently gained increasing importance. Brunsell and co-workers [193,194] applied entropy concepts for quantifying the information gained by surface vegetation as a function of the time-scale covered by the input precipitation field, revealing a clear relationship between information content and data resolution. In addition, they were able to characterize the sensitivity of evapotranspiration estimates to the spatial scales of vegetation and soil moisture dynamics obtained from satellite measurements that have been used for deriving these estimates. Stoy et al. [195] utilized the Shannon entropy, as well as relative entropy (Kullback-Leibler divergence) for assessing the amount of information contributed by the aggregation of scales in remote sensing data. Based on the inferred loss of information, they were able to define an “optimum” pixel size for working with such data. Subsequently, Brunsell [196] applied Shannon entropy, relative entropy and mutual information content for investigating the spatial variation in the temporal scaling of daily precipitation data, revealing the existence of distinct scaling regions in precipitation not discriminated before by traditional analysis techniques.

While in the aforementioned examples, information content and information transfer have been considered for different datasets with the same spatial and/or temporal resolution, a more general framework was put forward recently [197]. In this context, Shannon entropy and relative entropy have been used for identifying the dominant spatial scales in land-atmosphere H 2 O fluxes between surface and atmosphere obtained from remote sensing data and investigating the interactions of corresponding processes for different stages of the phenological cycle (i.e., the developmental stages of vegetation). The obtained results provided a deep and unique look into the multi-scale spatial structure of evapotranspiration, one of the most important climate variables determining vegetation growth. In a similar spirit, Cochran et al. [198] used relative entropy together with a wavelet-based multi-resolution analysis of ground-based eddy covariance measurements from so-called flux towers for studying the CO 2 exchange between the atmospheric boundary layer and the free troposphere. The aforementioned studies exemplify the relevance of entropic characteristics for studying various aspects of climate-vegetation feedback on local to global scales, thereby providing previously unrecognized information on the terrestrial carbon cycle and, hence, the Earth’s biogeochemistry.

As a complementary approach to studying interdependencies between the biosphere and climate system, Ruddell and Kumar put forward the idea of information process networks in ecohydrology, characterizing different types of coupling corresponding to synchronization, feedback and forcings [199,200]. Here, the quantification of information flows between simultaneously measured variables at various time lags by means of transfer entropy provides weighted network representations of the corresponding interaction structures, which can be further quantified using proper statistical measures. Kumar and Ruddell [201] successfully applied their formalism for inferring feedback loops from flux tower measurements in seven ecosystems in different climatic regions. As a particularly relevant result, they provided evidence that feedbacks tend to maximize information production, since variables participating in feedback loops exhibit “moderated” variability. The amount of information production was found to increase with gross ecosystem productivity and revealed a clear dependence of the corresponding processes on the specific ecological and climate setting, seasonal patterns, etc. Specifically, drier and colder ecosystems were found to respond more intensely and immediately to small changes in water and energy supply, which is in accordance with the general experience in ecology.

One important recent field of application of mutual information is the construction of climate networks from spatially distributed climate records. Here, individual time series corresponding to data from distinct measurement stations or grid points in a climate model are considered as variables, the strength of statistical associations between which can then be characterized by a variety of linear, as well as nonlinear measures. Selecting the strongest and/or statistically most significant pairwise associations provides a discrete set of spatial interdependencies, which can be further analyzed by network-theoretic concepts. Besides classical linear correlations and concepts referring to different notions of synchronization [202,203,204,205] (which have proven their potentials for bivariate analyses of climate data [206,207] before applications to network construction), application of mutual information based on symbolic dynamics [208,209] or order patterns [210,211] has attracted considerable interest and leads to a multi-scale description of the backbone of relevant spatial interdependencies in the climate system. Recently, the corresponding framework has been successfully generalized to studying directed interrelationships by making use of conditional mutual information [60]. We need to mention that the climate network approach based on information-theoretic characteristics is a subject of ongoing methodological discussions, referring to effects, such as the intrinsic dynamic complexity (which can be characterized itself by concepts, such as Gaussian process entropy rates [169]) or data artifacts resulting in spurious nonlinearities [212].

In addition to climate network construction and analysis, there have been first successful case studies applying information-theoretic concepts for coupling and causality analysis associated with some more specific research questions from climatology. Pompe and Runge [65] used a technique based on permutation entropy for studying the mutual interdependency between sea-surface temperature (SST) anomalies at two specific sites in the North Atlantic, motivated by the path of the North Atlantic Current. Their analysis revealed two distinct, relevant time-scales of oceanic coupling in addition to an instantaneous interrelationship; the latter is most likely mediated by atmospheric processes, namely the North Atlantic Oscillation (NAO) dipole pattern. These non-zero coupling delays were not as clearly detectable using the traditional mutual information or linear cross-correlation. In a similar study, graphical models extending the established transfer entropy framework have proven their skills in obtaining climatologically meaningful coupling patterns from the station data of daily sea-level pressure over Central and Eastern Europe during winter [59]. Another recent study making use of the same formalism investigated the vertical heat transport between the sea surface and upper troposphere based on monthly air temperature anomalies at different altitudes, having distinct implications for our mechanistic picture of the functioning of the atmospheric Walker circulation [64].

As an illustrative case study, we recall here the example originally provided by [213], who demonstrated the estimation algorithm of MIT using the linear partial correlation and its application to studying the Pacific Walker circulation based on reanalysis data with monthly resolution. In the following, we re-examine this example using the nonlinear conditional mutual information estimated with the k-nearest neighbor approach described in Section 2.4.1.

The basic mechanism of the Walker circulation [214,215,216,217] suggests that heating of the western equatorial Pacific, which is accompanied by low surface air pressure, promotes upwelling of waters in the eastern Pacific. This leads to easterly trade winds along the surface of the equatorial Pacific. The moist air then rises above the western Pacific, and the circulation is closed by the dry air masses descending along the central and eastern Pacific.

To test whether the feedback between the eastern Pacific (EPAC, represented here by the average SST over 80 o – 100 o W, 5 o S– 5 o N) and western Pacific (WPAC, average sea-level pressure anomalies over the region 130 o – 150 o E, 5 o S– 5 o N) was mediated via the surface of the central equatorial Pacific (CPAC, average SST over 150 o – 120 o W, 5 o S– 5 o N), we study the time series graph and MIT of the three-variable process (EPAC, CPAC, WPAC).

Figure 4 shows the lagged MI and the significant values of MIT after the time series graph was estimated using a significance level of α = 95 %. The MI lag function is significant between all pairs of variables, making a precise assessment of the causal structure impossible. For example, we find broad peaks between EPAC → CPAC and CPAC → WPAC, while the MIT values are significant only at lags 1 and 0 (indicating a contemporaneous dependency). Very interestingly, while MI features a broad peak in EPAC → WPAC, the time series graph shows no link (even for a lower significance level of 90%). Indeed, this link is obviously mediated via the surface of the central equatorial Pacific. In turn, the MIT of the link WPAC → EPAC at lag 1 is still significant, which demonstrates that the link back takes a different path, not via the surface of the equatorial central Pacific, which is in line with the climatological theory that the air masses return into the high troposphere.

In summary, the time series graph approach confirms the Walker circulation pattern, and the MIT further gives an estimate of the strengths of the underlying causal dependencies. We find generally very strong auto-MIT values, especially for the temperature variables, CPAC and EPAC, which indicates their strong inertia due to the large specific heat capacity of the ocean. Furthermore, the MIT values of the couplings EPAC → CPAC and CPAC → WPAC are comparable, whereas the MI peak of the former is significantly larger. This effect can be explained by the strong auto-dependencies within EPAC and CPAC, which increase MI, but are `filtered out’ in MIT. Another interesting point is that the link WPAC → CPAC that describes the descending mechanism is recovered here, while it was missing in the linear analysis in [213]. This finding suggests a nonlinear character of this dependency. However, in the simple Walker circulation picture, one would not expect the link CPAC → EPAC, which could be explained by the fact that the above described Walker circulation pattern is different during El Niño events, where the region of atmospheric updrafts shifts towards the central Pacific and also broadens markedly. Here, we used the whole time sample to test the hypothesis that the “average” influence is mediated via the central Pacific; a time-resolved analysis will be the subject of future research.