A General Symbolic Approach to Kolmogorov-Sinai Entropy

Abstract

It is popular to study a time-dependent nonlinear system by encoding outcomes of measurements into sequences of symbols following certain symbolization schemes. Mostly, symbolizations by threshold crossings or variants of it are applied, but also, the relatively new symbolic approach, which goes back to innovative works of Bandt and Pompe—ordinal symbolic dynamics—plays an increasing role. In this paper, we discuss both approaches novelly in one breath with respect to the theoretical determination of the Kolmogorov-Sinai entropy (KS entropy). For this purpose, we propose and investigate a unifying approach to formalize symbolizations. By doing so, we can emphasize the main advantage of the ordinal approach if no symbolization scheme can be found that characterizes KS entropy directly: the ordinal approach, as well as generalizations of it provide, under very natural conditions, a direct route to KS entropy by default.

1. Introduction

Using symbolizations to study observed data plays an important role in today’s time series analysis (see for instance the review papers of Daw et al. [1], Zanin et al. [2], Amigó et al. [3], and the examples in biology, medicine, artificial intelligence and data mining, just to mention a few, given therein). Thereby, it is assumed that time series, given by measurements of a real-world time-dependent system, store information about the complexity of the underlying system, which can be accessed by symbolic dynamics. In this paper, we assume further that measurements provide n-dimensional real-valued outcomes, that is a measuring process provides n time series.

Knowing the complexity is a key to classify systems and to predict future developments. A data analyst can for instance quantify complexity by empirical entropy measures, in particular by estimating the well-defined Kolmogorov-Sinai entropy (KS entropy). In order to estimate the KS entropy, however, a data analyst is always faced with the problem of choosing an adequate symbolization scheme.

Symbolizing a time series could be done in a “classical” manner for example by subdividing the data range into a finite number of intervals (see Section 2.1, often called the threshold crossing method in symbolic dynamics) or in an ordinal manner for example by considering the up and down behavior of subsequent measured values (see Section 2.2). The most ideal, however unrealistic, case is given if the analyst knows the underlying dynamics and picks a generating (under the dynamics) partition (see Section 1.1 and Section 1.4 for the mathematical formulation of the general problem, as well as for instance Crutchfield and Packard [4], Bollt et al. [5] and Kennel and Buhl [6]).

In the present paper, we show, by proposing a unifying approach to formalize symbolizations, that under relatively week assumptions, the search for a generating partition can be skipped if one chooses a symbolization scheme that regards a dependency between two measured values (see Section 2.2). In fact, following some rules by picking such a symbolization scheme, a generating sequence of finite partitions (see Section 1.1 and Section 1.4) is provided by default and needs no further attention (see Section 2.3 for an overview and Section 3 and Section 4.1, as well as the Appendix for the mathematics behind this). Moreover, the unifying approach allows one to consider “classical” and the relatively new ordinal symbolic dynamics [3] hand in hand and therefore to study respective assets and drawbacks.

In terms of the analyst, we propose a supplementing pool of complexity measures, which are in a certain sense approximations of the KS entropy and may be worth being compared in the finite setting of application (see Figure 7). Moreover, the relatively new ordinal approach could benefit from results achieved in “classical” symbolic dynamics, for instance to estimate a good symbolization scheme (see our ending remarks of the paper in Section 5 and for instance Steuer et al. [7], Letellier [8] and, published most recently, Li and Ray [9], as well as the references given therein). However, such topics exceed the scope of this paper.

1.1. Mathematical Formulation of the General Problem

Let us describe the central problems of determining KS entropy and give the main concepts of the paper without going into too much detail. The mathematical formulation is necessary at this point in order to state the results of the paper adequately.

We model a real-world time-dependent system by a state space $\mathsf{\Omega }$, that is states $\omega$ of the system are taken from the set $\mathsf{\Omega }$ and events on the system from a $\sigma$-algebra $\mathcal{A}$ on $\mathsf{\Omega }$. We assume that the states are distributed according to a probability distribution $\mu$ on $(\mathsf{\Omega },\mathcal{A})$. Moreover, considering states of the system at times in ${\mathbb{N}}_0=\{0,1,2,\dots \}$, the dynamics of the system is described by a map T with the interpretation that the system is in state $T\left(\omega \right)$ at time $t+1$ if it is in state $\omega$ at time t. For mathematical correctness, T is required to be measurable with respect to $\mathcal{A}$. We assume that the distribution of the states does not change in time, meaning T is $\mu$-invariant, which is defined by $T^{-1}\left(A\right)\in \mathcal{A}$ for all sets $A\in \mathcal{A}$.

The KS entropy is based on entropy rates of finite partitions of the state space. Given a finite partition $\mathcal{C}=\left\{C^{\left(1\right)},C^{\left(2\right)},\dots ,C^{\left(q\right)}\right\}\subset \mathcal{A}$ of $\mathsf{\Omega }$, the entropy rate $h_{\mu }(T,\mathcal{C})$, roughly speaking, measures the complexity of possible symbolic paths (see Section 4.1). A symbolic path is given by assigning to each state of the orbit:

$$\begin{array}{c}\hfill \omega ,T\left(\omega \right),T^{\circ 2}\left(\omega \right),T^{\circ 3}\left(\omega \right),\dots \end{array}$$

a symbol a when the state is contained in $C^{\left(a\right)}$. Here, $T^{\circ t}\left(\omega \right)$ denotes the t-th iterate of $\omega$ under T. We emphasize that starting with a partition $\mathcal{C}=\left\{C^{\left(1\right)},C^{\left(2\right)},\dots ,C^{\left(q\right)}\right\}$ is equivalent to a start where to each state in $\mathsf{\Omega }$ a symbol in $\{1,2,\dots ,q\}$ is assigned (in a measurable way). That is why we use the term symbolic approach.

In order to obtain a complexity measure that is independent of the discretization determined by a finite partition, one takes the supremum of the entropy rate $h_{\mu }(T,\mathcal{C})$ over all finite partitions $\mathcal{C}\subset \mathcal{A}$ of $\mathsf{\Omega }$, that is the KS entropy $h_{\mu }^{\mathrm{KS}}\left(T\right)$ of T:

$$h_{\mu }^{\mathrm{KS}}\left(T\right):=\underset{\mathcal{C}\mathrm{finite}\mathrm{partition}}{sup}{h}_{\mu }(T,\mathcal{C}).$$

Since usually there are uncountably many finite partitions, the determination of KS entropy on the basis of the definition is not feasible, so one is interested in finding natural partitions “carrying” the KS entropy.

In the case of a generating partition $\mathcal{C}$ under T (see Section 4.1), KS entropy is already characterized by this partition, meaning that:

$$h_{\mu }^{\mathrm{KS}}\left(T\right)=h_{\mu }(T,\mathcal{C})$$

(1)

(see, e.g., Walters [10], Theorem 4.18). Finding such suitable partitions, however, is impossible in most cases. A more realistic way of approaching KS entropy is to look for a generating and increasing, i.e., refining (see Section 4.1), sequence ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ of finite partitions ${\mathcal{C}}_d\subset \mathcal{A}$ of $\mathsf{\Omega }$, where:

$$h_{\mu }^{\mathrm{KS}}\left(T\right)=\underset{d\to \infty }{lim}{h}_{\mu }(T,{\mathcal{C}}_d)=\underset{d\in \mathbb{N}}{sup}{h}_{\mu }(T,{\mathcal{C}}_d)$$

(2)

(see, e.g., Walters [10], Theorem 4.22).

In the present paper, we discuss this countable increasing route to KS entropy in a framework where all partitions considered are derived from a natural real-valued “measuring process” and a symbolization scheme determined by a finite partition of the two-dimensional Euclidean space. The discussion includes and generalizes ideas from “classical” symbolic dynamics and from ordinal symbolic dynamics related to permutation entropy and sheds some new light on the latter one.

1.2. Observables and the Measuring Process

The modeling is completed by assuming that an n-dimensional outcome (here, $n\in \mathbb{N}=\{1,2,3,\dots \}$) of the system for each time is provided by observables $X_1,X_2,\dots ,X_n$, which mathematically are random variables on the probability space $(\mathsf{\Omega },\mathcal{A},\mu )$ with values in the real numbers $\mathbb{R}$. It provides the link between the dynamical model and the given n-dimensional time series data.

Fixing some state $\omega \in \mathsf{\Omega }$, we interpret the real numbers:

$$X_i\left(\omega \right),X_i\left(T\left(\omega \right)\right),X_i\left(T^{\circ 2}\left(\omega \right)\right),X_i\left(T^{\circ 3}\left(\omega \right)\right),\dots$$

as the values measured by $X_i$ at times $0,1,2,3,\dots$ when the given system is in state $\omega$ at the beginning.

Therefore, the random vector $\mathbf{X}={\left(X_i\right)}_{i=1}^n$ for the time-developing system provides random vectors:

$$\mathbf{X}={\left(X_i\right)}_{i=1}^n,\mathbf{X}\circ T={(X_i\circ T)}_{i=1}^n,\mathbf{X}\circ T^{\circ 2}={\left(X_i\circ T^{\circ 2}\right)}_{i=1}^n,\mathbf{X}\circ T^{\circ 3}={\left(X_i\circ T^{\circ 3}\right)}_{i=1}^n,\dots$$

forming the measuring process:

$${\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in {\mathbb{N}}_0}={\left({\left(X_i\circ T^{\circ t}\right)}_{i=1}^n\right)}_{t\in {\mathbb{N}}_0}$$

(3)

with the n time series ${\left(X_i\left(T^{\circ t}\left(\omega \right)\right)\right)}_{t\in {\mathbb{N}}_0}$ for $i=1,2,\dots ,n$ as outcomes. Note that the symbolizations we consider in the following are given at the observational level, i.e., with respect to the values of $X_i$; this complies with symbolizing a time series in real-world data analysis.

Let us regard $(\mathsf{\Omega },\mathcal{A},\mu )$, T, $n\in \mathbb{N}$ and $\mathbf{X}={\left(X_i\right)}_{i=1}^n$ as fixed in the following.

1.3. Information Contents in the Language of Event Systems

It is a central question of the given paper whether a description of a system, for instance by a measurement or by a symbolization, provides the same information as another one. In information theory, this is a matter of the richness of the event systems associated with the descriptions, more precisely a relation between sub-$\sigma$-algebras $\mathcal{F}$ and ${\mathcal{F}}^{\prime }$ of $\mathcal{A}$ defined by (compare to Walters [10], Definition 4.5):

$$\begin{array}{c}\hfill {\mathcal{F}}^{\prime }\stackrel{\mu }{\subset }\mathcal{F}\mathrm{if}\mathrm{for}\mathrm{each}{F}^{\prime }\in {\mathcal{F}}^{\prime }\mathrm{there}\mathrm{exists}\mathrm{some}F\in \mathcal{F}\mathrm{with}\mu (F▵F^{\prime })=0.\end{array}$$

The inclusion ${\mathcal{F}}^{\prime }\stackrel{\mu }{\subset }\mathcal{F}$ means that for each event in ${\mathcal{F}}^{\prime }$, there exists an event in $\mathcal{F}$ being distinct from the first one with probability zero and that is interpreted as meaning that $\mathcal{F}$ preserves all information contained in ${\mathcal{F}}^{\prime }$.

The $\sigma$-algebra $\mathcal{A}$ on $\mathsf{\Omega }$ consists of all events related to the given system, those events accessed by the given observables and the whole measuring process (3) form the sub-$\sigma$-algebras $\sigma \left(\mathbf{X}\right)$ and $\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in \mathbb{N}}\right)$ of $\mathcal{A}$, respectively. Mathematically, $\sigma \left(\mathbf{X}\right)$ is the smallest $\sigma$-algebra built from all preimages of Borel sets in $\mathbb{R}$ for:

$$X_1,X_2,\dots ,X_n,\dots ,$$

and $\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in \mathbb{N}}\right)$ is the smallest $\sigma$-algebra built from all preimages of Borel sets in $\mathbb{R}$ for:

$$X_1,X_2,\dots ,X_n,X_1\circ T,X_2,\circ T\dots ,X_n\circ T,X_1\circ T^{\circ 2},X_2\circ T^{\circ 2}\dots ,X_n\circ T^{\circ 2},\dots .$$

In these definitions, it is enough to take only intervals I instead of Borel sets. Here, ${(X_i\circ T^{\circ t})}^{-1}\left(I\right)$ describes the event that the value of the i-th measurement at time t is in I.

The sub-$\sigma$-algebra $\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)$, which is the smallest $\sigma$-algebra built from all events contained in some of the partitions ${\mathcal{C}}_d$ for $d\in \mathbb{N}$, provides the events accessed by the corresponding symbolization (see Section 4.1). Our goal is to construct an increasing sequence ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ of finite partitions, i.e., $C_{d+1}$ refines $C_d$ (see Section 4.1), which preserves the information given by the measuring process (3), i.e.,

$$\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in \mathbb{N}}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right),$$

(4)

or weaker by the observables themselves, i.e.,

$$\sigma \left(\mathbf{X}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right).$$

(5)

If (4) holds and the measuring process preserves the information of the original system, i.e., if:

$$\begin{array}{c}\hfill \mathcal{A}\stackrel{\mu }{\subset }\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in \mathbb{N}}\right),\end{array}$$

(6)

or if just (5) holds, but the observables preserve already the information of the original system, i.e., if:

$$\begin{array}{c}\hfill \mathcal{A}\stackrel{\mu }{\subset }\sigma \left(\mathbf{X}\right),\end{array}$$

(7)

then:

$$\begin{array}{c}\hfill \mathcal{A}\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)\subset \mathcal{A},\end{array}$$

meaning that ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ is generating (see Section 4.1 and compare to Walters [10]), which provides (2).

Conditions (6) and (7) are not as artificial as they appear at first glance:

• There is a very natural set of observables satisfying (7), hence (6). If $\mathsf{\Omega }$ is a Borel subset of ${\mathbb{R}}^n$, it is very plausible to assume that states and vectors of measured values are coinciding. This can be modeled by observables $X_i$; $i=1,2\dots ,n$ with $X_i$ being the i-th coordinate projection, i.e., $X_i\left(\omega \right)=x_i$ for $\omega =(x_1,x_2,\dots ,x_n)$. Clearly, in this simplest variant of modeling measurements, observables basically are superfluous in the modeling.
• In the case of only one observable, the separation of states is natural in a certain sense according to Takens’ theory (see Takens [11] and Gutman [12]).

1.4. A “Two-Dimensional” Way of Symbolizations

The partitions ${\mathcal{C}}_d$ with $d\in \mathbb{N}$ that we want to study in the following are formed on the basis of a finite partition $\mathcal{R}$ of the two-dimensional Euclidean space ${\mathbb{R}}^2$ and finite sets $E_d\subset {\mathbb{N}}_0\times {\mathbb{N}}_0$ of time pairs:

$${\mathcal{C}}_d={\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}):=\underset{i=1}{\overset{n}{\bigvee }}\underset{(s,t)\in E_d}{\bigvee }{\left(X_i\circ T^{\circ s},X_i\circ T^{\circ t}\right)}^{-1}\left(\mathcal{R}\right)\subset \mathcal{A},$$

(8)

i.e., ${\mathcal{C}}_d$ is the coarsest partition refining all partitions:

$${\left(X_i\circ T^{\circ s},X_i\circ T^{\circ t}\right)}^{-1}\left(\mathcal{R}\right)=\left\{\left\{\omega \in \mathsf{\Omega }\mid \left(X_i\left(T^{\circ s}\left(\omega \right)\right),X_i\left(T^{\circ t}\left(\omega \right)\right)\right)\in R\right\}\mid R\in \mathcal{R}\right\}$$

for $(s,t)\in E_d$ (see Section 4.1 for the definition of the join ${\bigvee }_{r=1}^m{\mathcal{C}}_r$ of finite partitions ${\mathcal{C}}_r\subset \mathcal{A}$ of $\mathsf{\Omega }$).

Here, $\mathcal{R}$ specifies the symbolization scheme for classifying the mutual position of measurements by $X_i$ at two times s and t (see Figure 1 and Figure 2, as well as the next section). We call $\mathcal{R}$ the basic symbolization scheme in the following. Note that we display the two-dimensional Euclidean space ${\mathbb{R}}^2$ by a square for illustrative purposes.

Further, the choice of ${\left(E_d\right)}_{d\in \mathbb{N}}$ complies with Definition 1, in particular in order to realize that ${\mathcal{C}}_{d+1}$ refines ${\mathcal{C}}_d$ (see Section 4.1), ${\mathcal{C}}_d$ is finite (see Definition 1(i)), and each time point that is relevant for the symbolization is accessed (see Definition 1(ii)).

Definition 1.

We call a sequence ${\left(E_d\right)}_{d\in \mathbb{N}}$ of sets $E_d$ with $E_1\subset E_2\subset \cdots \subset \{(s,t)\mid s,t\in {\mathbb{N}}_0,s<t\}$ a timing if there exists a set $\{v_0,v_1,\dots \}\subseteq {\mathbb{N}}_0$ with $v_0<v_1<v_2<\dots$ such that:

(i) 

for each $d\in \mathbb{N}$, it holds $E_d\subset {\{v_0,v_1,\dots ,v_d\}}^2$, and

(ii) 

for each $s\in \{v_0,v_1,\dots ,v_d\}$, there exists some $t\in \{v_0,v_1,\dots ,v_d\}$ such that $(s,t)\in E_d$ or $(t,s)\in E_d$.

A timing is for instance given by the sets:

$$E_d=\left\{(s,t)\mid s,t\in \left\{0,1,2,\cdots ,d\right\}\mathrm{with}s<t\right\};d\in \mathbb{N},$$

(9)

or by (11) and (17) (see below). It is suggestive to call the timing defined by (9) full timing in the following.

Subsequently, we discuss the following two questions:

• Why is the approach given natural and sufficiently general?
• Under which conditions on the basic symbolization scheme $\mathcal{R}$ and the timing $E_d$ does the sequence ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}={\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X})\right)}_{d\in \mathbb{N}}$ satisfy Statement (5) or even (4)?

Section 2 is devoted to the first question. In the first part of Section 3, we summarize our results to the second question and give some examples of basic symbolization schemes. Sufficient conditions that answer the second question including known results are presented for the interested reader in the second part of Section 3 and proven in Section 4 (see also the Appendix). We close this paper with some remarks about further theoretical and practical scientific issues (see Section 5).

2. Two Examples

At first glance, the above approach gives a rather exaggerated impression. The aim of the following examples is to convince the reader that sequences ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ formed on the basis of basic symbolization schemes $\mathcal{R}$ and timings ${\left(E_d\right)}_{d\in \mathbb{N}}$ are natural and are unifying known symbolic approaches.

2.1. “Classical” Symbolic Dynamics

First, we discuss “classical” symbolic dynamics with a fixed partition (see for instance Daw et al. [1], Kurths et al. [13] and the references given therein). For convenience, we assume that the dynamics is living on the real line, i.e., $\mathsf{\Omega }=\mathbb{R}$, and restrict ourselves to the simple case that $\mathbb{R}$ is subdivided into a finite number of intervals $I_1,I_2,\dots ,I_k$ (see for instance Figure 1).

The determination of the entropy rate of the partition $\mathcal{C}=\{I_1,I_2,\dots ,I_k\}$ is based on the partitions:

$${\left(\mathcal{C}\right)}_t=\{C^{(a_1,a_2,\dots ,a_t)}\mid a_1,a_2,\dots ,a_t\in \{1,2,\dots ,k\}\}$$

increasing with $t\in \mathbb{N}$, where:

$$C^{(a_1,a_2,\dots ,a_t)}=\{\omega \in \mathsf{\Omega }\mid \omega \in I_{a_1},T\left(\omega \right)\in I_{a_2},\dots ,T^{\circ t-1}\left(\omega \right)\in I_{a_t}\}$$

consists of those states $\omega \in \mathsf{\Omega }$ successively visiting $I_{a_1},I_{a_2},\dots ,I_{a_t}$, i.e., having the symbolic itinerary $a_1,a_2,\dots ,a_t$. Compare to Section 4.1, in particular for the deliberate notation used in this subsection, and see Figure 3, where we illustrate the symbolization process underlying the determination of the entropy rate for $t=3$.

For all $t\in \mathbb{N}$, it holds:

$${\left(\mathcal{C}\right)}_t=\underset{s=0}{\overset{t-1}{\bigvee }}{\left(T^{\circ s}\right)}^{-1}\left(\mathcal{C}\right).$$

In order to rewrite the “classical” approach into a form compatible with the proposed one, we need an artificial two-dimensional “blow-up” of the partitions ${\left(\mathcal{C}\right)}_t$; $t\in \mathbb{N}$, which is given by the partition:

$$\mathcal{R}:=\{I_1\times \mathbb{R},I_2\times \mathbb{R},\dots ,I_k\times \mathbb{R}\}(\mathrm{see}\mathrm{Figure} 1),$$

(10)

of ${\mathbb{R}}^2$ and the sets of time pairs:

$$E_t:=\{(0,1),(1,2),(2,3),\dots ,(t-1,t)\};t\in \mathbb{N},$$

(11)

that is:

$${\left(\mathcal{C}\right)}_t=\underset{s=0}{\overset{t-1}{\bigvee }}{(X\circ T^{\circ s})}^{-1}\left({\left(\mathcal{C}\right)}_1\right)=\underset{(s,u)\in E_t}{\bigvee }{(X\circ T^{\circ s},X\circ T^{\circ u})}^{-1}\left(\mathcal{R}\right)={\mathcal{C}}^{\mathcal{R},E_t}(T,X).$$

Here, we consider, motivated by our general procedure, the single observable X (meaning $\mathbf{X}=X$ in the general framework) with $X\left(\omega \right)=\omega$ for all $\omega \in \mathsf{\Omega }$, which fits the situation described at the end of Section 1.3.

This means in the other direction that the partitions ${\mathcal{C}}_d={\mathcal{C}}^{\mathcal{R},E_d}(T,X)$ are coinciding with the partitions ${\left(\mathcal{C}\right)}_d$. In particular, it holds:

$$h_{\mu }(T,{\mathcal{C}}_d)=h_{\mu }(T,{\left(\mathcal{C}\right)}_d)$$

for all $d\in \mathbb{N}$, and since $h_{\mu }(T,{\left(\mathcal{C}\right)}_t)=h_{\mu }(T,\mathcal{C})$ for all $t\in \mathbb{N}$ (see for instance Einsiedler and Schmidt [14], Satz 3.13), we obtain:

$$h_{\mu }(T,{\mathcal{C}}_d)=h_{\mu }(T,\mathcal{C})$$

for all $d\in \mathbb{N}$. This fact implying that the sequence ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}={\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X)\right)}_{d\in \mathbb{N}}$ is generating iff $\mathcal{C}$ is generating under T (see the final remarks in Section 4.1) says that $\mathcal{R}$ as given by (10) has no generating potential when $\mathcal{C}$ fails to be generating under T. This is not surprising since $\mathcal{R}$ is no more than a two-dimensional “blow-up” of $\mathcal{C}$. The second example shows the existence of good choices of $\mathcal{R}$ with generating properties under certain assumptions.

In Figure 4, we study (compare to Section 4.1) two different initial partitions, that is $\mathcal{C}=\left\{\right[0,1/2),[1/2,1\left)\right\}$ and $\mathcal{D}=\left\{\right[0,1/4),[1/4,1\left)\right\}$, under the transformation $T:\left(\right[0,1\left)\right)\hookleftarrow$, defined by:

$$T\left(\omega \right)=\left\{\begin{array}{cc}2\omega \hfill & \mathrm{if}0\le \omega <\frac{1}{2},\hfill \\ 2-2\omega \hfill & \mathrm{if}\frac{1}{2}\le \omega <1,\hfill \end{array}\right.$$

(12)

i.e., T is the full tent map on $\mathsf{\Omega }:=[0,1)$. The KS entropy is $ln\left(2\right)$ (see for instance Bollt et al. [5] and the references given therein), and in fact, it holds:

$$H_{\mu }\left(\mathcal{C}\right)=\frac{1}{2}{H}_{\mu }\left({\left(\mathcal{C}\right)}_2\right)=\frac{1}{3}{H}_{\mu }\left({\left(\mathcal{C}\right)}_3\right)=\cdots =\frac{1}{t}{H}_{\mu }\left({\left(\mathcal{C}\right)}_t\right)=\cdots =ln\left(2\right),$$

whereby:

$$H_{\mu }\left(\mathcal{D}\right)=-\frac{1}{4}ln\left(\frac{1}{4}\right)-\frac{3}{4}ln\left(\frac{3}{4}\right)<ln\left(2\right)$$

(see Section 4.1). Since $\frac{1}{t}{H}_{\mu }\left({\left(\mathcal{C}\right)}_t\right)$ decreases to $h_{\mu }(T,\mathcal{C})$ for any finite partition $\mathcal{C}\subset \mathcal{A}$ of $\mathsf{\Omega }$ (see Walters [10], Chapter 4) and by Theorem 4.18 of [10] (see Equation (1)), we have by $\mathcal{C}$ a generating and by $\mathcal{D}$ a non-generating (misplaced) partition under T (see Bollt et al. [5] and Steuer et al. [7] for detailed information about possible consequences if a non-generating partition is used in time series analysis, as well as Figure 7).

2.2. Ordinal Symbolic Dynamics

Ordinal symbolic dynamics is a relatively new symbolic approach going back to Bandt and Pompe [15] and applied in various fields (see for instance Zanin et al. [2], Amigó et al. [3] and the references given therein). The idea of the symbolization scheme is to partition the state space according to ordinal patterns of orders $d\in \mathbb{N}$. For fixed d and a random vector $\mathbf{X}={\left(X_i\right)}_{i=1}^n$, two states ${\omega }_1,{\omega }_2\in \mathsf{\Omega }$ belong to the same part of a partition if for each $i=1,2,\dots ,n$, the observable $X_i$ provides the same order relations on the orbits of length $d+1$ of ${\omega }_1$ and ${\omega }_2$:

For all $s,t$ with $0\le s<t\le d$, it holds:

$$X_i\left(T^{\circ s}\left({\omega }_1\right)\right)\ge X_i\left(T^{\circ t}\left({\omega }_1\right)\right)\mathrm{iff}{X}_i\left(T^{\circ s}\left({\omega }_2\right)\right)\ge X_i\left(T^{\circ t}\left({\omega }_2\right)\right).$$

One easily sees that the obtained partitions can be written in the form ${\mathcal{C}}_d={\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X})$ with:

$$\mathcal{R}=\left\{\left\{(x,y)\in \mathbb{R}\times \mathbb{R}\mid x\ge y\right\},\left\{(x,y)\in \mathbb{R}\times \mathbb{R}\mid x<y\right\}\right\}\left(\mathrm{see}\mathrm{Figure}2\right)$$

(13)

and the full timing (see Equation (9)).

The sequence ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ is obviously increasing. Antoniouk et al. [16] show the following statement for $\mathcal{R}$ as given by (13) and the full timing ${\left(E_d\right)}_{d\in \mathbb{N}}$ given by (9), here formulated in the language of our general approach:

$$\begin{array}{c}\mathrm{If} T \mathrm{is} \mathrm{ergodic} \mathrm{and} \mathbf{X} \mathrm{satisfies} \left(6\right) \mathrm{or} \mathrm{weaker} \left(7\right), \mathrm{then}\hfill \\ {\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}={\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X})\right)}_{d\in \mathbb{N}}\mathrm{is}\mathrm{generating},\mathrm{hence}{h}_{\mu }^{\mathrm{KS}}\left(T\right)=\underset{d\to \infty }{lim}{h}_{\mu }(T,{\mathcal{C}}_d)=\underset{d\in \mathbb{N}}{sup}{h}_{\mu }(T,{\mathcal{C}}_d).\hfill \end{array}$$

(14)

Unlike the “classical” approach, the basic symbolization scheme $\mathcal{R}$, as given by (13), regards a kind of dependency between two measurements by $X_i$. The statement (14) shows the substantial difference between “classical” and ordinal symbolic dynamics: by using $\mathcal{R}$ as given in (13), we obtain a generating sequence ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$, regardless of whether ${\mathcal{C}}_1=\mathcal{C}$ is generating under T.

2.3. An Extension of Ordinal Symbolic Dynamics

In the rest of the paper, we discuss for which $\mathcal{R}$ and ${\left(E_d\right)}_{d\in \mathbb{N}}$ the statement (14) remains true; whereby, this section is dedicated to those readers who are mainly interested in the idea and results of our study. Since the corresponding considerations to validate our subsequent statement are fairly technical and due to even more general results, we refer here only to the statements and proofs in the later discussion (Section 3 and the Appendix). First of all, obviously, (14) remains true when $\mathcal{R}$ is substituted by a refinement of (13) (see Figure 5). Moreover, in the case that $\mathsf{\Omega }$ is a Borel subset of ${\mathbb{R}}^n$ and $X_i$, $i=1,2,\cdots ,n$ is the i-th coordinate projection (see the closing remarks of Section 1.3), as well as $\mu \left(A\right)>0$ for all open subsets A of $\mathsf{\Omega }$, Statement (14) remains true if we modify $\mathcal{R}$ even more (see the closing remarks of Section 3.1):

Theorem 1.

Let Ω be a Borel subset of ${\mathbb{R}}^n$ and $X_i$; $i=1,2,\cdots ,n$ be the i-th coordinate projection. Further, let $\mathcal{R}$ be a basic symbolization scheme defined by:

$$\mathcal{R}=\left\{\left\{(x,y)\in \mathbb{R}\times \mathbb{R}\mid g\left(x\right)\ge y\right\},\left\{(x,y)\in \mathbb{R}\times \mathbb{R}\mid g\left(x\right)<y\right\}\right\},$$

(15)

or finer, where $g:\mathbb{R}\hookleftarrow$ is a one-to-one $\mathcal{B}\left(\mathbb{R}\right)$-$\mathcal{B}\left(\mathbb{R}\right)$ measurable map with $\mathcal{B}\left(\mathbb{R}\right)$ being the Borel σ-algebra on $\mathbb{R}$ (see for instance Figure 6). If $\mu \left(A\right)>0$ for all open subsets A of Ω, then for the full timing ${\left(E_d\right)}_{d\in \mathbb{N}}$ (see Equation (9)), Statement (14) is fulfilled.

3. Main Mathematical Results

Antoniouk et al. [16] show that the search for a generating partition under T can be bypassed by choosing ordinal symbolic dynamics. It namely provides, in the ergodic case and if $\mathbf{X}$ satisfies (6) or weaker (7), a generating sequence of finite partitions by default, that is the generating property is valid regardless of the properties of the original system considered (see Statement (14) in Section 2.2).

The question arises if other symbolic approaches deliver similar results. In fact, by generalizing the ideas and results of Antoniouk et al. [16], we give sufficient conditions on $\mathcal{R}$ and ${\left(E_d\right)}_{d\in \mathbb{N}}$ for (14). This we present the interested reader in the next sections.

3.1. Preserving the Information of Observables

The following is quite technical, but shows under which conditions the information given by the observables is preserved if basic symbolization schemes $\mathcal{R}$ such as given by (15), or finer, are considered. Hence, if Theorem 2 holds and $\mathcal{A}\stackrel{\mu }{\subset }\sigma \left(\mathbf{X}\right)$, then ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ is generating.

Theorem 2.

Let T be ergodic, $\mathbf{X}={\left(X_i\right)}_{i=1}^n$ a random vector, $\mathcal{R}$ a basic symbolization scheme, ${\left(E_d\right)}_{d\in \mathbb{N}}$ a timing and ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ a sequence of finite partitions constructed from $\mathcal{R}$ and ${\left(E_d\right)}_{d\in \mathbb{N}}$ (see Equation (8)). If further:

(i) 

g is admissible with respect to $X_i$ for all $i=1,2,\dots ,n$,

(ii) 

$F_{X_i}$ is admissible with respect to $g\circ X_i$ for all $i=1,2,\dots ,n$ and

(iii) 

$\left\{g\circ X_i\ge X_i\circ T^{\circ t}\right\}\in \sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right)$ for all $i=1,2,\dots ,n$ and $t\in {\mathbb{N}}_0$,

then:

$$\sigma \left(\mathbf{X}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right).$$

We call a function $\varphi :\mathbb{R}\hookleftarrow$ admissible with respect to a random variable Y on $\mathsf{\Omega }$ if $\sigma \left(Y\right)\stackrel{\mu }{\subset }\sigma (\varphi \circ Y);$ this is for example the case if $\varphi$ is a one-to-one $\mathcal{B}\left(\mathbb{R}\right)$-$\mathcal{B}\left(\mathbb{R}\right)$ measurable map (see the closing remarks of the Appendix on one-to-one maps and Lemma A3 for general conditions on $\varphi$ such that $\varphi$ is admissible). Requiring that $F_{X_i}$ has to be admissible with respect to $g\circ X_i$ means that g has to be constructed in such a way that:

$$\sigma (g\circ X_i)\stackrel{\mu }{\subset }\sigma (F_{X_i}\circ g\circ X_i)$$

holds (compare to the proof of Lemma A3 and subsequent remarks). This assumption is redundant if $\mathsf{\Omega }$ is a Borel subset of ${\mathbb{R}}^n$; each $X_i$; $i=1,2,\cdots ,n$ is the i-th coordinate projection (see the closing remarks of Section 1.3); and $\mu \left(A\right)>0$ for all open subsets A of $\mathsf{\Omega }$, because then, $F_{X_i}$ is one-to-one (see the closing remarks of the Appendix on one-to-one maps). Finally, note that symbolizations based on $\mathcal{R}$ as given by (15), or finer, have the property (iii) of Theorem 2 (in particular, compare (iii) of Theorem 2 to the structure of (15)). Summarizing, the assumptions of Theorem 2 are generalizations of the assumptions of Theorem 1, and thus, Theorem 1 follows by Theorem 2.

3.2. Preserving the Information of the Measuring Process

In this section, we state sufficient conditions such that the information given by the measuring process is preserved. Therefore, if these conditions are fulfilled and $\mathcal{A}\stackrel{\mu }{\subset }\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in \mathbb{N}}\right)$, then ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ is generating. Define:

$${\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{Y}):=\underset{i=1}{\overset{n}{\bigvee }}\underset{(s,t)\in E_d}{\bigvee }{\left(Y_i\circ T^{\circ s},Y_i\circ T^{\circ t}\right)}^{-1}\left(\mathcal{R}\right)\subset \mathcal{A},$$

where $\mathbf{Y}={\left(Y_i\right)}_{i=1}^n$ is an arbitrary random vector (compare to Equation (8)), and consider the special case $\mathbf{Y}={\left(Y_i\right)}_{i=1}^n=\mathbf{X}\circ T^{\circ l}={\left(X_i\circ T^{\circ l}\right)}_{i=1}^n$ for some $l\in \mathbb{N}$ in the following.

Definition 2.

Let $\mathcal{R}$ be a basic symbolization scheme and ${\left(E_d\right)}_{d\in \mathbb{N}}$ be a timing. We call the tuple $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ consistent if for all $t>1$ and $d\in \mathbb{N}$, it holds:

$$\underset{s=0}{\overset{t-1}{\bigvee }}{\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ s})\prec \underset{s=0}{\overset{t-2}{\bigvee }}{\mathcal{C}}^{\mathcal{R},E_{d+1}}(T,\mathbf{X}\circ T^{\circ s}).$$

(16)

Compare to Keller et al. [17], who regard the ordinal approach: observe that, in the consistent case, by applying (16) repeatedly, one shows that:

$$\underset{s=0}{\overset{t-1}{\bigvee }}{\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ s})\prec {\mathcal{C}}^{\mathcal{R},E_{d+t-1}}(T,\mathbf{X})$$

for all t, $d\in \mathbb{N}$. Consistency ensures that for all $t\in {\mathbb{N}}_0$, it holds:

$$\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ t})\right)}_{d\in \mathbb{N}}\right)\subset \sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right).$$

In other words, if the information given by a measurement at a time $t\in \mathbb{N}$ is preserved by ${\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ t})\right)}_{d\in \mathbb{N}}$, then it also is preserved by ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ (compare to the proof of Theorem 3).

Consistency depends on the interplay of the underlying system, the considered random vector $\mathbf{X}={\left(X_i\right)}_{i=1}^n$, $\mathcal{R}$ and ${\left(E_d\right)}_{d\in \mathbb{N}}$; however, a skillful choice of the timing guarantees that $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is consistent independent of the system and $\mathbf{X}$. This we discuss in Section 4.3. Note here that $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is always consistent if ${\left(E_d\right)}_{d\in \mathbb{N}}$ is the full timing (see Equation (9)); however, the tuple is not consistent in general if a timing given by:

$$E_d=\left\{(0,t)\mid t\in \left\{0,1,2,\cdots ,d\right\}\right\};d\in \mathbb{N}$$

(17)

is considered.

Theorem 3.

Let $\mathbf{X}={\left(X_i\right)}_{i=1}^n$ be a random vector, $\mathcal{R}$ a basic symbolization scheme, ${\left(E_d\right)}_{d\in \mathbb{N}}$ a timing and ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ a sequence of finite partitions constructed by $\mathcal{R}$ and ${\left(E_d\right)}_{d\in \mathbb{N}}$ (see Equation (8)). If further:

(i) 

$\sigma \left(\mathbf{X}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)$ and

(ii) 

$(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is consistent,

then:

$$\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in \mathbb{N}}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right).$$

Recall that (i) of Theorem 3 particularly holds if Conditions (i)–(iii) of Theorem 2 are fulfilled.

4. Proofs

We begin by summarizing some basic notations and concepts. Thereafter, we prove Theorem 2 and Theorem 3, whereby quite technical and complicated lemmas can be found in the Appendix.

4.1. Preliminaries

Our whole discussion is concerned with finite partitions $\mathcal{C}\subset \mathcal{A}$ of $\mathsf{\Omega }$ and with sequences of finite partitions ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$. A finite partition $\mathcal{C}$ of $\mathsf{\Omega }$ is a set system:

$$\mathcal{C}:=\left\{C^{\left(1\right)},C^{\left(2\right)},\dots ,C^{\left(q\right)}\right\}\subset \mathcal{A};q\in \mathbb{N},$$

where:

$$\bigcup _{l=1}^q{C}^{\left(l\right)}=\mathsf{\Omega }\mathrm{and}{C}^{\left(l\right)}\cap C^{\left(k\right)}=\varnothing \mathrm{for}\mathrm{any}l\ne k\in \{1,2,\cdots ,q\}.$$

Particularly, we are interested in such sequences that are increasing, meaning that ${\mathcal{C}}_{d+1}$ is finer than ${\mathcal{C}}_d$ for all $d\in \mathbb{N}$:

A partition $\mathcal{D}=\left\{D^{\left(1\right)},D^{\left(2\right)},\cdots ,D^{\left(p\right)}\right\}$ is finer than a partition $\mathcal{C}=\left\{C^{\left(1\right)},C^{\left(2\right)},\cdots ,C^{\left(q\right)}\right\}$; we write $\mathcal{C}\prec \mathcal{D}$, or, equivalently, $\mathcal{C}$ is coarser than $\mathcal{D}$ if for all $l\in \{1,2,\cdots ,q\}$ there exists a nonempty $K\subset \{1,2,\cdots ,p\}$ such that:

$$C^{\left(l\right)}=\bigcup _{k\in K}{D}^{\left(k\right)}.$$

Moreover, we consider the join ${\bigvee }_{r=1}^m{\mathcal{C}}_r$ of finite partitions ${\mathcal{C}}_r\subset \mathcal{A}$ of $\mathsf{\Omega }$ with $m\in \mathbb{N}$ and $r=1,2,\dots ,m$, which is defined by:

$$\underset{r=1}{\overset{m}{\bigvee }}{\mathcal{C}}_r=\left\{\bigcap _{r=1}^m{C}_r^{\left(l_r\right)}\ne \varnothing |l_r\in \{1,2,\dots ,|{\mathcal{C}}_r\left|\right\}\mathrm{for}r=1,2,\dots ,m\right\},$$

that is the coarsest partition refining all ${\mathcal{C}}_r$.

Furthermore, sub-$\sigma$-algebras of $\mathcal{A}$ are central to us, especially the ones $\sigma \left(\mathcal{M}\right)$ generated by subsets $\mathcal{M}$ of $\mathcal{A}$, i.e., the smallest sub-$\sigma$-algebra containing $\mathcal{M}$:

$$\sigma \left(\mathcal{M}\right)=\bigcap _{\begin{array}{c}\mathcal{F}\subset \mathcal{A}\mathrm{is}\sigma -\mathrm{algebra}\\ \mathrm{and}\mathcal{M}\subset \mathcal{F}\end{array}}\mathcal{F}.$$

We consider also the join of $\sigma$-algebras ${\mathcal{F}}_i\subset \mathcal{A};i\in I$ defined by:

$$\underset{i\in I}{\bigvee }{\mathcal{F}}_i=\sigma \left(\bigcup _{i\in I}{\mathcal{F}}_i\right).$$

Overall, we have a special interest in the following sub-$\sigma$-algebras of $\mathcal{A}$:

$$\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)=\underset{d\in \mathbb{N}}{\bigvee }\sigma \left({\mathcal{C}}_d\right)$$

for ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ being a finite partition sequence of $\mathsf{\Omega }$,

$$\begin{array}{c}\sigma \left(\mathbf{X}\right)=\underset{i=1}{\overset{n}{\bigvee }}\sigma \left(X_i\right)=\underset{i=1}{\overset{n}{\bigvee }}\left\{X_i^{-1}\left(B\right)\mid B\in \mathcal{B}\left(\mathbb{R}\right)\right\},\hfill \\ \sigma (\mathbf{X}\circ T^{\circ s})=\underset{i=1}{\overset{n}{\bigvee }}\sigma (X_i\circ T^{\circ s})=\underset{i=1}{\overset{n}{\bigvee }}\left\{{(X_i\circ T^{\circ s})}^{-1}\left(B\right)\mid B\in \mathcal{B}\left(\mathbb{R}\right)\right\},\hfill \\ \sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in {\mathbb{N}}_0}\right)=\underset{t\in {\mathbb{N}}_0}{\bigvee }\sigma \left(\mathbf{X}\circ T^{\circ t}\right)=\underset{i=1}{\overset{n}{\bigvee }}\underset{t\in {\mathbb{N}}_0}{\bigvee }\sigma \left(X_i\circ T^{\circ t}\right)\hfill \end{array}$$

for $\mathbf{X}={\left(X_i\right)}_{i=1}^n$ being a random vector on $\mathsf{\Omega }$ and $s\in \mathbb{N}$.

We close this subsection by giving an exact definition of the entropy rate $h_{\mu }(T,C)$ of a finite partition $\mathcal{C}$ (see Figure 3): one assigns to every part $C^{\left(i\right)}$ of the partition $\mathcal{C}=\left\{C^{\left(1\right)},C^{\left(2\right)},\cdots ,C^{\left(q\right)}\right\}$ the letter i of the alphabet $A=\{1,2,\dots ,q\}$. Each word $(a_1,a_2,\dots ,a_t)$ of length $t\in \mathbb{N}$ over A defines a set:

$$\begin{array}{c}\hfill C^{(a_1,a_2,\dots ,a_t)}=\left\{\omega \in \mathsf{\Omega }|\left(\omega ,T\left(\omega \right),\dots ,T^{\circ t-1}\left(\omega \right)\right)\in C^{\left(a_1\right)}\times C^{\left(a_2\right)}\times \dots \times C^{\left(a_t\right)}\right\}.\end{array}$$

All non-empty sets $C^{(a_1{a}_2\dots a_t)}$ provide a partition ${\left(\mathcal{C}\right)}_t\subset \mathcal{A}$ of $\mathsf{\Omega }$. We use the notation ${\left(\mathcal{C}\right)}_t$ to emphasize that the partition is constructed with respect to T. In particular, ${\left(\mathcal{C}\right)}_1=\mathcal{C}$. The entropy rate of T with respect to $\mathcal{C}$ is given by:

$$h_{\mu }(T,\mathcal{C})=\underset{t\to \infty }{lim}\frac{1}{t}{H}_{\mu }\left({\left(\mathcal{C}\right)}_t\right)=\underset{t\to \infty }{lim}\left(H_{\mu }\left({\left(\mathcal{C}\right)}_t\right)-H_{\mu }\left({\left(\mathcal{C}\right)}_{t-1}\right)\right),$$

(18)

where $H_{\mu }\left({\left(\mathcal{C}\right)}_t\right)$ is the Shannon entropy of ${\left(\mathcal{C}\right)}_t$, that is for a finite partition $\mathcal{D}=\left\{D^{\left(1\right)},D^{\left(2\right)},\cdots ,D^{\left(p\right)}\right\}$:

$$H_{\mu }\left(\mathcal{D}\right)=-\sum _{l=1}^p\mu \left(D^{\left(l\right)}\right)ln\left(\mu \left(D^{\left(l\right)}\right)\right)(\mathrm{with}0ln\left(0\right):=0).$$

For a fuller treatment, e.g., for statements that the limit in Equation (18) exists and $\frac{1}{t}{H}_{\mu }\left({\left(\mathcal{C}\right)}_t\right)$, as well as $H_{\mu }\left({\left(\mathcal{C}\right)}_t\right)-H_{\mu }\left({\left(\mathcal{C}\right)}_{t-1}\right)$ decreases to $h_{\mu }(T,\mathcal{C})$, we refer the reader to Chapter 4 of Walters [10]. Note that for all $t\in \mathbb{N}$, it holds ${\left(\mathcal{C}\right)}_{t-1}\prec {\left(\mathcal{C}\right)}_t$, and:

$$H_{\mu }\left({\left(\mathcal{C}\right)}_t\right)-H_{\mu }\left({\left(\mathcal{C}\right)}_{t-1}\right)\le \frac{1}{t}\sum _{s=1}^t{H}_{\mu }\left({\left(\mathcal{C}\right)}_s\right)-H_{\mu }\left({\left(\mathcal{C}\right)}_{s-1}\right)=\frac{1}{t}{H}_{\mu }\left({\left(\mathcal{C}\right)}_s\right).$$

(19)

Moreover, we say that $\mathcal{C}$ is generating under T if:

$$\mathcal{A}\stackrel{\mu }{\subset }\sigma \left({\left({\left(\mathcal{C}\right)}_t\right)}_{t\in \mathbb{N}}\right).$$

If we consider instead an arbitrary sequence of finite partitions ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ for which:

$$\mathcal{A}\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)$$

holds, then we call ${\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}$ just generating.

4.2. Proof of Theorem 2

In order to prove Theorem 2, we generalize the results of Antoniouk et al. [16] (Lemmas 3.2, 3.3 and Corollary 3.4) and extend their proofs. Thereby, we utilize properties of the distribution function $F_{X_i}:\mathbb{R}\to [0,1]$ of a random variable $X_i:\mathsf{\Omega }\to \mathbb{R}$, i.e.,

$$F_{X_i}\left(a\right)=\mu \left(\{\omega \in \mathsf{\Omega }\mid X_i\left(\omega \right)\le a\}\right)$$

for all $a\in \mathbb{R}$ (see Lemmas A1 and A2 in the Appendix) and show:

$$\sigma \left(X_i\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right)$$

by the detour:

$$\sigma \left(X_i\right)\stackrel{\mu }{\subset }\sigma (g\circ X_i)\stackrel{\mu }{\subset }\sigma (F_{X_i}\circ g\circ X_i)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right).$$

Proof of Theorem 2.

Note that it is enough to show that:

$$\sigma \left(X_i\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right)$$

holds for all $i=1,2,\cdots ,n$ since the sub-$\sigma$-algebras $\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right)$ generate the sub-$\sigma$-algebra:

$$\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)=\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X})\right)}_{d\in \mathbb{N}}\right).$$

Thus, let us regard $i\in \mathbb{N}$ as fixed. By the assumptions Theorem 2(i) and (ii) (compare also to Lemma A3), we obtain:

$$\sigma \left(X_i\right)\stackrel{\mu }{\subset }\sigma (g\circ X_i)\stackrel{\mu }{\subset }\sigma (F_{X_i}\circ g\circ X_i).$$

Moreover, by Theorem 2(iii) and Lemma A2, we have that:

$$\sigma (F_{X_i}\circ g\circ X_i)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right).$$

Hence:

$$\sigma \left(X_i\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,X_i)\right)}_{d\in \mathbb{N}}\right),$$

which completes the proof. ☐

4.3. Proof of Theorem 3

The main additional property needed in Theorem 3 is that $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is consistent (see Definition 2). In order to see how a timing has to be constructed such that $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is generally consistent, let us regard $t,d\in \mathbb{N}$ as fixed and consider states ${\omega }_1$, ${\omega }_2\in \mathsf{\Omega }$, which are in different parts of:

$$\underset{s=0}{\overset{t-1}{\bigvee }}{\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ s}).$$

Then, there exists at least one $s\in \{0,1,\cdots ,t-1\}$, one $i=1,2,\cdots ,n$ and a time pair $(u,v)\in E_d$ such that:

$$\left(X_i\left(T^{\circ s+u}\left({\omega }_1\right)\right),X_i\left(T^{\circ s+v}\left({\omega }_1\right)\right)\right)\mathrm{and}\left(X_i\left(T^{\circ s+u}\left({\omega }_2\right)\right),X_i\left(T^{\circ s+v}\left({\omega }_2\right)\right)\right)$$

are in different parts of $\mathcal{R}$. If now for any $s\in \{0,1,\cdots ,t-1\}$ and $(u,v)\in E_d$, it holds that $(s+u,s+v)\in E_{d+t-1}$, then:

$${\left(X_i\circ T^{\circ s+u},X_i\circ T^{\circ s+v}\right)}^{-1}\left(\mathcal{R}\right)\prec {\mathcal{C}}_{d+t-1}.$$

Hence, $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is consistent if the previous holds for all $t,d\in \mathbb{N}$.

Proof of Theorem 3.

Compare to [16] (Corollary 3.5). By Theorem 3(ii), i.e., $(\mathcal{R},{\left(E_d\right)}_{d\in \mathbb{N}})$ is consistent, it holds:

$${\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T)\prec {\mathcal{C}}^{\mathcal{R},E_{d+1}}(T,\mathbf{X})$$

for all $d\in \mathbb{N}$. These refinements imply:

$$\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ t})\right)}_{d\in \mathbb{N}}\right)\subset \sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right)$$

for all $t\in {\mathbb{N}}_0$ (see Walters [10], Chapter 4, Section 1). Moreover, by Theorem 3(i), it holds:

$$\sigma \left(\mathbf{X}\circ T^{\circ t}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}^{\mathcal{R},E_d}(T,\mathbf{X}\circ T^{\circ t})\right)}_{d\in \mathbb{N}}\right)$$

for all $t\in {\mathbb{N}}_0$. Hence:

$$\sigma \left({\left(\mathbf{X}\circ T^{\circ t}\right)}_{t\in {\mathbb{N}}_0}\right)\stackrel{\mu }{\subset }\sigma \left({\left({\mathcal{C}}_d\right)}_{d\in \mathbb{N}}\right),$$

which proves the theorem. ☐

5. Some Remarks

In proposing and studying our universal symbolic approach toward KS entropy, we have restricted ourselves to the kernel ideas. In particular, we have attached importance to point out that a “two-dimensional” symbolization, that is linking two observations, can provide a better basic symbolization scheme than symbolizing only on a one-dimensional observational level. The obtained results can be simply generalized in two directions:

On the one hand, infinitely many observables instead of finitely many ones can be considered. Here, the results obtained by Keller et al. [18] (see also the references given in the paper) can directly be adapted, which leads to a description of the KS entropy by a double limit substituting the limit in (2). On the other hand, some of our results remain true when relaxing ergodicity by some rather general conditions on the dynamics considered. For this, the ergodic decomposition theorem can be utilized. We refer to the discussion in Keller et al. [18] and the references given therein.

Our study does not touch aspects as the speed of convergence in (2), as a general comparison of basic symbolic schemes and as an entropy estimation, which are, incontestably, very interesting both from the theoretical and practical viewpoint. Our approach provides some theoretical framework within concrete methods for time series and system analysis and can be specified in accordance with requirements given in practice.

In order to give a brief perspective on the matter, we decoded a finite orbit ${\left(x_t\right)}_{t=0}^{\mathcal{T}}$; $\mathcal{T}\in \mathbb{N}$ of the tent map (see Section 2.1), that is:

$$x_{t+1}=\left\{\begin{array}{cc}2x_t\hfill & \mathrm{if}0\le x_t<\frac{1}{2},\hfill \\ 2-2x_t\hfill & \mathrm{if}\frac{1}{2}\le x_t<1\hfill \end{array}\right.$$

(20)

for all $t=0,1,\cdots ,\mathcal{T}-1$ and $x_0$ uniformly distributed, into a sequence of symbols, fixed a word length $t\in \mathbb{N}$ and naively estimated the difference $H_{\mu }\left({\left(\mathcal{C}\right)}_t\right)-H_{\mu }\left({\left(\mathcal{C}\right)}_{t-1}\right)$ by replacing the probabilities by relative frequencies of symbol word occurrences.

Figure 7 shows the results for different $t\in \mathbb{N}$ and symbolization schemes in dependence of the orbit length $\mathcal{T}$, here between ${10}^2$ and ${10}^6$. We chose to take the difference because of (19), i.e., the difference for fixed t is a better approximation of the entropy rate $h_{\mu }(T,\mathcal{C})$ than $\frac{1}{t}{H}_{\mu }\left({\left(\mathcal{C}\right)}_t\right)$. For a fuller treatment, we refer the reader to Keller et al. [19], in particular for more information of how to construct a symbol sequence with respect to ordinal symbolic dynamics with fixed order $d\in \mathbb{N}$ and word length $t\in \mathbb{N}$.

Clearly, Figure 7 emphasizes common problems of time series analysis, which have to be faced, as, for example, the trade-off between computational capacity and computation accuracy, which includes undersampling problems, the choice of parameters, stationarity assumptions, and so forth (see for instance Keller et al. [19]). A discussion of all these aspects would be beyond the scope of this paper, but is planned for the future.