All articles published by MDPI are made immediately available worldwide under an open access license. No special
permission is required to reuse all or part of the article published by MDPI, including figures and tables. For
articles published under an open access Creative Common CC BY license, any part of the article may be reused without
permission provided that the original article is clearly cited. For more information, please refer to
Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature
Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for
future research directions and describes possible research applications.
Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive
positive feedback from the reviewers.
Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world.
Editors select a small number of articles recently published in the journal that they believe will be particularly
interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the
most exciting work published in the various research areas of the journal.
Different authors have shown strong relationships between ordinal pattern based entropies and the Kolmogorov–Sinai entropy, including equality of the latter one and the permutation entropy, the whole picture is however far from being complete. This paper is updating the picture by summarizing some results and discussing some mainly combinatorial aspects behind the dependence of Kolmogorov–Sinai entropy from ordinal pattern distributions on a theoretical level. The paper is more than a review paper. A new statement concerning the conditional permutation entropy will be given as well as a new proof for the fact that the permutation entropy is an upper bound for the Kolmogorov–Sinai entropy. As a main result, general conditions for the permutation entropy being a lower bound for the Kolmogorov–Sinai entropy will be stated. Additionally, a previously introduced method to analyze the relationship between permutation and Kolmogorov–Sinai entropies as well as its limitations will be investigated.
The Kolmogorov–Sinai entropy is a central measure for quantifying the complexity of a measure-preserving dynamical system. Although it is easy from the conceptional viewpoint, its determination and its estimation from given data can be challenging. Since Bandt, Keller, and Pompe showed the coincidence of Kolmogorov–Sinai entropy and permutation entropy for interval maps (see ), there have been different attempts to approach the Kolmogorov–Sinai entropy by ordinal pattern based entropies (see e.g., [2,3,4,5,6] and references therein), leading to a nice subject of study. In this paper, we want to discuss the relationship of the Kolmogorov–Sinai entropy to the latter kind of entropies. We respond to the state of the art and give some generalizations and new results, mainly emphasizing combinatorial aspects.
For this, let be a measure-preserving dynamical system, which we think to be fixed in the whole paper. Here, is a probability space equipped with a -- measurable map satisfying for all . Certain properties of the system will be specified at the places where they are of interest. It is suggested for the following to interpret as the set of states of a system, as their distribution, and T as a description of the dynamics underlying the system and saying that the system is in state at time if it is in state at time t.
In the following, we give the definitions of the central entropies considered in this paper.
1.1. The Kolmogorov–Sinai Entropy
The base of quantifying dynamical complexity is to consider the development of partitions and their entropies under the given dynamics. Recall that the coarsest partitions refining given partitions and of are defined by
respectively. The entropy of a finite or countably infinite partition of is given by
For a finite or countably infinite partition of and some , consider the partition
for each multiindex . The entropy rate of T with regard to a finite or countably infinite partition with is defined by
The Kolmogorov–Sinai entropy is then defined as
where the supremum is taken over all finite or over all countably finite partitions with .
1.2. Ordinal Pattern Based Entropy Measures
As the determination and estimation of the Kolmogorov–Sinai entropy based on the given definition are often not easy, there are many different alternative approaches to it, among them the permutation entropy approach by Bandt and Pompe . The latter is built up on the concept of ordinal patterns, which we describe in a general manner now.
For this, let be a random vector for . Here, each of the random variables can be interpreted as an observable measuring some quantity in the following sense: If the system is in state at time 0, then the arschvalue of the quantity mesured at time t provides . This general approach includes the one-dimensional case that states and measurements coincide, and this is that and is the identical map on . This case, originally considered in  and subsequent papers, is discussed in Section 3. We refer to it as the simple one-dimensional case.
be the set of all permutations of length n. We say that a vector has ordinal pattern if
holds true for all . The possible ordinal patterns (compare Figure 1) provide a classification of the vectors. We denote the set of points with ordinal pattern with regard to , respectively, by
the partition of into those sets.
We are especially interested in three ordinal pattern based entropy measures. These are the lower and upper permutation entropies defined as
respectively, and the conditional entropy of ordinal patterns defined by
We speak of the permutation entropy if the upper and lower permutation entropies coincide.
1.3. Outline of This Paper
In Section 2, we will focus on the relationship between permutation and Kolmogorov–Sinai entropies in the general setting. With Theorems 1 and 3, we will restate two known statements. A new proof of Theorem 1 will be given in Appendix A.2. Theorem 3 is stated for completeness. Theorem 2 establishes a new relationship between the conditional permutation entropy and the Kolmogorov–Sinai entropy.
In Section 3, the relationship between permutation and Kolmogorov–Sinai entropies in the one-dimensional case is investigated. Conditions are introduced, under which the permutation entropy is equal to the Kolmogorov–Sinai entropy. The given conditions allow for a generalization of previous results. We will explain why (countably) piecewise monotone functions satisfy these conditions and consider two examples.
In Section 4, we will investigate a method to analyze the relationship between permutation and Kolmogorov–Sinai entropies that was first introduced in . We will use this method to relate two different kinds of conditional permutation entropies in the general setting. Theorem 5 shows that this method cannot be used directly to prove equality between permutation and Kolmogorov–Sinai entropies.
The results of the paper are summarized in Section 5. The proofs for all new results can be found in the Appendix A.
2. Relating Entropies
2.1. Partitions via Ordinal Patterns
Given some and some random vector , the partition described above can be defined in an alternative way, which is a bit more abstract but better fitting for the approach used in the proof of Theorem 4:
We can determine to which set a point belongs to if we know whether holds true for all with . Therefore, we can write
Throughout this paper, we will use the set
to describe the order relation between two points. This notation allows us to write
2.2. Ordinal Characterization of the Kolmogorov–Sinai Entropy
To be able to reconstruct all information of the given system via quantities based on the random vector , we need to assume that the latter itself does not reduce information. From the mathematical viewpoint, this means that the -algebra generated by is equivalent to the originally given -algebra , i.e., that
holds true, which is roughly speaking that orbits are separated by the given random vector. For definitions and some more details concerning -algebras and partitions, see Appendix A.1.
The following statement saying that, under (1), ordinal patterns entailing the complete information of the given system have been shown in  in a s slightly weaker form than given here.
Note that the inequality in (2) is a relatively simple fact: Since the partition is finer than the partition for all , we have
Dividing both sides by n and taking n and subsequently k to infinity proves this inequality.
Proofs of the inequality are also implicitly given in [1,8]. One-dimensional systems with direct observation as considered there are discussed in Section 3 in detail.
We will give a proof of the equality in (2) in Appendix A.2 being alternative to that in .
2.3. Conditional Entropies
In the case that (1) holds and that and coincide, in Appendix A.3, we will prove different representations of the Kolmogorov–Sinai entropy by ordinal pattern based conditional entropies as they are given in the following theorem.
Let be a random vector satisfying (1). If is true, then
holds true for all , in particular, in the case , one has .
2.4. Amigó’S Approach
Amigó et al. [2,8] describe an alternative ordinal way to the Kolmogorov–Sinai entropy, which is based on a refining sequence of finite partitions. We present it in a slightly more general manner as originally given and in the language of finite-valued random variables. Note that the basic result behind Amigo’s approach in [2,8] is that the Kolmogorov–Sinai entropy of a finite alphabet source and its permutation entropy given some order on the alphabet coincide (see also  for an alternative algebraic proof of the statement).
Given a sequence of -valued random variables satisfying
for all ,
for all ,
then it holds
3. The Simple One-Dimensional Case
In the following, we consider the case that is a subset of with coinciding with the Borel -algebra on , and with being the identical map on . The is superfluous here, which is why we leave out each superscript . For example, we write instead of .
3.1. (Countably) Piecewise Monotone Maps
We discuss some generalization of the results of Bandt, Keller, and Pompe that Kolmogorov–Sinai entropy and permutation entropy coincide for interval maps (see ) on the basis of a statement given in the paper . The discussion sheds some light on structural aspects of the proofs given in that paper with some potential for further generalizations.
Let Ω be a subset of and be the Borel σ-algebra on Ω and μ be a probability measure on . Then, we call a partition of Ω ordered (with regard to μ), if and
holds true for all with . Here, denotes the product measure of μ with itself.
We call a map (countably) piecewise monotone (with regard to μ) if there exists a finite (or countably infinite) ordered partition of Ω with such that
holds true for all .
Given a probability space , for two families of sets , we write
if, for all , there exists a with . If and are partitions of in , is equivalent to the fact that for every there exists a set such that and are equal up to some set with measure 0.
Let Ω be a subset of and be the Borel σ-algebra on Ω, and assume that the following conditions are satisfied:
Condition 1:There exists a finite or countably infinite ordered partition with and some with
Condition 2:For all , there exists a finite or countably infinite ordered partition with and
Theorem 4 extracts the two central arguments in proving the main statement of  in the form of Conditions 1 and 2. This statement is given in a slightly stronger form in Corollary 1. In the proof of , the m in Condition 1 is equal to 1. We will discuss in Section 3.2 a situation where Condition 1 with is of interest.
Let Ω be a compact subset of and be the Borel σ-algebra on Ω. If T is (countably) piecewise monotone, then
Since below we directly refer to the main statement in , which assumes compactness, and for simplicity, the Theorem is formulated under this assumption, we however will discuss a relaxation of the assumption in Remark A1.
To prove the above corollary, one needs to verify that Conditions 1 and 2 are satisfied for one-dimensional systems if T is piecewise monotone. It is easy to see that Condition 2 holds true for T being aperiodic and ergodic: If T is aperiodic, for any , one can choose a finite ordered partition such that holds true for all . The ergodicity then implies
One can also show that Condition 2 is true for non-ergodic aperiodic compact systems (see Remark A1 and ).
If T is (countably) piecewise monotone, there exists a finite (or countable infinite) ordered partition with satisfying (4), which is equivalent to
for all . Therefore,
is true for all . Because is an ordered partition, we have
for all . This implies
Hence, Condition 1 holds true if T is (countably) piecewise monotone. To show that Corollary 1 holds true if the dynamical system is not aperiodic, one splits the system into a periodic part and an aperiodic part in the following way:
be the set of periodic points. Assume that is true. Then,
holds true, where (9) is the periodic part of the upper permutation entropy and (10) the aperiodic part. One can use the aperiodic version of Corollary 1 to show that the Kolmogorov–Sinai entropy is an upper bound for (10). The proof of Corollary 1 for non-aperiodic dynamical systems is complete with Lemma A5 in Appendix A.4, which shows that (9) is equal to 0.
In order to illustrate the discussion in Section 3.1, we consider two examples. The first one reflects the situation in Corollary 1, and the second one discusses the case in Condition 1 in Theorem 4.
(Gaussian map).The map with
is called a Gaussian map (see Figure 2a). This map is measure-preserving with regard to the measure μ, which is defined by for all . The partition of is a countably infinite partition into monotony intervals of T satisfying . This map is countably piecewise monotone and ergodic. Thus, its Kolmogorov–Sinai entropy is equal to its permutation entropy.
Consider and the Borel σ-algebra on Ω. Set
The map is defined as piecewise linear on each set (see Figure 2b) by
Let λ be the one-dimensional Lebesgue measure. Define a measure μ on by
for all .
One can verify that T is measure-preserving and ergodic with regard to μ. The partition does satisfy (5) for , but holds true. Therefore, Condition 1 does not hold true for . However, one can show that Condition 1 holds true for and the partition , which implies that the Kolmogorov–Sinai entropy is equal to the permutation entropy of this map due to Theorem 4.
4. A Supplementary Aspect
To determine under what conditions the Kolmogorov–Sinai entropy and the upper or lower permutation entropies coincide remains an open problem in the general case, and in the simple one-dimensional case of maps not being (countably) piecewise monotone the relation of Kolmogorov–Sinai and upper and lower permutation entropies is not completely understood. There is not even known an example where the entropies differ. Finally, we shortly want to discuss a further approach for discussing the relationship of Kolmogorov–Sinai entropy and upper and lower permutation entropies.
In , it was shown that under (1) the Kolmogorov–Sinai entropy is equal to the permutation entropy if roughly speaking the information contents of ‘words’ of k ’successive’ ordinal patterns of large length n is not too far from the information contents of ordinal patterns of length . We want to explain this for the simple one-dimensional case and .
The ordinal pattern of some contains all information on the order relation between the points . When considering the ‘overlapping’ ordinal patterns of and , one has the same information with one exception: The order relation between and is not known a priori. Looking at the related partitions, the missing information is quantified by the conditional entropy . There is one situation reducing this missing information, namely that one of the lies between and . Then, the order relation between and is known by knowing the ordinal patterns of and . Therefore, the following set is of some special interest:
Let Ω be a subset of and be the Borel σ-algebra on Ω. Then,
holds true for all .
This indicates that analyzing the measure of as defined in (11) can be a useful approach to gain inside into the relationship between different kinds of entropies based on ordinal patterns. In particular, the behavior of for is of interest.
Let Ω be a subset of and be the Borel σ-algebra on Ω. If T is ergodic, then
holds true, and, if (stronger) T is mixing, then
The statement under the assumption of mixing has been shown in , and the proof in the ergodic case is given in Appendix A.5.
One can show that in the simple one-dimensional case the Kolmogorov–Sinai entropy is equal to the permutation entropy if
holds true. Using (12), this is the case when is finite, providing a fast decay of the . However, we have as stated in Theorem 5, which will be proved in Appendix A.6.
Let Ω be a subset of , be the Borel σ-algebra on Ω and T be aperiodic and ergodic. Then,
Although formula is false, we cannot answer the question of whether or when (13) is valid. Possibly, an answer to this question, and a better understanding of the kind of decay of the , could be helpful in further investigating the relationship of Kolmogorov–Sinai entropy and upper and lower permutation entropies, at least in the simple one-dimensional ergodic case.
With Theorem 1, we have slightly generalized a statement given in  by removing a technical assumption and using more basic combinatorial arguments. The remaining assumption (1) on the random vector cannot be weakened in general.
In Section 2.3, we have shown that the equality of the permutation entropy and the Kolmogorov–Sinai entropy implies the equality of conditional permutation entropy and Kolmogorov–Sinai entropies as well. We considered two different kinds of conditional permutation entropy, which have turned out to be equal in the cases considered in Section 2.3; it is however not clear whether these two kinds of conditional permutation entropy are equal in the general.
In Section 4, we have established some condition under which these two kinds of conditional entropy are equal, independently from of the equality between permutation and Kolmogorov–Sinai entropies. This condition is based on a concept introduced in  that was originally introduced as a tool for better understanding the relationship between permutation and Kolmogorov–Sinai entropies in a general setting. However, with Theorem 5, we have shown that this tool cannot directly be used to show the equality between permutation and Kolmogorov–Sinai entropies. It is an interesting question of whether and how a clever adaption and improvement of it can allow for new insights in the relationship between permutation and Kolmogorov–Sinai entropies.
In Section 3, we considered the simpler one-dimensional case. With Theorem 4, we have given two conditions under which the permutation entropy is a lower bound for the Kolmogorov–Sinai entropy. This theorem generalizes previous statements in  and slightly generalizes a statement in . One of the conditions (Condition 2) holds true for a large class of dynamical systems, while, for the other one (Condition 1) to hold true, it is necessary that the system is in some sense ’order preserving’. It is still an unsolved and interesting question, whether Condition 1 can be weakened, especially since, to the best of our knowledge, there does not exist a counterexample to the equality of permutation entropy and Kolmogorov–Sinai entropies. Finding a generalization of Theorem 4 to a multidimensional setting is a further interesting question one could ask.
Conceptualization, K.K. and T.G.; Formal analysis, T.G.; Investigation, K.K. and T.G.; Methodology, T.G.; Supervision, K.K.; Visualization, T.G.; Writing—original draft, K.K. and T.G. All authors have read and agreed to the published version of the manuscript.
This research received no external funding.
Conflicts of Interest
The authors declare no conflict of interest.
Appendix A. Proofs
Appendix A.1. Preliminaries
Given a probability space and two -algebras , we write
if for all there exists with . We write
if both and hold true.
For a collection of -valued random variables defined on some measure space , we denote by
the smallest -algebra containing all sets in , where is the Borel -algebra on .
Given a family of disjoint sets and some set , we define
as those sets in that are intersecting Q almost surely.
For two partitions and of in , the number of elements in can be used for an upper bound of the conditional entropy :
Consider the function with . Since f is convex, Jensen’s inequality provides
for all . Using the above inequality implies
This is equivalent to
Appendix A.2. Proof of the Equality in Formula (2)
The proof is based on the following Lemma A1 and Corollary A1.
Let be a random variable and a finite ordered partition of with regard to the image measure . Then, for , and all
Set and label the sets with in such a way that
holds true for all . Since is assumed to be an ordered partition, this is always possible. Set for all so that .
Using the future formula for the entropy rate (see e.g., ), we can write
for all . This implies
for all . □
Now, Lemmas A2 and A3 provide
The assumption then implies
for all .
Appendix A.4. Proofs for the Simple One-Dimensional Case
This subsection is mainly devoted to the proofs of Theorem 4 and to the proof of Lemma A5 mentioned at the end of Section 3.1. Recall the assumption that is a subset of and the Borel -algebra on . The following lemma is a step to the proof of Theorem 4.
Let Ω be a subset of and be the Borel σ-algebra on Ω. Suppose, there exist an ordered partition and an satisfying (5). Then, for all with and multiindices
Fix and with and . We will show that
holds true for all with using induction over s:
The above statement is trivial for . Suppose that (A5) holds true for some with . We will show that (A5) then holds true for :
In (A6), the induction hypotheses was used.
hold true. This implies
holds true. Notice that
For , we have
If is true, using the fact that is an ordered partition yields
For all other cases, we have
The above observations can be summarized as
In combination with (A7), this provides
Notice that the above Lemma immediately implies
We come now to the proof of Theorem 4, which slightly generalizes a proof given in , where the case was considered. For better readability, we restate this proof with the generalization to arbitrary at the appropriate places within the proof.
Take . According to Conditions 1 and 2, there exist finite or countably infinite ordered partitions , and with and satisfying (5) and (6). Consider the partition
Notice that is again a finite or countably infinite ordered partition with . Using (A1), this implies
where we consider itself as one multiindex and as one index set. Thus,
for all . Lemma A4 provides
Combining (A9) and (A10) yields
For each , we have
The statement of the theorem follows from the fact that can be chosen arbitrarily close to 0.
Let Ω be a subset of and be the Borel σ-algebra on Ω. If T is (countably) piecewise monotone and completely periodic, i.e.,
be the set of all points with period smaller or equal to and
the set of all periodic points. Since , for all there exists with
This provides because can be chosen arbitrarily close to 0. □
To be able to show that Condition 2 holds true under the assumptions of Corollary 1 for non-ergodic systems via ergodic decomposition, one needs to require that is a Lebesgue space. A probability space is called a Lebesgue space if is isomorph to some probability space , where is a complete separable metric space and the completion of the Borel σ-algebra on , i.e., contains additionally all subsets of Borel sets with measure 0. If is a Borel subset, is a Lebesgue space if is complete with regard to μ (see e.g., ).
Alternatively, one can use Rokhlin–Halmos towers to show that Condition 2 holds true for non-ergodic systems (see ). For this approach, it is only necessary to require that Ω is a separable metric space, the Borel σ-algebra on Ω, and an aperiodic map .
Moreover, notice that, in , it was required that Ω is a compact metric space so that μ is regular, which allowed for approximating any set of by a finite union of intervals. However, this is not necessary because the Borel σ-algebra is generated by the algebra containing all sets of the type , where I is an open or closed interval, and every set of a σ-algebra can be approximated by a set of the algebra that generates that σ-algebra (see, e.g., ).
Appendix A.5. Proof of Lemma 1 and the ‘Ergodic Part’ of Lemma 2
Let be a subset of and be the Borel -algebra on . We start with showing Lemma 1. For this, fix some . By its definition, the set can be written as a union of sets in . Notice that
For , consider some . If is true, we can use the transitivity of the order relation to determine the order relation of and from the ordering given by Q. This implies
for all . Thus,
This shows Lemma 1.
To prove the ‘ergodic part’ of Lemma 2, take . Choose an ordered partition of such that holds true for all . This is always possible because was assumed to be aperiodic. Label the sets with in such a way that
holds true for all . Since T is ergodic, there exists an such that
The set consists of all with orbit visiting each of the sets in . Thus, if such lies in with and in for , by definition of , the point must belong to . With a similar argumentation for or , one obtains the following:
holds true for all with . Using the ergodicity of T implies
The Stolz–Cesàro theorem then provides
Since can be choosen arbitrarily close to 0, this implies .
Appendix A.6. Proof of Theorem 5
Let be a subset of and be the Borel -algebra on and consider the set
Since T is -almost surely aperiodic, we have .
We will prove the statement of the theorem by contradiction. Suppose holds true. Using the Borel–Cantelli lemma, this implies
Therefore, there exists a with
for all . Notice that every aperiodic point satisfies . Thus,
Thus, there exists some such that
has a strictly positive measure. Because there exists a countable set with
for some . Using the ergodicity of T, this implies
Thus, in particular, is not empty. Now take some
We have . Additionally, there exists with such that holds true, which is equivalent to . As a consequence,
holds true. This implies that
is smaller or equal to . In particular, is well defined and not infinite. On the other hand,
implies . By construction of m, we have
for all . Hence, holds true, which is a contradiction to
Therefore, cannot be true.
Bandt, C.; Keller, G.; Pompe, B. Entropy of interval maps via permutations. Nonlinearity2002, 15, 1595. [Google Scholar] [CrossRef]
Amigó, J.M. The equality of Kolmogorov–Sinai entropy and metric permutation entropy generalized. Physica D2012, 241, 789–793. [Google Scholar] [CrossRef]
Antoniouk, A.; Keller, K.; Maksymenko, S. Kolmogorov–Sinai entropy via separation properties of order-generated σ-algebras. Discret. Cont. Dyn.-A2014, 34, 1793. [Google Scholar]
Fouda, E.; Koepf, W.; Jacquir, S. The ordinal Kolmogorov–Sinai entropy: A generalized approximation. Commun. Nonlinear Sci.2017, 46, 103–115. [Google Scholar] [CrossRef]
Unakafova, V.; Unakafov, A.; Keller, K. An approach to comparing Kolmogorov–Sinai and permutation entropy. Eur. Phys. J. Spec. Top.2013, 222, 353–361. [Google Scholar] [CrossRef][Green Version]
Gutjahr, T.; Keller, K. Equality of Kolmogorov–Sinai and permutation entropy for one-dimensional maps consisting of countably many monotone parts. Discret. Cont. Dyn.-A2019, 39, 4207. [Google Scholar] [CrossRef][Green Version]
Einsiedler, M.; Ward, T. Ergodic Theory: With a View Towards Number Theory; Graduate Texts in Mathematics; Springer: London, UK, 2010. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely
those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or
the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas,
methods, instructions or products referred to in the content.