Is Natural Language a Perigraphic Process? The Theorem about Facts and Words Revisited

As we discuss, a stationary stochastic process is nonergodic when a random persistent topic can be detected in the infinite random text sampled from the process, whereas we call the process strongly nonergodic when an infinite sequence of independent random bits, called probabilistic facts, is needed to describe this topic completely. Replacing probabilistic facts with an algorithmically random sequence of bits, called algorithmic facts, we adapt this property back to ergodic processes. Subsequently, we call a process perigraphic if the number of algorithmic facts which can be inferred from a finite text sampled from the process grows like a power of the text length. We present a simple example of such a process. Moreover, we demonstrate an assertion which we call the theorem about facts and words. This proposition states that the number of probabilistic or algorithmic facts which can be inferred from a text drawn from a process must be roughly smaller than the number of distinct word-like strings detected in this text by means of the Prediction by Partial Matching (PPM) compression algorithm. We also observe that the number of the word-like strings for a sample of plays by Shakespeare follows an empirical stepwise power law, in a stark contrast to Markov processes. Hence, we suppose that natural language considered as a process is not only non-Markov but also perigraphic.


Introduction
One of the motivating assumptions of information theory [1][2][3] is that communication in natural language can be reasonably modeled as a discrete stationary stochastic process, namely, an infinite sequence of discrete random variables with a well defined time-invariant probability distribution. The same assumption is made in several practical applications of computational linguistics, such as speech recognition [4] or part-of-speech tagging [5]. Whereas state-of-the-art stochastic models of natural language are far from being satisfactory, we may ask a more theoretically oriented question, namely: What can be some general mathematical properties of natural language treated as a stochastic process, in view of empirical data?
In this paper, we will investigate a question of whether it is reasonable to assume that natural language communication is a perigraphic process.
To recall, a stationary process is called ergodic if the relative frequencies of all finite substrings in the infinite text generated by the process converge in the long run with probability one to some constants-the probabilities of the respective strings. Now, some basic linguistic intuition suggests that natural language does not satisfy this property, cf. ( [3], Section 6.4). Namely, we can probably minimal grammar-based codes used in the statement of the theorem about facts and words are not computable in a polynomial time [26]. This precluded an empirical verification of the theory.
To state the relative novelty, in this paper, we are glad to announce a new stronger version of the theorem about facts and words for a somewhat more elegant definition of inferrable facts and the PPM code, which is computable almost in a linear time. For the first time, we also present two cases of the theorem: one for strongly nonergodic processes, applying Shannon information theory, and one for general stationary processes, applying algorithmic information theory. Having these results, we can supplement them finally with a rudimentary discussion of some empirical data.
The organization of this paper is as follows. In Section 2, we discuss some properties of ergodic and nonergodic processes. In Section 3, we define strongly nonergodic processes and we present some examples of them. Analogically, in Section 4, we discuss perigraphic processes. In Section 5, we discuss two versions of the theorem about facts and words. In Section 6, we discuss some empirical data and we suppose that natural language may be a perigraphic process. In Section 7, we offer concluding remarks. Moreover, three appendices follow the body of the paper. In Appendix A, we prove the first part of the theorem about facts and words. In Appendix B, we prove the second part of this theorem. In Appendix C, we show that that the number of inferrable facts for the Santa Fe processes follows a power law.

Ergodic and Nonergodic Processes
We assume that the reader is familiar with some probability measure theory [30]. For a real-valued random variable Y on a probability space (Ω, J , P), we denote its expectation Consider now a discrete stochastic process (X i ) ∞ i=1 = (X 1 , X 2 , . . . ), where random variables X i take values from a set X of countably many distinct symbols, such as letters with which we write down texts in natural language. We denote blocks of consecutive random variables X k j := (X j , . . . , X k ) and symbols x k j := (x j , . . . , x k ). Let us define a binary random variable telling whether some string x n 1 has occurred in sequence (X i ) ∞ i=1 on positions from i to i + n − 1, where The expectation of this random variable, is the probability of the chosen string according to the considered probability measure P, whereas the arithmetic average of consecutive random variables 1 m ∑ m i=1 Φ i (x n 1 ) is the relative frequency of the same string in a finite sequence of random symbols X m+n−1 is called stationary (with respect to a probability measure P) if expectations E Φ i (x n 1 ) do not depend on position i for any string x n 1 . In this case, we have the following well known theorem, which establishes that the limiting relative frequencies of strings x n 1 in infinite sequence (X i ) ∞ i=1 exist almost surely, i.e., with probability 1: Theorem 1 (ergodic theorem, cf. e.g., [31]). For any discrete stationary process (X i ) ∞ i=1 , there exist limits In general, limits Φ(x n 1 ) are random variables depending on a particular value of infinite sequence It is quite natural, however, to require that the relative frequencies of strings Φ(x n 1 ) are almost surely constants, equal to the expectations E Φ i (x n 1 ). Subsequently, process (X i ) ∞ i=1 will be called ergodic (with respect to a probability measure P) if limits Φ(x n 1 ) are almost surely constant for any string x n 1 . The standard definition of an ergodic process is more abstract but is equivalent to this statement ([31], Lemma 7.15).
The following examples of ergodic processes are well known: All IID processes are ergodic.

2.
Process A Markov process is ergodic in particular if For a sufficient and necessary condition, see ( [32], Theorem 7.16).

3.
Process (X i ) ∞ i=1 is called hidden Markov if X i = g(S i ) for a certain Markov process (S i ) ∞ i=1 and a function g. A hidden Markov process is ergodic in particular if the underlying Markov process is ergodic.
Whereas IID and Markov processes are some basic models in probability theory, hidden Markov processes are of practical importance in computational linguistics [4,5]. Hidden Markov processes as considered there usually satisfy condition (8) and therefore they are ergodic.
Let us call a probability measure P stationary or ergodic, respectively, if the process (X i ) ∞ i=1 is stationary or ergodic with respect to the measure P. Suppose that we have a stationary measure P that generates some data (X i ) ∞ i=1 . We can define a new random measure F equal to the relative frequencies of blocks in the data (X i ) ∞ i=1 . It turns out that the measure F is almost surely ergodic. Formally, we have this proposition.
Theorem 2 (cf. ( [33], Theorem 9.10)). Any process (X i ) ∞ i=1 with a stationary measure P is almost surely ergodic with respect to the random measure F given by Moreover, from the random measure F, we can obtain the stationary measure P by integration, P(X n 1 = x n 1 ) = E F(X n 1 = x n 1 ). The following result asserts that this integral representation of measure P is unique. Theorem 3 (ergodic decomposition, cf. ( [33], Theorem 9.12)). Any stationary probability measure P can be represented as where ν is a unique measure on stationary ergodic measures.
In other words, stationary ergodic measures are some building blocks from which we can construct any stationary measure. For a stationary probability measure P, the particular values of the random ergodic measure F are called the ergodic components of measure P.
Consider for instance, a Bernoulli(θ) process with measure where x i ∈ {0, 1} and θ ∈ [0, 1]. This measure will be contrasted with the measure of a mixture Bernoulli process with parameter θ uniformly distributed on interval [0, 1], Measure (11) is a measure of an IID process and is therefore ergodic, whereas measure (12) is a mixture of ergodic measures and hence it is nonergodic.

Strongly Nonergodic Processes
According to our definition, a process is ergodic when the relative frequencies of any strings in a random sample in the long run converge to some constants. Consider now the following thought experiment. Suppose that we select a random book from a library. In [34], it was observed that there is hardly any book that contains both the word lemma and the word love, namely, there are some keywords that are specific to particular topics of texts. We can pursue this idea one little step farther. Counting the relative frequencies of keywords, such as lemma for a text on mathematics and love for a romance, we can effectively recognize the topic of the book. Simply put, the relative frequencies of some keywords will be higher for books concerning some topics, whereas they will be lower for books concerning other topics. Hence, in our thought experiment, we expect that the relative frequencies of keywords are some random variables with values depending on the particular topic of the randomly selected book. Since keywords are just some particular strings, we may conclude that the stochastic process that models natural language should be nonergodic.
The above thought experiment provides another perspective onto nonergodic processes. According to the following theorem, a process is nonergodic when we can effectively distinguish in the limit at least two random topics in it. In the statement, function f : X * → {0, 1, 2} assumes values 0 or 1 when we can identify the topic, whereas it takes value 2 when we are not certain which topic a given text is about.
Theorem 4 (cf. [24]). A stationary discrete process (X i ) ∞ i=1 is nonergodic if and only if there exists a function f : X * → {0, 1, 2} and a binary random variable Z such that 0 < P(Z = 0) < 1 and for any position i ∈ N.
A binary variable Z satisfying condition (13) will be called a probabilistic fact. A probabilistic fact tells which of two topics the infinite text generated by the stationary process is about. It is a kind of a random switch which is preset before we start scanning the infinite text; compare a similar wording in [35]. To keep the proofs simple, here we only give a new elementary proof of the " =⇒ " statement of Theorem 4. The proof of the " ⇐= " part applies some measure theory and follows the idea of Theorem 9 from [24] for strongly nonergodic processes, which we will discuss in the next paragraph.
Proof. (only =⇒ ) Suppose that process (X i ) ∞ i=1 is nonergodic. Then, there exists a string x k 1 such that Φ = E Φ for Φ := Φ(x k 1 ) with some positive probability. Hence, there exists a real number y such that P(Φ = y) = 0 and Define Z := 1{Φ > y} and f (X i+n−1 Since lim n→∞ Φ in = Φ almost surely and Φ satisfies (14), convergence lim n→∞ Z in = Z also holds almost surely. Applying the Lebesgue dominated convergence theorem, we obtain As for books in the natural language, we may have an intuition that the pool of available book topics is extremely large and contains many more topics than just two. For this reason, we may need not a single probabilistic fact Z but rather a sequence of probabilistic facts Z 1 , Z 2 , . . . to specify the topic of a random book completely. Formally, stationary processes requiring an infinite sequence of independent uniformly distributed probabilistic facts to describe the topic of an infinitely long text will be called strongly nonergodic.
As we have stated above, for a strongly nonergodic process, there is an infinite number of independent probabilistic facts (Z k ) ∞ k=1 with a uniform distribution on the set {0, 1}. Formally, these probabilistic facts can be assembled into a single real random variable T = ∑ ∞ k=1 2 −k Z k , which is uniformly distributed on the unit interval [0, 1]. The value of variable T identifies the topic of a random infinite text generated by the stationary process. Thus, for a strongly nonergodic process, we have a continuum of available topics which can be incrementally identified from any sufficiently long text. Put formally, according to Theorem 9 from [24], a stationary process is strongly nonergodic if and only if its shift-invariant σ-field contains a nonatomic sub-σ-field. We note in passing that in [24] strongly nonergodic processes were called uncountable description processes.
In view of Theorem 9 from [24], the mixture Bernoulli process (12) is some example of a strongly nonergodic process. In this case, the parameter θ plays the role of the random variable T = ∑ ∞ k=1 2 −k Z k .
Showing that condition (17) is satisfied for this process in an elementary fashion is a tedious exercise. Hence, let us present now a simpler guiding example of a strongly nonergodic process, which we introduced in [24,25] and called the Santa Fe process. Let (Z k ) ∞ k=1 be a binary IID process with P(Z k = 0) = P(Z k = 1) = 1/2. Let (K i ) ∞ i=1 be an IID process with K i assuming values in natural numbers with a power-law distribution The Santa Fe process with exponent α is a sequence ( are pairs of a random number K i and the corresponding probabilistic fact Z K i . The Santa Fe process is strongly nonergodic since condition (17) holds for example for Simply speaking, function g(k; ·) returns 0 or 1 when an unambiguous value of the second constituent can be read off from pairs x i = (k, ·) and returns 2 when there is some ambiguity. Condition (17) is satisfied since Some salient property of the Santa Fe process is the power law growth of the expected number of probabilistic facts, which can be inferred from a finite text drawn from the process. Consider a strongly nonergodic process (X i ) ∞ i=1 . The set of initial independent probabilistic facts inferrable from a finite text X n 1 will be defined as In other words, we have U(X n 1 ) = {1, 2, . . . , l}, where l is the largest number such that g(k; X n 1 ) = Z k for all k ≤ l. To capture the power-law growth of an arbitrary function s : N → R, we will denote the Hilberg exponent defined hilb n→∞ s(n) := lim sup where log + x := log(x + 1) for x ≥ 0 and log + x := 0 for x < 0, cf. [36]. In contrast to Ref. [36], for technical reasons, we define the Hilberg exponent only for an exponentially sparse subsequence of terms s(2 n ) rather than all terms s(n). Moreover, in [36], the Hilberg exponent was considered only for mutual information s(n) = I(X n 1 ; X 2n n+1 ), defined later in Equation (51). We observe that for the exact power law growth s(n) = n β with β ≥ 0 we have hilb n→∞ s(n) = β. More generally, the Hilberg exponent captures an asymptotic power-law growth of the sequence. As shown in Appendix C, for the Santa Fe process with exponent α, we have the asymptotic power-law growth This property distinguishes the Santa Fe process from the mixture Bernoulli process (12), for which the respective Hilberg exponent is zero, as we discuss in Section 6.

Perigraphic Processes
Is it possible to demonstrate by a statistical investigation of texts that natural language is really strongly nonergodic and satisfies a condition similar to (24)? In the thought experiment described in the beginning of the previous section, we have ignored the issue of constructing an infinitely long text. In reality, every book with a well defined topic is finite. If we want to obtain an unbounded collection of texts, we need to assemble a corpus of different books and it depends on our assembling criteria whether the books in the corpus will concern some persistent random topic. Moreover, if we already have a single infinite sequence of books generated by some stationary source and we estimate probabilities as relative frequencies of blocks of symbols in this sequence, then, by Theorem 2, we will obtain an ergodic probability measure almost surely.
In this situation, we may ask whether the idea of the power-law growth of the number of inferrable probabilistic facts can be translated somehow to the case of ergodic measures. Some straightforward method to apply is to replace the sequence of independent uniformly distributed probabilistic facts (Z k ) ∞ k=1 , being random variables, with an algorithmically random sequence of particular binary digits (z k ) ∞ k=1 . Such digits z k will be called algorithmic facts in contrast to variables Z k being called probabilistic facts.
Let us recall some basic concepts. For a discrete random variable X, let P(X) denote the random variable that takes value P(X = x) when X takes value x. We will introduce the pointwise entropy where log stands for the natural logarithm. The prefix-free Kolmogorov complexity K(u) of a string u is the length of the shortest self-delimiting program written in binary digits that prints out string u ( [37], Chapter 3). K(u) is the founding concept of the algorithmic information theory and is an analogue of the pointwise entropy. To keep our notation analogical to (25), we will write the algorithmic entropy If the probability measure is computable, then the algorithmic entropy is close to the pointwise entropy. On the one hand, by the Shannon-Fano coding for a computable probability measure, the algorithmic entropy is less than the pointwise entropy plus a constant which depends on the probability measure and the dimensionality of the distribution ( [37], Corollary 4.3.1). Formally, where C P ≥ 0 is a certain constant depending on the probability measure P. On the other hand, since the prefix-free Kolmogorov complexity is also the length of a prefix-free code, we have It is also true that H a (X n 1 ) ≥ H(X n 1 ) for sufficiently large n almost surely ( [38], Theorem 3.1). Thus, we have shown that the algorithmic entropy is in some sense close to the pointwise entropy, for a computable probability measure.
Next, we will discuss the difference between probabilistic and algorithmic randomness. Whereas for an IID sequence of random variables (Z k ) ∞ k=1 with P(Z k = 0) = P(Z k = 1) = 1/2 we have similarly an infinite sequence of binary digits (z k ) ∞ k=1 is called algorithmically random (in the Martin-Löf sense) when there exists a constant C ≥ 0 such that for all k ∈ N ( [37], Theorem 3.6.1). The probability that the aforementioned sequence of random variables (Z k ) ∞ k=1 is algorithmically random equals 1-for example by ( [38], Theorem 3.1), so algorithmically random sequences are typical realizations of sequence be a stationary process. We observe that generalizing condition (17) in an algorithmic fashion does not make much sense. Namely, condition is trivially satisfied for any stationary process for a certain computable function g : N × X * → {0, 1, 2} and an algorithmically random sequence (z k ) ∞ k=1 . It turns out so since there exists a computable function ω : is the binary expansion of the halting probability Ω = ∑ ∞ k=1 2 −k Ω k , which is a lower semi-computable algorithmically random sequence ( [37], Section 3.6.2).
In spite of this negative result, the power-law growth of the number of inferrable algorithmic facts corresponds to some nontrivial property. For a computable function g : N × X * → {0, 1, 2} and an algorithmically random sequence of binary digits (z k ) ∞ k=1 , which we will call algorithmic facts, the set of initial algorithmic facts inferrable from a finite text X n 1 will be defined as Subsequently, we will call a process perigraphic if the expected number of algorithmic facts which can be inferred from a finite text sampled from the process grows asymptotically like a power of the text length.
Perigraphic processes can be ergodic. The proof of Theorem A10 from Appendix C can be easily adapted to show that some example of a perigraphic process is the Santa Fe process with sequence (Z k ) ∞ k=1 replaced by an algorithmically random sequence of binary digits (z k ) ∞ k=1 . To be very concrete, the example of a perigraphic process can be process ( where (Ω k ) ∞ k=1 is the binary expansion of the halting probability and (K i ) ∞ i=1 is an IID process with K i assuming values in natural numbers with the power-law distribution (18). This process is not only perigraphic but also IID and hence ergodic.
We can also easily show the following proposition.
Proof. Assume that a perigraphic process (X i ) ∞ i=1 has a computable measure P. By inequalities (A25) and (A26) from Appendix A, we have Since, for a computable measure P, we also have inequality (27), then Since we have obtained a contradiction with the assumption that the process is perigraphic, measure P cannot be computable.

Theorem about Facts and Words
In this section, we will present a result about stationary processes, which we call the theorem about facts and words. This proposition states that the expected number of independent probabilistic or algorithmic facts inferrable from the text drawn from a stationary process must be roughly less than the expected number of distinct word-like strings detectable in the text by a simple procedure involving the PPM compression algorithm. This result states, in particular, that an asymptotic power law growth of the number of inferrable probabilistic or algorithmic facts as a function of the text length produces a statistically measurable effect, namely, an asymptotic power law growth of the number of word-like strings.
To state the theorem about facts and words formally, we need first to discuss the PPM code. The general idea of the PPM code comes from Refs. [6,7], developed independently. This compression scheme was called the PPM code in [7], which stands for "Prediction by Partial Matching" and prevails in the literature, whereas it was called measure R in [6,39]. Whereas Ref. [7] focused on practical applications to data compression and earned most of the fame, in Refs. [6,39], one can find a few results that matter for theoretical considerations. Let us denote strings of symbols x k j := (x j , . . . , x k ), adopting an important convention that x k j is the empty string for k < j. In the following, we consider strings over a finite alphabet, say, x i ∈ X = {1, . . . , D}. We define the frequency of a substring w k 1 in a string x n 1 as Now, we will define the PPM probabilities in a way that is closer to the conventions of paper [6,39] than to the conventions of Ref. [7]. In particular, in Equation (38), we consider frequencies of strings x i i−k and x i−1 i−k in different strings, x i−1 1 and x i−2 1 , respectively, in the numerator and in the denominator to guarantee the proper normalization according to our definition of N(w k 1 |x n 1 ).
Definition 3 (cf. [6,7]). For x n 1 ∈ X n and k ∈ {−1, 0, 1, . . .}, we put Quantity PPM k (x i |x i−1 1 ) is called the conditional PPM probability of order k of symbol x i given string x i−1 1 . Next, we put Quantity PPM k (x n 1 ) is called the PPM probability of order k of string x n 1 . Finally, we put Quantity PPM(x n 1 ) is called the (total) PPM probability of the string x n 1 .
Quantity PPM k (x n 1 ) is an incremental approximation of the unknown true probability of the string x n 1 , assuming that the string has been generated by a Markov process of order k. In contrast, quantity PPM(x n 1 ) is a mixture of such Markov approximations for all finite orders. In general, the PPM probabilities are probability distributions over strings of a fixed length. That is: PPM(x n 1 ) > 0 and ∑ x n 1 ∈X n PPM(x n 1 ) = 1. In the following, we define an analogue of the pointwise entropy Quantity H PPM (x n 1 ) will be called the length of the PPM code for the string x n 1 . By nonnegativity of the Kullback-Leibler divergence, we have for any random block X n 1 that The length of the PPM code or the PPM probability, respectively, have two notable properties. First, the PPM probability is a universal probability, i.e., in the limit, the length of the PPM code consistently estimates the entropy rate of a stationary source. Second, the PPM probability can be effectively computed, i.e., the summation in definition (40) can be rewritten as a finite sum. Let us state these two results formally.
For stationary ergodic processes, the above claim follows by an iterated application of the ergodic theorem as shown, e.g., in Theorem 1.1 from [39] for the measure R, which is a slight modification of the PPM probability. To generalize the claim for nonergodic processes, one can use the ergodic decomposition theorem, but the exact proof requires too large of a theoretical overload to be presented within the framework of this paper. Theorem 7. The PPM probability can be effectively computed, i.e., we have where L(x n 1 ) = max k : N(w k 1 |x n 1 ) > 1 for some w k

(45)
is the maximal repetition of string x n 1 .

Proof.
We have N(x i−1 i−k |x i−2 1 ) = 0 for k > L(x i 1 ). Hence, PPM k (x n 1 ) = D −n for k > L(x n 1 ) and, in view of this, we obtain the claim.
Maximal repetition as a function of a string was studied, e.g., in [40,41]. Since the PPM probability is a computable probability distribution, then, by (27) for a certain constant C PPM , we have H a (X n 1 ) ≤ H PPM (X n 1 ) + 2 log n + C PPM .
Let us denote the length of the PPM code of order k, As we can easily see, the code length H PPM (x n 1 ) is approximately equal to the minimal code length H PPM k (x n 1 ) where the minimization goes over k ∈ {−1, 0, 1, . . .}. Thus, it is meaningful to consider this definition of the PPM order of an arbitrary string.

Proof. It follows by PPM
Let us divert for a short while from the PPM code definition. The set of distinct substrings of length m in string x n 1 is V(m|x n 1 ) := y m 1 : x t+m t+1 = y m 1 for some 0 ≤ t ≤ n − m .
The cardinality of set V(m|x n 1 ) as a function of substring length m is called the subword complexity of string x n 1 [40]. Now let us apply the concept of the PPM order to define some special set of substrings of an arbitrary string x n 1 . The set of distinct PPM words detected in x n 1 will be defined as the set V(m|x n 1 ) for m = G PPM (x n 1 ), i.e., V PPM (x n 1 ) := V(G PPM (X n 1 )|x n 1 ).
Let us define the pointwise mutual information and the algorithmic mutual information Now, we may write down the theorem about facts and words. The theorem states that the Hilberg exponent for the expected number of initial independent inferrable facts is less than the Hilberg exponent for the expected mutual information and this is less than the Hilberg exponent for the expected number of distinct detected PPM words plus the PPM order. (The PPM order is usually much less than the number of distinct PPM words.) Theorem 9 (facts and words I, cf. [25]). Let (X i ) ∞ i=1 be a stationary strongly nonergodic process over a finite alphabet. We have inequalities Proof. The claim follows by conjunction of Theorem A2 from Appendix A and Theorem A8 from Appendix B.
Theorem 9 also has an algorithmic version, for ergodic processes in particular.
Theorem 10 (facts and words II). Let (X i ) ∞ i=1 be a stationary process over a finite alphabet. We have inequalities Proof. The claim follows by conjunction of Theorem A3 from Appendix A and Theorem A8 from Appendix B.
The theorem about facts and words previously proven in [25] differs from Theorem 9 in three aspects. First of all, the theorem in [25] did not apply the concept of the Hilberg exponent and compared lim inf n→∞ with lim sup n→∞ rather than lim sup n→∞ with lim sup n→∞ . Second, the number of inferrable facts was defined as a functional of the process distribution rather than a random variable depending on a particular text. Third, the number of words was defined using a minimal grammar-based code rather than the concept of the PPM order. Minimal grammar-based codes are not computable in a polynomial time in contrast to the PPM order. Thus, we may claim that Theorem 9 is stronger than the theorem about facts and words previously proven in [25]. Moreover, applying Kolmogorov complexity and algorithmic randomness to formulate and prove Theorem 10 is a new idea.
It is an interesting question whether we have an almost sure version of Theorems 9 and 10, namely, whether hilb n→∞ card U(X n 1 ) ≤ hilb n→∞ I(X n 1 ; X 2n n+1 ) ≤ hilb n→∞ G PPM (X n 1 ) + card V PPM (X n 1 ) almost surely for strongly nonergodic processes, or for general stationary processes. We leave this question as an open problem.

Hilberg Exponents and Empirical Data
It is advisable to show that the Hilberg exponents considered in Theorem 9 can assume any value in range [0, 1] and the difference between them can be arbitrarily large. We adopt a convention that the set of inferrable probabilistic facts is empty for ergodic processes, U(X n 1 ) = ∅. With this remark in mind, let us inspect some examples of processes.
First of all, for Markov processes and their strongly nonergodic mixtures, of any order k, but, over a finite alphabet, we have hilb n→∞ E card U(X n 1 ) = hilb n→∞ E I(X n 1 ; X 2n n+1 ) = 0.
This happens to be so since the sufficient statistic of text X n 1 for predicting text X 2n n+1 is the maximum likelihood estimate of the transition matrix, the elements of which can assume at most (n + 1) distinct values. Hence, E I(X n 1 ; X 2n n+1 ) ≤ D k+1 log(n + 1), where D is the cardinality of the alphabet and k is the Markov order of the process. Similarly, it can be shown for these processes that the PPM order satisfies lim n→∞ G PPM (X n 1 ) ≤ k. Hence, the number of PPM words, which satisfies inequality card V PPM (X n 1 ) ≤ D G PPM (X n 1 ) , is also bounded above. In consequence, for Markov processes and their strongly nonergodic mixtures, of any order but over a finite alphabet, we obtain In contrast, Santa Fe processes are strongly nonergodic mixtures of some IID processes over an infinite alphabet. Being mixtures of IID processes over an infinite alphabet, they need not satisfy condition (58). In fact, as shown in [25,29] and Appendix C, for the Santa Fe process with exponent α, we have the asymptotic power-law growth The same equality for the number of inferrable probabilistic facts and the mutual information is also satisfied by a stationary coding of the Santa Fe process into a finite alphabet (see [29]).
Let us also note that, whereas the theorem about facts and words provides an inequality of Hilberg exponents, this inequality can be strict. To provide some substance, in [29], we have constructed a modification of the Santa Fe process that is ergodic and over a finite alphabet. For this modification, we have only the power-law growth of mutual information hilb n→∞ E I(X n 1 ; X 2n n+1 ) = 1/α ∈ (0, 1).
Since, in this case, hilb n→∞ E card U(X n 1 ) = 0, then the difference between the Hilberg exponents for the number of inferrable probabilistic facts and the number of PPM words can be an arbitrary number in range (0, 1). Now, we are in a position to discuss some empirical data. In this case, we cannot directly measure the number of facts and the mutual information, but we can compute the PPM order and count the number of PPM words. In Figure 1, we have presented data for a collection of 35 plays by William Shakespeare (downloaded from the Project Gutenberg, https://www.gutenberg.org/) and a random permutation of characters appearing in this collection of texts. The random permutation of characters is an IID process over a finite alphabet, so, in this case, we obtain In contrast, for the plays of Shakespeare, we seem to have a stepwise power law growth of the number of distinct PPM words. Thus, we may suppose that, for natural language, we have more generally If relationship (62) holds true, then natural language cannot be a Markov process of any order. Moreover, in view of the striking difference between observations (61) and (62), we may suppose that the number of inferrable probabilistic or algorithmic facts for texts in natural language also obeys a power-law growth. Formally speaking, this condition would translate to natural language being strongly nonergodic or perigraphic. We note that this hypothesis arises only as a form of a weak inductive inference since formally we cannot deduce condition (33) from mere condition (62), regardless of the amount of data supporting condition (62).

Conclusions
In this article, a stationary process has been called strongly nonergodic if some persistent random topic can be detected in the process and an infinite number of independent binary random variables, called probabilistic facts, is needed to describe this topic completely. Replacing probabilistic facts with an algorithmically random sequence of bits, called algorithmic facts, we have adapted this property back to ergodic processes. Subsequently, we have called a process perigraphic if the number of algorithmic facts which can be inferred from a finite text sampled from the process grows like a power of the text length.
We have demonstrated an assertion, which we call the theorem about facts and words. This proposition states that the number of independent probabilistic or algorithmic facts which can be inferred from a text drawn from a process must be roughly smaller than the number of distinct word-like strings detected in this text by means of the PPM compression algorithm. We have exhibited two versions of this theorem: one for strongly nonergodic processes, applying the Shannon information theory, and one for ergodic processes, applying the algorithmic information theory.
Subsequently, we have exhibited an empirical observation that the number of distinct word-like strings grows like a stepwise power law for a collection of plays by William Shakespeare, in stark contrast to Markov processes. This observation does not rule out that the number of probabilistic or algorithmic facts inferrable from texts in natural language also grows like a power law. Hence, we have supposed that natural language is a perigraphic process.
We suppose that the path of the future related research should lead through a further analysis of the theorem about facts and words, and demonstrating an almost sure version of this statement. It is also an important, still unresolved question whether theoretical analysis of effective universal coding algorithms and their rates of convergence to the entropy rate can contribute to some definite statements about natural language treated as a stochastic process. We realize that the results of this paper as far as the linguistic theory is concerned may be still too inconclusive. As we see it, the main merit of this paper lies in linking some concepts in the Shannon information theory and the algorithmic information theory and providing some linguistic interpretations of them. where K(x) is the prefix-free Kolmogorov complexity of an object x and K(x|z) is the prefix-free Kolmogorov complexity of an object x given an object z. In the above definitions, x and y must be finite objects (finite texts), whereas z can be also an infinite object (an infinite sequence). If In the following, we will prove a result for Hilberg exponents.
Theorem A1. For a function G : N → R, define J(n) := 2G(n) − G(2n). If the limit lim n→∞ G(n)/n = g exists and is finite, then with an equality if J(2 n ) + > 0 for all but finitely many n.
For any stationary process (X i ) ∞ i=1 over a finite alphabet, there exists a limit called the entropy rate of process (X i ) ∞ i=1 [3]. By (28), (43), and (46), we also have Moreover, for a stationary process, the mutual information satisfies Hence, by Theorem A1, we obtain Subsequently, we will prove the initial parts of Theorems 9 and 10, i.e., the two versions of the theorem about facts and words. The probabilistic statement for strongly nonergodic processes goes first.
Theorem A2 (facts and mutual information I). Let (X i ) ∞ i=1 be a stationary strongly nonergodic process over a finite alphabet. We have inequality Proof. Let us write S n := card U(X n 1 ). Observe that E H(Z S n 1 |S n ) = − ∑ s,w P(S n = s, Z s 1 = w) log P(Z s 1 = w|S n = s) ≥ − ∑ s,w P(S n = s, Z s 1 = w) log P(Z s 1 =w) P(S n =s) = − ∑ s,w P(S n = s, Z s 1 = w) log 2 −s P(S n =s) where the second row of inequalities follows by the maximum entropy bound from ( [3], Lemma 13.5.4). Hence, by the inequality for a measurable function f , we obtain that since the sequence of random variables Z ∞ 1 is a measurable function of the sequence of random variables X ∞ n+1 , as shown in [24,25]. Hence, we have By inequalities (A17) and (A18) and equality (A10), we obtain inequality (A12).
The algorithmic version of the theorem about facts and words follows roughly the same idea, with some necessary adjustments.
Theorem A3 (facts and mutual information II). Let (X i ) ∞ i=1 be a stationary process over a finite alphabet. We have inequality Proof. Let us write S n := card U a (X n 1 ). Observe that H a (S n ) + < 2 log(S n + 1), H a (K(z S n 1 )) + < 2 log(K(z S n 1 ) + 1) where the first row of inequalities follows by the algorithmic randomness of z ∞ 1 , whereas the second and the third row of inequalities follow by the bounds K(n) + < 2 log 2 (n + 1) for n ≥ 0 and K(z k 1 ) + < 2k. Moreover, for any a computable function f , there exists a constant C f ≥ 0 such that H a (x|y) Hence, we obtain that Since −E log(S n + 1) ≥ − log(E S n + 1) by the Jensen inequality, then Now, we observe that since the conditional prefix-free Kolmogorov complexity with the second argument fixed is the length of a prefix-free code. Hence, we have By inequalities (A25) and (A27) and equality (A11), we obtain inequality (A19).

Appendix B. Mutual Information and PPM Words
In this appendix, we will investigate some algebraic properties of the length of the PPM code to be used for proving the second part of the theorem about facts and words. First of all, it can be seen that Expression (A28) can be further rewritten using notation log * n := 0, n = 0, log n! − n log n + n, n ≥ 1, H(n 1 , . . . , n l ) := Then, for k ≥ 0, we define As a result for k ≥ 0, we obtain In the following, we will analyze the terms on the right-hand side of (A34).

Proof.
Observe that H(0, D − 1) = K(0, . . . , 0, D − 1) = 0. Hence, the summation in H PPM 1 k (x n 1 ) can be restricted to u ∈ X k such that N(u|x n−1 1 ) ≥ 1. Consider such a u and write N = N(u|x n−1 1 ) and N a = N(ua|x n 1 ). Since H(n 1 , . . . , n l ) ≥ 0 and K(n 1 , . . . , n l ) ≥ 0 (the second inequality follows by subadditivity of log * n), we obtain first where we use log(1 + x) ≤ x and N < n. On the other hand, function log * n is concave so by ∑ D a=1 N a = N and the Jensen inequality for log * n, we obtain Moreover, function F (N, D) is growing in argument N. Hence, Summing inequalities (A36) and (A39) over u ∈ X k such that N(u|x n 1 ) ≥ 1, we obtain the claim.
The mutual information is defined as a difference of entropies. Replacing the entropy with an arbitrary function H Q (u), we obtain this quantity: Definition A1. The Q pointwise mutual information is defined as We will show that the PPM 0 k pointwise mutual information cannot be positive.
Proof. Write N := ∑ k i=1 ∑ l j=1 n ij , p ij := n ij /N, q i := ∑ l j=1 p ij , and r j := ∑ k i=1 p ij . We observe that which is N times the Kullback-Leibler divergence between distributions p ij and q i r j and thus is nonnegative.
Theorem A6. For k ≥ 0, we have Proof. Consider k ≥ 0. For u ∈ X k and a ∈ X, we have Since the second term on the right-hand side is greater than or equal zero, we may omit it and summing the remaining terms over all u ∈ X k we obtain the claim. Now, we will show that the PPM pointwise mutual information between two parts of a string is roughly bounded above by the cardinality of the PPM vocabulary of the string multiplied by the logarithm of the string length.
Consequently, we may prove the second part of Theorems 9 and 10, i.e., the theorems about facts and words.

Appendix C. Hilberg Exponents for Santa Fe Processes
We begin with a general observation for Hilberg exponents. In [36], this result was discussed only for the Hilberg exponent of mutual information.
Theorem A9 (cf. [36]). For a sequence of random variables Y n ≥ 0, we have hilb n→∞ Y n ≤ hilb n→∞ E Y n almost surely. (A54) Proof. Denote δ := hilb n→∞ E Y n . From the Markov inequality, we have where A < ∞. Hence, by the Borel-Cantelli lemma, we have Y 2 k < 2 k(δ+ ) for all but finitely many n almost surely. Since we can choose arbitrarily small, in particular, we obtain inequality (A54).
In [29,36], it was shown that the Santa Fe process with exponent α satisfies equalities hilb n→∞ I(X 0 −n+1 ; X n 1 ) = 1/α almost surely, hilb n→∞ E I(X 0 −n+1 ; X n 1 ) = 1/α. (A57) We will now show a similar result for the number of probabilistic facts inferrable from the Santa Fe process almost surely and in expectation. Since Santa Fe processes are processes over an infinite alphabet, we cannot apply the theorem about facts and words.
To obtain equality (A59), we invoke Theorem A9 for the lower bound, whereas, for the upper bound, we observe that E card U(X n 1 ) ≤ M n + nP(card U(X n 1 ) ≥ M n ), where the last term decays according to the stretched exponential bound (A63) for M n = Cn 1/α .