Approximating Information Measures for Fields

Abstract

We supply corrected proofs of the invariance of completion and the chain rule for the Shannon information measures of arbitrary fields, as stated by Dębowski in 2009. Our corrected proofs rest on a number of auxiliary approximation results for Shannon information measures, which may be of an independent interest. As also discussed briefly in this article, the generalized calculus of Shannon information measures for fields, including the invariance of completion and the chain rule, is useful in particular for studying the ergodic decomposition of stationary processes and its links with statistical modeling of natural language.

1. Introduction

As it was noticed by Dębowski [1,2,3], a generalized calculus of Shannon information measures for arbitrary fields—initiated by Gelfand et al. [4] and later developed by Dobrushin [5], Pinsker [6], and Wyner [7]—is useful in particular for studying the ergodic decomposition of stationary processes and its links with statistical modeling of natural language. Fulfilling this need, Dębowski [1] has developed the calculus of Shannon information measures for arbitrary fields, relaxing the requirement of regular conditional probability, assumed implicitly by Dobrushin [5] and Pinsker [6]. He has done it unaware of the classical paper by Wyner [7], which pursued exactly the same idea, with some differences due to an independent interest.

Compared to exposition [7], the added value of the paper [1] was considering continuity and invariance of Shannon information measures with respect to completion of fields. Unfortunately, the proof of Theorem 2 in [1] establishing this invariance and the generalized chain rule contains some mistakes and gaps, which we have discovered recently. For this reason, in this article, we would like to provide a correction and a few new auxiliary results which may be of an independent interest. In this way, we will complete the full generalization of Shannon information measures and their properties, which was developed step-by-step by Gelfand et al. [4], Dobrushin [5], Pinsker [6], Wyner [7], and Dębowski [1]. By the way, we will also rediscuss the linguistic motivations of our results.

The preliminaries are as follows. Fix a probability space $(\mathsf{\Omega },\mathcal{J},P)$. Fields are set algebras closed under finite Boolean operations, whereas $\sigma$-fields are assumed to be closed also under countable unions and products. A field is called finite if it has finitely many elements. A finite partition is a finite collection of events ${\left\{B_j\right\}}_{j=1}^J\subset \mathcal{J}$ which are disjoint and whose union equals $\mathsf{\Omega }$. The definition proposed by Wyner [7] and Dębowski [1] independently reads as follows:

Definition 1.

For finite partitions $\alpha ={\left\{A_i\right\}}_{i=1}^I$ and $\beta ={\left\{B_j\right\}}_{j=1}^J$ and a probability measure P, the entropy and mutual information are defined as

$$\begin{array}{cccc}\hfill H_P\left(\alpha \right)& :=\sum _{i=1}^{I}P\left(A_i\right)log\frac{1}{P\left(A_i\right)},\hfill & \hfill I_P(\alpha ;\beta )& :=\sum _{i=1}^I\sum _{j=1}^{J}P(A_i\cap B_j)log\frac{P(A_i\cap B_j)}{P\left(A_i\right)P\left(B_j\right)}.\hfill \end{array}$$

(1)

Subsequently, for an arbitrary field $\mathcal{C}$ and finite partitions α and β, we define the pointwise conditional entropy and mutual information as

$$\begin{array}{cccc}\hfill H_P\left(\alpha \right|\left|\mathcal{C}\right)& :=H_{P(·|\mathcal{C})}\left(\alpha \right),\hfill & \hfill I_P(\alpha ;\beta |\left|\mathcal{C}\right)& :=I_{P(·|\mathcal{C})}(\alpha ;\beta ),\hfill \end{array}$$

(2)

where $P\left(E\right|\mathcal{C})$ is the conditional probability of event $E\in \mathcal{J}$ with respect to the smallest complete σ-field containing $\mathcal{C}$. Subsequently, for arbitrary fields $\mathcal{A}$, $\mathcal{B}$, and $\mathcal{C}$, the (average) conditional entropy and mutual information are defined as

$$\begin{array}{cccc}\hfill H_P\left(\mathcal{A}\right|\mathcal{C})& :=\underset{\alpha \subset \mathcal{A}}{sup}\mathit{E}{}_P{H}_P\left(\alpha \right|\left|\mathcal{C}\right),\hfill & \hfill I_P(\mathcal{A};\mathcal{B}|\mathcal{C})& :=\underset{\alpha \subset \mathcal{A},\beta \subset \mathcal{B}}{sup}\mathit{E}{}_{P}I(\alpha ;\beta |\left|\mathcal{C}\right),\hfill \end{array}$$

(3)

where the supremum is taken over all finite subpartitions and $\mathit{E}{}_{P}X:=\int XdP$ is the expectation. Finally, we define the unconditional entropy $H_P\left(\mathcal{A}\right):=H_P\left(\mathcal{A}\right|\left\{\varnothing ,\mathsf{\Omega }\right\})$ and mutual information $I_P(\mathcal{A};\mathcal{B}):=I_P(\mathcal{A};\mathcal{B}|\left\{\varnothing ,\mathsf{\Omega }\right\})$, as it is generally done in information theory. When the probability measure P is clear from the context, we omit subscript P from all above notations.

Although the above measures, called Shannon information measures, have usually been discussed for $\sigma$-fields, the defining equations (3) also make sense for fields. We observe a number of identities, such as $H\left(\mathcal{A}\right)=I(\mathcal{A};\mathcal{A})$ and $H\left(\mathcal{A}\right|\mathcal{C})=I(\mathcal{A};\mathcal{A}\left|\mathcal{C}\right)$. It is important to stress that Definition 1, in contrast to the earlier expositions by Dobrushin [5] and Pinsker [6], is simpler—as it applies one Radon–Nikodym derivative less—and does not require regular conditional probability, i.e., it does not demand that conditional distribution ${\left(P\left(E\right|\mathcal{C})\right)}_{E\in \mathcal{J}}$ be a probability measure almost surely. In fact, the expressions on the right-hand sides of the equations in (3) are defined for all $\mathcal{A}$, $\mathcal{B}$, and $\mathcal{C}$. No problems arise when conditional probability is not regular since conditional distribution ${\left(P\left(E\right|\mathcal{C})\right)}_{E\in \mathcal{E}}$ restricted to a finite field $\mathcal{E}$ is a probability measure almost surely [8] (Theorem 33.2).

We should admit that in the context of statistical language modeling, the respective probability space is countably generated so regular conditional probability is guaranteed to exist. Thus, for linguistic applications, one might think that expositions [5,6] are sufficient, although for a didactic reason, the approaches proposed by Wyner [7] and Dębowski [1] lead to a simpler and more general calculus of Shannon information measures. Yet, there is a more important reason for Definition 1. Namely, to discuss the ergodic decomposition of entropy rate and excess entropy—some highly relevant results for statistical language modeling, developed in [1] and to be briefly recalled in Section 3—we need the invariance of Shannon information measures with respect to completion of fields. But within the framework of Dobrushin [5] and Pinsker [6], such invariance of completion does not hold for strongly nonergodic processes, which seem to arise quite naturally in statistical modeling of natural language [1,2,3]. Thus, the approach proposed by Wyner [7] and Dębowski [1] is in fact indispensable.

Thus, let us inspect the problem of invariance of Shannon information measures with respect to completion of fields. A $\sigma$-field is called complete, with respect to a given probability measure P, if it contains all sets of outer P-measure 0. Let $\sigma \left(\mathcal{A}\right)$ denote the intersection of all complete $\sigma$-fields containing class $\mathcal{A}$, i.e., $\sigma \left(\mathcal{A}\right)$ is the completion of the generated $\sigma$-field. Let $\mathcal{A}\wedge \mathcal{B}$ denote the intersection of all fields that contain $\mathcal{A}$ and $\mathcal{B}$. Assuming Definition 1, the following statement has been claimed true by Dębowski [1] (Theorem 2):

Theorem 1.

Let $\mathcal{A}$, $\mathcal{B}$, $\mathcal{C}$, and $\mathcal{D}$ be subfields of $\mathcal{J}$.

1. 

$I(\mathcal{A};\mathcal{B}|\mathcal{C})=I(\mathcal{A};\sigma \left(\mathcal{B}\right)\left|\mathcal{C}\right)=I(\mathcal{A};\mathcal{B}|\sigma \left(\mathcal{C}\right))$ (invariance of completion);

2. 

$I(\mathcal{A};\mathcal{B}\wedge \mathcal{C}|\mathcal{D})=I(\mathcal{A};\mathcal{B}\left|\mathcal{D}\right)+I(\mathcal{A};\mathcal{C}|\mathcal{B}\wedge \mathcal{D})$ (chain rule).

The property stated in Theorem 1.1 will be referred to as the invariance of completion. It was not discussed by Wyner [7]. The property stated in Theorem 1.2 is usually referred to as the chain rule or the polymatroid identity. It was proved independently by Wyner [7].

As we have mentioned, the invariance of completion is crucial to prove the ergodic decomposition of the entropy rate and excess entropy of stationary processes. But the proof of the invariance of completion given by Dębowski [1] contains a mistake in the order of quantifiers, and the respective proof of the chain rule is too laconic and contains a gap. For this reason, we would like to supplement the corrected proofs in this article. As we have mentioned, the chain rule was proved by Wyner [7], using an approximation result by Dobrushin [5] and Pinsker [6]. For completeness, we would like to provide a different proof of this approximation result—which follows easily from the invariance of completion—and to supply proofs of both parts of Theorem 1.

The corrected proofs of Theorem 1, to be presented in Section 2, are much longer than the original proofs by Dębowski [1]. In particular, for the sake of proving Theorem 1, we will discuss a few other approximation results, which seem to be of an independent interest. To provide more context for our statements, in Section 3, we will also recall the ergodic decomposition of excess entropy and its application to statistical language modeling.

2. Proofs

Let us write ${\mathcal{B}}_n\uparrow \mathcal{B}$ for a sequence ${\left({\mathcal{B}}_n\right)}_{n\in \mathbb{N}}$ of fields such that ${\mathcal{B}}_1\subset {\mathcal{B}}_2\subset \cdots \subset \mathcal{B}={\bigcup }_{n\in \mathbb{N}}{\mathcal{B}}_n$. ($\mathcal{B}$ need not be a $\sigma$-field.) Our proof of Theorem 1 will rest on a few approximation results and this statement by Dębowski [1] (Theorem 1):

Theorem 2.

Let $\mathcal{A}$, $\mathcal{B}$, ${\mathcal{B}}_n$, and $\mathcal{C}$ be subfields of $\mathcal{J}$.

1. 

$I(\mathcal{A};\mathcal{B}|\mathcal{C})=I(\mathcal{B};\mathcal{A}\left|\mathcal{C}\right)$;

2. 

$I(\mathcal{A};\mathcal{B}|\mathcal{C})\ge 0$ with the equality if and only if $P(A\cap B|\mathcal{C})=P(A\left|\mathcal{C}\right)P\left(B\right|\mathcal{C})$ almost surely for all $A\in \mathcal{A}$ and $B\in \mathcal{B}$;

3. 

$I(\mathcal{A};\mathcal{B}|\mathcal{C})\le min(H\left(\mathcal{A}\right|\mathcal{C}),H(\mathcal{B}\left|\mathcal{C}\right))$;

4. 

$I(\mathcal{A};{\mathcal{B}}_1|\mathcal{C})\le I(\mathcal{A};{\mathcal{B}}_2|\mathcal{C})$ if ${\mathcal{B}}_1\subset {\mathcal{B}}_2$;

5. 

$I(\mathcal{A};{\mathcal{B}}_n|\mathcal{C})\uparrow I(\mathcal{A};\mathcal{B}|\mathcal{C})$ for ${\mathcal{B}}_n\uparrow \mathcal{B}$.

Let $A^c=\mathsf{\Omega }\backslash A$. Subsequently, let us denote the symmetric difference

$$\begin{array}{c}\hfill A▵B:=(A\backslash B)\cup (B\backslash A)=(A\cup B)\backslash (A\cap B).\end{array}$$

(4)

Symmetric difference satisfies the following identities, which will be used:

$$\begin{array}{cc}\hfill A^c▵B^c& =A▵B,\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill A▵B& \subset (A▵C)\cup (C▵B),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill (A\backslash C)▵B& \subset (A▵B)\cup (C\cap B),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill \left(\bigcup _{i\in C}{A}_i\right)▵\left(\bigcup _{i\in C}{B}_i\right)& \subset \bigcup _{i\in C}(A_i▵B_i).\hfill \end{array}$$

(8)

Moreover, we will apply the Bonferroni inequalities

$$\begin{array}{c}\hfill 0\le \sum _{1\le i\le n}P\left(A_i\right)-P\left(\bigcup _{1\le i\le n}{A}_i\right)\le \sum _{1\le i<j\le n}P(A_i\cap A_j)\end{array}$$

(9)

and inequality $P\left(A\right)\le P\left(B\right)+P(A▵B)$.

In the following, we will derive the necessary approximation results. Our point of departure is the following folklore fact.

Theorem 3

(approximation of $\sigma$-fields). For any field $\mathcal{K}$ and any event $G\in \sigma \left(\mathcal{K}\right)$, there is a sequence of events $K_1,K_2,\cdots \in \mathcal{K}$ such that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}P(G▵K_n)=0.\end{array}$$

(10)

Proof. 

Denote the class of sets G that satisfy (10) as $\mathcal{G}$. It is sufficient to show that $\mathcal{G}$ is a complete $\sigma$-field that contains the field $\mathcal{K}$. Clearly, all $G\in \mathcal{K}$ satisfy (10) so $\mathcal{G}\supset \mathcal{K}$. Now, we verify the conditions for $\mathcal{G}$ to be a $\sigma$-field.

• We have $\mathsf{\Omega }\in \mathcal{K}$. Hence, $\mathsf{\Omega }\in \mathcal{G}$.
• For $A\in \mathcal{G}$, consider $K_1,K_2,\cdots \in \mathcal{K}$ such that ${lim}_{n\to \infty }P(A▵K_n)=0$. Then, $A▵K_n=A^c▵K_n^c$, where $K_1^c,K_2^c,\cdots \in \mathcal{K}$. Hence, $A^c\in \mathcal{G}$.
• For $A_1,A_2,\cdots \in \mathcal{G}$, consider events $K_i^n\in \mathcal{K}$ such that $P(A_i▵K_i^n)\le 2^{-n}$. Then,

  $$\begin{array}{c}\hfill P\left(\left(\bigcap _{i=1}^n{A}_i\right)▵\left(\bigcap _{i=1}^n{K}_i^{i+n}\right)\right)\le \sum _{i=1}^{n}P(A_i▵K_i^{i+n})\le 2^{-n}.\end{array}$$

  (11)

  Moreover,

  $$\begin{array}{c}\hfill P\left(\left(\bigcap _{i=1}^{\infty }{A}_i\right)▵\left(\bigcap _{i=1}^n{A}_i\right)\right)=P\left(\bigcap _{i=1}^n{A}_i\right)-P\left(\bigcap _{i=1}^{\infty }{A}_i\right).\end{array}$$

  (12)

  Hence,

  $$\begin{array}{cc}\hfill & P\left(\left(\bigcap _{i=1}^{\infty }{A}_i\right)▵\left(\bigcap _{i=1}^n{K}_i^{i+n}\right)\right)\hfill \\ \hfill & \le P\left(\left(\bigcap _{i=1}^{\infty }{A}_i\right)▵\left(\bigcap _{i=1}^n{A}_i\right)\right)+P\left(\left(\bigcap _{i=1}^n{A}_i\right)▵\left(\bigcap _{i=1}^n{K}_i^{i+n}\right)\right)\hfill \\ \hfill & \le P\left(\bigcap _{i=1}^n{A}_i\right)-P\left(\bigcap _{i=1}^{\infty }{A}_i\right)-2^{-n},\hfill \end{array}$$

  (13)

  which tends to 0 for n going to infinity. Since ${\bigcap }_{i=1}^n{K}_i^{i+n}\in \mathcal{K}$, we thus obtain that ${\bigcap }_{i=1}^{\infty }{A}_i\in \mathcal{G}$.

Completeness of $\sigma$-field $\mathcal{G}$ is straightforward since, for any $A\in \mathcal{G}$ and $P(A▵A^{\prime })=0$, we obtain $A^{\prime }\in \mathcal{G}$ using the same sequence of approximating events in field $\mathcal{K}$ as for event A.  □

The second approximation result is the following bound:

Theorem 4

(continuity of entropy). Fix an $ϵ\in (0,e^{-1}]$ and a field $\mathcal{C}$. For finite partitions $\alpha ={\left\{A_i\right\}}_{i=1}^I$ and ${\alpha }^{\prime }={\left\{A_i^{\prime }\right\}}_{i=1}^I$ such that $P(A_i▵A_i^{\prime })\le ϵ$ for all $i\in \left\{1,\cdots ,I\right\}$, we have

$$\begin{array}{c}\hfill |H\left(\alpha \right|\mathcal{C})-H\left({\alpha }^{\prime }\right|\mathcal{C})|\le I\sqrt{ϵ}log\frac{I}{\sqrt{ϵ}}.\end{array}$$

(14)

Proof. 

We have the expectation $\int P(A_i▵A_i^{\prime }|\mathcal{C})dP=P(A_i▵A_i^{\prime })\le ϵ$. Hence, by the Markov inequality we obtain

$$\begin{array}{c}\hfill P(P(A_i▵A_i^{\prime }|\mathcal{C})\ge \sqrt{ϵ})\le \sqrt{ϵ}.\end{array}$$

(15)

Denote

$$\begin{array}{c}\hfill B=\left(P(A_i▵A_i^{\prime }|\mathcal{C})<\sqrt{ϵ})\mathrm{for}\mathrm{all}i\in \left\{1,\cdots ,I\right\}\right).\end{array}$$

(16)

From the Bonferroni inequality, we obtain $P\left(B^c\right)\le I\sqrt{ϵ}$. Subsequently, we observe that $|H\left(\alpha \right|\left|\mathcal{C}\right)-H({\alpha }^{\prime }\left|\right|\mathcal{C})|\le logI$ holds almost surely. Hence,

$$\begin{array}{cc}|H\left(\alpha \right|\mathcal{C})-H\left({\alpha }^{\prime }\right|\mathcal{C})|\hfill & =|\int \left[H\left(\alpha \right|\mathcal{C})-H\left({\alpha }^{\prime }\right|\mathcal{C})\right]dP|\hfill \\ \hfill & \le P\left(B^c\right)logI+{\int }_B\left|H\left(\alpha \right|\left|\mathcal{C}\right)-H({\alpha }^{\prime }\left|\right|\mathcal{C})\right|dP\hfill \\ \hfill & \le I\sqrt{ϵ}logI+{\int }_B\left|H\left(\alpha \right|\left|\mathcal{C}\right)-H({\alpha }^{\prime }\left|\right|\mathcal{C})\right|dP.\hfill \end{array}$$

(17)

Function $-xlogx$ is subadditive and increasing for $x\in (0,e^{-1}]$. In particular, we have $\left|(x+y)log(x+y)-xlogx\right|\le -ylogy$ for $x,y\ge 0$. Thus, on the event B we obtain

$$\begin{array}{cc}|H\left(\alpha \right|\left|\mathcal{C}\right)-H({\alpha }^{\prime }\left|\right|\mathcal{C})|\hfill & =|\sum _{i=1}^{I}P\left(A_i^{\prime }\right|\mathcal{C})logP\left(A_i^{\prime }\right|\mathcal{C})-\sum _{i=1}^{I}P\left(A_i\right|\mathcal{C})logP\left(A_i\right|\mathcal{C})|\hfill \\ \hfill & \le -\sum _{i=1}^I|P(A_i|\mathcal{C})-P(A_i^{\prime }|\mathcal{C})|log|P(A_i|\mathcal{C})-P(A_i^{\prime }|\mathcal{C})|\hfill \\ \hfill & \le -\sum _{i=1}^{I}P(A_i▵A_i^{\prime }|\mathcal{C})logP(A_i▵A_i^{\prime }|\mathcal{C})\hfill \\ \hfill & \le -I\sqrt{ϵ}log\sqrt{ϵ}\hfill \end{array}$$

(18)

Plugging (18) into (17) yields the claim.  □

Now, we can prove the invariance of completion. Note that

$$\begin{array}{c}\hfill I(\alpha ;\beta |\mathcal{C})=H(\alpha \left|\mathcal{C}\right)+H\left(\beta \right|\mathcal{C})-H(\alpha \wedge \beta \left|\mathcal{C}\right).\end{array}$$

(19)

Proof of Theorem 1.

1 (invariance of completion): Consider some measurable fields $\mathcal{A}$, $\mathcal{B}$, and $\mathcal{C}$. We are going to demonstrate

$$\begin{array}{c}\hfill I(\mathcal{A};\mathcal{B}|\mathcal{C})=I(\mathcal{A};\sigma \left(\mathcal{B}\right)\left|\mathcal{C}\right)=I(\mathcal{A};\mathcal{B}|\sigma \left(\mathcal{C}\right)).\end{array}$$

(20)

Equality $I(\mathcal{A};\mathcal{B}|\mathcal{C})=I(\mathcal{A};\mathcal{B}\left|\sigma \right(\mathcal{C}\left)\right)$ is straightforward since $P\left(A\right|\mathcal{C})=P(A\left|\sigma \right(\mathcal{C}\left)\right)$ almost surely for all $A\in \mathcal{J}$. It remains to prove $I(\mathcal{A};\mathcal{B}|\mathcal{C})=I(\mathcal{A};\sigma \left(\mathcal{B}\right)\left|\mathcal{C}\right)$. For this goal, it suffices to show that for any $ϵ>0$ and any finite partitions $\alpha \subset \mathcal{A}$ and ${\beta }^{\prime }\subset \sigma \left(\mathcal{B}\right)$ there exists a finite partition $\beta \subset \mathcal{B}$ such that

$$\begin{array}{c}\hfill |I(\alpha ;\beta |\mathcal{C})-I(\alpha ;{\beta }^{\prime }|\mathcal{C})|<ϵ.\end{array}$$

(21)

Fix then some $ϵ>0$ and finite partitions $\alpha :={\left\{A_i\right\}}_{i=1}^I\subset \mathcal{A}$ and ${\beta }^{\prime }:={\left\{B_j^{\prime }\right\}}_{j=1}^J\subset \sigma \left(\mathcal{B}\right)$. Invoking Theorem 3, we know that for each $\eta >0$ there exists a class of sets ${\left\{C_j\right\}}_{j=1}^J\subset \mathcal{B}$ which need not be a partition, such that

$$\begin{array}{c}\hfill P(C_j▵B_j^{\prime })\le \eta \end{array}$$

(22)

for all $j\in \left\{1,\cdots ,J\right\}$. Let us put $B_{J+1}^{\prime }:=\varnothing$ and let us construct sets $D_0:=\varnothing$ and $D_j:={\bigcup }_{k=1}^j{C}_k$ for $j\in \left\{1,\cdots ,J\right\}$. Subsequently, we put $B_j:=C_j\backslash D_{j-1}$ for $j\in \left\{1,\cdots ,J\right\}$ and $B_{J+1}:=\mathsf{\Omega }\backslash D_J$. In this way, we obtain a partition $\beta :={\left\{B_j\right\}}_{j=1}^{J+1}\subset \mathcal{B}$.

The next step of the proof is showing an analogue of bound (22) for partitions $\beta$ and ${\beta }^{\prime }$. To begin, for $j\in \left\{1,\cdots ,J\right\}$, we have

$$\begin{array}{cc}P(B_j▵B_j^{\prime })\hfill & =P((C_j\backslash D_{j-1})▵B_j^{\prime })\le P(C_j▵B_j^{\prime })+P(D_{j-1}\cap B_j^{\prime })\hfill \\ \hfill & \le \eta +\sum _{k=1}^{j-1}P(C_k\cap B_j^{\prime })\hfill \\ \hfill & \le \eta +\sum _{k=1}^{j-1}\left[P(B_k^{\prime }\cap B_j^{\prime })+P((C_k\cap B_j^{\prime })▵(B_k^{\prime }\cap B_j^{\prime }))\right]\hfill \\ \hfill & \le \eta +\sum _{k=1}^{j-1}\left[0+P(C_k▵B_k^{\prime })\right]\le j\eta .\hfill \end{array}$$

(23)

Now, we observe for $j,k\in \left\{1,\cdots ,J\right\}$ and $j\ne k$ that

$$\begin{array}{cc}\hfill P(C_j)& \ge P(B_j^{\prime })-P(C_j▵B_j^{\prime })\ge P(B_j^{\prime })-\eta \hfill \\ \hfill P(C_j\cap C_k)& \le P(B_j^{\prime }\cap B_k^{\prime })+P((C_j\cap C_k)▵(B_j^{\prime }\cap B_k^{\prime }))\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & \le 0+P(C_j▵B_j^{\prime })+P(C_k▵B_k^{\prime })\le 2\eta .\hfill \end{array}$$

(25)

Hence, by the Bonferroni inequality we derive

$$\begin{array}{cc}P(B_{J+1}▵B_{J+1}^{\prime })\hfill & =P((\mathsf{\Omega }\backslash D_J)▵\varnothing )=P(\mathsf{\Omega }\backslash D_J)=1-P\left(D_J\right)\hfill \\ \hfill & \le 1-\sum _{1\le j\le J}P\left(C_j\right)+\sum _{1\le j<k\le J}P(C_j\cap C_k)\hfill \\ \hfill & \le 1-\sum _{1\le j\le J}P(B_j^{\prime })+J\eta +\sum _{1\le j<k\le J}2\eta =J^2\eta .\hfill \end{array}$$

(26)

Resuming our bounds, we obtain

$$\begin{array}{cc}\hfill P((A_i\cap B_j)▵(A_i\cap B_j^{\prime }))& \le P(B_j▵B_j^{\prime })\le J^2\eta \hfill \end{array}$$

(27)

for all $i\in \left\{1,\cdots ,I\right\}$ and $j\in \left\{1,\cdots ,J+1\right\}$. Then, invoking Theorem 4 yields

$$\begin{array}{cc}\hfill |I(\alpha ;\beta |\mathcal{C})-I(\alpha ;{\beta }^{\prime }|\mathcal{C})|& \le |H(\alpha \wedge \beta |\mathcal{C})-H(\alpha \wedge {\beta }^{\prime }|\mathcal{C})|+|H\left(\beta \right|\mathcal{C})-H\left({\beta }^{\prime }\right|\mathcal{C})|\hfill \\ \hfill & \le I(J+1)\sqrt{J^2\eta }log\frac{I(J+1)}{\sqrt{J^2\eta }}+(J+1)\sqrt{J^2\eta }log\frac{J+1}{\sqrt{J^2\eta }}.\hfill \end{array}$$

(28)

Taking $\eta$ sufficiently small, we obtain (21), which is the desired claim.  □

Some consequence of the above result is this approximation result proved by Dobrushin [5] and Pinsker [6] and used by Wyner [7] to demonstrate the chain rule. Applying the invariance of completion, we supply a different proof than Dobrushin [5] and Pinsker [6].

Theorem 5

(split of join). Let $\mathcal{A}$, $\mathcal{B}$, $\mathcal{C}$, and $\mathcal{D}$ be subfields of $\mathcal{J}$. We have

$$\begin{array}{c}\hfill I(\mathcal{A};\mathcal{B}\wedge \mathcal{C}|\mathcal{D})=\underset{\alpha \subset \mathcal{A},\beta \subset \mathcal{B},\gamma \subset \mathcal{C}}{sup}\mathit{E}I(\alpha ;\beta \wedge \gamma |\left|\mathcal{D}\right),\end{array}$$

(29)

where the supremum is taken over all finite subpartitions.

Proof. 

Define class

$$\begin{array}{c}\hfill \mathcal{E}:=\bigcup _{\beta \subset \mathcal{B},\gamma \subset \mathcal{C}}\sigma (\beta \wedge \gamma ).\end{array}$$

(30)

It can be easily verified that $\mathcal{E}$ is a field such that $\sigma \left(\mathcal{E}\right)=\sigma (\mathcal{B}\wedge \mathcal{C})$. Thus, for all finite partitions $\beta \subset \mathcal{B}$ and $\gamma \subset \mathcal{C}$ we have $\beta \wedge \gamma \subset \mathcal{E}$. Moreover, by definition of $\mathcal{E}$, for each finite partition $\epsilon \subset \mathcal{E}$ there exists finite partitions $\beta \subset \mathcal{B}$ and $\gamma \subset \mathcal{C}$ such that partition $\beta \wedge \gamma$ is finer than $\epsilon$. Hence, by Theorem 2.4, we obtain in this case,

$$\begin{array}{c}\hfill \mathbf{E}I(\alpha ;\epsilon |\left|\mathcal{D}\right)\le \mathbf{E}I(\alpha ;\beta \wedge \gamma |\left|\mathcal{D}\right)\le I(\alpha ;\mathcal{E}|\mathcal{D}).\end{array}$$

(31)

In consequence, by Theorem 1.1, we obtain the claim

$$\begin{array}{cc}I(\mathcal{A};\mathcal{B}\wedge \mathcal{C}|\mathcal{D})\hfill & =I(\mathcal{A};\mathcal{E}|\mathcal{D})=\underset{\alpha \subset \mathcal{A},\epsilon \subset \mathcal{E}}{sup}\mathbf{E}I(\alpha ;\epsilon |\left|\mathcal{D}\right)\hfill \\ \hfill & =\underset{\alpha \subset \mathcal{A},\beta \subset \mathcal{B},\gamma \subset \mathcal{C}}{sup}\mathbf{E}I(\alpha ;\beta \wedge \gamma |\left|\mathcal{D}\right).\hfill \end{array}$$

(32)

  □

The final approximation result which we need to prove the chain rule is as follows:

Theorem 6

(convergence of conditioning). Let $\alpha ={\left\{A_i\right\}}_{i=1}^I$ be a finite partition and let $\mathcal{C}$ be a field. For each $ϵ>0$, there exists a finite partition ${\gamma }^{\prime }\subset \sigma \left(\mathcal{C}\right)$ such that for any partition $\gamma \subset \sigma \left(\mathcal{C}\right)$ finer than ${\gamma }^{\prime }$ we have

$$\begin{array}{c}\hfill \left|H\left(\alpha \right|\mathcal{C})-H(\alpha \left|\gamma \right)\right|\le ϵ.\end{array}$$

(33)

Proof. 

Fix an $ϵ>0$. For each $n\in \mathbb{N}$ and $A\in \mathcal{J}$, partition

$$\begin{array}{c}\hfill {\gamma }_A:=\left\{\left((k-1)/n<P\left(A\right|\mathcal{C})\le k/n\right):k\in \left\{0,1,\cdots ,n\right\}\right\}\end{array}$$

(34)

is finite and belongs to $\sigma \left(\mathcal{C}\right)$. If we consider partition ${\gamma }^{\prime }:={\bigwedge }_{i=1}^I{\gamma }_{A_i}$, it remains finite and still satisfies ${\gamma }^{\prime }\subset \sigma \left(\mathcal{C}\right)$. Let a partition $\gamma \subset \sigma \left(\mathcal{C}\right)$ be finer than ${\gamma }^{\prime }$. Then,

$$\begin{array}{c}\hfill |P\left(A_i\right|\mathcal{C})-P\left(A_i\right|\gamma )|\le 1/n\end{array}$$

(35)

almost surely for all $i\in \left\{1,\cdots ,I\right\}$. We also observe

$$\begin{array}{cc}\hfill \left|H\left(\alpha \right|\mathcal{C})-H(\alpha \left|\gamma \right)\right|& \le \int \left|H\right(\alpha \left|\right|\mathcal{C})-H(\alpha \left|\right|\gamma )|dP.\hfill \end{array}$$

(36)

We recall that function $-xlogx$ is subadditive and increasing for $x\in (0,e^{-1}]$. In particular, we have $\left|(x+y)log(x+y)-xlogx\right|\le -ylogy$ for $x,y\ge 0$. Hence, for $n\ge e$ we obtain almost surely

$$\begin{array}{cc}\hfill \left|H\left(\alpha \right|\left|\mathcal{C}\right)-H\left(\alpha \right|\left|\gamma \right)\right|& =|\sum _{i=1}^{I}P\left(A_i\right|\mathcal{C})logP\left(A_i\right|\mathcal{C})-\sum _{i=1}^{I}P\left(A_i\right|\gamma )logP\left(A_i\right|\gamma )|\hfill \\ \hfill & \le -\sum _{i=1}^I|P(A_i|\mathcal{C})-P(A_i|\gamma )|log|P\left(A_i\right|\mathcal{C})-P(A_i|\gamma )|\hfill \\ \hfill & \le \frac{Ilogn}{n}.\hfill \end{array}$$

(37)

Taking n so large that $n^{-1}Ilogn\le ϵ$ yields the claim.  □

Taking the above into account, we can demonstrate the chain rule. Our proof essentially follows the ideas of Wyner [7], except for invoking Theorem 6.

Proof of Theorem 1.

2 (chain rule): Let $\mathcal{A}$, $\mathcal{B}$, $\mathcal{C}$, and $\mathcal{D}$ be arbitrary fields, and let $\alpha$, $\beta$, $\gamma$, and $\delta$ be finite partitions. The point of our departure is the chain rule for finite partitions [9] (Equation 2.60)

$$\begin{array}{c}\hfill I(\alpha ;\beta \wedge \gamma )=I(\alpha ;\beta )+I(\alpha ;\gamma |\beta ).\end{array}$$

(38)

By Definition 1 and Theorems 1.1, 5, and 6, conditional mutual information $I(\mathcal{A};\mathcal{B}|\mathcal{C})$ can be approximated by $I(\alpha ;\beta |\gamma )$, where we take appropriate limits of refined finite partitions with a certain care.

In particular, by Theorems 1.1, 5, and 6, taking sufficiently fine finite partitions of arbitrary fields $\mathcal{B}$ and $\mathcal{C}$, the chain rule (38) for finite partitions implies

$$\begin{array}{c}\hfill I(\alpha ;\mathcal{B}\wedge \mathcal{C})=I(\alpha ;\mathcal{B})+I(\alpha ;\mathcal{C}|\mathcal{B}),\end{array}$$

(39)

where all expressions are finite. Hence, we also obtain

$$\begin{array}{cc}\hfill 0& =\left[I(\alpha ;\mathcal{B}\wedge \mathcal{C}\wedge \mathcal{D})-I(\alpha ;\mathcal{D})-I(\alpha ;\mathcal{B}\wedge \mathcal{C}|\mathcal{D})\right]\hfill \\ \hfill & \phantom{=}-\left[I(\alpha ;\mathcal{B}\wedge \mathcal{D})-I(\alpha ;\mathcal{D})-I(\alpha ;\mathcal{B}|\mathcal{D})\right]\hfill \\ \hfill & \phantom{=}-\left[I(\alpha ;\mathcal{B}\wedge \mathcal{C}\wedge \mathcal{D})-I(\alpha ;\mathcal{B}\wedge \mathcal{D})-I(\alpha ;\mathcal{C}|\mathcal{B}\wedge \mathcal{D})\right]\hfill \\ \hfill & =I(\alpha ;\mathcal{B}|\mathcal{D})+I(\alpha ;\mathcal{C}|\mathcal{B}\wedge \mathcal{D})-I(\alpha ;\mathcal{B}\wedge \mathcal{C}|\mathcal{D}),\hfill \end{array}$$

where all expressions are finite. Having established the above claim for a finite partition $\alpha$, we generalize it to

$$\begin{array}{c}\hfill I(\mathcal{A};\mathcal{B}\wedge \mathcal{C}|\mathcal{D})=I(\mathcal{A};\mathcal{B}\left|\mathcal{D}\right)+I(\mathcal{A};\mathcal{C}|\mathcal{B}\wedge \mathcal{D})\end{array}$$

(40)

for an arbitrary field $\mathcal{A}$, taking its appropriately fine finite partitions.  □

3. Applications

This section borrows its statements largely from Dębowski [1,2,3] and is provided only to sketch some context for our research and justify its applicability to statistical language modeling. Let ${\left(X_i\right)}_{i\in \mathbb{Z}}$ be a two-sided infinite stationary process over a countable alphabet $\mathbb{X}$ on a probability space $({\mathbb{X}}^{\mathbb{Z}},{\mathcal{X}}^{\mathbb{Z}},P)$, where $X_k\left({\left({\omega }_i\right)}_{i\in \mathbb{Z}}\right):={\omega }_k$. We denote random blocks $X_j^k:={\left(X_i\right)}_{j\le i\le k}$ and complete $\sigma$-fields ${\mathcal{G}}_j^k:=\sigma (X_j^k)$ generated by them. By the generalized calculus of Shannon information measures, i.e., Theorems 1 and 2, we can define the entropy rate $h_P$ and the excess entropy $E_P$ of process ${\left(X_i\right)}_{i\in \mathbb{Z}}$ as

$$\begin{array}{cc}\hfill h_P& :=\underset{n\to \infty }{lim}{H}_P\left({\mathcal{G}}_0\right|{\mathcal{G}}_{-n}^{-1})=H_P\left({\mathcal{G}}_0\right|{\mathcal{G}}_{-\infty }^{-1})\mathrm{if}\mathbb{X}\mathrm{is}\mathrm{finite},\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill E_P& :=\underset{n\to \infty }{lim}{I}_P({\mathcal{G}}_{-n}^{-1};{\mathcal{G}}_0^{n-1})=I_P({\mathcal{G}}_{-\infty }^{-1};{\mathcal{G}}_0^{\infty }),\hfill \end{array}$$

(42)

see [10] for more background.

Let $T\left({\left({\omega }_i\right)}_{i\in \mathbb{Z}}\right):={\left({\omega }_{i+1}\right)}_{i\in \mathbb{Z}}$ be the shift operation and let $\mathcal{I}:=\left\{A\in {\mathcal{X}}^{\mathbb{Z}}:T^{-1}\left(A\right)=A\right\}$ be the invariant $\sigma$-field. By the Birkhoff ergodic theorem [11], we have $\sigma \left(\mathcal{I}\right)\subset \sigma \left({\mathcal{G}}_{-\infty }\right)\cap \sigma \left({\mathcal{G}}_{\infty }\right)$ for the tail $\sigma$-fields ${\mathcal{G}}_{-\infty }:={\bigcap }_{n=1}^{\infty }{\mathcal{G}}_{-\infty }^{-n}$ and ${\mathcal{G}}_{\infty }:={\bigcap }_{n=1}^{\infty }{\mathcal{G}}_n^{\infty }$. Hence, by Theorems 1 and 2 we further obtain expressions

$$\begin{array}{cc}\hfill h_P& =H_P\left({\mathcal{G}}_0\right|{\mathcal{G}}_{-\infty }^{-1})=H_P\left({\mathcal{G}}_0\right|{\mathcal{G}}_{-\infty }^{-1}\wedge \mathcal{I})\mathrm{if}\mathbb{X}\mathrm{is}\mathrm{finite},\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill E_P& =I_P({\mathcal{G}}_{-\infty }^{-1};{\mathcal{G}}_0^{\infty })=H_P\left(\mathcal{I}\right)+I_P({\mathcal{G}}_{-\infty }^{-1};{\mathcal{G}}_0^{\infty }|\mathcal{I}).\hfill \end{array}$$

(44)

Denoting the conditional probability $F\left(A\right):=P\left(A\right|\mathcal{I})$, which is a random stationary ergodic measure by the ergodic decomposition theorem [12], we notice that $H_P\left({\mathcal{G}}_0\right|{\mathcal{G}}_{-\infty }^{-1}\wedge \mathcal{I})=\mathbf{E}{}_P{H}_F\left({\mathcal{G}}_0\right|{\mathcal{G}}_{-\infty }^{-1})$ and $I_P({\mathcal{G}}_{-\infty }^{-1};{\mathcal{G}}_0^{\infty }|\mathcal{I})=\mathbf{E}{}_P{I}_F({\mathcal{G}}_{-\infty }^{-1};{\mathcal{G}}_0^{\infty })$, and consequently we obtain the ergodic decomposition of the entropy rate and excess entropy, which reads

$$\begin{array}{cc}\hfill h_P& =\mathbf{E}{}_P{h}_F\mathrm{if}\mathbb{X}\mathrm{is}\mathrm{finite},\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill E_P& =H_P\left(\mathcal{I}\right)+\mathbf{E}{}_P{E}_F.\hfill \end{array}$$

(46)

Formulae (45) and (46) were derived by Gray and Davisson [13] and Dębowski [1] respectively. The ergodic decomposition of the entropy rate (45) states that a stationary process is asymptotically deterministic, i.e., $h_P=0$, if and only if almost all its ergodic components are asymptotically deterministic, i.e., $h_F=0$ almost surely. In contrast, the ergodic decomposition of the excess entropy (46) states that a stationary process is infinitary, i.e., $E_P=\infty$, if some of its ergodic components are infinitary, i.e., $E_F=\infty$ with a nonzero probability, or if $H_P\left(\mathcal{I}\right)=\infty$, i.e., if the process is strongly nonergodic in particular, see [14,15].

The linguistic interpretation of the above results is as follows. There is a hypothesis by Hilberg [16] that the excess entropy of natural language is infinite. This hypothesis can be partly confirmed by the original estimates of conditional entropy by Shannon [17], by the power-law decay of the estimates of the entropy rate given by the PPM compression algorithm [18], by the approximately power-law growth of vocabulary called Heaps’ or Herdan’s law [2,3,19,20], and by some other experiments applying neural statistical language models [21,22]. In parallel, Dębowski [1,2,3] supposed that the very large excess entropy in natural language may be caused by the fact that texts in natural language describe some relatively slowly evolving and very complex reality. Indeed, it can be mathematically proved that if the abstract reality described by random texts is unchangeable and infinitely complex, then the resulting stochastic process is strongly nonergodic, i.e., $H_P\left(\mathcal{I}\right)=\infty$ in particular [1,2,3]. Consequently, its excess entropy is infinite by formula (46). We suppose that a similar mechanism may work for natural language, see [23,24,25,26] for further examples of abstract stochastic mechanisms leading to infinitary processes.