On Asymptotic Equipartition Property for Stationary Process of Moving Averages

Abstract

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be a stationary process with values in a finite set. In this paper, we present a moving average version of the Shannon–McMillan–Breiman theorem; this generalize the corresponding classical results. A sandwich argument reduced the proof to direct applications of the moving strong law of large numbers. The result generalizes the work by Algoet et. al., while relying on a similar sandwich method. It is worth noting that, in some kind of significance, the indices $a_n$ and $\varphi \left(n\right)$ are symmetrical, i.e., for any integer n, if the growth rate of ${\left(a_n\right)}_{n\in \mathbb{Z}}$ is slow enough, all conclusions in this article still hold true.

1. Introduction

Information theory is mainly concerned with stationary random processes $X={\left\{X_n\right\}}_{n\in \mathbb{Z}}$, where $X_n$ takes values in a set $\mathcal{X}$, with cardinality $|\mathcal{X}|<\infty$. The strong convergence of the entropy at time n of a random processes divided by n to a constant limit called the entropy rate of the process is known as the ergodic theorem of information theory or the asymptotic equipartition property (AEP) [1], in some sense, of the expression

$$\underset{n\to \infty }{lim}[-\frac{1}{n}logp(X_0,\cdots ,X_{n-1})]\to \mathrm{a}\mathrm{constant}.$$

Its original version proven in the 1950s for ergodic stationary processes is known as the Shannon–McMillan theorem for the convergence in mean and as the Shannon–Breiman–McMillan theorem [2,3,4] for the almost everywhere convergence. Since then, generalized versions of Shannon–McMillan–Breiman’s limit theorem were developed by many authors [1,2,4,5]. Extensions have been made in the direction of weakening the assumptions on the reference measure, state space, index set and required properties of the process. For the general development, please see Girardin [6] and the references therein.

In statistics, smoothing data is to create an approximating function that attempts to capture important patterns in the data, while leaving out noise phenomena. One of the most used smoothing methods is moving average (MA). A number of authors have studied the question of almost everywhere convergence for an invertible transformation of X, which is measure preserving the moving averages, e.g., Akcoglu and Del Junco [7]; Bellow, Jones, and Rosenblatt [8]; Junco and Steele [9]; Schwartz [10]; and Haili and Nair [11]. Recently, Wang and Yang [12,13] proposed a new concept of the generalized entropy density, and established a generalized entropy ergodic theorem for time-nonhomogeneous Markov chain and for non-null stationary processes. Shi, Wang et al. [14] studied the generalized entropy ergodic theorem for nonhomogeneous Markov chains indexed by a binary tree.

Motivated by the work above, in this paper we will give a moving average version of the Shannon–McMillan–Breiman theorem. The results in this paper generalize the results of those in [2]. It is worth noting that, in some sense, the indices $a_n$ and $\varphi \left(n\right)$ are symmetrical. In this paper, we are discussing the so-called forward moving average; if the growth rate of ${\left(a_n\right)}_{n\in \mathbb{Z}}$ w.r.p. to integer n is slow enough, all conclusions in this article still hold true, i.e., the backward moving average is still established.

The method used in showing the main results is the “sandwich” approximation approach of Algoet and Cover [2], which depends strongly on the moving strong law of large numbers: sample entropy is asymptotically sandwiched between two functions whose limits can be determined from the moving SLLN theorem.

This paper is organized as follows. In Section 2, we introduced some necessary preparatory knowledge. To distinguish this from the main conclusion theorem names, we present some required preliminaries and three lemmas. In Section 3, we give the main results and some properties of them are studied in the same section. Also, we give examples of applications.

2. Preliminaries

Throughout this section, let $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$ denote a fixed probability space and let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be a stationary sequence taking values from a finite set $\mathcal{X}=\{1,2,\cdots ,b\}$. For the sequence ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$, denote the partial sequence $X_i,\cdots ,X_j$ by $X_i^j$ and $x_i,\cdots ,x_j$ by $x_i^j$ for $i<j$. Likewise, we write $X_{-\infty }^n$, $x_{-\infty }^n$ for the sequence of ${\left\{X_i\right\}}_{i⩽n}$ and ${\left\{x_i\right\}}_{i⩽n}$, respectively. Let

$$p\left(x_i^j\right)=\mathbb{P}(X_i^j=x_i^j)$$

and

$$p(x_j|x_i^{j-1})=\mathbb{P}(X_j=x_j|X_i^{j-1}=x_i^{j-1})$$

wherever the conditioning event has positive probability. Define random variables

$$p\left(X_i^j\right)\mathrm{and}\mathbb{P}(X_j|X_i^{j-1})$$

by setting $X_j=X_j\left(\omega \right)$ in the corresponding definitions. Since

$$\mathbb{P}(p\left(X_i^j\right)=0)=0$$

the conditional probability makes sense $\mathbb{P}-a.e.$ (i.e., almost everywhere holds true under measure $\mathbb{P}$).

Definition 1

(see e.g., [2]). The canonical Markov approximation of order m to the probability is defined for large $j>m$ as

$$p^{\left[m\right]}\left(X_i^{i+j-1}\right)=p\left(X_i^{i+m-1}\right)\prod _{k=i+m}^{i+j-1}p(X_k|X_{k-m}^{k-1})$$

We will prove a new version of AEP for a stationary process ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$. Before developing the main theme of the paper, we shall need to derive some basic lemmas. Let ${\{a_n,\varphi \left(n\right)\}}_{n\in \mathbb{Z}}$ be a pair of positive integers such that $\varphi \left(n\right)\to \infty$ as $n\to \infty$, and for every $\epsilon >0$, ${\Sigma }_{n=1}^{\infty }{2}^{-\epsilon \varphi \left(n\right)}<\infty$.

Lemma 1.

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be a stationary process with values from a finite set $\mathcal{X}$; then, we have

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}log\frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩽0,\mathbb{P}-a.e.\end{array}$$

(1)

and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}log\frac{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}⩽0,\mathbb{P}-a.e.\end{array}$$

(2)

where the base of the logarithm is taken to be 2.

Proof. 

Let A be the support set of $p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)$; then,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\mathbb{P}}\left[\frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}\right]=& \sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A}\frac{p^{\left[m\right]}\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}·p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\hfill \\ \hfill =& \sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A}{p}^{\left[m\right]}\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\hfill \\ \hfill =& p^{\left[m\right]}\left(A\right)\hfill \\ \hfill ⩽& 1.\hfill \end{array}$$

(3)

where ${\mathbb{E}}_{\mathbb{P}}$ indicates taking expectation under measure $\mathbb{P}$.

Similarly, let $B\left(X_{-\infty }^{a_n-1}\right)$ denote the support set of $p(·|X_{-\infty }^{a_n-1})$. Then, we have

$$\begin{array}{cc}& {\mathbb{E}}_{\mathbb{P}}\left\{\frac{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}\right\}\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\left\{{\mathbb{E}}_{\mathbb{P}}\left[\frac{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}|X_{-\infty }^{a_n-1}\right]\right\}\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\left[\sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in B\left(X_{-\infty }^{a_n-1}\right)}\frac{p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p(x_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}·p(x_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})\right]\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\left[\sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in B\left(X_{-\infty }^{a_n-1}\right)}p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\right]\hfill \\ \hfill ⩽& 1\hfill \end{array}$$

(4)

By Markov’s inequality and Equation (4), we have, for any $\epsilon >0$,

$$\mathbb{P}\left\{\frac{1}{\varphi \left(n\right)}\log \frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩾\epsilon \right\}⩽\frac{1}{2^{\epsilon \varphi \left(n\right)}}$$

Noting that ${\sum }_{n=1}^{\infty }{2}^{-\epsilon \varphi \left(n\right)}<\infty$, we see by the Borel–Cantelli lemma that the event

$$\mathbb{P}\left\{\omega :\frac{1}{\varphi \left(n\right)}log\frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩾\epsilon i.o.\right\}=0$$

By the arbitrariness of $\epsilon$, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\log \frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩽0,\mathbb{P}-a.e.$$

Applying the same arguments using Markov’s inequality to Equation (3), we obtain

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\log \frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}⩽0,\mathbb{P}-a.e.$$

This proves the lemma. □

Lemma 2.

(SLLN for MA): For a stationary stochastic process ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}-\frac{1}{\varphi \left(n\right)}log{p}^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)=H^m,\mathbb{P}-a.e.\end{array}$$

(5)

and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}-\frac{1}{\varphi \left(n\right)}logp(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})=H^{\infty },\mathbb{P}-a.e.\end{array}$$

(6)

where $H^m={\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}\{-logp(X_0|X_{-m}^{-1})\}$, $H^{\infty }={\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}\{-logp(X_0|X_{-\infty }^{-1})\}$.

Proof. 

It is not difficult to verify that ${\mathbb{E}}_{\mathbb{P}}p\left(X_{a_n}^{a_n+m-1}\right)⩽1$. An argument similar to the one used in Lemma 1 shows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+m-1}\right)=0,\mathbb{P}-a.e.\end{array}$$

(7)

Let $s\in [-\frac{1}{2},\frac{1}{2}]\setminus \left\{0\right\}$, and define

$${\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)}(s,\omega )=\frac{2^{-s{\sum }_{k=a_n+m}^{a_n+\varphi \left(n\right)}logp(X_k|X_{k-m}^{k-1})}}{{\prod }_{k=a_n+m}^{a_n+\varphi \left(n\right)-1}{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}[2^{-slog(X_k|X_{k-m}^{k-1})}|X_{k-m}^{k-1}]},n\in \mathbb{Z}$$

Since

$$\begin{array}{cc}& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}{\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)}(s,\omega )\hfill \\ \hfill =& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}\left[{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}({\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)}(s,\omega )|X_{a_n}^{a_n+\varphi \left(n\right)-1})\right]\hfill \\ \hfill =& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}\left[{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}({\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)-1}(s,\omega )·\frac{2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})}}{{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})}|X_{a_n+\varphi \left(n\right)-m}^{a_n+\varphi \left(n\right)-2})}|X_{a_n}^{a_n+\varphi \left(n\right)-2})\right]\hfill \\ \hfill =& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}\left[\frac{{\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)-1}(s,\omega )}{{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})}\right]\hfill \\ & ·{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})}|X_{a_n}^{a_n+\varphi \left(n\right)-2})\hfill \\ \hfill =& \frac{{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}{\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)-1}(s,\omega )}{{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_{a_n+\varphi \left(n\right)}|X_{a_n+\varphi \left(n\right)-m}^{a_n+\varphi \left(n\right)-1})}|X_{a_n+\varphi \left(n\right)-m}^{a_n+\varphi \left(n\right)-1})}\hfill \\ & ·{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})}|X_{a_n+\varphi \left(n\right)-m-1}^{a_n+\varphi \left(n\right)-2})(\mathrm{by}\mathrm{Markov}\mathrm{property})\hfill \\ \hfill =& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}{\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)-1}(s,\omega )\hfill \\ \hfill =& \cdots =1.\hfill \end{array}$$

It is straightforward to show that

$$\begin{array}{c}\hfill \underset{n}{lim\; sup}\frac{1}{\varphi \left(n\right)}log{\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)}(s,\omega )⩽0,{\mathbb{P}}^{\left[m\right]}-a.e.\end{array}$$

(8)

Note that

$$\begin{array}{cc}& \frac{1}{\varphi \left(n\right)}log{\mathrm{\Lambda }}_{a_n,\varphi \left(n\right)}(s,\omega )\hfill \\ \hfill =& \frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}[-slogp(X_k|X_{k-m}^{k-1})-log{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_k|X_{k-m}^{k-1})}|X_{k-m}^{k-1})],\hfill \end{array}$$

(9)

By Equations (8) and (9) and and the property of superior limits, we have

$$\begin{array}{cc}& \underset{n}{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}\left[-slogp(X_k|X_{k-m}^{k-1})\right]\hfill \\ \hfill ⩽& \underset{n}{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}log{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_k|X_{k-m}^{k-1})}|X_{k-m}^{k-1}),{\mathbb{P}}^{\left[m\right]}-a.e.\hfill \end{array}$$

(10)

Setting $s\in (0,\frac{1}{2}]$, dividing both sides of Equation (10) by s, we obtain

$$\begin{array}{cc}& \underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}[-logp(X_k|X_{k-m}^{k-1})]=\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}[-log\prod _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})]\hfill \\ \hfill ⩽& \underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}\frac{1}{s}log{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_k|X_{k-m}^{k-1})}|X_{k-m}^{k-1}),{\mathbb{P}}^{\left[m\right]}-a.e.\hfill \end{array}$$

(11)

Using the inequalities $logx<\frac{x-1}{ln2}(x>0)$ and $0<2^x-1-xln2<\frac{1}{2}{(xln2)}^2{e}^{|xln2|}$,$x\in \mathbf{R}$.

• It follows from Equation (11) that

$$\begin{array}{cc}& \underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}\frac{1}{s}log{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_k|X_{k-m}^{k-1})}|X_{k-m}^{k-1})\hfill \\ \hfill ⩽& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(-logp(X_0|X_{-m}^{-1})\hfill \\ \hfill & +\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}\left[\frac{{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(2^{-slogp(X_k|X_{k-m}^{k-1})}|X_{k-m}^{k-1})-1}{sln2}+{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}(logp(X_k|X_{k-m}^{k-1})|X_{k-m}^{k-1})\right]\hfill \\ \hfill =& H^m+\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}\left\{\frac{{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}\left[(2^{-slogp(X_k|X_{k-m}^{k-1})}-1+sln2logp(X_k|X_{k-m}^{k-1})|X_{k-m}^{k-1})\right]}{sln2}\right\}\hfill \\ \hfill ⩽& H^m+\frac{s}{2ln2}\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+1}^{a_n+\varphi \left(n\right)-1}{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}[{ln}^{2}2{log}^{2}p(X_k|X_{k-m}^{k-1})e^{s|ln2logp(X_k|X_{k-m}^{k-1})|}|X_{k-m}^{k-1}]\hfill \\ \hfill =& H^m+\frac{s}{2ln2}\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+1}^{a_n+\varphi \left(n\right)-1}{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}[{ln}^{2}p(X_k|X_{k-m}^{k-1})e^{s|lnp(X_k|X_{k-m}^{k-1})|}|X_{k-m}^{k-1}]\hfill \\ \hfill ⩽& H^m+\frac{s}{2ln2}\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}\sum _{k=a_n+1}^{a_n+\varphi \left(n\right)-1}{\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}[{ln}^{2}p(X_k|X_{k-m}^{k-1})p^{-\frac{1}{2}}(X_k|X_{k-m}^{k-1})|X_{k-m}^{k-1}],{\mathbb{P}}^{\left[m\right]}-a.e.\hfill \end{array}$$

(12)

By the fact that $max\{t^{\frac{1}{2}}{ln}^{2}t,0⩽t⩽1\}=16e^{-2}$, we have

$$\begin{array}{cc}& {\mathbb{E}}_{{\mathbb{P}}^{\left[m\right]}}[{ln}^{2}p(X_k|X_{k-m}^{k-1})p^{-\frac{1}{2}}(X_k|X_{k-m}^{k-1})|X_{k-m}^{k-1}]\hfill \\ \hfill =& \sum _{j=1}^b{ln}^{2}p(j|X_{k-m}^{k-1})p^{-\frac{1}{2}}(j|X_{k-m}^{k-1})·p(j|X_{k-m}^{k-1})\hfill \\ \hfill =& \sum _{j=1}^b{ln}^{2}p(j|X_{k-m}^{k-1})·p^{\frac{1}{2}}(j|X_{k-m}^{k-1})\hfill \\ \hfill ⩽& 16b{e}^{-2}\hfill \end{array}$$

(13)

From Equations (11)–(13), we have

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}[-log\prod _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})]⩽H^m+8loge·sb{e}^2,{\mathbb{P}}^{\left[m\right]}-a.e.\end{array}$$

(14)

Putting $s\downarrow 0$ in Equation (14), we obtain

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}[-log\prod _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})]⩽H^m,{\mathbb{P}}^{\left[m\right]}-a.e.$$

Replacing $s\in (0,\frac{1}{2}]$ by $s\in [-\frac{1}{2},0)$ in the above argument, we can obtain

$$\underset{n\to \infty }{lim\; inf}\frac{1}{\varphi \left(n\right)}[-log\prod _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})]⩾H^m,{\mathbb{P}}^{\left[m\right]}-a.e.$$

These imply that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{\varphi \left(n\right)}[-log\prod _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})]=H^m,{\mathbb{P}}^{\left[m\right]}-a.e.\end{array}$$

(15)

Note that $\mathbb{P}\ll {\mathbb{P}}^{\left[m\right]}$; therefore, we have, by Equation (15),

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{\varphi \left(n\right)}[-log\prod _{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})]=H^m,\mathbb{P}-a.e.\end{array}$$

(16)

Since ${\mathbb{P}}^{\left[m\right]}(X_{a_n}^{a_n+\varphi \left(n\right)}-1)=p\left(X_{a_n}^{a_n+m-1}\right){\prod }_{k=a_n+m}^{a_n+\varphi \left(n\right)-1}p(X_k|X_{k-m}^{k-1})$, Equation (5) follows immediately from Equations (7) and (16).

• Similarly, let s be a nonzero real number, and define

$${\Delta }_{a_n,\varphi \left(n\right)}(s,\omega )=\frac{2^{-s{\sum }_{k=a_n}^{a_n+\varphi \left(n\right)-1}logp(X_k|X_{-\infty }^{k-1})}}{{\left[{\mathbb{E}}_{\mathbb{P}}(2^{-slog(X_0|X_{-\infty }^{-1})}|X_{-\infty }^{-1})\right]}^{\varphi \left(n\right)}},n\in \mathbb{Z}$$

Note that

$$\begin{array}{cc}& {\mathbb{E}}_{\mathbb{P}}{\Delta }_{a_n,\varphi \left(n\right)}(s,\omega )\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\left[{\mathbb{E}}_{\mathbb{P}}({\Delta }_{a_n,\varphi \left(n\right)}(s,\omega )|X_{-\infty }^{a_n+\varphi \left(n\right)-2})\right]\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\left[{\mathbb{E}}_{\mathbb{P}}({\Delta }_{a_n,\varphi \left(n\right)-1}(s,\omega )·\frac{2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n+\varphi \left(n\right)-2})}}{{\mathbb{E}}_{\mathbb{P}}(2^{-slogp(X_0|X_{-\infty }^{-1})}|X_{-\infty }^{-1})}|X_{-\infty }^{a_n+\varphi \left(n\right)-2})\right]\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\left[\frac{{\Delta }_{a_n,\varphi \left(n\right)-1}(s,\omega )}{{\mathbb{E}}_{\mathbb{P}}(2^{-slogp(X_0|X_{-\infty }^{-1})}|X_{-\infty }^{-1})}\right]\hfill \\ & ·{\mathbb{E}}_{\mathbb{P}}(2^{-slogp(X_{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n+\varphi \left(n\right)-1})}|X_{-\infty }^{a_n+\varphi \left(n\right)-2})\hfill \\ \hfill =& \left[\frac{{\mathbb{E}}_{\mathbb{P}}{\Delta }_{a_n,\varphi \left(n\right)-1}(s,\omega )}{{\mathbb{E}}_{\mathbb{P}}(2^{-slogp(X_0|X_{-\infty }^{-1})}|X_{-\infty -1})}\right]\hfill \\ & ·{\mathbb{E}}_{\mathbb{P}}(2^{-slogp(X_0|X_{-\infty }^{-1})}|X_{-\infty }^{-1})(\mathrm{by}\mathrm{stationary})\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}{\Delta }_{a_n,\varphi \left(n\right)-1}(s,\omega )\hfill \\ \hfill =& \cdots =1.\hfill \end{array}$$

The remainder of the argument is analogous to that in proving Equation (5) and is left to the reader. □

Lemma 3.

(No gap): $H^m\searrow H^{\infty }$ and $H=H^{\infty }$.

Proof. 

We know that for stationary precesses $H^m\searrow H$, so it remains to show that $H^m\searrow H^{\infty }$.

Let $Z_0=-logp\left(X_0\right),Z_n=-logp(X_0|X_{-n}^{-1}),n⩾1$. Since $\mathbb{E}\left(Z_n\right)=H(X_0|X_{-n}^{-1})⩽H\left(X_0\right)<\infty ,$ $Z_n$ is integrable. Now, since all random variables are discrete, we may write

$$\begin{array}{cc}\hfill & {\mathbb{E}}_{\mathbb{P}}(Z_{n+1}|X_{-n}^0=x_{-n}^0)\hfill \\ \hfill =& -\sum _{x_{-(n+1)}}p(x_{-(n+1)}|x_{-n}^0)logp(x_0|x_{-(n+1)}^{-1})\hfill \\ \hfill & +\sum _{x_{-(n+1)}}\frac{p\left(x_{-(n+1)}^0\right)}{p\left(x_{-n}^0\right)}log\frac{p\left(x_{-(n+1)}^{-1}\right)}{p\left(x_{-(n+1)}^0\right)}\hfill \\ \hfill ⩽& log\left[\sum _{x_{-(n+1)}}\frac{p\left(x_{-(n+1)}^{-1}\right)}{p\left(x_{-n}^0\right)}\right]\left(Jense{n}^{\prime }sinequality\right)\hfill \\ \hfill =& log\frac{p\left(x_{-n}^{-1}\right)}{p\left(x_{-n}^0\right)}\hfill \\ \hfill =& -logp(x_0|x_{-n}^{-1})\hfill \end{array}$$

Therefore, ${\mathbb{E}}_{\mathbb{P}}(Z_{n+1}|X_{-n}^0)⩽Z_n$ and $Z_n$ is measurable relative to $\sigma -$ field $\sigma (X_0,X_{-1},\cdots ,X_{-n})$; ${\left\{Z_n\right\}}_{n⩾1}$ is a non-negative supermartingale, hence converges $a.e.$ to an integrable limit function for all $x_0\in \mathcal{X}$.

Note that, for any m,

$$\begin{array}{cc}\hfill H^m=& {\mathbb{E}}_{\mathbb{P}}\{-\log p(X_{m+a_n}|X_{a_n}^{a_n+m-1})\}\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\{-logp(X_0|X_{-m}^{-1})\}\hfill \end{array}$$

where the last equation follows from stationarity.

Since $\mathcal{X}$ is finite and $plogp$ is bounded and continuous in p for all $0⩽p⩽1$, the bounded convergence theorem allows interchange of expectation and limit, yielding

$$\begin{array}{cc}& \underset{m\to \infty }{lim}{H}^m\hfill \\ \hfill =& \underset{m\to \infty }{lim}{\mathbb{E}}_{\mathbb{P}}\{-\sum _{x_0\in \mathcal{X}}p(x_0|X_{-m}^{-1})logp(x_0|X_{-m}^{-1})\}\hfill \\ \hfill =& {\mathbb{E}}_{\mathbb{P}}\{-\sum _{x_0\in \mathcal{X}}p(x_0|X_{-\infty }^{-1})logp(x_0|X_{-\infty }^{-1})\}\hfill \\ \hfill =& H^{\infty }\hfill \end{array}$$

Thus, $H^m\searrow H=H^{\infty }$. □

3. Main Results

With the preliminary accounted for, we wish to use the Lemma 1 to conclude that

$$\begin{array}{cc}\hfill -\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1)}\right)& =-\frac{1}{\varphi \left(n\right)}\sum _{i=0}^{\varphi \left(n\right)-1}logp(X_{a_n+i}|X_{a_n}^{a_n+i-1})\hfill \\ & \to \underset{n\to \infty }{lim}{\mathbb{E}}_{\mathbb{P}}(-logp(X_n|X_0^{n-1})\hfill \end{array}$$

(17)

It is not easy to prove Equation (17). However, the closely related quantities $p(X_{a_n+\varphi \left(n\right)}|$$X_{a_n+\varphi \left(n\right)-m}^{a_n+\varphi \left(n\right)-1})$ and $p(X_{a_n+\varphi \left(n\right)}|X_{-\infty }^{a_n+\varphi \left(n\right)-1})$ are easily identified as entropy rates.

Recall that the entropy rate is given by

$$H=\underset{m\to \infty }{lim}{H}^m=\underset{n\to \infty }{lim}\frac{1}{n}\sum _{m=0}^{n-1}{H}^m$$

Of course, $H^m\searrow H$ by stationarity and the fact that conditioning does not increase entropy. It will be crucial that $H^m\searrow H=H^{\infty }$.

With the help of the preceding lemmas, we can now prove the following theorem:

Theorem 1.

(AEP) If H is the entropy rate of a finite-valued stationary process ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$, then it holds that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}-\frac{1}{\varphi \left(n\right)}\log p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)=H,\mathbb{P}-a.e.\end{array}$$

(18)

Remark 1.

In the case $a_n\equiv 1,\varphi \left(n\right)=n$, Theorem 1 reduces to the famous Shannon–McMillan–Breiman theorem, which is the fundamental theorem of information theory. Let $a_n=n$; it gives a delayed average version of AEP.

Proof. 

We argue that the sequence of random variables $-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1)}\right)$ is asymptotically sandwiched between the upper bound $H^m$ and the lower bound $H^{\infty }$ for all $m\ge 0$. The AEP will follow since $H^m\to H^{\infty }$ and $H^{\infty }=H$. □

From Lemma 1, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}log\frac{p^{\left[m\right]}\left(X_{a_n}^{a_n+m-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩽0,\mathbb{P}-a.e.$$

which we rewrite, taking the existence of the ${lim}_{n\to \infty }\frac{1}{\varphi \left(n\right)}log{p}^{\left[m\right]}\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)$ into account, as

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}log\frac{1}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩽\underset{n\to \infty }{lim}\frac{1}{\varphi \left(n\right)}log\frac{1}{p^m\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}=H^m$$

for $m=1,2,\cdots .$ Also, from Lemma 1, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}log\frac{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}⩽0,\mathbb{P}-a.e.$$

which we rewrite as

$$\underset{n\to \infty }{lim\; sup}\frac{1}{\varphi \left(n\right)}log\frac{1}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}⩾\underset{n\to \infty }{lim}\frac{1}{\varphi \left(n\right)}\log \frac{1}{p(X_{a_n}^{a_n+\varphi \left(n\right)-1}|X_{-\infty }^{a_n-1})}=H^{\infty }$$

From the definition of $H^{\infty }$ in Lemma 2, we have, by putting together Equations (6) and (7),

$$H^{\infty }⩽\underset{n\to \infty }{lim\; inf}-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)⩽\underset{n\to \infty }{lim\; sup}-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)⩽H^m$$

for all m.

• But, by Lemma 3, $H^m\searrow H^{\infty }=H$. Consequently,

$$\underset{n\to \infty }{lim}-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)=H,\mathbb{P}-a.e.$$

Now, we give some interesting applications of our main results in the next examples.

Example 1.

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be independent, identically distributed random variables drawn from the probability mass function $p\left(x\right)$; then,

$$\underset{n\to \infty }{lim}-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)=H\left(X\right),a.e.$$

Example 2.

Let

$$X=\left\{\begin{array}{cc}1,\hfill & \frac{1}{2};\hfill \\ 2,\hfill & \frac{1}{4};\hfill \\ 3,\hfill & \frac{1}{4}.\hfill \end{array}\right.$$

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be drawn i.i.d. according to this distribution; then,

$$\underset{n\to \infty }{lim}p{\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}^{\frac{1}{\varphi \left(n\right)}}=2^{\frac{1}{4}log6},a.e.$$

Example 3.

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be independent identically distributed random variables drawn according to the probability mass function $p\left(x\right),x\in \mathcal{X}$. Thus, $p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)={\prod }_{i=a_n}^{a_n+\varphi \left(n\right)-1}p\left(x_i\right)$. Let $q\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)={\prod }_{i=a_n}^{a_n+\varphi \left(n\right)-1}q\left(x_i\right)$, where q is another probability mass function on $\mathcal{X}$; then,

$$\underset{n\to \infty }{lim}\frac{1}{\varphi \left(n\right)}log\frac{q\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}{p\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}=D(\mathbf{p}\Vert \mathbf{q}),a.e.$$

where $D(\mathbf{p}\Vert \mathbf{q})$ is the informational divergence between two probability distributions $\mathbf{p}$ and $\mathbf{q}$ on a common alphabet $\mathcal{X}$.

Since convergence almost everywhere implies convergence in probability, Theorem 1 has the following implication:

Definition 2.

The typical set $A_{\epsilon }^{a_n,\varphi \left(n\right)}$ with respect to $\mathbb{P}$ is the set of sequence $\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\in {\mathcal{X}}^{\varphi \left(n\right)}$ with the following properties:

$$\begin{array}{c}\hfill 2^{-\varphi \left(n\right)(H+\epsilon )}⩽p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)⩽2^{-\varphi \left(n\right)(H-\epsilon )}\end{array}$$

(19)

As a consequence of the Theorem 1, we can show that the set $A_{\epsilon }^{(a_n,\varphi \left(n\right))}$ has the following properties:

Proposition 1.

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be independent, identically distributed random variables drawn from the probability mass function $p\left(x\right)$; then,

(1).

If $\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\in A_{\epsilon }^{(a_n,\varphi \left(n\right))}$, then

$$H-\epsilon ⩽-\frac{1}{\varphi \left(n\right)}logp\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)⩽H+\epsilon .$$

(2).

$\mathbb{P}\left(A_{\epsilon }^{(a_n,\varphi \left(n\right))}\right)>1-\epsilon$ for sufficiently large n.

(3).

$|A_{\epsilon }^{(a_n,\varphi \left(n\right))}|⩽2^{\varphi \left(n\right)(H-\epsilon )}$, where $|A|$ denotes the number of elements in set A.

(4).

$|A_{\epsilon }^{(a_n,\varphi \left(n\right))}|⩾(1-\epsilon )2^{\varphi \left(n\right)(H-\epsilon )}$ for sufficiently large n.

Proof. 

The property (1) is immediate from the definition of $A_{\epsilon }^{(a_n,\varphi \left(n\right))}$.

Property (2) follows directly from Theorem 1, since the probability of the event $\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\in A_{\epsilon }^{(a_n,\varphi \left(n\right))}$ tends to 1 as $n\to \infty$.

Thus, for any $\delta >0$, there exists an $n_0$ such that for all $n\ge n_0$, we have

$$\mathbb{P}\{|-\frac{1}{\varphi \left(n\right)}logp\left(X_{a_n}^{a_n+\varphi \left(n\right)-1}\right)-H\left(X\right)|<\epsilon \}>1-\delta .$$

Setting $\delta =\epsilon$, we have the following:

To prove property (3), noticing that

$$\begin{array}{cc}\hfill 1=& \sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in {\mathcal{X}}^{\varphi \left(n\right)}}p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)}\right)\hfill \\ \hfill \ge & \sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A_{\epsilon }^{(a_n,\varphi \left(n\right))})}{A}_{\epsilon }^{(a_n,\varphi \left(n\right))}\hfill \\ \hfill \ge & \sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A_{\epsilon }^{(a_n,\varphi \left(n\right))})}{2}^{-\varphi \left(n\right)\left[H\right(X)+\epsilon ]}\hfill \\ \hfill =& 2^{-\varphi \left(n\right)\left[H\right(X)+\epsilon ]}|A_{\epsilon }^{(a_n,\varphi \left(n\right))}|,\hfill \end{array}$$

where the second inequality follows from Equation (19),

$$|A_{\epsilon }^{(a_n,\varphi \left(n\right))}|⩽2^{\varphi \left(n\right)(H-\epsilon )}.$$

Finally, for sufficiently large n, $\mathbb{P}\left(A_{\epsilon }^{(a_n,\varphi \left(n\right))}\right)>1-\epsilon$,

$$\begin{array}{cc}\hfill 1-\epsilon <& \mathbb{P}\left(A_{\epsilon }^{(a_n,\varphi \left(n\right))}\right)\hfill \\ \hfill \le & \sum _{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A_{\epsilon }^{(a_n,\varphi \left(n\right))})}{2}^{-\varphi \left(n\right)\left[H\right(X)-\epsilon ]}\hfill \\ \hfill =& 2^{-\varphi \left(n\right)\left[H\right(X)-\epsilon ]}|A_{\epsilon }^{(a_n,\varphi \left(n\right))}|,\hfill \end{array}$$

where the second inequality follows from Definition 2. Therefore,

$$|A_{\epsilon }^{(a_n,\varphi \left(n\right))}|⩾(1-\epsilon )2^{\varphi \left(n\right)(H-\epsilon )}.$$

These complete the proof of the proposition. □

Example 4.

Let ${\left\{X_n\right\}}_{n\in \mathbb{Z}}$ be i.i.d. $\sim p\left(x\right),x\in \mathcal{X}$. Let $H=-\sum p\left(x\right)logp\left(x\right)$. Let $A^{a_n,\varphi \left(n\right)}=\{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in {\mathcal{X}}^{\varphi \left(n\right)}:|-\frac{1}{\varphi \left(n\right)}logp\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)-H|⩽ϵ\}$ and $B^{a_n,\varphi \left(n\right)}=\{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in {\mathcal{X}}^{\varphi \left(n\right)}:|\frac{1}{\varphi \left(n\right)}{\Sigma }_{i=a_n}^{a_n+\varphi \left(n\right)-1}{X}_i-E{X}_i|⩽ϵ\}$. Then we have the following:

(1) ${lim}_{n\to \infty }\mathbb{P}\{X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\}=1$;

(2) ${lim}_{n\to \infty }\mathbb{P}\{X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}\}=1$;

(3) $|A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}|\le 2^{\varphi \left(n\right)(H+ϵ)}$, for all n;

(4) $|A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}|\ge \left(\frac{1}{2}\right)2^{\varphi \left(n\right)(H-ϵ)}$, for sufficiently large n.

Proof. 

(1) By Theorem 1, the probability $X_{a_n}^{a_n+\varphi \left(n\right)-1}$ is typical goes to 1.

(2) By the Strong Law of Large Numbers for moving average, we have $\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in B^{a_n,\varphi \left(n\right)})\to 1$. So there exists $ϵ>0$ and $N_1$ such that $\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)})>1-\frac{ϵ}{2}$ for all $n>N_1$, and there exists $N_2$ such that $\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in B^{a_n,\varphi \left(n\right)})>1-\frac{ϵ}{2}$ for all $n>N_2$. So for all $n>max(N_1,N_2)$,

$$\begin{array}{cc}& \mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)})\hfill \\ \hfill =& \mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)})+\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in B^{a_n,\varphi \left(n\right)})\hfill \\ \hfill & -\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\cup B^{a_n,\varphi \left(n\right)})\hfill \\ \hfill >& 1-\frac{ϵ}{2}+1-\frac{ϵ}{2}-1\hfill \\ \hfill =& 1-ϵ\hfill \end{array}$$

So for any $ϵ>0$ there exists $N=max(N_1,N_2)$ such that $\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)})>1-ϵ$ for all $n>N$; therefore, $\mathbb{P}(X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)})\to 1$.

(3) By the law of total probability ${\Sigma }_{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}}p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\le 1$. Also, for $x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}$, from Theorem 1 in the text, $p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\ge 2^{-\varphi \left(n\right)(H+ϵ)}$. Combining these two equations gives

$$\begin{array}{cc}\hfill 1\ge & {\Sigma }_{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}}p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\hfill \\ \hfill \ge & {\Sigma }_{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}}{2}^{-\varphi \left(n\right)(H+ϵ)}\hfill \\ \hfill =& |A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}|2^{-\varphi \left(n\right)(H+ϵ)}.\hfill \end{array}$$

Multiplying through by $2^{\varphi \left(n\right)(H+ϵ)}$ gives the result $|A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}|\le 2^{\varphi \left(n\right)(H+ϵ)}$.

(4) Since from (2) $\mathbb{P}\{X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}\}\to 1$, there exists N such that $\mathbb{P}\{X_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}\}\ge \frac{1}{2}$ for all $n>N$. From Theorem 1 in the text, for $x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}$, $p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\le 2^{-n(H-ϵ)}$. So, combining these two gives

$$\begin{array}{cc}\hfill \frac{1}{2}& \le {\Sigma }_{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}}p\left(x_{a_n}^{a_n+\varphi \left(n\right)-1}\right)\hfill \\ & \le {\Sigma }_{x_{a_n}^{a_n+\varphi \left(n\right)-1}\in A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}}{2}^{-\varphi \left(n\right)(H-ϵ)}\hfill \\ & =|A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}|2^{-\varphi \left(n\right)(H-ϵ)}.\hfill \end{array}$$

Multiplying through by $2^{\varphi \left(n\right)(H-ϵ)}$ gives the result $|A^{a_n,\varphi \left(n\right)}\bigcap B^{a_n,\varphi \left(n\right)}|\ge \left(\frac{1}{2}\right)2^{\varphi \left(n\right)(H-ϵ)}$ for sufficiently large n. □