On a New Characterization of Harris Recurrence for Markov Chains and Processes

Abstract

This paper shows that Harris recurrent Markov chains and processes can be characterized as the class of Markov chains and processes for which there exists a random time T at which the distribution of the chain or process does not depend on its initial condition. In particular, no independence assumptions concerning the post-T process or T play a role in the characterization. Since Harris chains and processes are known to contain infinite sequences of regeneration times exhibiting various independence properties, it follows that the existence of this single T implies the existence of infinitely many times at which regeneration occurs.

1. Introduction

The exploitation of the regenerative structure has a long and successful history in the development of both theory and algorithms for Markov chains and processes, going back to the pioneering work of Doeblin [1] in which the central limit theorem for Markov chains was studied. In the 1970s, Athreya and Ney [2] and Nummelin [3] independently showed how Harris recurrent Markov chains can be viewed as regenerative stochastic processes. Sigman [4] later extended these regeneration ideas to the setting of Harris recurrent Markov processes in continuous time. Important contributions were also made to this literature on regeneration by Vladimir Kalashnikov, both in book form [5] and in various papers published over the course of his research life [6,7,8,9,10,11,12].

In view of Professor Kalashnikov’s major contributions to this research domain, this note also discusses regeneration. In particular, we provide a new characterization of the class of regenerative Markov processes. Specifically, we recall in Section 2 that such processes can be identified with the class of chains and processes that are recurrent in the sense of Harris. The main contribution of the current note is that this class of Markov processes is exactly the class for which there exists a single random time T (not necessarily a randomized stopping time) at which the chain or process has a distribution that does not depend on its initial state, see Theorems 3 and 5. With only this property assumed, the Markov process must in fact then be wide-sense regenerative in discrete time, and one-dependent regenerative in continuous time. A useful review of some of these regeneration concepts can be found in [13]. In particular, when a single such time T exists, such processes then necessarily possess an infinite sequence of randomized stopping times at which the process is identically distributed, and at which the process also exhibits various forms of “cycle independence” relative to that sequence of times. In other words, this seemingly weak property involving a single T is equivalent to the much stronger property that the process regenerates at an infinite sequence of randomized stopping times.

In this sense, there is some similarity to the results of [14], in which it is shown that for general (possibly non-Markov) stochastic processes, the existence of a single wide-sense (or classical) regeneration time implies the existence of an infinite sequence of such wide-sense (or classical) regeneration times. In contrast to their results, the current paper assumes a Markov structure, but makes no independence assumptions related to either T or the post-T process, so makes much weaker demands on T.

2. Main Results

We start with a discussion in the setting of discrete-time Markov chains. Let S be a separable metric space, and suppose that $\mathcal{S}$ is its associated Borel $\sigma$-algebra. Put $\Omega ={\mathcal{S}}^{\infty }$, and let $\mathcal{F}$ be the associated $\sigma$-algebra on $\Omega$ induced by the product topology. For $\omega =(x_i:i\ge 0)\in \Omega$, let $X_i(\omega )=x_i$ (for $i\ge 0$) be the i’th coordinate projection. Given a one-step transition kernel $P:S\times \mathcal{S}\to [0,1]$ and $x\in S$, let $P_x(·)$ be the probability on $\Omega$ under which

$$P_x(X_i\in A_i,1\le i\le n)={\int }_{A_1}\cdots {\int }_{A_n}P(x,d{x}_1)P(x_1,d{x}_2)\cdots P(x_{n-1},d{x}_n)$$

for $A_i\in \mathcal{S}$ ($1\le i\le n$), so that $X=(X_n:n\ge 0)$ is a Markov chain with transition kernel P starting from x.

Definition 1.

We say that P induces a Harris recurrent Markov chain on S if there exists a non-trivial non-negative σ-finite measure η on $(S,\mathcal{S})$ for which $\eta (A)>0$ implies that

$$P_x(X_n\in A\mathit{infinitely}\mathit{often})=1$$

for $x\in S.$

The theory developed by [2,3] showed that Harris recurrence is equivalent to wide-sense regeneration. To state this result, let $\tilde{S}=S\times [0,1]$ and let $\tilde{\mathcal{S}}$ be the associated Borel product $\sigma$-algebra. Put $\tilde{\Omega }={\tilde{S}}^{\infty }$, and let $\tilde{F}$ be the associated Borel product $\sigma$-algebra. For $\tilde{\omega }=((x_i,u_i),i\ge 0)\in \tilde{\Omega },$ let ${\tilde{X}}_i(\tilde{\omega })=x_i$ and $U_i(\tilde{\omega })=u_i$ for $i\ge 0$. Given a transition kernel P on $(S,\mathcal{S})$, we say that the family $({\tilde{P}}_x:x\in S)$ of probabilities on $(\tilde{\Omega },\tilde{\mathcal{F}})$ is consistent with P if for each $x\in S$ and $n\ge 0$,

(i)

${\tilde{P}}_x(({\tilde{X}}_0,\cdots ,{\tilde{X}}_n)\in ·)=P_x((X_0,\cdots ,X_n)\in ·);$

(ii)

$(U_i:i\ge 0)$ is a sequence of independent and identically distributed (iid) random variables (rv’s) under ${\tilde{P}}_x.$

The existence of the sequence $(U_i:i\ge 0)$ on the same probability space $(\tilde{\Omega },\tilde{\mathcal{F}},{\tilde{P}}_x)$ that supports the Markov chain $({\tilde{X}}_n:n\ge 0)$ allows the possibility of constructing random times $(R_n:n\ge 1)$ that exhibit regeneration structure.

Definition 2.

We say that P induces wide-sense regeneration if for some family $({\tilde{P}}_x:x\in S)$ of probabilities on $\tilde{\Omega }$ consistent with P, there exist strictly increasing random times $(R_n:n\ge 1)$ and a probability $\lambda (·)$ on S such that for each $x\in S$ and $n\ge 1,$

(i) 

$R_n$ is independent of $({\tilde{X}}_{R_n+k}:k\ge 0)$ under ${\tilde{P}}_x$;

(ii) 

${\tilde{P}}_x(({\tilde{X}}_{R_n+k}:k\ge 0)\in ·)={\int }_S\lambda (dy)P_y((X_k:k\ge 0)\in ·).$

In the presence of wide-sense regeneration, one may analyze $({\mathbb{E}}_{x}f(X_n):n\ge 0)$ via the use of renewal equations, thereby greatly simplifying the theory of such Markov chains. Refs. [2,3] essentially proved the following result. (They proved the “only if” direction. The proof uses the existence of C-sets to construct the wide-sense regeneration. The C-set construction, in turn, uses the fact that $\mathcal{S}$ is countably generated. This is why we assume that $\mathcal{S}$ is the Borel $\sigma$-algebra of a separable metric space. The converse follows from, for example, our Theorem 3).

Theorem 1.

The transition kernel P induces a Harris recurrent Markov chain if and only if P induces wide-sense regeneration.

A related, but different, form of regeneration is the following.

Definition 3.

We say that P induces one-dependent regeneration if for some family $({\tilde{P}}_x:x\in S)$ on $\tilde{\Omega }$ consistent with P, there exist strictly increasing random times $(R_n:n\ge 1)$ with $R_0=0$, such that for each $x\in S$,

$$((({\tilde{X}}_{R_n+k}:0\le k<R_{n+1}-R_n),R_{n+1}-R_n):n\ge 0)$$

is one-dependent (in n) under ${\tilde{P}}_x$ and

$${\tilde{P}}_x((({\tilde{X}}_{R_n+k}:0\le k<R_{n+1}-R_n),R_{n+1}-R_n)\in ·)$$

is independent of $x\in S$ and $n\ge 1.$

Using the Athreya–Ney–Nummelin regenerative construction, ref. [15] established the “only if” direction of the following result. As for Theorem 1, the other direction follows, for example, from Theorem 3.

Theorem 2.

The transition kernel P induces a Harris recurrent Markov chain if, and only if, P induces one-dependent regeneration.

We now turn to our first new result.

Definition 4.

We say that the random time T on $\tilde{\Omega }$ exhibits distributional invariance with respect to the transition kernel P if there exists a family $({\tilde{P}}_x:x\in S)$ of probabilities on $\tilde{\Omega }$ consistent with P such that the distributions ${\tilde{P}}_x({\tilde{X}}_T\in ·)$ do not depend on $x\in S$.

It is obvious that if P induces either wide-sense regeneration or one-dependent regeneration, then $R_n$ exhibits distributional invariance for $n\ge 1$.

Theorem 3.

The transition kernel P induces a Harris recurrent Markov chain if, and only if, there exists a random time $T\ge 1$ that exhibits distributional invariance with respect to P.

Proof. 

If P induces Harris recurrence, then the random time $R_n$ of Theorem 1 exhibits distributional invariance. On the other hand, if T exhibits distributional invariance with respect to P, then

$$\eta (·)\stackrel{\Delta }{=}{\tilde{P}}_x({\tilde{X}}_T\in ·)$$

does not depend on $x\in S$. Suppose $\eta (A)>0$. Then, for each $x\in S$, there exists $n(x)$ such that

$${\tilde{P}}_x({\tilde{X}}_T\in A,T\le n(x))\ge \eta (A)/2$$

Furthermore, we may select $n(·)$ to be $\mathcal{S}$ measurable. Hence, if $\tilde{T}(A)=inf\{n\ge 0:{\tilde{X}}_n\in A\}$, it follows that

$${\tilde{P}}_x(1\le \tilde{T}(A)\le n(x))\ge {\tilde{P}}_x({\tilde{X}}_T\in A,T\le n(x))\ge \eta (A)/2,$$

for each $x\in S$. Consequently, if $T(A)=inf\{n\ge 0:X_n\in A\}$,

$$P_x(1\le T(A)\le n(x))\ge \eta (A)/2.$$

Put $N_1=n(X_0)$ and $N_{i+1}=n(X_{N_i})$ for $i\ge 1$. Then, $(N_i:i\ge 1)$ is a strictly increasing sequence for which

$$P_x(X_m\in A\mathrm{for}\mathrm{some}m\in \{N_i+1,\cdots ,N_{i+1}\}|X_j:0\le j\le N_i)\ge \eta (A)/2$$

for each $i\ge 1$. It follows from the conditional Borel–Cantelli lemma (see, for example, [16], Corollary 2, p. 324) that

$$P_x(X_m\in A\mathrm{infinitely}\mathrm{often})=1,$$

proving that P induces Harris recurrence. □

As noted in the Introduction, the seemingly weak assumption of existence of a distributionally invariant random time T already implies existence of an entire sequence of wide-sense or one-dependent regeneration times.

We now extend this theory to continuous time. Given a separable metric space S, let $\Omega =D[0,\infty )$ be the space of functions $\omega :[0,\infty )\to S$, such that $\omega (·)$ is right continuous everywhere, with left limits at $t\in (0,\infty )$ (RCLL). For $t\ge 0$, put $X(t,\omega )=\omega (t)$, $X=(X(t):t\ge 0)$, and $({\theta }_{t}X)(\omega )=(\omega (t+s):s\ge 0).$ Let $(P_x:x\in S)$ be a family of probabilities on $\Omega$ for which $P_x(X(0)=x)=1$ for $x\in S$. Furthermore, we require that for each $x\in S$ and non-negative (measurable) f,

$${\mathbb{E}}_x\left[f({\theta }_{t}X)|X(u):0\le u\le t\right]=u(X(t))P_{x}a.s.,$$

where $u(y)={\mathbb{E}}_{y}f(X)$ for $y\in S$ and ${\mathbb{E}}_y(·)$ is the expectation associated with $P_y(·).$ It follows that X is a time-homogeneous Markov process under $P_x.$ We further assume that X is a Feller process (i.e., for each bounded continuous $f:S\to \mathbb{R}$ and $t\ge 0$, ${\mathbb{E}}_{x}f(X(t))$ is continuous in x).

We now state the definition of Harris recurrence in continuous time; see, for example, [17].

Definition 5.

The process X and its associated probabilities $(P_x:x\in S)$ on Ω is said to be Harris recurrent in continuous time if there exists a non-trivial non-negative σ-finite measure η for which

$$P_x\left({\int }_0^{\infty }I(X(t)\in A)dt=\infty \right)=1$$

for each $x\in S$ whenever $\eta (A)>0$. (Here, $I(B)$ is the indicator rv corresponding to B).

It has been known since the 1990s that such Harris recurrence implies the existence of one-dependent regeneration times (see [4]). As in discrete time, the statement of this result requires the use of a probability space upon which auxiliary randomization can be defined. To this end, let $\tilde{\Omega }=\Omega \times {[0,1]}^{\infty }.$ For $\tilde{\omega }=(\omega ,(u_i:i\ge 0)),$ let $\tilde{X}(t,\tilde{\omega })=\omega (t)$ and $U_i(\tilde{\omega })=u_i$ for $t\ge 0$ and $i\ge 0$.

We say that the family of probabilites $({\tilde{P}}_x:x\in S)$ on $\tilde{\Omega }$ is consistent with $(P_x:x\in S)$ if for each $x\in S$,

(i)

${\tilde{P}}_x((\tilde{X}(t):t\ge 0)\in ·)=P_x((X(t):t\ge 0)\in ·);$

(ii)

$(U_i:i\ge 0)$ is a sequence of iid rv’s on $[0,1]$ under ${\tilde{P}}_x.$

Definition 6.

We say that X and $(P_x:x\in S)$ induce one-dependent regeneration in continuous time if for some family $({\tilde{P}}_x:x\in S)$ of probabilities on $\tilde{\Omega }$ consistent with $(P_x:x\in S)$, there exists a sequence of strictly increasing random times $(R_n:n\ge 0)$ (with $R_0=0$), such that for each $x\in S$,

$$(((\tilde{X}(R_n+t):0\le t<R_{n+1}-R_n),R_{n+1}-R_n):n\ge 0)$$

is a one-dependent sequence under ${\tilde{P}}_x,$ and

$${\tilde{P}}_x(((\tilde{X}(R_n+t):0\le t<R_{n+1}-R_n),R_{n+1}-R_n)\in ·)$$

is independent of $x\in S$ and $n\ge 1.$

As noted earlier, ref. [4] proved the forward implication of the next theorem. As in discrete time, the separability of S is used to ensure that $\mathcal{S}$ is countably generated (so that the “splitting construction” of Athreya–Ney–Nummelin applies.) The reverse implication is a consequence of (for example) Theorem 5.

Theorem 4.

The process X and the family $(P_x:x\in S)$ is Harris recurrent in continuous time if, and only if, X and $(P_x:x\in S)$ induce one-dependent regeneration in continuous time.

In view of our discrete time discussion, an obvious question is whether Harris recurrence in continuous time implies wide-sense regeneration. This issue, first raised in the 1990s, remains an open question (see, for example, [18]).

Our second major result is the extension of Theorem 3 to continuous time.

Definition 7.

We say that the random time T on $\tilde{\Omega }$ exhibits distributional invariance in continuous time with respect to X and $(P_x:x\in S)$ if there exists a family $({\tilde{P}}_x:x\in S)$ of probabilities on $\tilde{\Omega }$ consistent with $(P_x:x\in S)$ such that the distributions ${\tilde{P}}_x(\tilde{X}(T)\in ·)$ do not depend on $x\in S.$

Theorem 5.

The process X and its probabilities $(P_x:x\in S)$ is Harris recurrent in continuous time if, and only if, there exists a random time T on $\tilde{\Omega }$ exhibiting distributional invariance.

Proof. 

The proof is similar to that of Theorem 3. Put $\eta (·)={\tilde{P}}_x(\tilde{X}(T)\in ·)$. Given A for which $\eta (A)>0$, let $(t(x):x\in S)$ be a measurable function for which

$${\tilde{P}}_x(\tilde{X}(T)\in A,T\le t(x))\ge \eta (A)/2.$$

(1)

Set $T_0=0$ and $T_{i+1}=t(X(T_i))$ for $i\ge 0$. As in discrete time, (1) implies that for each $x\in S$,

$$P_x(X(·)\mathrm{visits}A\mathrm{in}[T_i,T_{i+1})|X(u):u\le T_i)\ge \eta (A)/2$$

for $i\ge 0$; see [19] for a discussion of the measurability of hitting times of generic Borel sets A. The conditional Borel–Cantelli lemma again implies that

$$P_x(X(·)\mathrm{visits}A\mathrm{infinitely}\mathrm{often})=1$$

(2)

for each $x\in S.$ Given (2), Theorem 1 of [20] then implies that X and its probabilities $(P_x:x\in S)$ are Harris recurrent in continuous time. Note that Theorem 1 of [20] requires that X is a Borel right process (strong Markov process with right continuous sample paths). Since every Feller process with RCLL sample paths satisfies the strong Markov property (see, for example, [21], Theorem 3.1, p. 102), the requirement is met.

The forward direction is an immediate consequence of [4]. □

3. A Further Generalization

We conclude this paper with a generalization of Theorem 3, which generalizes the well-known minorization-type characterization of Harris recurrence in [2,3].

Definition 8.

Given a set $C\in \mathcal{S}$ and a random time T on $\tilde{\Omega }$, we say that $(C,T)$ is a minorization pair for the transition kernel P if $P_x(X_n\in C\mathit{for}\mathit{some}n)=1$ for all $x\in S$, and there exists a family $({\tilde{P}}_x:x\in S)$ of probabilities on $\tilde{\Omega }$ consistent with P, such that there exists a non-trivial non-negative measure η, such that ${\tilde{P}}_x({\tilde{X}}_T\in ·)\ge \eta (·)$ for all $x\in C$.

Note that the characterization of Harris recurrence in [2,3] is actually a minorization pair with constant T.

Theorem 6.

The transition kernel P induces a Harris recurrent Markov chain if, and only if, there exists a minorization pair $(C,T)$ for P.

Its proof is similar to that of Theorem 3 and, hence, omitted.

4. Conclusions

We have shown that a Markov chain or process is Harris recurrent if, and only if, there exists a single random time at which the distribution of the chain or process does not depend on its initial condition. This characterization shows that the regeneration property of Markov chains or processes can be extracted from a much weaker property, the existence of a single random time that exhibits distributional invariance. Then we further weaken the assumption so that the well-known minorization-type characterization becomes a special case of our new characterization of Harris recurrence. This implies that one strategy for verifying Harris recurrence when wide-sense regeneration fails is to construct a random time T satisfying Definition 4. Practical strategies for identifying such random times is a direction for future work in the area.