Rate of Convergence and Periodicity of the Expected Population Structure of Markov Systems that Live in a General State Space

: In this article we study the asymptotic behaviour of the expected population structure of a Markov system that lives in a general state space (MSGS) and its rate of convergence. We continue with the study of the asymptotic periodicity of the expected population structure. We conclude with the study of total variability from the invariant measure in the periodic case for the expected population structure of an MSGS.


Introductory Notes
Ref [1] introduced the stochastic process non-homogeneous Markov system (NHMS) with countable state space. Ref [2] introduced the stochastic process non-homogeneous semi-Markov system (NHSMS) in countable spaces. The theory and applications of both processes have seen the realization of an interesting growth and have found many applications in fields with great diversity. The latest reviews on the theory are given in [3,4]. The motives of this theory go back in the homogeneous Markov chain models for manpower systems in [5,6] and the evolution of which, in a variety of populations, are well described in [7]. The stochastic process of an NHMS represents a general frame work for a large variety of applied probability models to be accommodated in. However, real motives were the works on non-homogeneous Markov models for manpower systems such as [8][9][10]. We could selectively only supply some applications of the theory due to their large numbers. We start with the evolution of the HIV virus within the human body of T-cells in [11][12][13]; asthma was studied in [14]; reliability applications exist in [15] for example; examples of biomedical studies exist in [16,17]; applications in gene sequences ( [18]); in DNA and web navigation ( [19]); in manpower systems in [20][21][22][23][24][25]; in Physical Chemistry ( [26]); examples from ecological modelling ( [27]). Finally, there is the work of the research school of Prof McClean in the health systems (see, for example, [28][29][30][31][32]). Note that health systems are large manpower systems and each member has many parameters that are used to categorize him/her into the different groups. This characteristic of the health systems is what makes it an area of potential application of the present results (see [33]).
The rigorous foundation of Markov systems in general spaces (MSGS) was introduced in [33]. Also, the problem of asymptotic behaviour or ergodicity of Markov systems was studied. Important theorems were proved on the ergodicity of the distribution of expected population structure of a Markov system, which lives in the general space (X, B (X)). In addition, the total variability from the invariant measure was studied given that the Markov system is ergodic. It is shown that the total variation is finite.
Markov chains in general spaces have very important applications in the areas of Markov models in time series, models in control and systems theory, Markov models with regenerations times, Moran dams, Markov chains Monte Carlo simulation algorithms, etc. For more details, see [34] Chapter 2. There is a belief that the introduction in the presence of a population which is modulated as a Markov chain with general state space will increase the dynamics for new applications considerably, as it was described above in the countable case. Hence, although the results in the present are purely theoretical, the prospects for interesting applications are present and promising.
In Sections 2 and 2.1, we start with the formal definition of a Markov system in a general state space, as it was introduced in [33]. In Section 2.2, we provide an abstract image for an MSGS in an attempt to help the reader in understanding the present paper. In Section 2.3, we introduce some known concepts and results useful in the foundation of the novel results that follow from this point to the end. In Section 3, we study the rate of convergence of the expected population structure in an MSGS. Two theorems are provided where in the first we provide conditions under which the MSGS is uniformly ergodic, and in the second we provide conditions under which the MSGS is V-uniformly ergodic. In Section 4, we proceed to study the asymptotic periodicity of the expected population structure, for a population that lives in a general state space. A basic theorem is proved, where it is shown that if d is the period of the inherent Markov chain, then the sequence of the expected population structures splits into d converging subsequences. The asymptotic periodicity of non-homogeneous Markov systems in countable spaces was studied by [35,36]. In the present study, the basic tools and methodology are different since, among other reasons, the equations are of completely different nature. That is, we move from difference equations to integral equations for the expected population structure. In Section 5, we study the total variability from the invariant measure in the periodic case for the expected population structure of an MSGS. In the form of two theorems, we provide conditions under which the total variability in the periodic case is finite.

The Markov System in a General State Space
We will start with a formal definition of a Markov system in a general state space ( [33]) and then we provide an explanatory example. The reader could choose his own suitable order of reading the two subsections.

The Foundation of an MSGS
Let (X, B (X)) be the state space where X is a general set and B (X) denote a countably generated σ-field on X. Assume that small letters are elements of X and A, B, C elements of B (X). Let us denote by T (t) the population of the system at time t, which lives in (X, B (X)). We assume that {T (t)} ∞ t=0 is a known stochastic process with state space Z + and in discrete time.
Let that λ : X → R + be a positive σ-finite measure λ : X → R + , that is, for which we have that λ (X) = 1. We assume that λ (.) represents the initial distribution of a member of the population in the space (X, B (X)). We also assume that there is a set W ∈ B (X) for which λ (W) = 0. We will refer to W as the "gate" of the space (X, B (X)).
In the space (X, B (X)) where the population lives, it is assumed that each member has a membership, and that the leavers are leaving their memberships at the "gate". Therefore, at any time t at the "gate" W, the number of memberships are those left by leavers, and the necessary ∆T (t) = T (t) − T (t − 1) memberships to complete the desired population T (t). We will only work the cases for which ∆T (t) ≥ 0. New members of the population take their memberships at the gate "gate" W from which they then make a transition to any A ∈ B (X). Now let be the transition probability kernel from x ∈ X to the set A ∈ B (X) in one time step for a member of the population. Denote by Φ p = Φ (n) p n∈N + the Markov chain with state space (X, B (X)), which is defined uniquely by the probability kernel (2.1). We now make the following: That is, there exists a measure µ (.) on B (X) such that with µ (X) = 1. Now, define by The transition probability kernel Q (x, A) in (3) is the probability of a membership to make a transition from point x into the set A in one time step, either by direct transition of the member holding the membership with probability P (x, A) , or by the member leaving the system through the gate W with probability P (x, W) and the entrance of a new member, who gets the membership of the member from gate W, into set A with probability µ (A). The transition probability kernel Q (x, A) in (4) is the probability of new memberships entering the space (X, B (X)) at each time interval (t − 1, t] . From the gate of the atom, W is then distributed according to the σ-finite probability measure µ (.) : We will use the notation by Φ Q = Φ (n) Q n∈N + for the Markov chain with state space (X, B (X)) defined uniquely by Q (x, A) in (3) and (4). We also use the notation Q n (x, A) for probability that a membership in x will move in n time steps to the set A ∈ B (X). Now, define: N (x, t, A) : the number of memberships of the population that are in set A ∈ B (X) at time t given that were initially at x ∈ X.
Assume, as it is usual in many applications, a natural partition of the space (X, B (X)), that is, , with A i ∈ B (X) and A i ∩ A j = ∅ for i = j and such that (5) ∪ k+1 i=1 A i = X and with no loss of generality A k+1 = W.
It is of interest the expected population structure defined as: where N (t, X) is a non-negative measurable function on X. We are intersted also for the relative expected population structure defined by It could be easily seen that q : X → R + is a positive σ-finite probability measure since Of central importance is the evolution of the expected population structure E P [N (t, X)] or the relative expected population structure E P [q (t, X)] in the study of Markov systems in general state space. From Vassiliou (2014) we get that Note that is the expected number of memberships from the initial population T (0) that survived up to time t and are in set A. The part is the expected number of memberships that entered the system in the interval [0, t] and survived up to time t and are in set A.
We call the random process described above with state space (X, B (X)) a Markov system in a general state space. In addition, we will call the Markov chain Φ Q = Φ (n) Q n∈N + the inherent Markov chain of the Markov system.

An Abstract Image for MSGS
In the present subsection, in order to understand the various concepts and results, we will use the analogy of the example of the frog who jumps on lilies in a pond. This example of an abstract image was firstly used in [33] and the reader is referred to that study for a more extensive description and an explanation of the various concepts. In summary, imagine a lily pond that is covered by the leaves of the lilies. A population of frogs is living on the pond and a number of them are in every lily at each point in time and hence we have an expected population of frogs in the lilies. There are also leavers and newcomers in the pond at every point in time due to the antagonistic mating situation that exists between the males.

Introducing Some Important Concepts and Known Results
Throughout the paper, we will assume that the Markov chain Φ Q is ψ-irreducible. That is, there is a σ-finite measure ψ on (X, B (X)) such that, for any A ∈ B (X) with ψ (A) > 0, and any x ∈ X, Q n (x, A) > 0, for all n sufficiently large. (11) Assume that ψ is maximal with respect to ψ. We will also assume that the Markov chain is aperiodic, apart from the last section, where we will study periodicity. The concept of ψ−irreducibility for Markov chains on general state spaces was introduced by [37,38], and followed by many authors [39][40][41][42][43][44].
We will adopt the term ergodicity for the Markov chains for which the limit, lim n→∞ Q n (x, A) exists and is equal with π (A), where π is the invariant measure or stationary distribution of Φ Q . As a mode of convergence in the space (X, B (X)), the total variation norm will be used. For a a signed measure µ on (X, B (X)), the total variation norm µ is given by The key limit of interest in the present paper will be of the form For more on the spectral theory and rates of convergence for Markov chains in general spaces, see [34]. Research is motivated by elegant mathematics as well as a range of applications. Note that, for countable inhomogeneous Markov systems it was proved that the space of all possible expected population structures is a Hilbert space ( [45]). Now, for a ψ−irreducible Markov chain who lives in (X, B (X)), denoted by Let Q 1 and Q 2 be transition kernels for Markov chains, then for a positive function ∞ > V ≥ 1, define the V-norm distance between Q 1 and Q 2 as We define the outer product of the function 1 and the invariant measure π the kernel In many applications, we consider the distance |||Q n − 1 ⊗ π||| V for large n. We provide the following definition from [34]: Of great interest is the special case when V = 1. In this case, we provide the following definition: The concept of a small set is very useful in order to have a finite break up into cyclic sets for ψ-irreducible chains.

Definition 3.
For C ∈ B (X) we say that it is a small set if there exists an m > 0, and a non-trivial measure ν m on B (X), such that for all x ∈ C, B ∈ B (X) In such a case we say that C is ν m -small.
From [34] p. 105 we get the following Theorem: then for every A ∈ B + (X), there exists m ≥ 1 and a ν m -small set C ⊆ A such that C ∈ B + (X) and ν m (C) > 0.
Given the existence of just one small set from the previous Theorem, the Proposition in [34] p. 106 states that it is further possible for set X to be covered by small sets in the ψ-irreducible case.
Assume that C is any ν M -small set, and ν M (C) > 0, without loss of generality. With the use of the set C ν M we define a cycle for a general irreducible Markov chain. We will suppress the subscript of ν for simplicity.
We have that hence, when the chain starts in C, there is a positive probability that the chain will return to C at time M. Let E C = {n ≥ 1 : the set C is ν n = δ n ν small for some δ n > 0} .
Then the "period" for the set C, is given by the greatest common divisor of E C . The following Theorem ( [34] p. 113) will be useful in what follows: If C ∈ B + (X) is ν M -small and d the greatest common divisor of the set E C . Then there exist disjoint sets D 0 , ..., It is obvious from the above that any small set must be essentially contained inside one specific member of the d-cycle {D i } cyclic class. From [34] p. 115, we need the following Proposition: Then each of the sets D i is an absorbing ψ-irreducible set for the chain Φ d corresponding to the transition kernel P d , and Φ d on each D i is aperiodic.
For any set A ∈ B (X), we denote by oc (A) the number of visits by Φ to A after time zero and it is given by We define the kernel The chain Φ is called recurrent if it is ψ-irreducible and U (x, A) = ∞ for every x ∈ X and every A ∈ B + (X). If the chain Φ is irreducible and admits an atom a ∈ B + (X) then if a is recurrent, then every set in B + (X) is recurrent.
A chain Φ is called Harris recurrent if it is ψ-irreducible and every set in B + (X) is Harris recurrent.

Definition 5.
A σ-finite measure π on B (X) with the property will be called invariant.
Suppose that Φ is irreducible, and admits an invariant probability measure π. Then Φ is called a positive chain. If Φ is Harris recurrent and positive, then Φ is called a positive Harris chain. From [34] p. 328 and p. 204, we get the following two results: Theorem 3. If Φ is positive Harris and aperiodic, then for any initial distribution λ For any Harris recurrent set H, we write where H is a non-empty maximal Harris set and N is transient.

Rate of Convergence of MSGS
In the present section, we will study the rate of convergence of Markov systems in general spaces. We will provide conditions under which the rate of convergence of the expected population structure of an ergodic Markov system is uniformly ergodic and V-uniformly ergodic. From [34] p. 393, we get the following two Theorems: Theorem 5. Suppose that Φ is ψ-irreducible and aperiodic with transition kernel Q. Then the following are equivalent for V ≥ 1 : (i) Φ is V-uniformly ergodic.
(ii) There exists r > 1 and R < ∞ such that for all n ∈ N + Theorem 6. For any Markov chain Φ with transition kernel Q, the following are equivalent: (i) Φ is uniformly ergodic.
(ii) There exists r > 1 and R < ∞ such that for all x that is, the convergence takes place at a uniform geometric rate.
(iv) The chain is aperiodic and Doeblin condition holds: that is, there is a probability measure φ on B (X) and ε < 1, (v) The set X for some m is ν m -small. (vi) For every set A ∈ B + (X) and an aperiodic Markov chain there is a petit set C with (vii) For every set A ∈ B + (X) and an aperiodic Markov chain there is a petite set C and κ > 1 with for some β > 0, b < ∞, and some petit set C, Under (v) , we get that for any x, where ρ = 1 − ν m (X) . Now, from [46], we will borrow the following theorem adapted for an MSGS: Theorem 7. Let a Markov system in a general state space (X, B (X)), which is expanding (∆T (t) = T (t) − T (t − 1) ≥ 0). The following two statements are equivalent (i) The sequence {T (t)} ∞ t=0 converges, that is, lim t→∞ T (t) = T geometrically. (ii) The non-negative sequence {∆T (t)} ∞ t=0 tends to zero geometrically.
We will start by studying Uniform ergodicity for an MSGS for simplicity reasons and then proceed to study V-uniform ergodicity. Definition 6. (Uniform ergodicity for an MSGS). Let a Markov system in a general state space (X, B (X)) . Also let {T (t)} ∞ t=0 be the total population of memberships with lim t→∞ T (t) = T; Q (x, A) with x ∈ X and A ∈ B (X) the transition kernel of the inherent Markov chain of memberships and E Q [N (x, A)] the expected population structure. We say that the MSGS is uniformly ergodic if and only if there exists a C < ∞ and an 0 < a < 1 such that We now provide the following theorem concerning uniform ergodicity for an MSGS.
Theorem 8. Let a Markov system that lives in (X, B (X)). Assume in addition that lim t→∞ T (t) = T at a geometric rate, ∆T (t) ≥ 0 for every t ∈ N + , and that the inherent Markov chain with transition kernel Q (x, A) is uniformly ergodic. Then the MSGS is uniformly ergodic.

Proof. We have that
From (27) and since lim t→∞ T (t) = T is geometrically fast and ∆T (t) ≥ 0 for every t ∈ N + from Theorem 6 we get that there exist a c > 0 and a 1 > b > 0 such that Also from (27), since X λ (dx) = 1 and since the inherent Markov chain with kernel Q (x, A) is uniformly ergodic, from Theorem 6 (ii) there exists R < ∞ and r > 1 such that with a = r −1 , 0 < a < 1. Now, again since lim t→∞ T (t) = T is geometrically fast and ∆T (t) ≥ 0 for every t ∈ N + from Theorem 6 there exist a c > 0 and a 1 > b > 0 with ∆T (t) ≤ cb t ; in addition since the inherent Markov chain with kernel Q (x, A) is uniformly ergodic, from Theorem 6 (ii) there exists R < ∞ and r > 1, that is, 0 < a = r −1 < 1 such that Now, from (27) Let a 1 = max {a, b} then 0 < a 1 < 1 and which concludes the proof that MSGS is uniformly ergodic.
Generalizing the above results to V−iniform ergodicity is now straight forward. We start with the definition Definition 7. (V-Uniform ergodicity for an MSGS). Let a Markov system in a general state space (X, B (X)) . Also let {T (t)} ∞ t=0 be the total population of memberships with lim t→∞ T (t) = T; Q (x, A) with x ∈ X and A ∈ B (X) the transition kernel of the inherent Markov chain of memberships and E Q [N (x, A)] the expected population structure. We say that the MSGS is V− uniformly ergodic if and only if there exists a C < ∞ and an 0 < a < 1 such that Following exactly the proof of Theorem 8, we arrive at the following result: Theorem 9. Let a Markov system that leaves in (X, B (X)). Assume in addition that lim t→∞ T (t) = T at a geometric rate, ∆T (t) ≥ 0 for every t ∈ N + , and that the inherent Markov chain with transition kernel Q (x, A) is V−uniformly ergodic. Then the MSGS is V−uniformly ergodic.

Asymptotic Periodicity of an MSGS
The study of the asymptotic behaviour of Markov chains has been very important for finite or countable spaces from the start of their history. The general state space results are due to [40,41,47]. In the non-homogeneous Markov chains when the state space is countable, the initial important results on asymptotic behaviour were achieved by [48][49][50][51][52]. For the semi Markov process analogue, a wealth of results exist in the books by [53,54] . For NHMS and NHSMS on countable spaces the asymptotic behaviour has also been a problem of central importance see for example [1,36,[55][56][57]. In the present section, we will assume that the Markov chain is periodic and study the asymptotic behaviour of the expected population structure, which apparently is an important problem for a population that lives in (X, B (X)). Let it be that the inherent Markov chain of the MSGS with kernel Q is positive Harris with period d ≥ 1, and let the d-cycle described in Theorem 2 be the set {D 0 , D 1 , ..., D d−1 }, where D i ∈ B (X) for i = 0, 1, ..., d − 1. We have assumed that λ (x) is the initial distribution on X. Then the distribution on the set D i is From Theorem 2, we have that if x ∈ D 0 then Q (x, D 1 ) = 1 and consequently Q 2 (x, Now, in general for x ∈ D i and for d > i + υ we have It is apparent from (35) that If x ∈ D i and for d > i + υ we have Q nd+υ x, D j = 0 for j = i + υ.
If x ∈ D i and for d ≤ i + υ we have From the fact that the inherent Markov chain with kernel Q is assumed to be positive Harris with period d we know that for each cyclic set is positive Harris on the d-skeleton and aperiodic, and by Theorem 3 it is straightforward to check the following Theorem: for any set A ⊆ D i+υ , and for any set A ⊆ D i+υ−d .
Now, from the above Theorem and the decomposition Theorem 4, we immediately get the following: Theorem 11. If the inherent Markov chain of the MSGS is positive recurrent and periodic with period d, then for the sets {D 0 , D 1 , ..., D d−1 } with D i ∈ B (X), then for each i there exist a π i −null set N i which for every initial distribution for any set A ⊆ D i+υ , and for any set A ⊆ D i+υ−d .
We will now start building the proof for the periodicity Theorem for an MSGS and we will state the theorem in the end. The important question is the periodicity of the expected population structure when the inherent Markov chain with kernel Q is positive Harris and periodic with period d or is positive recurrent and periodic with period d. With no loss of generality and for simplicity reasons, in what follows we will use the set A instead of any of the sets A i , (i = 1, 2, ..., k) . From Equation (10) we have that We will work with each part of the right hand side of Equation (37) separately. We have that We could write the set A in general as follows Let without loss of generality that t = πd + υ and consider in what follows that for all i = 0, 1, ..., d − 1, d > i + υ. Then we have that Now, from (40) and (41) we get that (from Theorem 10 and relation (36) we get that) ≤∈ .
Hence, we proved that when t = πd + υ and d > i + υ then the limit of Similarly, following the same steps we could prove that for t = πd + υ and d ≤ i + υ then From (42) and (43) it is not difficult to check that Assume now that lim t→∞ T (t) = T and ∆T (t) ≥ 0 for every t. Let again that t = πd + υ then Now, since lim t→∞ ∆T (t) = 0 we have Hence, from (46) and (47) we get that It is apparent that since ∆T (t) ≥ 0 and thus the series is bounded by T − T (0), its elements are non-negative and thus it converges; so let Now, let us define by Now from (49) we get that We now provide the following useful Lemma: Lemma 1. If a n / n ∑ k=0 a k → 0 as n → ∞ and {a n } non-negative, and {b n } → b then lim n→∞ b 0 a n + b 1 a n−1 + ...
From Lemma 1, (52) and the result in (45) where we may replace λ (dx) by µ (dx) it is not difficult to check that the following holds Define by Then similarly we could prove that Therefore from (37) , (45) , (48) , (54) and (56) for t = πd + υ we get that for υ = 0, 1, ..., d − 1. Hence, we have proved the following Theorem: Theorem 12. Let an MSGS and let that the inherent Markov chain of the MSGS be positive Harris and periodic with period d. OR let it be that the inherent Markov chain of the MSGS is positive recurrent and periodic with period d, and let the sets {D 0 , D 1 , ..., D d−1 } with D i ∈ B (X). Then the expected population structure splits into d converging subsequences, that is, for υ = 0, 1, 2, ..., d − 1.

Total Variability from the Invariant Measures in the Periodic Case. Coupling Theorems
In this section, we study the total variability from the invariant measures in the periodic case for an MSGS. We show that the total variation in the periodic case is finite. This is also known as the coupling problem (see [58]). From [34] (p. 332) we get the following: Theorem 13. Suppose that Φ is positive Harris and aperiodic chain and assume that the chain has an atom a ∈ B + (X). Then for any λ, µ regular initial distributions and in the case that Φ is regular, then for any x, y ∈ X ∞ ∑ n=1 P n (x, .) − P n (y, .) < ∞.
(b) If for the accessible atom W we have that E w τ 2 w < ∞, then for any regular initial distribution λ we have for d > i + υ, x ∈ D i and i, υ = 0, 1, ..., d − 1 We will now prove the following coupling theorem for an MSGS with inherent Markov chain Φ Q , which is assumed to be positive Harris and periodic. Then for any regular distributions λ 1 , µ 1 , λ 2 , µ 2 we have (58) With no loss of generality assume that t = πd + υ. Denoted by From (58) and (59) we have that We have that From (61) it is apparent that with no loss of generality we may assume that Also, without loss of generality again we may assume that d > j + υ, then from (60) and (62) Now, it is not difficult to check that and Therefore from (63)-(65) we get that From Theorem 16 and (66) we get that In a similar way we get that U λ (t) < ∞ and consequently from (63) we get that The case for d ≤ j + υ is proved in an analogous way.
We have that We start with U (υ) µ (t) and we have that Denote by Then with no loss of generality we may assume that there is an r such that From (70)-(72) and using Theorem 16 we get that We now move to U (π) µ (t) and we have that We start first with U (π,1) µ (t) and we get that Now, using Theorem 17 and from (75) we have that Similarly, we get also that U (π,1,2) µ (t) < ∞ and U (π,2) µ (t) < ∞. Similarly, and with the use of Theorem 16.b we could prove the following Theorem Theorem 18. Assume that the inherent Markov chain Φ Q of a Markov system that lives in (X, B (X)) with kernel Q (x, .) is positive Harris with period d. Assume, in addition, that for the accessible atom W, we have E ω τ 2 ω < ∞ and that lim t→∞ T (t) = T, ∆T (t) ≥ 0 for every t ∈ Z + .
Then for any regular distribution λ we have

Conclusions and Further Research
We studied a population that lives in a general state space that at every instant has leavers and new entrants, which evolves under the Markov property. We proved that under certain conditions the expected population structure converges with geometrical rate to an invariant population structure with loss of memory. We also provided the analogous result for the V-uniformly ergodicity of MSGS. Moreover, under the assumption that the inherent Markov chain is periodic with period d, we proved that the sequence of the expected population structure splits into d convergent subsequences and provided their limits in closed forms. Finally, we proved that the total variability from the invariant measure in the periodic case is finite. We concluded the novel results by proving two coupling theorems under the assumption that the inherent Markov chain is positive Harris with period d using different additional conditions in each case. The above theoretical results have important applications both in the classical areas where Markov chains in general spaces have already been applied but especially to the important area of the health systems that, nowadays, are in crisis all over the world.
The results in the present paper together with the ones introduced in [33] provide the nucleus for a new path for extensive research in the area. Immediate further research of interest is to establish Laws of Large numbers for MSGS as a possible extension of the Laws of Large numbers for populations that live in countable spaces [59].