Controlled Discrete-Time Semi-Markov Random Evolutions and Their Applications

Abstract

In this paper, we introduced controlled discrete-time semi-Markov random evolutions. These processes are random evolutions of discrete-time semi-Markov processes where we consider a control. applied to the values of random evolution. The main results concern time-rescaled weak convergence limit theorems in a Banach space of the above stochastic systems as averaging and diffusion approximation. The applications are given to the controlled additive functionals, controlled geometric Markov renewal processes, and controlled dynamical systems. We provide dynamical principles for discrete-time dynamical systems such as controlled additive functionals and controlled geometric Markov renewal processes. We also produce dynamic programming equations (Hamilton–Jacobi–Bellman equations) for the limiting processes in diffusion approximation such as controlled additive functionals, controlled geometric Markov renewal processes and controlled dynamical systems. As an example, we consider the solution of portfolio optimization problem by Merton for the limiting controlled geometric Markov renewal processes in diffusion approximation scheme. The rates of convergence in the limit theorems are also presented.

1. Introduction

Random evolutions were introduced over 40 years ago, see, e.g., in [1,2] and for its asymptotic theory in [3,4,5,6] and references therein. Discrete-time random evolutions, induced by discrete-time Markov chains, are introduced by Cohen [7] and Keepler [8], and discrete-time semi-Markov random evolutions (DTSMRE) by Limnios [9]. See also [10]. Koroliuk and Swishchuk [4], Swishchuk and Wu [5], Anisimov [11,12,13,14], and Koroliuk and Limnios [3], studied discrete-time random evolutions induced by the embedded Markov chains of continuous time semi-Markov processes. This is equivalent to discrete-time Markov random evolution stopped at random time (continuous). One of the examples of discrete-time random evolutions is the geometric Markov renewal process (GMRP). Applications of GMRP in finance have been considered in [15,16,17]. Optimal stopping of GMRP and pricing of European and American options for underlying assets modelled by GMRP have been studied in [18].

Discrete-time semi-Markov chains (SMC) have only recently been used in applications. Especially, in DNA analysis, image and speech processing, reliability theory, etc., see in [19] and references therein. These applications have stimulated a research effort in this area. While the literature in discrete-time Markov chains theory and applications is quite extensive, there is only a small amount of the literature on SMC and most of them are related to hidden semi-Markov models for estimation.

The present article is a continuation of our previous work [20]. Thus, we keep all our notation and definitions the same as in the latter paper. Compared with our previous work [20], where we studied random evolutions of semi-Markov chains, here we considered additionally a control on the random evolution, which we call controlled discrete-time semi-Markov random evolution (CDTSMRE) in a Banach space, and we presented time-rescaled convergence theorems. In particular, we get weak convergence theorems in Skorokhod space $D[0,\infty )$ for càdlàg stochastic processes, see, e.g., in [21]. The limit theorems include averaging, diffusion approximation, and diffusion approximation with equilibrium. For the above limit theorems we also presented rates of convergence results. Finally, we give some applications regarding the above mentioned results, especially to controlled additive functionals (CAF), CGMRP, and controlled dynamical systems (CDS), and optimization problems.

Regarding the optimization problems, we provide dynamical principles for discrete-time dynamical systems such as CAF and CGMRPs (see Section 2.4), see, e.g., [22,23,24]. We also produce dynamic programming equations (Hamilton–Jacobi–Bellman equations) for the limiting processes in diffusion approximation such as CAF, CGMRP, and CDS. As an example, we consider the solution of portfolio optimization problem by Merton for the limiting CGMRP in DA (see Section 4.4). Merton problem, or Merton portfolio’s problem, is a problem in continuous-time finance associated with portfolio choice. In $(B,S)$-security market, which consists of a stock and a risk-free asset, an investor must choose how much to consume, and must allocate his wealth between the stock and the risk-free asset in a such way that maximizes expected utility. The problem was formulated and first solved by Robert Merton in 1969, and published in 1971 [25].

Results presented here are new and deals with CDTSMRE on Banach spaces. This paper contains new and original results on dynamical principle for CDTSMRE and DPE (HJB equations) for the limiting processes in DA. One of the new remarkable results is the solution of Merton portfolio problem for the limiting CGMRP in DA. The method of proofs was based on the martingale approach together with convergence of transition operators of the extended semi-Markov chain via a solution of a singular perturbation problem [3,4,26]. As in our previous work [20], the tightness of these processes is proved via Sobolev’s embedding theorems [27,28,29]. It is worth mentioning that, as in the Markov case, the results presented here cannot be deduced directly from the continuous-time case. We should also note that that DTSMREs have been completely studied in [20]. For semi-Markov processes see, e.g., [30,31,32,33]. For Markov chains and additive functionals see, e.g., [34,35,36,37,38].

The paper is organized as follows. Definition and properties of discrete-time semi-Markov random evolutions and Controlled DTSMREs, as well as particular stochastic systems as applications, are introduced in Section 2. The main results of this paper, limit theorems of CDTSMRE, as averaging, diffusion approximation and diffusion approximation with equilibrium of controlled DTSMREs are considered in Section 3. In Section 4, we provide three applications of averaging, diffusion approximation, and diffusion approximation with equilibrium of controlled DTSMREs: controlled additive functionals, controlled GMRP, and controlled dynamical systems. Section 5 deals with the analysis of the rates of convergence in the limit theorems, presented in the previous sections, for controlled DTSMREs and for CAF and CGMRP. In Section 6, we give the proofs of theorems presented in the previous sections. The last section concludes the paper and indicates some future works.

2. Controlled Discrete-Time Semi-Markov Random Evolutions

2.1. Semi-Markov Chains

The aim of this section is to present some notation and to make this paper as autonomous as possible. The reader may refer to our article in [20] for more details.

Let $(E,\mathcal{E})$ be a measurable space with countably generated $\sigma$-algebra and $(\mathsf{\Omega },\mathcal{F},{\left({\mathcal{F}}_n\right)}_{n\in \mathrm{I}\mathrm{N}},\mathbf{P})$ be a stochastic basis on which we consider a Markov renewal process $(x_n,{\tau }_n,n\in \mathrm{I}\mathrm{N})$ in discrete time $k\in \mathrm{I}\mathrm{N}$, with state space $(E,\mathcal{E})$. Notice that $\mathrm{I}\mathrm{N}$ is the set of non-negative integer numbers. The semi-Markov kernel q is defined by (see, e.g., in [9,19]),

$$\begin{array}{c}\hfill q(x,B,k):=\mathbf{P}(x_{n+1}\in B,{\tau }_{n+1}-{\tau }_n=k\mid x_n=x),x\in E,B\in \mathcal{E},k,n\in \mathrm{I}\mathrm{N}.\end{array}$$

(1)

We will denote also $q(x,B,\Gamma )={\sum }_{n\in \Gamma }q(x,B,n)$, where $\Gamma \subset \mathrm{I}\mathrm{N}$. The process $\left(x_n\right)$ is the embedded Markov chain of the MRP $(x_n,{\tau }_n)$ with transition kernel $P(x,dy)$. The semi-Markov kernel q is written as

$$q(x,dy,k)=P(x,dy)f_{xy}\left(k\right),$$

where $f_{xy}\left(k\right):=\mathbf{P}({\tau }_{n+1}-{\tau }_n=k\mid x_n=x,x_{n+1}=y)$, the conditional distribution of the sojourn time in state x given that the next visited state is y.

Define also the counting process of jumps ${\nu }_k=max\{n:{\tau }_n\le k\}$, and the discrete-time semi-Markov chain $z_k$ by $z_k=x_{{\nu }_k}$, for $k\in \mathrm{I}\mathrm{N}$. Define now the backward recurrence time process ${\gamma }_k:=k-{\tau }_{{\nu }_k}$, $k\ge 0$, and the filtration ${\mathcal{F}}_k:=\sigma (z_{\ell },{\gamma }_{\ell };\ell \le k)$, $k\ge 0$.

Let us consider a separable Banach space B of real-valued measurable functions defined on $E\times \mathrm{I}\mathrm{N}$, endowed with the sup norm $∥·∥$ and denote by $\mathcal{B}$ its Borel $\sigma$-algebra. The Markov chain $(z_k,{\gamma }_k),k\ge 0$, has the following transition probability operator $P^{♯}$ on B

$$\begin{array}{c}\hfill P^{♯}\phi (x,k)=\frac{1}{{\overline{H}}_x\left(k\right)}{\int }_{E\backslash \left\{x\right\}}q(x,dy,k+1)\phi (y,0)+\frac{{\overline{H}}_x(k+1)}{{\overline{H}}_x\left(k\right)}\phi (x,k+1),\end{array}$$

(2)

where $\phi \in B$, and its stationary distribution, if there exist, is given by

$$\begin{array}{c}\hfill {\pi }^{♯}(dx\times \left\{k\right\})=\rho \left(dx\right){\overline{H}}_x\left(k\right)/m,\end{array}$$

where

$$\begin{array}{c}\hfill m:={\int }_E\rho \left(dx\right)m\left(x\right),m\left(x\right)=\sum _{k\ge 0}{\overline{H}}_x\left(k\right),\end{array}$$

and $\rho \left(dx\right)$ is the stationary distribution of the EMC $\left(x_n\right)$, $H_x\left(k\right):=q(x,E,[0,k])$, and ${\overline{H}}_x\left(k\right):=1-H_x\left(k\right)=q(x,E,[k+1,\infty ))$. The probability measure $\pi$ defined by $\pi \left(B\right)={\pi }^{♯}(B\times \mathrm{I}\mathrm{N})$ is the stationary probability of the SMC $\left(z_k\right)$. Define also the r-th moment of holding time in state $x\in E$,

$$m_r\left(x\right):=\sum _{k\ge 1}{k}^{r}q(x,E,k),r=1,2,\cdots$$

Of course, $m\left(x\right)=m_1\left(x\right)$, for any $x\in E$.

Define now the stationary projection operator $\Pi$ on the null space of the (discrete) generating operator $Q^{♯}:=P^{♯}-I$,

$$\begin{array}{c}\hfill \Pi \phi (x,s)=\sum _{\ell \ge 0}{\int }_E{\pi }^{♯}(dy\times \left\{\ell \right\})\phi (y,\ell )\mathbf{1}(x,s),\end{array}$$

where $\mathbf{1}(x,s)=1$ for any $x\in E$, and $s\in \mathrm{I}\mathrm{N}$. This operator satisfies the equations

$$\Pi Q^{♯}=Q^{♯}\Pi =0.$$

The potential operator of $Q^{♯}$, denoted by $R_0$, is defined by

$$\begin{array}{c}\hfill R_0:={(Q^{♯}+\Pi )}^{-1}-\Pi =\sum _{k\ge 0}[{(P^{♯})}^k-\Pi ].\end{array}$$

2.2. General Definition and Properties of DTSMREs

We define here controlled discrete-time semi-Markov random evolutions. Let U denote a compact Polish space representing the control, and let $u_k$ be U-valued control process and we suppose that it is a Markov chain. We note that we could also define the process $u_{{\nu }_k}$ which is a semi-Markov control process, considered in many papers (see, e.g., in [39,40]). We suppose that homogeneous Markov chain $u_k$ is independent of $z_k$, and transition probability kernel $P^u=P(u_{k+1}\in dy\mid u_k=u)=Q(u,dy)$.

Let us consider a family of bounded contraction operators $D(z,u),z\in E,u\in U$, defined on B, where the maps $D(z,u)\phi :E\times U\to B$ are $\mathcal{E}\times \mathcal{U}$-measurable, $\phi \in B$. Denote by I the identity operator on B. Let $\Pi B=\mathcal{N}\left(Q^{♯}\right)$ be the null space, and $(I-\Pi )B=\mathcal{R}\left(Q^{♯}\right)$ be the range values space of operator $Q^{♯}$. We will suppose here that the Markov chain $(z_k,{\gamma }_k,k\in \mathrm{I}\mathrm{N})$ is uniformly ergodic, that is, $∥({\left(P^{♯}\right)}^n-\Pi )\phi ∥\to 0$, as $n\to \infty$, for any $\phi \in B$. In that case, the transition operator is reducible-invertible on B. Thus, we have $B=\mathcal{N}\left(Q^{♯}\right)\oplus \mathcal{R}\left(Q^{♯}\right)$, the direct sum of the two subspaces. The domain of an operator A on B is denoted by $\mathcal{D}\left(A\right):=\{\phi \in B:A\phi \in B\}$.

Definition 1.

A controlled discrete-time semi-Markov random evolution (CDTSMRE) ${\mathsf{\Phi }}_k^u,k\in \mathrm{I}\mathrm{N}$, on the Banach space B, is defined by

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_k^u\phi =D(z_k,u_k)D(z_{k-1},u_{k-1})\cdots D(z_2,u_2)D(z_1,u_1)\phi ,\end{array}$$

(3)

for $k\ge 1,$${\mathsf{\Phi }}_0^u=I,u_0=u\in U$, and for any $\phi \in B_0:={\cap }_{x\in E,u\in U}\mathcal{D}\left(D(x,u)\right)$. Thus we have ${\mathsf{\Phi }}_k=D(z_k,u_k){\mathsf{\Phi }}_{k-1}.$

The process $(z_k,{\gamma }_k,u_k)$ is a Markov chain on $E\times \mathrm{I}\mathrm{N}\times U$, adapted to the filtration ${\mathcal{F}}_k^u:=\sigma (z_{\ell },{\gamma }_{\ell },u_{\ell };\ell \le k)$, $k\ge 0$. We also note that $({\mathsf{\Phi }}_k^u\phi ,z_k,{\gamma }_k,u_k)$ is a Markov chain on $B\times E\times \mathrm{I}\mathrm{N}\times U$ with discrete generator

$$\begin{array}{c}\hfill L^u\phi =[\tilde{P}+{\int }_E{\int }_U\tilde{P}(·,dv)(D(v,u)-I)]\phi ,\end{array}$$

(4)

where $\phi :=\phi (x,z,s,u)$, and

$$\tilde{P}\phi (z,s,u):=P^{♯}{P}^u\phi (z,s,u)=\sum _{s^{\prime }\in \mathrm{I}\mathrm{N}}{\int }_{E\times U}{P}^{♯}(z,s;d{z}^{\prime },s^{\prime })P^u(u;d{u}^{\prime })\phi (z^{\prime },s^{\prime },u^{\prime }).$$

The process $M_k^u$ defined by

$$\begin{array}{c}\hfill M_k^u:={\mathsf{\Phi }}_k^u-I-\sum _{\ell =0}^{k-1}\mathbf{E}[{\mathsf{\Phi }}_{\ell +1}^u-{\mathsf{\Phi }}_{\ell }^u\mid {\mathcal{F}}_{\ell }^u],k\ge 1,M_0=0,\end{array}$$

(5)

on B, is an ${\mathcal{F}}_k^u$-martingale. The random evolution ${\mathsf{\Phi }}_k^u$ can be written as follows

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_k^u:=I+\sum _{\ell =0}^{k-1}[D(z_{\ell +1},u_{\ell +1})-I]{\mathsf{\Phi }}_{\ell }^u,\end{array}$$

and then, the martingale (5) can be written as follows,

$$\begin{array}{c}\hfill M_k^u:={\mathsf{\Phi }}_k^u-I-\sum _{\ell =0}^{k-1}\mathbf{E}[(D(z_{\ell +1},u_{\ell +1})-I){\mathsf{\Phi }}_{\ell }^u\mid {\mathcal{F}}_{\ell }^u],\end{array}$$

or

$$\begin{array}{c}\hfill M_k^u:={\mathsf{\Phi }}_k^u-I-\sum _{\ell =0}^{k-1}[\mathbf{E}(D(z_{\ell +1},u_{\ell +1})\mid {\mathcal{F}}_{\ell }^u)-I]{\mathsf{\Phi }}_{\ell }^u.\end{array}$$

Finally, as $\mathbf{E}[D(z_{\ell +1},u_{\ell +1}){\mathsf{\Phi }}_{\ell }^u\phi \mid {\mathcal{F}}_{\ell }^u]=[(P^{♯}+P^u)D(·){\mathsf{\Phi }}_{\ell }\phi ](z_{\ell },{\gamma }_{\ell },u_{\ell })$, one takes

$$\begin{array}{c}\hfill M_k^u:={\mathsf{\Phi }}_k^u-I-\sum _{\ell =0}^{k-1}[\tilde{P}D(·,u)-I]{\mathsf{\Phi }}_{\ell }^u.\end{array}$$

2.3. Some Examples

Example 1. 

Controlled Additive Functional or Markov Decision Process.

Let define the following controlled additive functional,

$$y_k^u=\sum _{l=0}^{k}a(z_l,u_l),k\ge 0,y_0=y.$$

If we define the operator $D(z,u)$ on $C_0\left(\mathrm{I}\mathrm{R}\right)$ in the following way,

$$D(z,u)\phi \left(y\right):=\phi (y+a(z,u\left)\right),$$

then the controlled discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_k\phi$ has the following presentation,

$${\mathsf{\Phi }}_k^u\phi \left(y\right)=\phi \left(y_k^u\right).$$

Process $y_k^u$ is usually called in the literature the Markov decision process (see, e.g., in [41,42,43,44]).

Example 2. 

Controlled geometric Markov renewal process.

The CGMRP is defined in the following way,

$$S_k^u:=S_0\prod _{l=1}^k(1+a(z_l,u_l)),k\in \mathrm{I}\mathrm{N},S_0=s.$$

We suppose that ${\prod }_{k=1}^0=1.$

If we define the operator $D(z,u)$ on $C_0\left(\mathrm{I}\mathrm{R}\right)$ in the following way,

$$D(z,u)\phi \left(s\right):=\phi \left(s\right(1+a(z,u)\left)\right),$$

then the controlled discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_k^u\phi$ can be given as follows,

$${\mathsf{\Phi }}_k^u\phi \left(s\right)=\phi \left(S_k^u\right).$$

To the authors opinion, this process is defined for the first time in the literature and the notion of controlled GMRP is a new one as well.

2.4. Dynamic Programming for Controlled Models

Here, we present dynamic programming for controlled models given in Examples in previous section. Let us consider a Markov control model (see in [45]) $(E,A,\{A\left(z\right)|z\in E\},Q,c).$ Here, E is the state space; A is the control or action set; Q is the transition kernel, i.e., a stochastic kernel on E given $K,$ where $K:=\left\{\right(z,u\left)\right|z\in E,u\in A\left(z\right)\}$; and $c:K\to R$ is a measurable function called the cost-per-stage function.

We are interested in is to minimize the finite-horizon performance criterion either (see Example 1)

$$J_1(\pi ,z):={\mathbf{E}}_z^{\pi }[\sum _{l=0}^{N-1}a(z_l,u_l)+a_N\left(z_N\right)]$$

or (see Example 2)

$$J_2(\pi ,z):={\mathbf{E}}_z^{\pi }[ln(\prod _{l=1}^k(1+a(z_l,u_l))(1+a_N\left(z_N\right)))],$$

where $a_N\left(z_N\right)$ is the terminal cost function, $\pi \in \Pi$ is the set of control policies.

In this way, denoting by $J^{*}$ the value function

$$J_i^{*}\left(z\right):=\underset{\Pi }{inf}{J}_i(\pi ,z),z\in E,i=1,2,$$

the problem is to find a policy ${\pi }^{*}\in \Pi$ such that

$$J_i({\pi }^{*},z)=J_i^{*}\left(z\right),z\in E,i=1,2,.$$

Example 3. 

Controlled Additive Functional.

Let us provide an algorithm for finding both the value function $J^{*}$ and an optimal policy ${\pi }^{*}$ for the example with function $J_1(\pi ,z)$ (see Example 1).

Let $J_{0,1},J_{1,1},\cdots ,J_{N,1}$ be the functions on E defined from $l=N$ to $l=0$ by (backwards)

$$J_{N,1}\left(z\right):=a_N\left(z\right),l=N$$

and

$$J_{l,1}\left(z\right):=\underset{A\left(z\right)}{min}[a(z,u)+{\int }_E{J}_{l+1,1}\left(y\right)Q(u,dy)],l=N-1,N-2,\cdots ,0.$$

Suppose that there is a selector $f_t\in F$ such that $f_l\left(z\right)\in A\left(z\right)$ attains the minimum in the above expression for $J_l\left(z\right)$ for all $z\in E,$ meaning for any $z\in E$ and $l=0,\cdots ,N-1,$

$$J_{l,1}\left(z\right)=a(z,f_l)+{\int }_E{J}_{l+1,1}\left(y\right)Q(f_l,dy).$$

Then, the optimal policy is the deterministic Markov one ${\pi }^{*}=\{f_0,\cdots ,f_{N-1}\}$, and the value function $J^{*}$ equals $J_0,$ i.e.,

$$J_1^{*}\left(z\right)=J_0\left(z\right)=J_1({\pi }^{*},z),z\in E.$$

Example 4. 

Controlled Geometric Markov Renewal Chain.

Let us provide an algorithm for finding both the value function $J^{*}$ and an optimal policy ${\pi }^{*}$ for the example with function $J_2(\pi ,z)$ (see Example 2). We will modify the expression for $S_k^u$ in Example 2. Let $ln\left(\frac{S_k^u}{S_0}\right)$ be a log-return, then

$$ln\left(\frac{S_k^u}{S_0}\right)=\sum _{l=1}^{k}ln(1+a(z_l,u_l).$$

Thus, we are interested in minimizing the finite-horizon performance criterion for

$$J_2(\pi ,z):={\mathbf{E}}_z^{\pi }[\sum _{l=0}^{N-1}ln(1+a(z_l,u_l)+ln(1+a_N\left(z_N\right)]$$

Let $J_{0,2},J_{1,2},\cdots ,J_{N,2}$ be the functions on E defined from $l=N$ to $l=0$ by (backwards)

$$J_{N,2}\left(z\right):=ln(1+a_N\left(z\right)),l=N$$

and

$$J_{l,2}\left(z\right):=\underset{A\left(z\right)}{min}[ln(1+a(z,u))+{\int }_E{J}_{l+1,1}\left(y\right)Q(u,dy)],l=N-1,N-2,\cdots ,0.$$

Suppose that there is a selector $f_t\in F$ such that $f_l\left(z\right)\in A\left(z\right)$ attains the minimum in the above expression for $J_l\left(z\right)$ for all $z\in E,$ meaning for any $z\in E$ and $l=0,\cdots ,N-1,$

$$J_{l,2}\left(z\right)=ln(1+a(z,f_l))+{\int }_E{J}_{l+1,2}\left(y\right)Q(f_l,dy|z).$$

Then, the deterministic Markov policy ${\pi }^{*}=\{f_0,\cdots ,f_{N-1}\}$ is optimal, and the value function $J^{*}$ equals $J_0,$ i.e.,

$$J_2^{*}\left(z\right)=J_0\left(z\right)=J_2({\pi }^{*},z),z\in E.$$

3. Limit Theorems for Controlled Semi-Markov Random Evolutions

In this section, we present averaging, diffusion approximation, and diffusion approximation with equilibrium results for the controlled discrete-time semi-Markov random evolutions. It is worth noticing that the main scheme of results are almost the same as in our previous works in particular [20]. Nevertheless, the additional component of the control allows us to study more interesting problems.

3.1. Averaging of CDTSMREs

We consider here CDTSMREs defined in Section 2. Let us now set $k:=[t/\epsilon ]$ and consider the continuous time process $M_t^{\epsilon }$

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}:=M_{[t/\epsilon ]}^u={\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-I-\sum _{\ell =0}^{[t/\epsilon ]-1}[\tilde{P}{D}^{\epsilon }(·,u)-I]{\mathsf{\Phi }}_{\ell }^{\epsilon ,u}\end{array}$$

We will prove here asymptotic results for this process as $\epsilon \to 0$.

The following assumptions are needed for averaging.

A1:

The MC $(z_k,{\gamma }_k,k\in \mathrm{I}\mathrm{N})$ is uniformly ergodic with ergodic distribution ${\pi }^{♯}(B\times \left\{k\right\}),B\in \mathcal{E},k\in \mathrm{I}\mathrm{N}$.

A2:

The moments $m_2\left(x\right),x\in E$, are uniformly integrable.

A3:

The perturbed operators $D^{\epsilon }\left(x\right)$ have the following representation on B

$$\begin{array}{c}\hfill D^{\epsilon }(x,u)=I+\epsilon D_1(x,u)+\epsilon D_0^{\epsilon }(x,u),\end{array}$$

where operators $D_1(x,u)$ on B are closed and $B_0:={\cap }_{x\in E,u\in U}\mathcal{D}\left(D_1(x,u)\right)$ is dense in B, ${\overline{B}}_0=B$. Operators $D_0^{\epsilon }(x,u)$ are negligible, i.e., ${lim}_{\epsilon \to 0}∥D_0^{\epsilon }(x,u)\phi ∥=0$ for any $\phi \in B_0$.

A4:

We have ${\int }_E{\int }_U[\pi \left(dx\right){\pi }_1\left(du\right)]{∥D_1(x,u)\phi ∥}^2<\infty .$ (See A7.)

A5:

There exists Hilbert spaces H and $H^{*}$ such that compactly embedded in Banach spaces B and $B^{*},$ respectively, where $B^{*}$ is a dual space to $B.$

A6:

Operators $D^{\epsilon }\left(x\right)$ and ${\left(D^{\epsilon }\right)}^{*}\left(x\right)$ are contractive on Hilbert spaces H and $H^{*},$ respectively.

A7:

The MC $(u_k,k\in \mathrm{I}\mathrm{N})$, is independent of $\left(z_k\right)$, and is uniformly ergodic with stationary distribution ${\pi }_1\left(du\right),k\in \mathrm{I}\mathrm{N}$.

We note that if $B=C_0\left(\mathrm{I}\mathrm{R}\right),$ then $H=W^{l,2}\left(\mathrm{I}\mathrm{R}\right)$ is a Sobolev space, and $W^{l,2}\left(\mathrm{I}\mathrm{R}\right)\subset C_0\left(\mathrm{I}\mathrm{R}\right)$ and this embedding is compact (see [29]). For the spaces $B=L_2\left(\mathrm{I}\mathrm{R}\right)$ and $H=W^{l,2}\left(\mathrm{I}\mathrm{R}\right)$ the situation is the same.

We also note, that semi-Markov chain $(z_k,u_k)$ is uniformly ergodic on $E\times U$ with stationary probabilities $\pi \left(dx\right){\pi }_1\left(du\right),$ which follows from conditions A1 and A7.

Theorem 1.

Under Assumptions A1–A7, the following weak convergence takes place,

$${\mathsf{\Phi }}_{[t/\epsilon ,u]}^{\epsilon }⟹\overline{\mathsf{\Phi }}\left(t\right),\epsilon \downarrow 0,$$

where the limit random evolution $\overline{\mathsf{\Phi }}\left(t\right)$ is determined by the following equation,

$$\begin{array}{c}\hfill \overline{\mathsf{\Phi }}\left(t\right)\phi -\phi -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}\overline{\mathsf{\Phi }}\left(s\right)\phi ds=0,0\le t\le T,\phi \in B_0,\end{array}$$

(6)

or, equivalently,

$$\begin{array}{c}\hfill \frac{d}{dt}\overline{\mathsf{\Phi }}\left(t\right)\phi =\widehat{\mathrm{I}\mathrm{L}}\overline{\mathsf{\Phi }}\left(t\right)\phi ,\end{array}$$

where the limit contracted operator is then given by

$$\begin{array}{c}\hfill \widehat{\mathrm{I}\mathrm{L}}={\widehat{D}}_1={\int }_E{\int }_U[\pi \left(dx\right){\pi }_1\left(du\right)]D_1(x,u).\end{array}$$

(7)

This result generalize the classical Krylov–Bogolyubov averaging principle [46] on a Banach and a controlled spaces.

3.2. Diffusion Approximation of DTSMREs

For the diffusion approximation of CDTSMREs, we will consider a different time-scaling and some additional assumptions.

D1:

Let us assume that the perturbed operators $D^{\epsilon }(x,u)$ have the following representation in B,

$$\begin{array}{c}\hfill D^{\epsilon }(x,u)=I+\epsilon D_1(x,u)+{\epsilon }^2{D}_2(x,u)+{\epsilon }^2{D}_0^{\epsilon }(x,u),\end{array}$$

where operators $D_2(x,u)$ on B are closed and $B_0:={\cap }_{x\in E,u\in U}\mathcal{D}\left(D_2(x,u)\right)$ is dense in B, ${\overline{B}}_0=B$; operators $D_0^{\epsilon }(x,u)$ are a negligible operator, i.e., ${lim}_{\epsilon \downarrow 0}∥D_0^{\epsilon }(x,u)\phi ∥=0$.

D2:

The following balance condition holds,

$$\begin{array}{c}\hfill \Pi D_1(x,u)\Pi =0,\end{array}$$

(8)

where

$$\begin{array}{c}\hfill \Pi \phi (x,k,u):=\sum _{l\ge 0}{\int }_E{\int }_U{\pi }^{♯}(dy\times \ell ){\pi }_1\left(du\right)\phi (y,\ell ,u)\mathbf{1}(x,k).\end{array}$$

(9)

D3:

The moments $m_3\left(x\right),x\in E$, are uniformly integrable.

Theorem 2.

Under Assumptions A1, A5–A7 (see Section 3.1), and D1-D3, the following weak convergence takes place,

$${\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon }⟹{\mathsf{\Phi }}_0\left(t\right),\epsilon \downarrow 0,$$

where the limit random evolution ${\mathsf{\Phi }}_0\left(t\right)$ is a diffusion random evolution determined by the following generator

$$\begin{array}{c}\hfill \mathrm{I}\mathrm{L}=\Pi D_2\left(x\right)\Pi +\Pi D_1\left(x\right)R_0{D}_1\left(x\right)\Pi -\Pi D_1^2\left(x\right)\Pi ,\end{array}$$

where

$$\begin{array}{c}\hfill R_0:={[\tilde{Q}+\Pi ]}^{-1}-\Pi ,\end{array}$$

(10)

and

$$\begin{array}{c}\hfill \tilde{Q}:=\tilde{P}-I.\end{array}$$

(11)

3.3. Diffusion Approximation with Equilibrium

The diffusion approximation with equilibrium or the normal deviation is obtained by considering the difference between the rescaled initial processes and the averaging limit process. This is of great interest when we have no balance condition as previously in the standard diffusion approximation scheme.

Consider now the controlled discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_{[t/\epsilon ,u]}^{\epsilon },$ averaged evolution $\overline{\mathsf{\Phi }}\left(t\right)$ (see Section 3.1) and the deviated evolution

$$\begin{array}{c}\hfill W_t^{\epsilon ,u}:={\epsilon }^{-1/2}[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-\overline{\mathsf{\Phi }}\left(t\right)].\end{array}$$

(12)

Theorem 3.

Under Assumptions A1, A5–A6 (see Section 3.1), and D3, with operators $D^{\epsilon }\left(x\right)$ in A3, instead of D1, the deviated controlled semi-Markov random evolution $W_t^{\epsilon ,u}$ weakly convergence, when $\epsilon \to 0$, to the diffusion random evolution $W_t^0$ defined by the following generator

$$\begin{array}{c}\hfill \mathrm{I}\mathrm{L}=\Pi (D_1(x,u)-{\overline{D}}_1)R_0(D_1(x,u)-{\overline{D}}_1)\Pi ,\end{array}$$

(13)

where Π is defined in (9).

4. Applications to Stochastic Systems

In this section, we give two applications in connection with the above results: additive functionals that has many application, e.g., in storage, reliability, and risk theories (see, e.g., in [3,4,19,47]), and to geometric Markov renewal processes, that also have many application including finance (see [15,16,17,18]). Our main goal here is to get the limiting processes and apply optimal control methods to receive the solutions of optimization problems. The limiting results for MC such as LLN and CLT were considered in [11,12].

4.1. Controlled Additive Functionals

Let us consider here the CAF, $\left(y_k^u\right)$, described previously in Example 1.

Averaging of CAF. Now, if we define the continuous time process

$$y_t^{\epsilon ,u}:=\epsilon \sum _{l=0}^{[t/\epsilon ]}a(z_l,u_l),$$

then from Theorem 1 it follows that this process has the following limit $y_0\left(t\right)={lim}_{\epsilon \to 0}{y}_t^{\epsilon }$

$$y_0\left(t\right)=y+\widehat{a}t,$$

where $\widehat{a}={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a(z,u).$ We suppose that ${\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)\left|a(z,u)\right|<+\infty .$

Diffusion Approximation of CAF. If we consider the continuous time process ${\xi }_t^{\epsilon ,u}$ as follows

$${\xi }_t^{\epsilon ,u}:=\epsilon \sum _{l=0}^{[t/{\epsilon }^2]}a(z_l,u_l),{\xi }_0^{\epsilon }=y,$$

then under balance condition ${\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a(z,u)=0$ and ${\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right){\left|a(z,u)\right|}^2<+\infty$ we get that the limit process ${\xi }_0\left(t\right)={lim}_{\epsilon \to 0}{\xi }_t^{\epsilon }$ has the following form,

$${\xi }_0\left(t\right)=y+b{w}_t,$$

where $b^2=2{\widehat{a}}_0-{\widehat{a}}_2$, and

$${\widehat{a}}_0={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a(z,u)R_{0}a(z,u),{\widehat{a}}_2={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a^2(z,u),$$

and $w_t$ is a standard Wiener process.

Diffusion Approximation with Equilibrium of CAF. Let us consider the following normalized additive functional,

$$w_t^{\epsilon ,u}:={\epsilon }^{-1/2}[y_t^{\epsilon ,u}-\widehat{a}t].$$

Then, this process converges to the following process, $\sigma w_t,$ where

$${\sigma }^2={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)(a(z,u)-\widehat{a})R_0(a(z,u)-\widehat{a}),$$

and $w_t$ is a standard Wiener process.

In this way, the AF $y_t^{\epsilon }$ may be presented in the following approximated form,

$$y_t^{\epsilon }\approx \widehat{a}t+\sqrt{\epsilon }\sigma w_t.$$

4.2. Controlled Geometric Markov Renewal Processes

The CGMRP is defined in the following way (see in [15,16]),

$$S_k^u:=S_0\prod _{l=1}^k(1+a(z_l,u_l)),k\in \mathrm{I}\mathrm{N},{\tau }_0=s.$$

We suppose that ${\prod }_{k=1}^0=1.$

If we define the operator $D\left(z\right)$ on $C_0\left(\mathrm{I}\mathrm{R}\right)$ in the following way,

$$D(z,u)\phi \left(s\right):=\phi \left(s\right(1+a(z,u)\left)\right),$$

then the discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_k^u\phi$ has the following presentation,

$${\mathsf{\Phi }}_k^u\phi \left(s\right)=\phi \left(S_k^u\right).$$

Averaging of CGMRP. Now, define the following sequence of processes,

$$S_t^{\epsilon ,u}:=S_0\prod _{k=1}^{[t/\epsilon ]}(1+\epsilon a(z_k,u_k)),t\in {\mathrm{I}\mathrm{R}}_{+},S_0=s.$$

Then, under averaging conditions the limit process ${\overline{S}}_t$ has the following form,

$${\overline{S}}_t=S_0{e}^{\widehat{a}t},$$

where $\widehat{a}={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a(z,u).$

Diffusion Approximation of CGMRP. If we define the following sequence of processes,

$$S^{\epsilon ,u}\left(t\right):=S_0\prod _{k=1}^{[t/{\epsilon }^2]}(1+\epsilon a(z_k,u_k)),t\in {\mathrm{I}\mathrm{R}}_{+},S_0=s,$$

then, in the diffusion approximation scheme, we have the following limit process, $S_0\left(t\right)$

$$S_0\left(t\right)=S_0{e}^{-t{\widehat{a}}_2/2}{e}^{{\sigma }_{a}w\left(t\right)},$$

where

$${\widehat{a}}_2:={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a^2(z,u),$$

$${\sigma }_a^2:={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)[a^2(z,u)/2+a(z,u)R_{0}a(z,u)].$$

It means that $S_0\left(t\right)$ satisfies the following stochastic differential equation,

$$\frac{d{S}_0\left(t\right)}{S_0\left(t\right)}=\frac{1}{2}({\sigma }_a^2-{\widehat{a}}_2)dt+{\sigma }_{a}d{w}_t,$$

where $w_t$ is a standard Wiener process.

Diffusion Approximation with Equilibrium of CGMRP. Let us consider the following normalized GMRP:

$$w_t^{\epsilon ,u}:={\epsilon }^{-1/2}[ln(S_t^{\epsilon ,u}/S_0)-\widehat{a}t].$$

It is worth noticing that in finance the expression $ln(S_t^{\epsilon ,u}/S_0)$ represents the log-return of the underlying asset (e.g., stock) $S_t^{\epsilon ,u}.$

Then, this process converges to the following process, $\sigma w_t,$ where

$${\sigma }^2={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)(a(z,u)-\widehat{a})R_0(a(z,u)-\widehat{a}),$$

and $w_t$ is a standard Wiener process.

In this way, the GMRP $S_t^{\epsilon }$ may be presented in the following approximated form,

$$S_t^{\epsilon }\approx S_0{e}^{\widehat{a}t+\sqrt{\epsilon }\sigma w_t}.$$

4.3. Controlled Dynamical Systems

We consider here discrete-time CDS and their asymptotic behaviour in series scheme: average and diffusion approximation ([9]).

Define the measurable function C on $\mathrm{I}\mathrm{R}\times E\times U$. Let us consider the difference equation

$$\begin{array}{c}\hfill y_{k+1}^{\epsilon ,u}=y_k^{\epsilon ,u}+\epsilon C(y_k^{\epsilon };z_{k+1},u_{k+1}),k\ge 0,\mathrm{and}{y}_0^{\epsilon }=u,\end{array}$$

(14)

switched by the SMC $\left(z_k\right)$.

The perturbed operators $D^{\epsilon }(z,u),x\in E$, are defined now by

$$D^{\epsilon }(z,u)\phi \left(u\right)=\phi (z+\epsilon C(z,x,u)).$$

Averaging of CDS. Under averaging assumptions the following weak convergence takes place,

$$\begin{array}{c}\hfill y_{[t/\epsilon ]}^{\epsilon ,u}\Rightarrow \overline{y}\left(t\right),\mathrm{as}\epsilon \downarrow 0,\end{array}$$

where $\overline{y}\left(t\right),t\ge 0$ is the solution of the following (deterministic) differential equation,

$$\begin{array}{c}\hfill \frac{d}{dt}\overline{y}\left(t\right)=\overline{C}\left(\overline{y}\left(t\right)\right),\mathrm{and}\overline{y}\left(0\right)=u,\end{array}$$

(15)

where $\overline{C}\left(z\right)={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C(z,x,u)$.

Diffusion Approximation of CDS. Under diffusion approximation conditions the following weak convergence takes place

$$\begin{array}{c}\hfill y_{[t/{\epsilon }^2]}^{\epsilon ,u}\Rightarrow x_t,\mathrm{as}\epsilon \downarrow 0,\end{array}$$

where $x_t,t\ge 0$, is a diffusion processes, with initial value $x_0=u$, determined by the operator

$$\begin{array}{c}\hfill \mathrm{I}\mathrm{L}\phi \left(z\right)=a\left(z\right){\phi }^{\prime }\left(z\right)+\frac{1}{2}{b}^2\left(z\right){\phi }^{″}\left(z\right),\end{array}$$

provided that $b^2\left(z\right)>0$, and drift and diffusion coefficients are defined as follows,

$$\begin{array}{ccc}& & b^2\left(z\right):=2{\overline{C}}_0\left(z\right)-{\overline{C}}_2\left(z\right),\hfill \\ & & a\left(z\right):={\overline{C}}_{01}\left(z\right)-{\overline{C}}_1\left(z\right),\hfill \end{array}$$

with:

${\overline{C}}_0\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C_0(z,x,u)$, $C_0(z,x,u):=C(z,x,u)R_{0}C(z,x,u)$,

${\overline{C}}_2\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C^{*}(z,x,u)C(z,x,u)$, where $C^{*}$ means transpose of the vector C,

${\overline{C}}_{01}\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C_{01}(z,x,u)$, $C_{01}(z,x,u):=C(z,x,u)R_0{C}_z^{\prime }(z,x,u)$,

${\overline{C}}_1\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C_1(z,x,u)$, $C_1(z,x,u):=C(z,x,u)C_z^{\prime }(z,x,u)$.

4.4. The Dynamic Programming Equations for Limiting Models in Diffusion Approximation

In this section, we consider the DPE, i.e., HJB Equations, for the limiting models in DA from Section 4.1, Section 4.2 andSection 4.3. As long as all limiting processes in DA in Section 4.1, Section 4.2 and Section 4.3 are diffusion processes, then we will set up a general approach to control for diffusion processes, see in [48].

Let $x_t^u$ be a diffusion process satisfying the following stochastic differential equation,

$$d{x}_t^u=\mu (x_t^u,u_t)dt+\sigma (x_t^u,u_t)d{w}_t,$$

where $u_t$ is the control process, $w_t$ is a standard Wiener process. Let us also introduce the following performance criterion function, $J^u(t,x)$

$$J^u(t,x):={\mathbf{E}}_{t,x}[G\left(x_T^u\right)+{\int }_t^{T}F(s,x_s^u,u_s)ds],$$

where $G\left(x\right):R\to R$ is a terminal reward function (uniformly bounded), $F(t,x,u):R_{+}\times R^2\to R$ is a running penalty/reward function (uniformly bounded), $0\le t\le T.$ The problem is to maximize this performance criteria, i.e., to find the value function

$$J(t,x):=\underset{u\in {\mathcal{U}}_{t,T}}{sup}{J}^u(t,x),$$

where ${\mathcal{U}}_{t,T}$ is the admissible set of strategies/controls which are $\mathcal{F}$-predictable, non-negative, and bounded.

The Dynamic Programming Principle (DPP) for diffusions states that the value function $J(t,x)$ satisfies the DPP

$$J(t,x)=\underset{u\in {\mathcal{U}}_{t,T}}{sup}{\mathbf{E}}_{t,x}[J^u(T,x_T^u)+{\int }_t^{T}F(s,x_s^u,u_s)ds]$$

for all $(t,x)\in [0,T]\times R.$

Moreover, the value function $J(t,x)$ above satisfies the Dynamic Programming Equation (DPE) or Hamilton–Jacobi–Bellman (HJB) Equation:

$$\begin{array}{ccc}\hfill & & \frac{\partial J(t,x)}{\partial t}+\underset{u\in {\mathcal{U}}_{t,T}}{sup}[L_t^{u}J(t,x)+F(t,x,u)]=0\hfill \\ & & J(T,x)=G\left(x\right),\hfill \end{array}$$

(16)

where $L_t^u$ is an infinitesimal generator of the diffusion process $x_t^u$ above, i.e.,

$$L_t^u=\mu (x,u)\frac{\partial }{\partial x}+\frac{{\sigma }^2(x,u)}{2}\frac{{\partial }^2}{\partial x^2}.$$

•DPE/HJB Equation for the Limiting CAF in DA (see Section 4.1)

We remind that the limiting process ${\xi }_0\left(t\right)={lim}_{\epsilon \to 0}{\xi }_t^{\epsilon }$ in this case has the following form

$${\xi }_0\left(t\right)=y+b{w}_t,$$

where $b^2=2{\widehat{a}}_0-{\widehat{a}}_2,$ and

$${\widehat{a}}_0={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a(z,u)R_{0}a(z,u),{\widehat{a}}_2={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a^2(z,u),$$

and $w_t$ is a standard Wiener process.

In this case, the DPE or HJB Equation (16) reads with the generator

$$L_t^u=\frac{1}{2}{b}^2\left(u\right)\frac{{\partial }^2}{\partial x^2},$$

with $b^2\left(u\right):=2{\widehat{a}}_0\left(u\right)-{\widehat{a}}_2\left(u\right),$ and

$${\widehat{a}}_0\left(u\right):={\int }_E\pi \left(dz\right)a(z,u)R_{0}a(z,u),{\widehat{a}}_2\left(u\right):={\int }_E\pi \left(dz\right)a^2(z,u).$$

•DPE/HJB Equation for the Limiting CGMRP in DA (see Section 4.2)

We recall that we have the following limiting process $S_0\left(t\right)$ in this case:

$$S_0\left(t\right)=S_0{e}^{-t{\widehat{a}}_2/2}{e}^{{\sigma }_{a}w\left(t\right)},$$

where

$${\widehat{a}}_2:={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a^2(z,u),$$

$${\sigma }_a^2:={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)[a^2(z,u)/2+a(z,u)R_{0}a(z,u)].$$

Furthermore, $S_0\left(t\right)$ satisfies the following stochastic differential equation (SDE),

$$\frac{d{S}_0\left(t\right)}{S_0\left(t\right)}=\frac{1}{2}({\sigma }_a^2-{\widehat{a}}_2)dt+{\sigma }_{a}d{w}_t,$$

where $w_t$ is a standard Wiener process.

In this case, the DPE or HJB Equation (16) reads with the generator

$$L_t^u=\frac{1}{2}({\sigma }_a^2\left(u\right)-a_2\left(u\right))\frac{\partial }{\partial s}+\frac{1}{2}{\sigma }_a^2\left(u\right)\frac{{\partial }^2}{\partial s^2},$$

and ${\widehat{a}}_2\left(u\right):={\int }_E\pi \left(dz\right)a^2(z,u),$${\sigma }_a^2\left(u\right):={\int }_E\pi \left(dz\right)[a^2(z,u)/2+a(z,u)R_{0}a(z,u)].$

•DPE/HJB Equation for the Limiting CDS in DA (see Section 4.3)

We remind that in the diffusion approximation the limiting process is a diffusion process $x_t$ with a generator

$$L\phi \left(z\right)=a\left(z\right){\phi }^{\prime }\left(z\right)+\frac{1}{2}{b}^2\left(z\right){\phi }^{″}\left(z\right),$$

provided that $b^2\left(z\right)>0$, and drift and diffusion coefficients are defined as follows,

$$\begin{array}{ccc}& & b^2\left(z\right):=2{\overline{C}}_0\left(z\right)-{\overline{C}}_2\left(z\right),\hfill \\ & & a\left(z\right):={\overline{C}}_{01}\left(z\right)-{\overline{C}}_1\left(z\right),\hfill \end{array}$$

with

${\overline{C}}_0\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C_0(z,x,u)$, $C_0(z,x,u):=C(z,x,u)R_{0}C(z,x,u)$,

${\overline{C}}_2\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C^{*}(z,x,u)C(z,x,u)$, where $C^{*}$ means transpose of the vector C,

${\overline{C}}_{01}\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C_{01}(z,x,u)$, $C_{01}(z,x,u):=C(z,x,u)R_0{C}_z^{\prime }(z,x,u)$,

${\overline{C}}_1\left(z\right):={\int }_E{\int }_U\pi \left(dx\right){\pi }_1\left(du\right)C_1(z,x,u)$, $C_1(z,x,u):=C(z,x,u)C_z^{\prime }(z,x,u)$.

In this case the DPE or HJB Equation (16) reads with the generator

$$L_t^u=a(z,u){\phi }^{\prime }\left(z\right)+\frac{1}{2}{b}^2(z,u){\phi }^{″}\left(z\right),$$

and

$$\begin{array}{ccc}& & b^2(z,u):=2{\overline{C}}_0(z,u)-{\overline{C}}_2(z,u),\hfill \\ & & a(z,u):={\overline{C}}_{01}(z,u)-{\overline{C}}_1(z,u),\hfill \end{array}$$

with:

${\overline{C}}_0(z,u):={\int }_E\pi \left(dx\right)C_0(z,x,u)$, $C_0(z,x,u):=C(z,x,u)R_{0}C(z,x,u)$,

${\overline{C}}_2(z,u):={\int }_E\pi \left(dx\right)C^{*}(z,x,u)C(z,x,u)$, where $C^{*}$ means transpose of the vector C,

${\overline{C}}_{01}(z,u):={\int }_E\pi \left(dx\right)C_{01}(z,x,u)$, $C_{01}(z,x,u):=C(z,x,u)R_0{C}_z^{\prime }(z,x,u)$,

${\overline{C}}_1(z,u):={\int }_E\pi \left(dx\right)C_1(z,x,u)$, $C_1(z,x,u):=C(z,x,u)C_z^{\prime }(z,x,u)$.

Remark 1.

Our construction here is equivalent to some extend to “Recurrent Processes of a semi-Markov type (RPSM)” studied first in [13,14] including limit theorems. Those results were described in more detail in [11,12]. In particular, “RPSM with Markov switching” reflects the case of independent Markov components $z_k$ and $u_k$, and “General case of RPSM” reflects the case when $u_k$ is dependent on $z_k$.

•The Merton Problem

This is an example of solution of DPE/HJB equation for the limiting CGMRP in DA. Let us consider the portfolio optimization problem proposed by Merton (1971), see in [25]. We will apply this approach to the limiting CGMRP in DA above. In this problem, the agent seeks to maximize expected wealth by trading in a risky asset and the risk-free bonds (or bank account). She/he places $\${\pi }_t$ for a total wealth $X_t$ in the risky asset $S_0\left(t\right)$ and looks to obtain the value function (performance criterion)

$$J^{\pi }(t,S,x):=\underset{\pi \in {\mathcal{U}}_{\mathcal{0},\mathcal{T}}}{sup}{\mathbf{E}}_{t,S,x}[U\left(X_T^{\pi }\right)],$$

which depends on the current wealth x and asset price $S,$ and the optimal trading strategy $\pi ,$ $U\left(x\right)$ is the agent’s utility function (e.g., exponential $(-e^{-\gamma x})$ or power $x^{\gamma }$). We suppose that the asset price $S_0\left(t\right)$ satisfies the following SDE

$$\frac{d{S}_0\left(t\right)}{S_0\left(t\right)}=(\mu -r)dt+{\sigma }_{a}d{w}_t,S_0\left(0\right)=S,$$

where

$$\mu :=\frac{1}{2}({\sigma }_a^2-{\widehat{a}}_2),$$

$${\widehat{a}}_2:={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)a^2(z,u),$$

$${\sigma }_a^2:={\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)[a^2(z,u)/2+a(z,u)R_{0}a(z,u)].$$

Here, $\mu$ represents the expected continuously compounded rate of growth of the traded asset, r is the continuously compounded rate of return of the risk-free asset (bond or bank account).

The wealth process $X_t^{\pi }$ follows the following SDE,

$$d{X}_t^{\pi }=({\pi }_t(\mu -r)+r{X}_t^{\pi })dt+{\pi }_t{\sigma }_{a}d{w}_t,X_0^{\pi }=x.$$

From the SDEs for $S_0\left(t\right)$ and for $X_t^{\pi }$ above we conclude that the infinitesimal generator for the pair $(S_0\left(t\right),X_t^{\pi })$ is

$$L_t^{\pi }=(rx+(\mu -r)\pi )\frac{\partial }{\partial x}+\frac{1}{2}{\sigma }_a^2\pi \frac{{\partial }^2}{\partial x^2}+(\mu -r)S\frac{\partial }{\partial S}+\frac{1}{2}{\sigma }_a^2{S}^2\frac{{\partial }^2}{\partial S^2}+{\sigma }_a\pi \frac{{\partial }^2}{\partial x\partial S}.$$

From HJB equation for the limiting CGRMP in DA it follows that the value function

$$J(t,S,x)=\underset{\pi \in {\mathcal{U}}_{t,T}}{sup}{J}^{\pi }(t,S,x)$$

should satisfy the equation

$$\frac{\partial J(t,S,x)}{\partial t}+\underset{\pi }{sup}[L_t^{\pi }J(t,S,x)]=0$$

with terminal condition $J(T,S,x)=U\left(x\right).$

The explicit solution of this PDE depends on the explicit form of the utility function $U\left(x\right).$ Let us take the exponential utility function

$$U\left(x\right)=-e^{-\gamma x},\gamma >0,x\in R.$$

In this case we can find that the optimal amount to invest in the risky asset is a deterministic function of time

$${\pi }_t^{*}=\frac{({\sigma }_a^2-{\widehat{a}}_2)/2-r}{\gamma {\sigma }_a^2}{e}^{-r(T-t)}.$$

5. Rates of Convergence in Averaging and Diffusion Approximations

The rate of convergence in a limit theorem is important in several ways, both theoretical and practical. We present here the rates of convergence of CDTSMRE in the averaging, diffusion approximation and diffusion approximation with equilibrium schemes and, as corollaries, we give the rates of convergence for CAF and CGMRP in the corresponding limits.

Proposition 1.

The Rate of Convergence of CDTSMRE in the Averaging has the following form,

$$\left|\right|\mathbf{E}[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi ]-\overline{\mathsf{\Phi }}\left(t\right)\phi \left|\right|\le \epsilon A(T,\phi ,||R_0\left|\right|,D_1(z,u)),$$

where $A(T,\phi ,|\left|R_0\right||,D_1(z,u))$ is a constant, and $0\le t\le T.$

The proof of this proposition is given in Section 6.4.

Proposition 2.

The Rate of Convergence of CDTSMRE in the Diffusion Approximation takes the following form,

$$\left|\right|\mathbf{E}[{\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}\phi ]-{\mathsf{\Phi }}_0\left(t\right)\phi \left|\right|\le \epsilon D(T,|\left|\phi \right||,||R_0\left|\right|,\left|\right|D_1\left|\right|,\left|\right|D_2\left|\right|),$$

where $D(T,|\left|\phi \right||,|\left|R_0\right||,|\left|D_1\right||,|\left|D_2\right|\left|\right)$ is a constant, and $0\le t\le T.$

Proposition 3.

The Rate of Convergence of CDTSMRE in Diffusion Approximation with Equilibrium has the following form,

$$\left|\right|\mathbf{E}[W_t^{\epsilon ,u}\phi ]-W_t^0\phi \left|\right|\le \sqrt{\epsilon }N(T,|\left|\phi \right||,||R_0\left|\right|,\left|\right|D_1\left|\right|,\left|\right|D_1^2\left|\right|),$$

where $N(T,|\left|\phi \right||,|\left|R_0\right||,|\left|D_1\right||,|\left|D_1^2\right|\left|\right)$ is a constant and $0\le t\le T.$

The proofs of the above Propositions 2 and 3 are similar as the proof of Proposition 1. We give in what follows some rate of convergence results (Corollaries 1 and 2) concerning applications.

Corollary 1.

The Rate of Convergence in the Limit Theorems for CAF:

- Rate of Convergence in Averaging:

$$\left|\right|\mathbf{E}{y}_t^{\epsilon ,u}-y_0\left(t\right)\left|\right|\le \epsilon a(T,||R_0\left|\right|,\left|\right|a\left|\right|),$$

where $a(T,|\left|R_0\right||,|\left|a\right|\left|\right)$ is a constant, and $0\le t\le T.$

- Rate of Convergence in Diffusion Approximation

$$\left|\right|\mathbf{E}{\xi }_t^{\epsilon ,u}-{\xi }_0\left(t\right)\left|\right|\le \epsilon d(T,||R_0\left|\right|,\left|\right|a\left|\right|,\left|\right|a^2\left|\right|),$$

where $d(T,|\left|R_0\right||,|\left|a\right||,|\left|a^2\right|\left|\right)$ is a constant, and $0\le t\le T.$

- Rate of Convergence in diffusion approximation with equilibrium for CAF

$$\left|\right|\mathbf{E}{W}_t^{\epsilon ,u}-w_t\left|\right|\le \sqrt{\epsilon }n(T,||R_0\left|\right|,\left|\right|a\left|\right|,\left|\right|a^2\left|\right|),$$

where $n(T,|\left|R_0\right||,|\left|a\right||,|\left|a^2\right|\left|\right)$ is a constant, and $0\le t\le T.$

Corollary 2.

The Rate of Convergence in the Limit Theorems for CGMRP:

- Rate of Convergence in Averaging

$$\left|\right|\mathbf{E}{S}_t^{\epsilon ,u}-{\overline{\tau }}_t\left|\right|\le \epsilon a(T,||R_0\left|\right|,\left|\right|a\left|\right|),$$

where $a(T,|\left|R_0\right||,|\left|a\right|\left|\right)$ is a constant, and $0\le t\le T.$

- Rate of Convergence in Diffusion Approximation

$$\left|\right|\mathbf{E}{S}_t^{\epsilon ,u}-{\tau }_0\left(t\right)\left)\right||\le \epsilon d(T,||R_0\left|\right|,\left|\right|a\left|\right|,\left|\right|a^2\left|\right|),$$

where $d(T,|\left|R_0\right||,|\left|a\right||,|\left|a^2\right|\left|\right)$ is a constant, and $0\le t\le T.$

- Rate of Convergence in diffusion approximation with equilibrium

$$\left|\right|\mathbf{E}{W}_t^{\epsilon ,u}-w_t\left|\right|\le \sqrt{\epsilon }n(T,||R_0\left|\right|,\left|\right|a\left|\right|,\left|\right|a^2\left|\right|),$$

where $n(T,|\left|R_0\right||,|\left|a\right||,|\left|a^2\right|\left|\right)$ is a constant, and $0\le t\le T.$

6. Proofs

The proofs here have almost the same general construction scheme as in our paper [20] except that we consider also the control process. Let $C_B[0,\infty )$ be the space of B-valued continuous functions defined on $[0,\infty )$.

6.1. Proof of Theorem 1

The proof of the relative compactness of CDTSMRE in the average approximation is based on the following four lemmas.

The CDTSMRE ${\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi$, see (3), is weakly compact in $D_B[0,\infty )$ with limit points into $C_B[0,\infty )$.

Lemma 1.

Under Assumptions A1–A7, the limit points of ${\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi$, $\phi \in B_0$, as $\epsilon \to 0$, belong to $C_B[0,\infty )$.

Proof. 

Assumptions A5–A6 imply that the discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_k^u\phi$ is a contractive operator in H and, therefore, $\left|\right|{\mathsf{\Phi }}_k^u\phi {\left|\right|}_H$ is a supermartingale for any $\phi \in H,$ where $\left|\right|·{\left|\right|}_H$ is a norm in Hilbert space H ([4,9]) Obviously, the same properties satisfy the following family ${\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}.$ Using Doob’s inequality for the supermartingale $\left|\right|{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}{\left|\right|}_H$ we obtain

$$\mathbf{P}\{{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\in K_{\Delta }\}\ge 1-\Delta ,$$

where $K_{\Delta }$ is a compact set in B and $\Delta$ is any small number. It means that sequence ${\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}$ is tight in $B.$ Taking into account conditions A1–A6, we obtain that discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}$ is weakly compact in $D_B[0,+\infty )$ with limit points in $C_B[0,+\infty ),$$\phi \in B_0.$

Let $J_t^{\epsilon ,u}:=J({\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u};[t/\epsilon ]):={sup}_{k\le [t/\epsilon ]}∥{\mathsf{\Phi }}_{[t/\epsilon ]+k}^{\epsilon ,u}\phi -{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi ∥$, and let $K_{\Delta }$ be a compact set from compact containment condition $\Delta >0$. It is sufficient to show that $J_t^{\epsilon ,u}$ weakly converges to zero. This is equivalent to the convergence of $J_t^{\epsilon ,u}$ in probability as $\epsilon \to 0$.

From the very definition of $J_t^{\epsilon ,u}$ and A3, we obtain

$$J_t^{\epsilon ,u}{\mathbf{1}}_{K_{\Delta }}\le \epsilon \underset{k\le [t/\epsilon ]}{sup}\underset{\phi \in S_{\Delta }}{sup}(∥D_1(z_k,u_k)\phi ∥+∥D_0^{\epsilon }(z_k,u_k)\phi ∥),$$

where ${\mathbf{1}}_{K_{\Delta }}$ is the indicator of the set $K_{\Delta }$, and $S_{\Delta }$ is the finite $\delta$-set for $K_{\Delta }$. Then, for $\delta <\Delta$, we have

$$\begin{array}{ccc}\hfill {\mathbf{P}}_{\pi \times {\pi }_1}(J_t^{\epsilon ,u}{\mathbf{1}}_{K_{\Delta }}>\Delta )& \le & {\mathbf{P}}_{\pi \times {\pi }_1}(\underset{k\le [t/\epsilon ]}{sup}{D}_k>(\Delta -\delta )/\epsilon )\hfill \\ & =& \sum _{i=1}^{[t/\epsilon ]}{\mathbf{P}}_{\pi \times {\pi }_1}(\{\underset{k\le [t/\epsilon ]}{sup}{D}_k>(\Delta -\delta )/\epsilon \}\cap D_i)\hfill \\ & \le & {\epsilon }^2[t/\epsilon ]\underset{\phi \in S_{\Delta }}{sup}[{\tilde{P}}^{[t/\epsilon ]}({∥D_1(x,u)\phi ∥}^2\hfill \\ & & +2∥D_1(x,u)\phi ∥∥D_0^{\epsilon }(x,u)\phi ∥+{∥D_0^{\epsilon }(x,u)\phi ∥}^2)],\hfill \end{array}$$

where $D_k:={sup}_{\phi \in S_{\Delta }}(∥D_1(z_k,u_k)\phi ∥+∥D_0^{\epsilon }(z_k,u_k)\phi ∥)$, and

$D_i:=\{\omega :D_k\mathrm{contains}\mathrm{the}\mathrm{maximum}\mathrm{for}\mathrm{the}\mathrm{first}\mathrm{time}\mathrm{on}\mathrm{the}\mathrm{variable}{D}_i\}$.

It is worth noticing that the operator ${\tilde{P}}^k$ is bounded when $k\to \infty$. So is the case for ${\tilde{P}}^{[t/\epsilon ]}$ when $\epsilon \to 0$.

Taking both $\epsilon$ and $\delta$ go to 0 we obtain the proof of the this lemma. □

Let us now consider the continuous time martingale

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}:=M_{[t/\epsilon ]}^{\epsilon }={\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-I-\sum _{k=0}^{[t/\epsilon ]-1}{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}-{\mathsf{\Phi }}_k^{\epsilon ,u}\mid {\mathcal{F}}_k].\end{array}$$

(17)

Lemma 2.

The process

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}:={\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-I-\sum _{\ell =0}^{[t/\epsilon ]-1}[\tilde{P}{D}^{\epsilon }(·,u)-I]{\mathsf{\Phi }}_{\ell }^{\epsilon ,u},\end{array}$$

is an ${\mathcal{F}}_{[t/\epsilon ]}^u$-martingale.

Proof. 

As long as $M_k^{\epsilon ,u}:={\mathsf{\Phi }}_k^{\epsilon ,u}-I-{\sum }_{\ell =0}^{k-1}[\tilde{P}{D}^{\epsilon }(·,u)-I]{\mathsf{\Phi }}_{\ell }^{\epsilon ,u}$, is a martingale $M_t^{\epsilon ,u}=M_{[t/\epsilon ]}^{\epsilon ,u}$ is an ${\mathcal{F}}_{[t/\epsilon ]}^u$-martingale. Here, we have ${\mathbf{E}}_{\pi \times {\pi }_1}[M_{k+1}^{\epsilon }\mid {\mathcal{F}}_k^u]=M_k^{\epsilon ,u}$ which can be easily checked. □

Lemma 3.

The family $\ell ({\sum }_{k=0}^{[t/\epsilon ]}{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi -{\mathsf{\Phi }}_k^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u])$ is relatively compact for all $\ell \in B_0^{*}$, dual of the space $B_0$.

Proof. 

Let

$$N_t^{\epsilon ,u}:=\sum _{k=0}^{[t/\epsilon ]}{\mathbf{E}}_{\pi \times {\pi }_1}[({\mathsf{\Phi }}_{k+1}^{\epsilon ,u}-{\mathsf{\Phi }}_k^{\epsilon ,u})\phi \mid {\mathcal{F}}_k^u].$$

Then,

$$N_t^{\epsilon ,u}=\sum _{k=0}^{[t/\epsilon ]}[\tilde{P}{D}^{\epsilon }(·,u)-I]{\mathsf{\Phi }}_k^{\epsilon ,u}.$$

As long as ${\mathsf{\Phi }}_{k+1}^{\epsilon ,u}=D^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u},$ we obtain

$${\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u]={\mathbf{E}}_{\pi \times {\pi }_1}[D^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u].$$

Then,

$$\begin{array}{ccc}& & \left|\ell (\sum _{k=[t/\epsilon ]+1}^{[(t+\eta )/\epsilon ]}{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi -{\mathsf{\Phi }}_k^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u])\right|\hfill \\ & =& \left|\ell (\sum _{k=[t/\epsilon ]+1}^{[(t+\eta )/\epsilon ]}[\tilde{P}{D}^{\epsilon }(z_{k+1},z_{k+1})-I]{\mathsf{\Phi }}_k^{\epsilon ,u}\phi )\right|\hfill \\ & \le & \epsilon ∥\ell ∥([(t+\eta )/\epsilon ]-[t/\epsilon ]-1)∥\tilde{P}(D_1(z_{k+1},u_{k+1})+D_0^{\epsilon }(z_{k+1},u_{k+1}))\phi ∥\hfill \\ & \le & \epsilon ∥\ell ∥\frac{\eta }{\epsilon }∥\tilde{P}(D_1(·,u)+D_0^{\epsilon }(·,u))\phi ∥\hfill \\ & =& \eta ∥\ell ∥∥\tilde{P}(D_1(·,u)+D_0^{\epsilon }(·,u))\phi ∥\to 0,\eta \to 0,\hfill \end{array}$$

as $∥\tilde{P}(D_1(·,u)+D_0^{\epsilon }(·,u))\phi ∥$ is bounded for any $\phi \in B_0$.

It means that the family $\ell ({\sum }_{k=0}^{[t/\epsilon ]}{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi -{\mathsf{\Phi }}_k^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u])$, is relatively compact for any $\ell \in B_0^{*}$. □

Lemma 4.

The family $\ell \left(M_{[t/\epsilon ]}^{\epsilon ,u}\phi \right)$ is relatively compact for any $\ell \in B_0^{*}$, and any $\phi \in B_0$.

Proof. 

It is worth noticing that the martingale $M_{[t/\epsilon ]}^{\epsilon ,u}$ can be represented in the form of the martingale differences

$$M_{[t/\epsilon ]}^{\epsilon ,u}=\sum _{k=0}^{[t/\epsilon ]-1}{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi -{\mathbf{E}}_{\pi \times {\pi }_1}({\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u)].$$

Then, using the equality

$${\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u]={\mathbf{E}}_{\pi \times {\pi }_1}[D^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u],$$

we get

$$\begin{array}{ccc}\hfill M_{[(t+\eta )/\epsilon ]}^{\epsilon ,u}\phi -M_{[t/\epsilon ]}^{\epsilon ,u}\phi & =& \sum _{k=[t/\epsilon ]+1}^{[(t+\eta )/\epsilon ]}[D^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u}\phi -{\mathbf{E}}_{\pi \times {\pi }_1}[D^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u}\phi \mid {\mathcal{F}}_k^u]\}\hfill \\ \hfill & =& \sum _{k=[t/\epsilon ]+1}^{[(t+\eta )/\epsilon ]}[D^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u}\phi -\tilde{P}{D}^{\epsilon }(z_{k+1},u_{k+1}){\mathsf{\Phi }}_k^{\epsilon ,u}\phi ]\hfill \\ \hfill & =& \sum _{k=[t/\epsilon ]+1}^{[(t+\eta )/\epsilon ]}[D^{\epsilon }(z_{k+1},u_{k+1})-\tilde{P}{D}^{\epsilon }(z_{k+1},u_{k+1})]{\mathsf{\Phi }}_k^{\epsilon ,u}\phi ,\hfill \end{array}$$

for any $\eta >0$. Now, from the above, we get

$$\begin{array}{ccc}& & {\mathbf{E}}_{\pi \times {\pi }_1}\left|\ell (M_{[(t+\eta )/\epsilon ]}^{\epsilon ,u}\phi -M_{[t/\epsilon ]}^{\epsilon ,u}\phi )\right|\hfill \\ & \le & ([t+\eta )/\epsilon ]-[t/\epsilon ])\epsilon {\mathbf{E}}_{\pi \times {\pi }_1}(∥D_1(z_{k+1},u_{k+1})\phi ∥+∥D_0^{\epsilon }(z_{k+1},u_{k+1})\phi ∥\hfill \\ & & +∥\tilde{P}{D}_1(·,u)\phi ∥+∥\tilde{P}{D}_0^{\epsilon }(·,u)\phi ∥)\hfill \\ & \le & 2\eta (∥\tilde{P}{D}_1(·,u)\phi ∥+∥\tilde{P}{D}_0^{\epsilon }(·,u)\phi ∥)\to 0,\eta \to 0,\hfill \end{array}$$

which proves the lemma. □

Now the proof of Theorem 1 is achieved as follows.

From Lemmas 2–4 and the representation (17) it follows that the family $\ell ({\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi )$ is relatively compact for any $\ell \in B_0^{*}$, and any $\phi \in B_0$.

Moreover, let ${\mathrm{I}\mathrm{L}}^{\epsilon }\left(x\right)$, $x\in E$, be a family of perturbed operators defined on B as follows,

$$\begin{array}{c}\hfill {\mathrm{I}\mathrm{L}}^{\epsilon ,u}\left(x\right):={\epsilon }^{-1}\tilde{Q}+\tilde{P}{D}_1(x,u)+\tilde{P}{D}_0^{\epsilon }(x,u).\end{array}$$

(18)

Then, the process

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}={\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-I-\epsilon \sum _{\ell =0}^{[t/\epsilon ]-1}{\mathrm{I}\mathrm{L}}^{\epsilon ,u}{\mathsf{\Phi }}_{\ell }^{\epsilon ,u},\end{array}$$

(19)

is an ${\mathcal{F}}_t^{\epsilon ,u}$-martingale.

The following singular perturbation problem, for the non-negligible part of compensating operator, ${\mathrm{I}\mathrm{L}}^{\epsilon ,u}$, denoted by ${\mathrm{I}\mathrm{L}}_0^{\epsilon ,u}\left(x\right):={\epsilon }^{-1}\tilde{Q}+\tilde{P}{D}_1(x,u)$,

$$\begin{array}{c}\hfill {\mathrm{I}\mathrm{L}}_0^{\epsilon ,u}{\phi }^{\epsilon }=\mathrm{I}\mathrm{L}\phi +\epsilon {\theta }^{\epsilon ,u},\end{array}$$

(20)

on the test functions ${\phi }^{\epsilon }(z,x)=\phi \left(z\right)+\epsilon {\phi }_1(z,x)$, has the solution (see [3] Proposition 5.1): $\phi \in \mathcal{N}(\tilde{Q})$, ${\phi }_1=R_0{\tilde{D}}_1\phi$, with ${\tilde{D}}_1(x,u)=\tilde{P}{D}_1(x,u)-{\widehat{D}}_1$, ${\widehat{D}}_1={\int }_E\pi \times {\pi }_1\left(dx\right)D_1(x,u)$, and ${\theta }^{\epsilon ,u}\left(x\right)=(P^{♯}\times P^u)D_1(x,u)R_0{\tilde{D}}_1(x,u)\phi$.

The limit operator is then given by

$$\begin{array}{c}\hfill \mathrm{I}\mathrm{L}\Pi =\Pi D_1(·,u)\Pi ,\end{array}$$

(21)

form which we get the contracted limit operator

$$\begin{array}{c}\hfill \widehat{\mathrm{I}\mathrm{L}}={\widehat{D}}_1.\end{array}$$

(22)

We note that martingale $M_t^{\epsilon ,u}$ has the following asymptotic representation,

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}={\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-I-\epsilon \sum _{\ell =0}^{[t/\epsilon ]-1}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_{\ell }^{\epsilon ,u}+O_{\phi }\left(\epsilon \right),\end{array}$$

(23)

where $\left|\right|O_{\phi }\left(\epsilon \right)\left|\right|\to 0$, as $\epsilon \to 0$. The families $l\left(M_{[t/\epsilon ]}\right)$ and $l({\sum }_{\ell =0}^{[t/\epsilon ]-1}[(P^{♯}\times P^u)D^{\epsilon }(·,u)-I]{\mathsf{\Phi }}_{\ell }^{\epsilon ,u})$ are weakly compact for all $l\in B_0^{*}$ in a dense subset $B_0^{*}\subset B.$ It means that family $l({\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u})$ is also weakly compact. In this way, the sum $\epsilon {\sum }_{\ell =0}^{[t/\epsilon ]-1}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_{\ell }^{\epsilon ,u}\phi$ converges, as $\epsilon \to 0$, to the integral ${\int }_0^t\widehat{\mathrm{I}\mathrm{L}}\overline{\mathsf{\Phi }}\left(s\right)\phi ds.$ The quadratic variation of the martingale $l\left(M_t^{\epsilon ,u}\phi \right)$ tends to zero when $\epsilon \to 0,$ thus, $M_t^{\epsilon ,u}\phi \to 0$ when $\epsilon \to 0,$ for any $f\in B_0$ and for any $l\in B_0^{*}.$ Passing to the limit in (23), when $\epsilon \to 0$, we get ${\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi {\to }_{\epsilon \to 0}\overline{\mathsf{\Phi }}\left(t\right)\phi ,$ where $\overline{\mathsf{\Phi }}\left(t\right)$ is defined in (6).

The quadratic variation of the martingale $M_t^{\epsilon ,u}$, in the average approximation, is

$$\begin{array}{c}\hfill \langle \ell (M_{[t/\epsilon ]}^{\epsilon ,u}\rangle =\sum _{k=0}^{[t/\epsilon ]}{\mathbf{E}}_{\pi \times {\pi }_1}[{\ell }^2(M_{k+1}^{\epsilon ,u}{\phi }^{\epsilon }-M_k^{\epsilon ,u}{\phi }^{\epsilon })\mid {\mathcal{F}}_k^u],\end{array}$$

(24)

where ${\phi }^{\epsilon }\left(x\right)=\phi \left(x\right)+\epsilon {\phi }_1\left(x\right)$. Hence

$$\begin{array}{c}\hfill \ell (M_{k+1}^{\epsilon ,u}{\phi }^{\epsilon }-M_k^{\epsilon ,u}{\phi }^{\epsilon })=\ell \left((M_{k+1}^{\epsilon ,u}-M_k^{\epsilon ,u})\phi \right)+\epsilon \ell \left((M_{k+1}^{\epsilon ,u}-M_k^{\epsilon ,u}){\phi }_1\right),\end{array}$$

and

$$\begin{array}{c}\hfill M_{k+1}^{\epsilon ,u}-M_k^{\epsilon ,u}={\mathsf{\Phi }}_{k+1}^{\epsilon ,u}-{\mathsf{\Phi }}_k^{\epsilon ,u}-{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}-{\mathsf{\Phi }}_k^{\epsilon ,u}\mid {\mathcal{F}}_k^u].\end{array}$$

(25)

Therefore,

$$\begin{array}{ccc}& & \ell (M_{k+1}^{\epsilon ,u}{\phi }^{\epsilon }-M_k^{\epsilon ,u}{\phi }^{\epsilon })\hfill \\ & =& \ell \left((D{(z_{k+1},u_{k+1})}^{\epsilon }-I){\mathsf{\Phi }}_k^{\epsilon ,u}\phi \right)-{\mathbf{E}}_{\pi \times {\pi }_1}[(D^{\epsilon }(z_{k+1},u_{k+1})-I)\phi \mid {\mathcal{F}}_k^u]\hfill \\ & & +\epsilon \ell \left((D^{\epsilon }(z_{k+1},u_{k+1})-I){\phi }_1\right)-{\mathbf{E}}_{\pi \times {\pi }_1}[(D{(z_{k+1},u_{k+1})}^{\epsilon }-I){\phi }_1\mid {\mathcal{F}}_k^u]\hfill \\ & =& \epsilon \ell \left((D_1(z_{k+1},u_{k+1})+D_0^{\epsilon }(z_{k+1},u_{k+1}))\phi \right)\hfill \\ & & -\epsilon {\mathbf{E}}_{\pi \times {\pi }_1}[(D_1(z_{k+1},u_{k+1})+D_0^{\epsilon }(z_{k+1},u_{k+1}))\phi \mid {\mathcal{F}}_k^u]\hfill \\ & & +{\epsilon }^2\ell \left((D_1(z_{k+1},u_{k+1})+D_0^{\epsilon }(z_{k+1},u_{k+1})){\phi }_1\right)\hfill \\ & & -{\epsilon }^2{\mathbf{E}}_{\pi }[(D_1(z_{k+1},u_{k+1})+D_0^{\epsilon }(z_{k+1},u_{k+1})){\phi }_1\mid {\mathcal{F}}_k^u].\hfill \end{array}$$

(26)

Now, from (24) and (26) and from boundedness of all operators in (26) with respect to ${\mathbf{E}}_{\pi \times {\pi }_1}$, it follows that $\langle \ell (M_{[t/\epsilon ]}^{\epsilon ,u}\rangle$ goes to 0 when $\epsilon \to 0$, and the quadratic variation of limit process $M_t^{0,u}$, for the martingale $M_t^{\epsilon ,u}$, is equals to 0.

In this case, the limit martingale $M_t^0$ equals to 0. Therefore, the limit equation for $M_t^{\epsilon ,u}$ has the form (6). As long as the solution of the martingale problem for operator $\widehat{\mathrm{I}\mathrm{L}}$ is unique, then it follows that the solution of the Equation (6) is unique as well [49,50]. It is worth noticing that operator $\widehat{\mathrm{I}\mathrm{L}}$ is a first order operator (${\widehat{D}}_1$, see (22)). Finally, the operator $\widehat{\mathrm{I}\mathrm{L}}$ generates a semigroup, then $\overline{\mathsf{\Phi }}\left(t\right)\phi =exp[\widehat{\mathrm{I}\mathrm{L}}t]\phi$ and the latter representation is unique.

6.2. Proof of Theorem 2

We can prove the relative compactness of the family ${\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}$ exactly on the same way, and following the same steps as above. However, in the case of diffusion approximation the limit continuous martingale $M_0\left(t\right)$ for the martingale $M_t^{\epsilon }$ has quadratic variation that is not zero, that is,

$$\begin{array}{c}\hfill M_0\left(t\right)\phi ={\mathsf{\Phi }}_0\left(t\right)\phi -\phi -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_0\left(s\right)ds,\end{array}$$

and so $\langle \ell \left(M_0\right)\rangle \ne 0$, for $\ell \in B_0^{*}$.

Moreover, operator $\widehat{\mathrm{I}\mathrm{L}}$ defined in Theorem 2 is a second-order kind operator as it contains operator ${\widehat{D}}_2$ and $\Pi D_1{R}_0\tilde{P}{D}_1\Pi$, compare with the first-order operator $\widehat{\mathrm{I}\mathrm{L}}$ in (7).

Let ${\mathrm{I}\mathrm{L}}^{\epsilon ,u}\left(x\right)$, $x\in E$, be a family of perturbed operators defined on B as follows,

$$\begin{array}{c}\hfill {\mathrm{I}\mathrm{L}}^{\epsilon ,u}\left(x\right):={\epsilon }^{-2}\tilde{Q}+{\epsilon }^{-1}\tilde{P}{D}_1(x,u)+\tilde{P}{D}_2(x,u)+\tilde{P}{D}_0^{\epsilon }(x,u).\end{array}$$

(27)

Then, the process

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}={\mathsf{\Phi }}_{[t/{\epsilon }^{\epsilon }]}^{\epsilon ,u}-I-{\epsilon }^2\sum _{k=0}^{[t/{\epsilon }^2]-1}{\mathrm{I}\mathrm{L}}^{\epsilon ,u}{\mathsf{\Phi }}_k^{\epsilon ,u},\end{array}$$

(28)

is an ${\mathcal{F}}_t^{\epsilon ,u}$-martingale with mean value zero.

For the non-negligible part of compensating operator, ${\mathrm{I}\mathrm{L}}^{\epsilon ,u}$, denoted by ${\mathrm{I}\mathrm{L}}_0^{\epsilon ,u}\left(x\right):={\epsilon }^{-2}\tilde{Q}+{\epsilon }^{-1}\tilde{P}{D}_1(x,u)+\tilde{P}{D}_2(x,u)$, consider the following singular perturbation problem,

$$\begin{array}{c}\hfill {\mathrm{I}\mathrm{L}}_0^{\epsilon ,u}{\phi }^{\epsilon }=\mathrm{I}\mathrm{L}\phi +\epsilon {\theta }^{\epsilon ,u}\left(x\right),\end{array}$$

(29)

where ${\phi }^{\epsilon }(z,x)=\phi \left(z\right)+\epsilon {\phi }_1(z,x)+{\epsilon }^2{\phi }_2(z,x)$. The solution of this problem is realized by the vectors (see in [3], Proposition 5.2)

$${\phi }_1=R_0\tilde{P}{D}_1(x,u)\phi ,{\phi }_2=R_0\tilde{A}\phi ,$$

with $\tilde{A}(x,u):=A(x,u)-\widehat{A}$. Finally, the negligible term ${\theta }^{\epsilon ,u}\left(x\right)$ is

$${\theta }^{\epsilon ,u}\left(x\right)=[\tilde{P}{D}_1(x,u)+\epsilon \tilde{P}{D}_2(x,u)]{\phi }_2+\tilde{P}{D}_2(x,u){\phi }_1.$$

Of course, $\phi \in \mathcal{N}\left(\tilde{Q}\right)$.

Now the limit operator $\mathrm{I}\mathrm{L}$ is given by

$$\begin{array}{c}\hfill \mathrm{I}\mathrm{L}=\tilde{P}{D}_2(·,u)+\tilde{P}{D}_1(·,u)R_0\tilde{P}{D}_1(·,u),\end{array}$$

(30)

from which, the contracted operator on the null space $\mathcal{N}\left(\tilde{Q}\right)$ is

$$\begin{array}{c}\hfill \widehat{\mathrm{I}\mathrm{L}}={\widehat{D}}_2\Pi +\Pi D_1(x,u)R_0\tilde{P}{D}_1(x,u)\Pi .\end{array}$$

(31)

Moreover, due to the balance condition (8) we get the limit operator.

We worth noticing that Assumptions A5–A7 and D1–D3 imply that discrete-time semi-Markov random evolution ${\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}\phi$ is a contractive operator in H and, therefore, $\left|\right|{\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}\phi {\left|\right|}_H$ is a supermartingale for any $\phi \in H,$ where $\left|\right|·{\left|\right|}_H$ is a norm in Hilbert space H ([4,9]). By Doob’s inequality for the supermartingale $\left|\right|{\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}{\left|\right|}_H$ we obtain

$$\mathbf{P}\{{\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}\in K_{\Delta }^1\}\ge 1-\Delta ,$$

where $K_{\Delta }^1$ is a compact set in B and $\Delta$ is any positive small real number.

We conclude that under Assumptions A5–A7 and D1–D3, the family $M_t^{\epsilon ,u}$ is tight and is weakly compact in $D_B[0,+\infty )$ with limit points in $C_B[0,+\infty ).$

Moreover, under Assumptions A5–A6 and D1–D2, the martingale $M_t^{\epsilon ,u}$ has the following asymptotic presentation:

$$\begin{array}{c}\hfill M_t^{\epsilon ,u}\phi ={\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}\phi -\phi -{\epsilon }^2\sum _{k=0}^{[t/{\epsilon }^2]-1}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_k^{\epsilon ,u}\phi +O_{\phi }\left(\epsilon \right),\end{array}$$

(32)

where $\left|\right|O_{\phi }\left(\epsilon \right)\left|\right|\to 0$, as $\epsilon \to 0$. The families $l\left(M_t^{\epsilon ,u}\varphi \right)$ and $l({\epsilon }^2{\sum }_{k=0}^{[t/{\epsilon }^{\epsilon }]-1}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_k^{\epsilon ,u}\phi )$ are weakly compact for all $l\in B^{*}$ and $\phi \in B_0.$ It means that ${\mathsf{\Phi }}_{[t/{\epsilon }^2]}^{\epsilon ,u}$ is also weakly compact and has a limit.

Let us denote the previous limit by ${\mathsf{\Phi }}_0\left(t\right)$, then the sum ${\epsilon }^2{\sum }_{k=0}^{[t/{\epsilon }^{\epsilon }]-1}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_k^{\epsilon ,u}\phi$ converges to the integral ${\int }_0^t\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_0\left(s\right)\phi ds.$ Let $M_0\left(t\right)$ also be a limit martingale for $M_t^{\epsilon ,u}$ when $\epsilon \to 0.$ Then, from the previous steps and (32), we obtain

$$\begin{array}{c}\hfill M_0\left(t\right)\phi ={\mathsf{\Phi }}_0\left(t\right)\phi -\phi -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_0\left(s\right)\phi ds.\end{array}$$

(33)

As long as martingale $M_t^{\epsilon ,u}$ has mean value zero, the martingale $M_0\left(t\right)$ has also mean value zero. If we take the mean value from both parts of (33) we get

$$\begin{array}{c}\hfill 0=\mathbf{E}{\mathsf{\Phi }}_0\left(t\right)\phi -\phi -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}\mathbf{E}{\mathsf{\Phi }}_0\left(t\right)\phi ds,\end{array}$$

(34)

or, solving it, we get

$$\begin{array}{c}\hfill \mathbf{E}{\mathsf{\Phi }}_0\left(t\right)\phi =exp[\widehat{\mathrm{I}\mathrm{L}}t]\phi .\end{array}$$

(35)

The last equality means that the operator $\widehat{\mathrm{I}\mathrm{L}}$ generates a semigroup, namely, $U\left(t\right):=\mathbf{E}{\mathsf{\Phi }}_0\left(t\right)\phi =exp[\widehat{\mathrm{I}\mathrm{L}}t]\phi .$ Now, the uniqueness of the limit evolution ${\mathsf{\Phi }}_0\left(t\right)$ in diffusion approximation approximation follows from the uniqueness of solution of the martingale problem for ${\mathsf{\Phi }}_0\left(t\right)$ (uniqueness of the limit process under weak compactness). As long as the solution of the martingale problem for operator $\widehat{\mathrm{I}\mathrm{L}}$ is unique, then it follows that the solution of the Equation (34) is unique as well [49,50].

6.3. Proof of Theorem 3

We note that $W_t^{\epsilon ,u}$ in (12) has the following presentation,

$$\begin{array}{c}\hfill W_t^{\epsilon ,u}={\epsilon }^{-1/2}\{\sum _{k=1}^{[t/\epsilon ]}[D^{\epsilon }(z_{k-1},u_{k-1})-I]{\mathsf{\Phi }}_k^{\epsilon ,u}-{\int }_0^t{\overline{D}}_1\overline{\mathsf{\Phi }}\left(s\right)ds\}.\end{array}$$

(36)

As the balance condition $\Pi (D_1-{\widehat{D}}_1)=0,$ holds, then we apply the diffusion approximation algorithm (see Section 3.2), i.e., to the right-hand side of (36) with the following operators, $D_2=0$ and $(D_1\left(z\right)-{\overline{D}}_1)$ instead of $D_1\left(z\right).$ It is worth mentioning that the family $W_t^{\epsilon ,u}$ is weakly compact and the result is proved (see Section 6.1 and Section 6.2).

6.4. Proof of Proposition 1

The proof of this proposition is based on the estimation of

$$\left|\right|{\mathbf{E}}_{\pi }[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}{\phi }^{\epsilon }]-\overline{\mathsf{\Phi }}\left(t\right)\phi \left|\right|,$$

for any $\phi \in B_0$, where ${\phi }^{\epsilon }\left(x\right)=\phi \left(x\right)+\epsilon {\phi }_1\left(x\right)$.

We note that

$$\begin{array}{c}\hfill (\tilde{P}-I){\phi }_1\left(x\right)=-({\widehat{D}}_1-\tilde{P}{D}_1(x,u))\phi .\end{array}$$

(37)

As long as $\Pi ({\widehat{D}}_1-(P^{♯}\times P^u)D_1(x,u))\phi =0$, $\phi \in B_0$, Equation (37) has the solution in domain $\mathcal{R}(\tilde{P}-I)$, ${\phi }_1\left(x\right)=R_0{\tilde{D}}_1\phi$.

In this way,

$$\begin{array}{c}\hfill {\mathbf{E}}_{\pi \times {\pi }_1}∥{\phi }_1\left(x\right)∥\le 2∥R_0∥{\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)∥\tilde{P}{D}_1(·,u)\phi ∥:=2C_1\left({\phi }_1∥R_0∥\right),\end{array}$$

(38)

where $R_0$ is a potential operator of $\tilde{Q}:=\tilde{P}-I$.

From here we obtain

$$\begin{array}{c}\hfill {\mathbf{E}}_{\pi \times {\pi }_1}∥({\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}-I){\phi }_1∥\le 4C_1\left({\phi }_1∥R_0∥\right),\end{array}$$

(39)

as ${\mathsf{\Phi }}_k^{\epsilon ,u}$ are contractive operators.

We note also that

$$\begin{array}{c}\hfill ∥{\mathbf{E}}_{\pi \times {\pi }_1}[\epsilon \sum _{k=0}^{[t/\epsilon ]}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_k^{\epsilon ,u}\phi -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}\overline{\mathsf{\Phi }}\left(s\right)\phi ds]∥\le \epsilon C_2(t,\phi ),\end{array}$$

(40)

where $C_2(t,\phi ):=4T{\int }_E{\int }_U\pi \left(dz\right){\pi }_1\left(du\right)∥\tilde{P}{D}_1(·,u)\phi ∥$, $t\in [0,T]$. This follows from standard argument about the convergence of Riemann sums in Bochner integral (see Lemma 4.14, p. 161, [4]).

We note that

$$\begin{array}{c}\hfill \left|\right|{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}{\phi }^{\epsilon }]-\overline{\mathsf{\Phi }}\left(t\right)\phi \left|\right|\le \left|\right|{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi -\overline{\mathsf{\Phi }}\left(t\right)\phi \left|\right|+\epsilon C_1\left({\phi }_1∥R_0∥\right),\end{array}$$

(41)

where we applied representation ${\phi }^{\epsilon }=\phi +\epsilon {\phi }_1$.

We also note that $\overline{\mathsf{\Phi }}\left(t\right)$ satisfies the equation

$$\begin{array}{c}\hfill \overline{\mathsf{\Phi }}\left(t\right)\phi -\phi -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}\overline{\mathsf{\Phi }}\left(s\right)\phi ds=0.\end{array}$$

(42)

Let us introduce the following martingale,

$$\begin{array}{c}\hfill M_{[t/\epsilon ]+1}^{\epsilon ,u}{\phi }^{\epsilon }:={\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}{\phi }^{\epsilon }-{\phi }^{\epsilon }-\sum _{k=0}^{[t/\epsilon ]}{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{k+1}^{\epsilon ,u}{\phi }^{\epsilon }-{\mathsf{\Phi }}_k^{\epsilon ,u}{\phi }^{\epsilon }\mid {\mathcal{F}}_k^u].\end{array}$$

(43)

This is of zero mean-value martingale

$$\begin{array}{c}\hfill {\mathbf{E}}_{\pi \times {\pi }_1}{M}_{[t/\epsilon ]}^{\epsilon ,u}{\phi }^{\epsilon }=0,\end{array}$$

(44)

which comes directly from (43).

Again, from (43), we get the following asymptotic representation

$$\begin{array}{ccc}\hfill M_{[t/\epsilon ]}^{\epsilon ,u}{\phi }^{\epsilon }& =& {\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi -\phi +\epsilon [{\mathsf{\Phi }}_{[t/\epsilon ]}-I]{\phi }_1-\epsilon \sum _{k=0}^{[t/\epsilon ]}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_k^{\epsilon ,u}\phi \hfill \\ & & -{\epsilon }^2\sum _{k=0}^{[t/\epsilon ]}[\tilde{P}{D}_1(·,u){\mathsf{\Phi }}_k^{\epsilon ,u}{\phi }_1+o_{\phi }\left(1\right)],\hfill \end{array}$$

(45)

where $o_{\phi }\left(1\right)\to 0$, as $\epsilon \to 0$, for any $\phi \in B_0$.

Now, from Equation (6) and expressions (44) and (45), we obtain the following representation

$$\begin{array}{ccc}\hfill {\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}\phi -\overline{\mathsf{\Phi }}\left(t\right)\phi ]& =& \epsilon {\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{{[t/\epsilon ]}^{\epsilon ,u}}-I]{\phi }_1+{\mathbf{E}}_{\pi \times {\pi }_1}[\epsilon \sum _{k=0}^{[t/\epsilon ]}\widehat{\mathrm{I}\mathrm{L}}{\mathsf{\Phi }}_k^{\epsilon ,u}\phi \hfill \\ & & -{\int }_0^t\widehat{\mathrm{I}\mathrm{L}}\overline{\mathsf{\Phi }}\left(s\right)\phi ds]+{\epsilon }^2{\mathbf{E}}_{\pi \times {\pi }_1}[\sum _{k=0}^{[t/\epsilon ]-1}{R}_k^u\left({\phi }_1\right)],\hfill \end{array}$$

(46)

where $R_k^u\left({\phi }_1\right):=\tilde{P}{D}_1(·,u){\mathsf{\Phi }}_k^{\epsilon ,u}{\phi }_1+o_{\phi }\left(1\right)$.

Let us estimate $∥R_k^u\left({\phi }_1\right)∥$ in (46).

$$\begin{array}{c}\hfill ∥R_k^u\left({\phi }_1\right)∥\le \underset{g\in K_{\Delta }}{sup}(∥\tilde{P}{D}_1(z,u)g∥+∥o_g\left(1\right)∥):=C_2(z,g,K_{\Delta },u),\end{array}$$

(47)

where $K_{\Delta }$ is a compact set, $\Delta >0$,because ${\mathsf{\Phi }}_k^{\epsilon ,u}{\phi }_1$ satisfies compactness condition for any $\epsilon >0$ and any k.

In this way, we get from (46) that

$$\begin{array}{c}\hfill ∥{\mathbf{E}}_{\pi \times {\pi }_1}[\sum _{k=0}^{[t/\epsilon ]-1}{R}_k^u\left({\phi }_1\right)]∥\le T{\int }_E\pi \left(dz\right){\pi }_1\left(du\right)C_3(z,g,K_{\Delta },u),t\in [0,T].\end{array}$$

(48)

Finally, from inequalities (38)–(41) and from (47)–(48), we obtain the desired rate of convergence of the CDTSMRE in averaging scheme

$$\begin{array}{c}\hfill \left|\right|{\mathbf{E}}_{\pi \times {\pi }_1}[{\mathsf{\Phi }}_{[t/\epsilon ]}^{\epsilon ,u}{\phi }^{\epsilon }]-\overline{\mathsf{\Phi }}\left(t\right)\phi \left|\right|\le \epsilon A(T,\phi ,∥R_0∥,D_1(z,u)),\end{array}$$

where the constant

$$\begin{array}{c}\hfill A(T,\phi ,∥R_0∥,D_1(z,u)):=5C_1(\phi ,∥R_0∥)+C_2(T,\phi )+T{\int }_E\pi \left(dz\right){\pi }_1\left(du\right)C_3(z,g,K_{\Delta },u),\end{array}$$

(49)

and $C_3(z,g,K_{\Delta },u)$ is defined in (48). Therefore, the proof of Proposition 1 is done.

Remark 2.

In a similar way, we can obtain the rate of convergence results in diffusion approximation (see Propositions 2–3).

7. Concluding Remarks and Future Work

In this paper, we introduced controlled semi-Markov random evolutions in discrete-time in Banach space. The main results concerned time-rescaled limit theorems, namely, averaging, diffusion approximation, and diffusion approximation with equilibrium by martingale weak convergence method. We applied these results to various important families of stochastic systems, i.e., the controlled additive functionals, controlled geometric Markov renewal processes, and controlled dynamical systems. We provided dynamical principles for discrete-time dynamical systems such as controlled additive functionals and controlled geometric Markov renewal processes. We also produced dynamic programming equations (Hamilton–Jacobi–Bellman equations) for the limiting processes in diffusion approximation such as CAF, CGMRP, and CDS. As an example, we considered the solution of portfolio optimization problem by Merton for the limiting CGMRP in DA. We also point out the importance of convergence rates and obtained them in the limit theorems for CDTSMRE and CAF, CGMRP, and CDS.

The future work will be associated with the study of optimal control for the initial, not limiting models, such as CAF in Section 4.1, CGMRP in Section 4.2, and CDS in Section 4.3. Other optimal control problems would be also interesting to consider for diffusion models with equilibrium, e.g., CAF in Section 4.1 and CGMRP in Section 4.2. In our future work, the latter models will be considered for solutions of Merton portfolio’s problems as well. We will also consider in our future research the case of dependent SMC $z_k$ and the MC $u_k$.