# Controlled Discrete-Time Semi-Markov Random Evolutions and Their Applications

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Controlled Discrete-Time Semi-Markov Random Evolutions

#### 2.1. Semi-Markov Chains

#### 2.2. General Definition and Properties of DTSMREs

**Definition**

**1.**

#### 2.3. Some Examples

**Example 1.**

**Controlled Additive Functional or Markov Decision Process**.

**Example 2.**

**Controlled geometric Markov renewal process**.

#### 2.4. Dynamic Programming for Controlled Models

**Example 3.**

**Controlled Additive Functional**.

**Example 4.**

**Controlled Geometric Markov Renewal Chain**.

## 3. Limit Theorems for Controlled Semi-Markov Random Evolutions

#### 3.1. Averaging of CDTSMREs

- A1:
- The MC $({z}_{k},{\gamma}_{k},k\in \mathrm{I}\phantom{\rule{-2.2pt}{0ex}}\mathrm{N})$ is uniformly ergodic with ergodic distribution ${\pi}^{\u266f}(B\times \left\{k\right\}),B\in \mathcal{E},k\in \mathrm{I}\phantom{\rule{-2.2pt}{0ex}}\mathrm{N}$.
- A2:
- The moments ${m}_{2}\left(x\right),x\in E$, are uniformly integrable.
- A3:
- The perturbed operators ${D}^{\epsilon}\left(x\right)$ have the following representation on B$$\begin{array}{c}\hfill {D}^{\epsilon}(x,u)=I+\epsilon {D}_{1}(x,u)+\epsilon {D}_{0}^{\epsilon}(x,u),\end{array}$$
- A4:
- We have ${\int}_{E}{\int}_{U}[\pi \left(dx\right){\pi}_{1}\left(du\right)]{\u2225{D}_{1}(x,u)\phi \u2225}^{2}<\infty .$ (See A7.)
- A5:
- There exists Hilbert spaces H and ${H}^{*}$ such that compactly embedded in Banach spaces B and ${B}^{*},$ respectively, where ${B}^{*}$ is a dual space to $B.$
- A6:
- Operators ${D}^{\epsilon}\left(x\right)$ and ${\left({D}^{\epsilon}\right)}^{*}\left(x\right)$ are contractive on Hilbert spaces H and ${H}^{*},$ respectively.
- A7:
- The MC $({u}_{k},k\in \mathrm{I}\phantom{\rule{-2.2pt}{0ex}}\mathrm{N})$, is independent of $\left({z}_{k}\right)$, and is uniformly ergodic with stationary distribution ${\pi}_{1}\left(du\right),k\in \mathrm{I}\phantom{\rule{-2.2pt}{0ex}}\mathrm{N}$.

**Theorem**

**1.**

#### 3.2. Diffusion Approximation of DTSMREs

- D1:
- Let us assume that the perturbed operators ${D}^{\epsilon}(x,u)$ have the following representation in B,$$\begin{array}{c}\hfill {D}^{\epsilon}(x,u)=I+\epsilon {D}_{1}(x,u)+{\epsilon}^{2}{D}_{2}(x,u)+{\epsilon}^{2}{D}_{0}^{\epsilon}(x,u),\end{array}$$
- D2:
- The following balance condition holds,$$\begin{array}{c}\hfill \Pi {D}_{1}(x,u)\Pi =0,\end{array}$$$$\begin{array}{c}\hfill \Pi \phi (x,k,u):=\sum _{l\ge 0}{\int}_{E}{\int}_{U}{\pi}^{\u266f}(dy\times \ell ){\pi}_{1}\left(du\right)\phi (y,\ell ,u)\mathbf{1}(x,k).\end{array}$$
- D3:
- The moments ${m}_{3}\left(x\right),x\in E$, are uniformly integrable.

**Theorem**

**2.**

#### 3.3. Diffusion Approximation with Equilibrium

**Theorem**

**3.**

## 4. Applications to Stochastic Systems

#### 4.1. Controlled Additive Functionals

**Averaging of CAF.**Now, if we define the continuous time process

**Diffusion Approximation of CAF.**If we consider the continuous time process ${\xi}_{t}^{\epsilon ,u}$ as follows

**Diffusion Approximation with Equilibrium of CAF.**Let us consider the following normalized additive functional,

#### 4.2. Controlled Geometric Markov Renewal Processes

**Averaging of CGMRP.**Now, define the following sequence of processes,

**Diffusion Approximation of CGMRP.**If we define the following sequence of processes,

**Diffusion Approximation with Equilibrium of CGMRP.**Let us consider the following normalized GMRP:

#### 4.3. Controlled Dynamical Systems

**Averaging of CDS.**Under averaging assumptions the following weak convergence takes place,

**Diffusion Approximation of CDS.**Under diffusion approximation conditions the following weak convergence takes place

#### 4.4. The Dynamic Programming Equations for Limiting Models in Diffusion Approximation

**DPE/HJB Equation for the Limiting CAF in DA (see Section 4.1)**

**DPE/HJB Equation for the Limiting CGMRP in DA (see Section 4.2)**

**DPE/HJB Equation for the Limiting CDS in DA (see Section 4.3)**

**Remark**

**1.**

**The Merton Problem**

## 5. Rates of Convergence in Averaging and Diffusion Approximations

**Proposition**

**1.**

**Proposition**

**2.**

**Proposition**

**3.**

**Corollary**

**1.**

**Corollary**

**2.**

## 6. Proofs

#### 6.1. Proof of Theorem 1

**Lemma**

**1.**

**Proof.**

**Lemma**

**2.**

**Proof.**

**Lemma**

**3.**

**Proof.**

**Lemma**

**4.**

**Proof.**

#### 6.2. Proof of Theorem 2

#### 6.3. Proof of Theorem 3

#### 6.4. Proof of Proposition 1

**Remark**

**2.**

## 7. Concluding Remarks and Future Work

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

SMC | Discrete-time semi-Markov chain; |

DTSMRE | Discrete-time semi-Markov random evolution; |

CDTSMRE | Controlled discrete-time semi-Markov random evolution; |

CGMRP | Controlled geometric Markov renewal processes; |

CAF | Controlled additive functionals; |

CDS | Controlled dynamical systems; |

HJB | Hamilton–Jacobi–Bellman (equation); |

DPE | Dynamic programming equation; |

DPP | Dynamic programming principle; |

DA | Diffusion approximation; |

SDE | Stochastic differential equation. |

## References

- Griego, R.; Hersh, R. Random evolutions, Markov chains, and Systems of partial differential equations. Proc. Natl. Acad. Sci. USA
**1969**, 62, 305–308. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Hersh, R. Random evolutions: A Survey of results and problems. Rocky Mountain J. Math.
**1974**, 4, 443–477. [Google Scholar] [CrossRef] - Koroliuk, V.S.; Limnios, N. Stochastic Systems in Merging Phase Space; World Scientific: Singapore, 2005. [Google Scholar]
- Korolyuk, V.S.; Swishchuk, A. Evolution of System in Random Media; CRC Press: Boca Raton, FL, USA, 1995. [Google Scholar]
- Swishchuk, A.; Wu, J. Evolution of Biological Systems in Random Media: Limit Theorems and Stability; Kluwer: Dordrecht, The Netherlands, 2003. [Google Scholar]
- Swishchuk, A.V. Random Evolutions and their Applications; Kluwer AP: Dordrecht, The Netherlands, 1995. [Google Scholar]
- Cohen, J.E. Random Evolutions in Discrete and Continuous Time. Stoch. Proc. Appl.
**1979**, 9, 245–251. [Google Scholar] [CrossRef] [Green Version] - Keepler, M. Random evolutions processes induced by discrete time Markov chains. Port. Math.
**1998**, 55, 391–400. [Google Scholar] - Limnios, N. Discrete-time random evolutions—Difference equations and additive functionals. Commun. Statist. Theor. Methods
**2011**, 40, 1–11. [Google Scholar] [CrossRef] - Yin, G.G.; Zhang, Q. Discrete-Time Markov Chains. Two-Time-Scale Methods and Applications; Springer: New York, NY, USA, 2005. [Google Scholar]
- Anisimov, V.V. Switching processes: Averaging principle, diffusion approximation and applications. Acta Appl. Math.
**1995**, 40, 95–141. [Google Scholar] [CrossRef] - Anisimov, V.V. Switching Processes in Queueing Models; J. Wiley & Sons: London, UK; ISTE: London, UK, 2008. [Google Scholar]
- Anisimov, V.V. Averaging principle for switching recurrent sequences. Theory Probab. Math. Stat.
**1991**, 45, 1–8. [Google Scholar] - Anisimov, V.V. Averaging principle for switching processes. Theory Probab. Math. Stat.
**1992**, 46, 1–10. [Google Scholar] - Swishchuk, A.V.; Islam, M.S. The Geometric Markov Renewal Processes with application to Finance. Stoch. Anal. Appl.
**2010**, 29, 4. [Google Scholar] [CrossRef] - Swishchuk, A.V.; Islam, M.S. Diffusion Approximations of the Geometric Markov Renewal Processes and option price formulas. Int. J. Stoch. Anal.
**2010**, 2010, 347105. [Google Scholar] [CrossRef] [Green Version] - Swishchuk, A.V.; Islam, M.S. Normal Deviation and Poisson Approximation of GMRP. Communic. Stat. Theory Methods
**2013**, 1488–1501. [Google Scholar] [CrossRef] - Swishchuk, A.V.; Limnios, N. Optimal stopping of GMRP and pricing of European and American options. In Proceedings of the 15th Intern. Congress on Insurance: Mathematics and Economics (IME 2011), Trieste, Italy, 14–17 June 2011. [Google Scholar]
- Barbu, V.; Limnios, N. Semi-Markov Chains and Hidden Semi-Markov Models. Toward Applications. Their Use in Reliability and DNA Analysis; Lecture Notes in Statistics; Springer: New York, NY, USA, 2008; Volume 191. [Google Scholar]
- Limnios, N.; Swishchuk, A. Discrete-time Semi-Markov Random Evolutions and their Applications. Adv. Appl. Probab.
**2013**, 45, 214–240. [Google Scholar] [CrossRef] - Jacod, J.; Shiryaev, A.N. Limit Theorems for Stochastic Processes; Springer: Berlin/Heidelberg, Germany, 1987. [Google Scholar]
- Bertsekas, D.; Shreve, S. Stochastic Optimal Control: The Discrete-Tiime Case; Athena Scientific Publisher: Belmont, MA, USA, 1996. [Google Scholar]
- Fleming, W.H.; Rishel, R.W. Deterministic and Stochastic Optimal Control; Springer: New York, NY, USA, 1975. [Google Scholar]
- Kushner, H.J. Introduction to Stochastic Control Theory; Holt, Rinehart, Winston: New York, NY, USA, 1971. [Google Scholar]
- Merton, R. Optimum consumption and portfolio rules in a continuous-time model. J. Econ. Theory
**1971**, 3, 373–413. [Google Scholar] [CrossRef] [Green Version] - Sviridenko, M.N. Martingale approach to limit theorems for semi-Markov processes. Theor. Probab. Appl.
**1986**, 34, 540–545. [Google Scholar] [CrossRef] - Adams, R. Sobolev Spaces; Academic Press: New York, NY, USA, 1979. [Google Scholar]
- Rudin, W. Functional Analysis; McGraw-Hill: New York, NY, USA, 1991. [Google Scholar]
- Sobolev, S. Some Applications of Functional Analysis in Mathematical Physics, 3rd ed.; American Mathematical Society: Providence, RI, USA, 1991; Volume 90. [Google Scholar]
- Pyke, R. Markov renewal processes: Definitions and preliminary properties. Ann. Math. Statist.
**1961**, 32, 1231–1242. [Google Scholar] [CrossRef] - Pyke, R. Markov renewal processes with finitely many states. Ann. Math. Statist.
**1961**, 32, 1243–1259. [Google Scholar] [CrossRef] - Pyke, R.; Schaufele, R. Limit theorems for Markov renewal processes. Ann. Math. Statist.
**1964**, 35, 1746–1764. [Google Scholar] [CrossRef] - Shurenkov, V.M. On the theory of Markov renewal. Theor. Probab. Appl.
**1984**, 19, 247–265. [Google Scholar] [CrossRef] - Maxwell, M.; Woodroofe, M. Central limit theorems for additive functionals of Markov chains. Ann. Probab.
**2000**, 28, 713–724. [Google Scholar] - Nummelin, E. General Irreducible Markov Chains and Non-Negative Operators; Cambridge University Press: Cambridge, UK, 1984. [Google Scholar]
- Revuz, D. Markov Chains; North-Holland: Amsterdam, The Netherlands, 1975. [Google Scholar]
- Skorokhod, A.V. Asymptotic Methods in the Theory of Stochastic Differential Equations; AMS: Providence, RI, USA, 1989; Volume 78. [Google Scholar]
- Skorokhod, A.V.; Hoppensteadt, F.C.; Salehi, H. Random Perturbation Methods with Applications in Science and Engineering; Springer: New York, NY, USA, 2002. [Google Scholar]
- Jaśkiewicz, A.; Nowak, A. Average optimality for semi-Markov control processes. Morfismos
**2007**, 11, 15–36. [Google Scholar] - Vega-Amaya, O.; Luque-Vásquez, F. Sample-path average cost optimality for semi-Markov control processes on Borel spaces: Unbounded cost and mean holding times. Appl. Math.
**2000**, 27, 343–367. [Google Scholar] [CrossRef] [Green Version] - Altman, E.; Shwartz, A. Markov decision problems and state-action frequencies. SIAM J. Control. Optimikzation
**1991**, 29, 786–809. [Google Scholar] [CrossRef] - Beutler, F.; Ross, K. Optimal policies for controlled Markov chains with a constraint. J. Mathem. Analysis Appl.
**1985**, 112, 236–252. [Google Scholar] [CrossRef] [Green Version] - Borkar, V. Dynamic programming for ergodic control with partial observations. Stoch. Proc. Appl.
**2003**, 103, 293–310. [Google Scholar] [CrossRef] - Boussement, M.; Limnios, N. Markov decision processes with asymptotic average failure rate constraint. Communic. Statis. Theory Methods
**2004**, 33, 1689–1714. [Google Scholar] [CrossRef] - Hernández-Lerma, O.; Lasserre, J.B. Discrete-Time Markov Control Processes; Springer: New York, NY, USA, 1996. [Google Scholar]
- Krylov, N.; Bogolyubov, N. Introduction to Non-linear Mechanics; Princeton University Press: Princeton, NJ, USA, 1947. [Google Scholar]
- Limnios, N.; Oprişan, G. Semi-Markov Processes and Reliability; Birkhäuser: Boston, MA, USA, 2001. [Google Scholar]
- Cartea, A.; Jaimungal, S.; Penalva, J. It Algorithmic and High-Frequency Trading; Cambridge University Press: Cambridge, UK, 2015. [Google Scholar]
- Ethier, S.N.; Kurtz, T.G. Markov Processes: Characterization and Convergence; J. Wiley: New York, NY, USA, 1986. [Google Scholar]
- Stroock, D.W.; Varadhan, S.R.S. Multidimensional Diffusion Processes; Springer: Berlin/Heidelberg, Germany, 1979. [Google Scholar]

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Swishchuk, A.; Limnios, N.
Controlled Discrete-Time Semi-Markov Random Evolutions and Their Applications. *Mathematics* **2021**, *9*, 158.
https://doi.org/10.3390/math9020158

**AMA Style**

Swishchuk A, Limnios N.
Controlled Discrete-Time Semi-Markov Random Evolutions and Their Applications. *Mathematics*. 2021; 9(2):158.
https://doi.org/10.3390/math9020158

**Chicago/Turabian Style**

Swishchuk, Anatoliy, and Nikolaos Limnios.
2021. "Controlled Discrete-Time Semi-Markov Random Evolutions and Their Applications" *Mathematics* 9, no. 2: 158.
https://doi.org/10.3390/math9020158