Stochastic Optimal Control of Averaged SDDE with Semi-Markov Switching and with Application in Economics

Abstract

This paper is devoted to the study of stochastic optimal control of averaged stochastic differential delay equations (SDDEs) with semi-Markov switchings and their applications in economics. By using the Dynkin formula and solution of the Dirichlet–Poisson problem, the Hamilton–Jacobi–Bellman (HJB) equation and the inverse HJB equation are derived. Applications are given to a new Ramsey stochastic models in economics, namely the averaged Ramsey diffusion model with semi-Markov switchings. A numerical example is presented as well.

1. Introduction

Many papers and books in the past were devoted to stochastic optimal control see, for example, [1,2,3]. This was not limited to the stochastic optimal control in infinite dimensions [4] and applied to stochastic control of jump diffusion [5]. In particular, stochastic delay equations were also a subject of studies in many papers, including [6,7,8,9,10]. For example, delay differential equations driven by Lévy processes were considered in [11]. However, no paper or book in this field considered stochastic optimal control problems for stochastic delay differential equations (SDDEs) with semi-Markov regime-switching or dealt with the Dynkin formula, the Dirichlet–Poisson problem, or the HJB and inverse HJB equations for such equations, or particularly their applications in economics. Therefore, the present paper is devoted to these topics, and as such, these results are new and original. We note that optimal control of stochastic differential delay equations with jumps and Markov switching and with an application in economics were studied in [12]. The stability of stochastic Ito equations with delay, Poisson jumps, and Markov switchings with applications to finance was considered in [13]. A survey of results on SDDE and their applications up to 2003 was presented in [14]. A good introduction to the theory of functional differential equation is [15].

In an earlier paper [16], the following controlled stochastic differential delay equation (SDDE) was introduced:

$$x\left(t\right)=x\left(0\right)+{\int }_0^{t}a(x(s-T),u\left(s\right))ds+{\int }_0^{t}b(x(s-T),u\left(s\right))dw\left(s\right),$$

where $x\left(t\right)=\varphi \left(t\right),t\in [-T,0],$$T>0,$ is a given continuous process, $u\left(t\right)$ is a control process, and $w\left(t\right)$ is a standard Wiener process.

We note that in the case of stochastic differential delay equations, the solution is not a Markov solution. However, it can be made a Markov solution by considering the pair $(x_t,x\left(t\right))$, where $x_t:=x(t+s),s\in [-T,0]$ is the path of the process on the interval $[t-T,t]$ (see details in [17]). The pair is a strong Markov process to which one can apply the theoretical basis of the corresponding weak infinitesimal generator (see, for example, [18] for more details).

The Ramsey growth model, or the Ramsey–Cass–Koopmans model is a neoclassical model of economic growth based primarily on the work of Frank P. Ramsey [19], with significant extensions by David Cass [20] and Tjalling Koopmans [21]. This model is very popular nowadays as well (see, for example, [22,23,24,25,26,27,28]).

The Ramsey diffusion model was described in [19] by the equation (see also [16,29])

$$dK\left(t\right)=\left[AK\right(t-1)-u(K\left(t\right)\left)C\right(t\left)\right]dt+\sigma \left(K\right(t-1\left)\right)dw\left(t\right)$$

where K is the capital, C is the production rate, u is a control process, A is a positive constant, and $\sigma$ is the diffusion coefficient of the SDDE. The “initial capital”

$$K\left(t\right)=\varphi \left(t\right),t\in [-1,0],$$

is a continuous, bounded positive function. For this stochastic economic model, the optimal control was found to be $u_{\min }=K\left(0\right)·C\left(0\right).$

Through time rescaling, the delay T was normalized to $T=1,$ which will be our assumption in the theoretical considerations that follow. The obtained results are valid, however, for the general delay $T>0.$

In this paper, we consider SDDEs with semi-Markov switching:

$$\begin{array}{ccc}\hfill x\left(t\right)& =& x\left(0\right)+{\int }_0^{t}a(x(s-1),u\left(s\right),r\left(s\right))ds\hfill \\ & +& {\int }_0^{t}b(x(s-1),u\left(s\right),r\left(s\right))dw\left(s\right),\hfill \end{array}$$

where $r\left(t\right)$ is a semi-Markov process (SMP) with a state space $X=\{1,2,...,N\}$ [30,31]. See the next section for the definition of an SMP.

We note that in the case of stochastic differential delay equations, the solution is not a Markov solution. However, it can be made a Markov solution by considering the vector $(x_t,x\left(t\right),r\left(t\right),\gamma \left(t\right))$, where $x_t:=x(t+s),s\in [-1,0]$ is the path of the process on the interval $[t-1,t]$ (see details in [17]). We mention that the process $\left(r\right(t),\gamma (t\left)\right)$ is a Markov process (see the next section). The vector is a strong Markov process to which one can apply the theoretical basis of the corresponding weak infinitesimal generator (see, for example, [18] for more details).

Due to the complicated expression associated with the generator of the vector $(x_t,x\left(t\right),r\left(t\right),\gamma \left(t\right))$ and generator Q of the SMP $r\left(t\right),$ it is not possible to solve the optimal control problem exactly. Thus, we consider our model for $x\left(t\right)$ in the series scheme

$$\begin{array}{ccc}\hfill x^{ϵ}\left(t\right)& =& x^{ϵ}\left(0\right)+{\int }_0^{t}a(x^{ϵ}(s-1),u\left(s\right),r(s/ϵ))ds\hfill \\ & +& {\int }_0^{t}b(x^{ϵ}(s-1),u\left(s\right),r(s/ϵ))dw\left(s\right).\hfill \end{array}$$

Therefore, to solve the stochastic optimal control problem, we take $x^{ϵ}\left(t\right)$ in a series scheme, where the semi-Markov process $r(t/ϵ)$ is considered in the long run (i.e., when $ϵ\to 0$), and instead of the time t, we have $t/ϵ.$

Using the averaging principle for an SDDEs-in-series scheme under some conditions (A.1–4 in this paper), we find that $x^{ϵ}\left(t\right){\to }_{ϵ\to 0}\hat{x}\left(t\right)$ weakly, where $\hat{x}\left(t\right)$ satisfies the following limiting controlled, averaged SDDE:

$$\begin{array}{ccc}\hfill \hat{x}\left(t\right)& =& x\left(0\right)+{\int }_0^t\hat{a}(\hat{x}(s-1),u\left(s\right))ds\hfill \\ & +& {\int }_0^t\hat{b}(\hat{x}(s-1),u\left(s\right))dw\left(s\right),\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill \hat{a}(x,u)& :=& {\displaystyle \frac{1}{m}}\left[{\int }_{X}a(x,u,r)m\left(r\right)\pi \left(dr\right)\right]\hfill \\ \hfill \hat{b}(x,u)& :=& {\displaystyle \frac{1}{m}}\left[{\int }_{X}b(x,u,r)m\left(r\right)\pi \left(dr\right)\right].\hfill \end{array}$$

Here, $m\left(r\right):={\int }_0^{+\infty }t{G}_r\left(dt\right),$$m:={\int }_{X}m\left(r\right)\pi \left(dr\right),$ and $\pi \left(dr\right)$ represents the stationary probabilities for a Markov chain $r_k$ (see Section 2).

An application in this paper is given to a stochastic model in economics, specifically a Ramsey model [19,29], which takes into account the delay and semi-Markov randomness in the production cycle.

Thus, the Ramsey model with semi-Markov switching in this paper is

$$\begin{array}{ccc}\hfill dK\left(t\right)& =& \left[A\right(r\left(t\right)\left)K\right(t-1)-u(K\left(t\right)\left)C\right(t\left)\right]dt\hfill \\ & +& \sigma \left(K\right(t-1),r(t\left)\right)dw\left(t\right),\hfill \end{array}$$

where $r\left(t\right)$ is a semi-Markov process.

The idea behind the semi-Markov switching in this Ramsey diffusion model is that we have different states of the economy, namely N states (e.g., if $N=2$, then they could be “bad” and “good” states), and in each state, the economy has not an exponential but an arbitrary distribution (Gamma, Beta, Weibull, etc.). In this paper, we consider a Weibull distribution.

We note that the averaged Ramsey diffusion model has the following form:

$$\begin{array}{ccc}\hfill d\hat{K}\left(t\right)& =& [\hat{A}\hat{K}(t-T)-u\left(\hat{K}\left(t\right)\right)C\left(t\right)]dt\hfill \\ & +& \hat{\sigma }\left(\hat{K}(t-T)\right)dw\left(t\right),\hfill \end{array}$$

where $\hat{A}:={\displaystyle \frac{1}{m}}\left[{\int }_X\pi \left(dr\right)m\left(r\right)A\left(r\right)\right],$$m\left(r\right):={\int }_0^{+\infty }t{G}_r\left(dt\right),$$m:={\int }_{X}m\left(r\right)\pi \left(dr\right)$ and $\hat{\sigma }\left(K(t-T)\right):={\displaystyle \frac{1}{m}}\left[{\int }_X\pi \left(dr\right)m\left(r\right)\sigma (K(t-T),r)\right].$

It should be mentioned that reducing the original SDDE with semi-Markov switching to the averaged SDDE and reducing the original Ramsey model with semi-Markov switching to the averaged Ramsey diffusion model does not limit both averaged or limiting models because all of the semi-Markov parameters and features are incorporated into the averaged parameters.

This paper is organized as follows. Section 2 introduces the semi-Markov process and its properties. Section 3 is devoted to the controlled stochastic differential delay equations with semi-Markov switching and a controlled, averaged SDDE. In Section 4, we consider a solution to the stochastic optimal control problem for the controlled, averaged SDDE which is based on the Hamilton–Jacobi–Bellman equation derived from the Dynkin formula and the solution to the Dirichlet–Poisson problem for the controlled, averaged SDDE. The economic Ramsey diffusion model with semi-Markov switching and the averaged Ramsey model are investigated in Section 5. We found optimal control for this model, and we present a numerical example for a two-state semi-Markov process with a Weibull distribution.

2. Semi-Markov Process

Let $(\mathrm{\Omega },F,{\left(F_t\right)}_{t\in {\mathbb{R}}_{+}},P)$ be a filtered probability space with a right-continuous filtration ${\left(F_t\right)}_{t\in {\mathbb{R}}_{+}}$ and a probability P.

Let $(X,\mathcal{X})$ be a measurable space and

$$Q_{SM}(r,B,t):=P(r,B)G_r\left(t\right)\mathrm{for}r\in X,B\in \mathcal{X},t\in {\mathbb{R}}_{+},$$

(1)

be a semi-Markov kernel. Let $(r_n,{\tau }_n;n\in \mathbb{N})$ be an $(X\times {\mathbb{R}}_{+},\mathcal{X}\otimes {\mathcal{B}}_{+})$-valued Markov renewal process with $Q_{SM}$ as the associated kernel; that is, we have

$$\mathbb{P}(r_{n+1}\in B,{\tau }_{n+1}-{\tau }_n\le t|F_n)=Q_{SM}(r_n,B,t).$$

Let us then define the process

$${\nu }_t:=sup\{n\in \mathbb{N}:{\tau }_n\le t\}$$

that gives the number of jumps of the Markov renewal process in the time interval $(0,t]$ and

$${\theta }_n:={\tau }_n-{\tau }_{n-1}$$

which gives the sojourn time of the Markov renewal process in the nth visited state. The semi-Markov process, associated with the Markov renewal process ${(x_n,{\tau }_n)}_{n\in \mathbb{N}}$, is defined by [31]

$$r\left(t\right):=r_{\nu \left(t\right)}\mathrm{for}t\in {\mathbb{R}}_{+}.$$

(2)

Associated with the semi-Markov process, it is possible to define some auxiliary processes. We are interested in the backward recurrence time (or lifetime) process defined by

$$\gamma \left(t\right):=t-{\tau }_{\nu \left(t\right)}\mathrm{for}t\in {\mathbb{R}}_{+}.$$

(3)

The next well-known result characterizes the backward recurrence time process [31]. The backward recurrence time ${\left(\gamma \left(t\right)\right)}_t$ in Equation (3) is a Markov process with a generator

$$Q_{\gamma }f\left(t\right)=f^{\prime }\left(t\right)+\lambda \left(t\right)[f\left(0\right)-f\left(t\right)],$$

where $\lambda \left(t\right)=-{\displaystyle \frac{{\overline{G_x}}^{\prime }\left(t\right)}{\overline{G_r}\left(t\right)}}$, $\overline{G_r}\left(t\right)=1-G_r\left(t\right)$ and $Domain\left(Q_{\gamma }\right)=C^1\left({\mathbb{R}}_{+}\right)$.

As is well known, semi-Markov processes preserve the lost memories property only at transition times, and thus ${\left(r\left(t\right)\right)}_{t\in {\mathbb{R}}_{+}}$ is not a Markov process. However, if we consider the joint process ${(r\left(t\right),\gamma \left(t\right))}_{t\in {\mathbb{R}}_{+}}$, then we record at any instant the time already spent by the semi-Markov process in the present state, which then results in the fact that ${(r\left(t\right),\gamma \left(t\right))}_{t\in {\mathbb{R}}_{+}}$ is a Markov process with a generator (see [31])

$$Qf(r,t)={\displaystyle \frac{df}{dt}}(r,t)+{\displaystyle \frac{g_r\left(t\right)}{{\overline{G}}_r\left(t\right)}}{\int }_{X}P(r,dy)[f(y,0)-f(r,t)],$$

(4)

where $G_r\left(t\right)$ is defined in Equation (1) and $g_r\left(t\right)=G_r^{\prime }\left(t\right).$

3. Controlled Stochastic Differential Delay Equations (SDDEs) with Semi-Markov Switching and Averaged Controlled SDDEs

3.1. Controlled SDDEs with Semi-Markov Switching

Consider the following SDDEs with semi-Markov switching:

$$\begin{array}{ccc}\hfill x\left(t\right)& =& x\left(0\right)+{\int }_0^{t}a(x(s-1),u\left(s\right),r\left(s\right))ds\hfill \\ & +& {\int }_0^{t}b(x(s-1),u\left(s\right),r\left(s\right))dw\left(s\right),\hfill \end{array}$$

(5)

where $r\left(t\right)$ is a semi-Markov process (SMP) in Equation (2) with the state space $X=\{1,2,...,N\}$ [31]. See the next section for the definition of an SMP.

It is important to mention again that in the case of stochastic differential delay equations, the solution is not a Markov solution, as was also indicated in the Introduction. However, it can made a Markov solution by considering the vector $(x_t,x\left(t\right),r\left(t\right),\gamma \left(t\right))$, where $x_t:=x(t+s),s\in [-1,0]$ is the path of the process on the interval $[t-1,t]$ (see details in [17]). We mention that the process $\left(r\right(t),\gamma (t\left)\right)$ (see Equations (2) and (3)) is a Markov process. The vector is a strong Markov process to which one can apply the theoretical basics of the corresponding weak infinitesimal generator (see, for example, [18] for more details).

3.2. Assumptions and Existence of Solutions

Below, we recall some basic notions and facts from [17,32,33] necessary for subsequent exposition in this paper. Let $x\left(t\right),t\in [-1,\infty )$ be a stochastic process and $F_{uv}\left(x\right)$ be a minimal $\sigma$-algebra with respect to which $x\left(t\right)$ is measurable for every $t\in [u,v]$. Let $w\left(t\right),t\in [0,\infty )$ be a Wiener process with $w\left(0\right)=0,$, and let $F_{uv}\left(dw\right)$ be a minimal Borel $\sigma$-algebra such that $w\left(t\right)-w\left(s\right)$ is measurable for all $t,s$ with $u\le s\le t\le v$. Also, let $F_{uv}\left(dr\right)$ be a minimal Borel $\sigma$-algebra such that $r\left(t\right)-r\left(s\right)$ is measurable for all $t,s$ with $u\le s\le t\le v,$ where $r\left(t\right)$ is a finite-state semi-Markov process with a state space $X=\{1,2,\dots ,N\}$ [31].

Finally, let $u\left(t\right)\in \mathcal{U},t\in [-1,\infty )$ be a stochastic process whose values can be chosen from the given Borel set $\mathcal{U}$ and such that $u\left(t\right)$ is $F_{uv}\left(u\right)$-adapted for all $t\in [u,v]$.

Let $\mathcal{D}$ denote the Banach space of all càdlàg functions defined on the interval $[-1,0]$ and equipped with the Skorokhod topology (see [34]). We note the initial process $x\left(t\right)=\varphi \left(t\right),t\in [-1,0]$, where $\varphi \in \mathcal{D}$ is a given càdlàg function. Therefore, we assume that the processes $\varphi \left(t\right),t\in [-1,0],w\left(t\right)$, and $u\left(t\right),t\ge 0$ are defined on the probability space $(\mathrm{\Omega },F,F_t,P)$ and $F_t:=F_{-1t}\left(x\right)\vee F_{0t}\left(dw\right)\vee F_{0t}\left(u\right)\vee F_{0t}\left(dr\right).$

Let the following conditions be satisfied for Equation (2):

A.1 $a(\varphi ,u,r)$ and $b(\varphi ,u,r)$ are continuous, real-valued functionals defined on $\mathcal{D}\times \mathcal{U}\times X$;

A.2 $\varphi \in \mathcal{D}$ is a càdlàg function with a probability of one in the interval $[-1,0],$ independent of $w\left(s\right),s\ge 0,$ and ${E\left|\varphi \left(t\right)\right|}^4<\infty$. Here, $|.|$ is the Skorokhod norm [34].

A.3 $\forall \varphi ,\psi \in \mathcal{D}$:

$$|a(\varphi ,u,r)-a(\psi ,u,r)|+|b(\varphi ,u,r)-b(\psi ,u,r)|\le K{\int }_{-1}^0|\varphi \left(\theta \right)-\psi \left(\theta \right)|d\theta ,$$

with $\left|a\right(\eta ,u,r\left)\right|+\left|b\right(\eta ,u,r\left)\right|\le M$ for some constants, $M,K>0$, and all $\eta \in \mathcal{D},u\in \mathcal{U},r=r\left(0\right)\in X$.

Under assumptions A.1–3, the solution to the initial value problem in Equation (2) exists, and the vector $(x_t,x\left(t\right),r\left(t\right),\gamma \left(t\right))$ is a unique stochastic Markov process [17,32,33]. The solution $x\left(t\right)$ can be viewed at time $t\ge 0$ as an element $x_t$ of the space $\mathcal{D}$ or as a point in $\mathbb{R}$.

Equation (5) is expressed in the integral form as follows

$$\begin{array}{ccc}\hfill x\left(t\right)& =& x\left(0\right)+{\int }_0^{t}a(x(s-1),u\left(s\right),r\left(s\right))ds\hfill \\ & +& {\int }_0^{t}b(x(s-1),u\left(s\right),r\left(s\right))dw\left(s\right),\hfill \end{array}$$

(6)

We note that the generator for the vector $(x_t,x\left(t\right),r\left(t\right),\gamma \left(t\right))$ is

$$\begin{array}{cc}\hfill A_G^{r}F(x(-1),x\left(0\right),r)& :=F(x\left(0\right),x\left(0\right),r)-F(x(-1),x\left(0\right),r)+\hfill \\ \hfill & +{\int }_{-1}^0{L}^{u}F(x\left(s\right),x\left(0\right),r)ds+\hfill \\ \hfill & +QF(x\left(0\right),x\left(0\right),r),\hfill \end{array}$$

(7)

where the operator Q is defined in Equation (4) and acts only on the r variable. The operator $L^u$ is defined by

$$\begin{array}{cc}{L}^u\hfill & F\left(x\right(-1),x(0),r):=\hfill \\ \hfill & =F_{x\left(0\right)}^{\prime }(x(-1),x\left(0\right),r)a(x(-1),u,r)+\hfill \\ & F_{x\left(0\right)x\left(0\right)}^{″}(x(-1),x\left(0\right),r){\displaystyle \frac{1}{2}}{b}^2(x(-1),u,r)\hfill \end{array}$$

(8)

and acts on F as a function of $x\left(0\right)$ only, while $u=u\left(0\right)$ (see [33]).

3.3. Controlled Averaged SDDE

Due to the complicated expression associated with the generator $A_G^r$ in Equation (7) above and the generator Q in Equation (4) of the SMP $r\left(t\right),$ we consider our model for $x\left(t\right)$ in Equation (5) in the series scheme:

$$\begin{array}{ccc}\hfill x^{ϵ}\left(t\right)& =& x^{ϵ}\left(0\right)+{\int }_0^{t}a(x^{ϵ}(s-1),u\left(s\right),r(s/ϵ))ds\hfill \\ & +& {\int }_0^{t}b(x^{ϵ}(s-1),u\left(s\right),r(s/ϵ))dw\left(s\right),\hfill \end{array}$$

(9)

Therefore, to solve the stochastic optimal control problem, we take $x^{ϵ}\left(t\right)$ in a series scheme, where the semi-Markov process $r(t/ϵ)$ is considered in the long run; in other words, instead of the time t, we have $t/ϵ.$

We need one more condition for this reason:

A.4. The embedded Markov chain $r_n$ of the semi-Markov process $r\left(t\right)$ is ergodic, with the transition probabilities $\pi \left(dr\right).$

Using the averaging principle for an SDDEs-in-series scheme under conditions A.1–4, we can find that $x^{ϵ}\left(t\right){\to }_{ϵ\to 0}\hat{x}\left(t\right)$ weakly (see [13,31,35]), where $\hat{x}\left(t\right)$ satisfied the following limiting controlled, averaged SDDE for the SDDE in Equation (9):

$$\begin{array}{ccc}\hfill \hat{x}\left(t\right)& =& x\left(0\right)+{\int }_0^t\hat{a}(\hat{x}(s-1),u\left(s\right))ds\hfill \\ & +& {\int }_0^t\hat{b}(\hat{x}(s-1),u\left(s\right))dw\left(s\right),\hfill \end{array}$$

(10)

where

$$\begin{array}{ccc}\hfill \hat{a}(x,u)& :=& {\displaystyle \frac{1}{m}}\left[{\int }_{X}a(x,u,r)m\left(r\right)\pi \left(dr\right)\right]\hfill \\ \hfill \hat{b}(x,u)& :=& {\displaystyle \frac{1}{m}}\left[{\int }_{X}b(x,u,r)m\left(r\right)\pi \left(dr\right)\right].\hfill \end{array}$$

(11)

Here, $m\left(r\right):={\int }_0^{+\infty }t{G}_r\left(dt\right),$$m:={\int }_{X}m\left(r\right)\pi \left(dr\right).$

Remark 1. 

In the case of a finite or infinite but countable X value, the integrals above become sums or series, respectively. We present the above formulas for the case of a finite-state semi-Markov process (i.e., $X=\{0,1,2,\dots ,N\}$):

$${\hat{\sigma }}^2:={\displaystyle \frac{1}{m}}\sum _{i=0}^N{\sigma }^2\left(i\right)m\left(i\right)\pi \left(i\right),m:=\sum _{i=0}^{N}m\left(i\right)\pi \left(i\right),m\left(i\right):={\int }_0^{+\infty }t{G}_i\left(dt\right).$$

In this way, the generator for the process $\hat{x}\left(t\right)$ in Equation (10) takes the following form:

$$\begin{array}{cc}{\hat{L}}^u\hfill & F\left(x\right(-1),x(0\left)\right):=\hfill \\ \hfill & =\hat{a}(x(-1),u)F_{x\left(0\right)}^{\prime }(x(-1),x\left(0\right))+\hfill \\ & {\displaystyle \frac{1}{2}}{\hat{b}}^2(x(-1),u),F_{x\left(0\right)x\left(0\right)}^{″}(x(-1),x\left(0\right))\hfill \end{array}$$

(12)

where $\hat{a}$ and $\hat{b}$ are defined in Equation (11).

We note that the pair $({\hat{x}}_t,\hat{x}\left(t\right))$ is now a strong Markov process with a generator ${\hat{L}}^u$ in Equation (12).

4. Solution to the Stochastic Optimal Control Problem for the Averaged SDDEs

The main idea of solving the stochastic optimal control problem for the averaged SDDEs with semi-Markov switching is to derive the Hamilton–Jacobi–Bellman (HJB) equation and the inverse HJB equation by applying the Dynkin formula and solution to the Dirichlet–Poisson problem for the averaged SDDE, As long as we have reduced the optimal control problem with semi-Markov switching in Equations (6) and (9) to the optimal control problem in diffusion and hence the Markov case, then we can apply the results from [16]. Below we list all of the results which are necessary for the solution of the stochastic optimal control problem in our setting.

4.1. Dynkin Formula for the SDDE with Semi-Markov Switching

Let $\tau$ be a stopping time for the strong Markov process $({\hat{x}}_t,\hat{x}\left(t\right))$ such that $E_{x,x\left(0\right)}\left|\tau \right|<\infty$. Then, we have the following Dynkin formula (see [16]):

$$\begin{array}{c}{E}_{x,x\left(0\right)}\hat{F}(x_{\tau },x\left(\tau \right))=\hfill \\ +E_{x,x\left(0\right)}{\int }_0^{\tau }\hat{F}(x\left(s\right),x\left(s\right))ds-{\int }_0^{\tau }\hat{F}(x(s-1),x\left(s\right)ds\hfill \\ +E_{x,x\left(0\right)}{\int }_0^{\tau }{\int }_{-1}^0{\hat{L}}^{u}F(\varphi \left(\theta \right),x\left(s\right))d\theta ds,\hfill \end{array}$$

(13)

where ${\hat{L}}^u$ is defined in Equation (12) and $\hat{F}:={\displaystyle \frac{1}{m}}\left[{\int }_{X}F(x(-1),x\left(0\right),r)m\left(r\right)\pi \left(dr\right)\right].$

4.2. Solution to the Dirichlet–Poisson Problem for the SDDE with Semi-Markov Switching

Let $\psi (x,x(0),r)\in C(\partial (H\times G\times X)$ be bounded, and let $F(x,x(0),u,r)\in \mathbf{C}(\mathcal{D}\times C\times \mathcal{U}\times X)$ be such that

$$E_{x,x\left(0\right)}\left[{\int }_{-1}^0{\int }_0^{{\tau }_{H\times G}}|F(\varphi \left(\theta \right),x\left(s\right),u\left(s\right),r\left(s\right))dsd\theta \right]<\infty$$

$$\mathrm{for} \mathrm{all}(x,x(0),r)\in H\times G\times X,$$

where ${\tau }_{H\times G\times X}=\inf \{t:(x_t,x\left(t\right),r\left(t\right))\notin H\times G\times X\}$, $\mathbf{C}$ is a set of continuous and bounded functions on $\mathcal{D}\times C\times \mathcal{U}\times X,$ and $\partial (H\times G\times X)$ is a regular boundary of $H\times G\times X$, $H\in C,G\in \mathcal{D}$ (see, for example, [18]).

Define

$$\begin{array}{c}J(x,x\left(0\right),u):=E_{x,x\left(0\right)}\left[{\int }_{-1}^0{\int }_0^{{\tau }_{H\times G}}\hat{F}(\varphi \left(\theta \right),x\left(s\right),u\left(s\right))dsd\theta \right]\hfill \\ +E_{x,x\left(0\right),r}\left[\hat{\psi }(x_{{\tau }_G},x\left({\tau }_G\right))\right],(x,x\left(0\right),r)\in H\times G\times X.\hfill \end{array}$$

Then (see [16]), it follows that

$${\hat{L}}^{r}J(x,x\left(0\right),u)=-{\int }_{-1}^{0}F(\varphi \left(\theta \right),x,u)d\theta ,\mathrm{in}H\times G\forall u\in \mathcal{U}$$

and

$$\underset{t\uparrow {\tau }_G}{lim}J(x_t,x\left(t\right),u)=\psi (x\left({\tau }_G\right),x\left({\tau }_G\right))\forall (x,x\left(0\right))\in H\times G,$$

where ${\hat{L}}^r$ is generator defined above in Equation (12).

4.3. Hamilton–Jacobi–Bellman (HJB) Equation for the SDDE with Semi-Markov Switching

We assume that the cost function is given in the form

$$\begin{array}{c}J(x,x\left(0\right),u):=E_{x,x\left(0\right)}\left[{\int }_{-1}^0{\int }_0^{{\tau }_{H\times G\times X}}\hat{F}(\varphi \left(\theta \right),x\left(s\right),u\left(s\right))dsd\theta \right.\hfill \\ \left.+\psi (x_{{\tau }_{H\times G\times X}},x\left({\tau }_{H\times G\times X}\right))\right],\hfill \end{array}$$

where $\psi$ is a bounded real function, F is bounded, real, and continuous, and ${\tau }_{H\times G\times X}$ is the exit time of the process $(x_t,x\left(t\right))$ defined in Equation (7) from the fixed open set $H\times G\times X\subset \mathcal{D}\times C\times X$. In particular, ${\tau }_{H\times G}$ can be a fixed time $t_0$. We assume that $E_{x,x\left(0\right),r}\left|{\tau }_{H\times G\times X}\right|<\infty ,\forall (x,x\left(0\right),r)\in H\times G\times X.$ The function $\hat{F}$ is defined in Equation (15).

The problem is as follows. For each $(x,x(0),r)\in H\times G\times X$, find the number ${\hat{J}}^{*}(x,x\left(0\right))$ and control $u^{*}=u^{*}(x,x\left(0\right),r),\omega )$ such that

$${\hat{J}}^{*}(x,x\left(0\right)):=\underset{u}{\inf }\left\{\hat{J}(x,x\left(0\right),u)\right\}=\hat{J}(x,x\left(0\right),u^{*},r),$$

where the infimum is taken over all $F_t$-adapted processes $u\left(t\right)\in \mathcal{U}$. Such a control $u^{*}$, if it exists, is called an optimal control, and ${\hat{J}}^{*}(x,x\left(0\right))$ is called the optimal performance.

We consider only the Markov controls $u\left(t\right).$ For every $\nu \in \mathcal{U}$, we define the following operator:

$$\begin{array}{c}\left({\hat{A}}^{\nu }J\right)(x,x\left(0\right))=\hat{F}(x\left(0\right),x\left(0\right),\nu \left(0\right))-\hat{F}(x(-1),x\left(0\right),\nu \left(0\right))+\hfill \\ +{\int }_{-1}^0{\hat{L}}^{\nu }F(\varphi \left(\theta \right),x\left(0\right),\nu \left(0\right),r)d\theta ,\hfill \end{array}$$

(14)

where the operator ${\hat{L}}^{\nu }$ is given by Equation (12) and

$$\hat{J}(x,x\left(0\right)):={\int }_{-1}^0\hat{F}(\varphi \left(\theta \right),x\left(0\right),\nu \left(0\right))d\theta$$

Theorem 1 

(HJB equation for SDDEs with semi-Markov switchings). Define

$${\hat{J}}^{*}(x,x\left(0\right),r)=\inf \{\hat{J}(x,x\left(0\right),u):uisaMarkovcontrol\}.$$

Suppose that $J\in C^2(H\times G\times X)$ and the optimal control $u^{*}$ exists. Then, it follows that

$$\begin{array}{c}{\displaystyle \underset{\nu \in \mathcal{U}}{\inf }}\left[{\int }_{-1}^0\hat{F}(\varphi \left(\theta \right),x,\nu )d\theta +\left({\hat{L}}^{\nu }{\hat{J}}^{*}\right)(x,x\left(0\right))\right]=0,\hfill \\ \forall (x,x\left(0\right))\in H\times G,\hfill \end{array}$$

(15)

and

$${\hat{J}}^{*}(x,x\left(0\right))=\hat{\psi }(x,x\left(0\right)),\forall (x,x\left(0\right),r)\in \partial (H\times G),$$

where functions F and ψ are given by Equation (14), the operator ${\hat{L}}^{\nu }$ is given by Equation (12), and $\partial (H\times G)$ is the boundary of set $H\times G$ for $(x_t,x\left(t\right)).$

The infimum in Equation (15) is achieved when $\nu =u^{*}$, where $u^{*}$ is an optimal control. In other words, there is

$$\begin{array}{c}{\int }_{-1}^0\hat{F}(\varphi \left(\theta \right),x,u^{*})d\theta +\left({\hat{L}}^{u^{*}}{J}^{*}\right)(x,x\left(0\right))=0,\hfill \\ \forall (x,x\left(0\right))\in H\times G,\hfill \end{array}$$

(16)

which is Equation (15).

Proof. 

Follow the steps of Theorem 1 [16] while replacing the operator $A^{\nu }$ with ${\hat{L}}^{\nu }$ in Equation (12). □

Theorem 2 

(Converse of the HJB equation for SDDEs with semi-Markov switchings). Let g be a bounded function in $C^2(H\times G)\cap C(\partial (H\times G))).$ Suppose that for all $u\in \mathcal{U}$, the inequality

$$\begin{array}{c}{\int }_{-1}^0\hat{F}(x,x\left(0\right),\varphi \left(\theta \right),u)d\theta +\left({\hat{L}}^{u}g\right)(x,x\left(0\right))\ge 0,\hfill \\ (x,x\left(0\right))\in H\times G\hfill \end{array}$$

and the boundary condition

$$g(x,x\left(0\right))=\hat{\psi }(x,x\left(0\right)),(x,x\left(0\right))\in \partial H\times G$$

(17)

are satisfied. Then, $g(x,x\left(0\right),r)\le \hat{J}(x,x\left(0\right),u$ for all Markov controls $u\in \mathcal{U}$ and for all $(x,x(0\left)\right)\in H\times G.$

Moreover, if for every $(x,x(0),\in H\times G$, there exists $u^0$ such that

$${\int }_{-1}^0\hat{F}(x,x\left(0\right),\varphi \left(\theta \right),u^0)d\theta +\left({\hat{L}}^{u^0}g\right)(x,x\left(0\right))=0,$$

(18)

then $u^0$ is a Markov control $g(x,x\left(0\right))=\hat{J}(x,x\left(0\right),u^0)=J^{*}(x,x\left(0\right)),$ and therefore $u^0$ is an optimal control.

Proof. 

Follow the steps of Theorem 2 [16] while replacing $A^{\nu ,r}$ with ${\hat{L}}^u$ in Equation (12). □

5. Ramsey Diffusion Model in Economics with Semi-Markov Switching

We recall that the Ramsey diffusion model with semi-Markov switching is

$$\begin{array}{ccc}\hfill dK\left(t\right)& =& \left[A\right(r\left(t\right)\left)K\right(t-T)-u(K\left(t\right)\left)C\right(t\left)\right]dt\hfill \\ & +& \sigma \left(K\right(t-T),r(t\left)\right)dw\left(t\right),\hfill \end{array}$$

(19)

where $r\left(t\right)$ is a semi-Markov process. We suppose that functions $A\left(r\right)>0$ and $\sigma (·,r)>0$ are bounded and continuous with r on $X.$ For this model, the initial condition is

$$K\left(t\right)=\varphi \left(t\right),t\in [-1,0].$$

(20)

We note that the averaged Ramsey diffusion model has the following form, which follows from our previous results (see Section 3.3, Equations (9)–(12)):

$$\begin{array}{ccc}\hfill d\hat{K}\left(t\right)& =& [\hat{A}\hat{K}(t-T)-u\left(\hat{K}\left(t\right)\right)C\left(t\right)]dt\hfill \\ & +& \hat{\sigma }\left(\hat{K}(t-T)\right)dw\left(t\right),\hfill \end{array}$$

(21)

where $\hat{A}:={\displaystyle \frac{1}{m}}\left[{\int }_X\pi \left(dr\right)m\left(r\right)A\left(r\right)\right],$ $\hat{\sigma }\left(K(t-T)\right):={\displaystyle \frac{1}{m}}\left[{\int }_X\pi \left(dr\right)m\left(r\right)\sigma (K(t-T),r)\right],$ $m\left(r\right):={\int }_0^{+\infty }t{G}_r\left(dt\right),$ $m:={\int }_{X}m\left(r\right)\pi \left(dr\right),$ and $\pi \left(dr\right)$ represents the stationary probabilities for the Markov chain $r_k.$

In the next section, we present the solution for the stochastic optimal control problem for this averaged diffusion Ramsey model and a numerical example.

5.1. Optimal Control for Ramsey Diffusion Model in Economics with Semi-Markov Switching

Here, we consider the averaged Ramsey diffusion model in Equation (21) with the boundary condition in Equation (20).

Let us choose the following cost function with the modification of the first term being ${\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}$:

$$J(K,u,r)={\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta +{\displaystyle \frac{u^2\left(0\right)}{2}}.$$

(22)

The operator $A_G^{u}J$ has the following form, taking into account Equation (22):

$$\begin{array}{cc}\hfill A^{u,r}J& ={\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\varphi }^2\left(0\right)+{\displaystyle \frac{u^2\left(0\right)}{2}}-\left[{\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\varphi }^2(-1)+{\displaystyle \frac{u^2\left(0\right)}{2}}\right]\hfill \\ \hfill & +\left[K\left(0\right)·(A\left(r\right)·K\left(0\right)-u\left(0\right)·C\left(0\right))+{\displaystyle \frac{1}{2}}{\sigma }^2\left(K\left(0\right)\right)\right]\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\int }_{-1}^{+\infty }[{(K\left(0\right)+yK\left(0\right))}^2-K^2\left(0\right)-2K\left(0\right)yK\left(0\right)]\mathrm{\Pi }\left(dy\right)\hfill \\ \hfill & ={\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\varphi }^2\left(0\right)+{\displaystyle \frac{u^2\left(0\right)}{2}}-\left[{\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\varphi }^2(-1)+{\displaystyle \frac{u^2\left(0\right)}{2}}\right]\hfill \\ \hfill & +\left[K\left(0\right)·(A\left(r\right)·K\left(0\right)-u\left(0\right)·C\left(0\right))+{\displaystyle \frac{1}{2}}{\sigma }^2\left(K\left(0\right)\right)\right]\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{K}^2\left(0\right){\int }_{-1}^{+\infty }{y}^2\mathrm{\Pi }\left(dy\right),\hfill \end{array}$$

since

$$\begin{array}{c}F(K\left(0\right),K\left(0\right),u\left(0\right),r)={\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\varphi }^2\left(0\right)+{\displaystyle \frac{u^2\left(0\right)}{2}},\hfill \\ F(K\left(0\right),K(-1),u\left(0\right),r)={\displaystyle \frac{A\left(r\right)K^2\left(0\right)}{2}}+{\varphi }^2(-1)+{\displaystyle \frac{u^2\left(0\right)}{2}},\hfill \\ L^{u}J(K,u,r)=K\left(0\right)\left(A\left(r\right)·K\left(0\right)-u\left(0\right)·C\left(0\right)+{\displaystyle \frac{1}{2}}{\sigma }^2\left(K\left(0\right)\right)\right).\hfill \end{array}$$

From Theorem 1, (16), we obtain the following HJB equation:

$$\begin{array}{cc}\hfill {\displaystyle \underset{u}{\inf }}& \left[{\displaystyle \frac{\hat{A}{K}^2\left(0\right)}{2}}+{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta +{\displaystyle \frac{u^2\left(0\right)}{2}}+{\varphi }^2\left(0\right)-{\varphi }^2(-1)+\hat{A}·K^2\left(0\right)\right.\hfill \\ \hfill & +\left.u\left(0\right)·K\left(0\right)·C\left(0\right)+{\displaystyle \frac{1}{2}}{\sigma }^2\left(K\left(0\right)\right)+{\displaystyle \frac{1}{2}}{K}^2\left(0\right){\int }_{-1}^{+\infty }{y}^2\mathrm{\Pi }\left(dy\right)\right]=0,\hfill \end{array}$$

or equivalently

$$\begin{array}{cc}\hfill {\displaystyle \underset{u}{\inf }}& \left[u^2\left(0\right)-2K\left(0\right)C\left(0\right)u\left(0\right)+(2{\varphi }^2\left(0\right)-2{\varphi }^2(-1)+2{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta \right.\hfill \\ \hfill & \left.+K^2\left(0\right)(1+2\hat{A})+{\sigma }^2\left(K\left(0\right)\right))+K^2\left(0\right){\int }_{-1}^{+\infty }{y}^2\mathrm{\Pi }\left(dy\right)\right]=0,\hfill \end{array}$$

where $\hat{A}:={\displaystyle \frac{1}{m}}\left[{\int }_X\pi \left(dx\right)m\left(x\right)A\left(x\right)\right],$ $m\left(x\right):={\int }_0^{+\infty }t{G}_x\left(dt\right),$ $m:={\int }_{X}m\left(x\right)\pi \left(dx\right),$ and $\pi \left(dx\right)$ represents the stationary probabilities for the Markov chain $r_k$ (see also Equation (21)). Let

$$\begin{array}{cc}\hfill 4K^2\left(0\right)C^2\left(0\right)& \ge 4(2{\varphi }^2\left(0\right)-2{\varphi }^2(-1)+2{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta \hfill \\ \hfill & +K^2\left(0\right)(1+2\hat{A})+{\sigma }^2\left(K\left(0\right)\right))+K^2\left(0\right){\int }_{-1}^{+\infty }{y}^2\mathrm{\Pi }\left(dy\right),\hfill \end{array}$$

or

$$\begin{array}{cc}\hfill K^2\left(0\right)·(C^2\left(0\right)& -3-2\hat{A})\ge 2{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta -2{\varphi }^2(-1)\hfill \\ \hfill & +{\sigma }^2\left(K\left(0\right)\right)+K^2\left(0\right){\int }_{-1}^{+\infty }{y}^2\mathrm{\Pi }\left(dy\right),\hfill \end{array}$$

since $K\left(0\right)=\varphi \left(0\right)$. Hence, the infinum is achieved when

$$u\left(0\right)=-\left(-{\displaystyle \frac{2K\left(0\right)·C\left(0\right)}{2}}\right)=K\left(0\right)·C\left(0\right).$$

Therefore, $u_{\min }=K\left(0\right)·C\left(0\right)$, and

$$\begin{array}{cc}\hfill J(K,u_{\min })=& ={\displaystyle \frac{\hat{A}{K}^2\left(0\right)}{2}}+{\displaystyle \frac{K^2\left(0\right)·C^2\left(0\right)}{2}}+{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta =\hfill \\ \hfill & ={\displaystyle \frac{K^2\left(0\right)}{2}}(\hat{A}+C^2\left(0\right))+{\int }_{-1}^0{\varphi }^2\left(\theta \right)d\theta .\hfill \end{array}$$

(23)

5.2. Numerical Example for Ramsey Diffusion Model in Economics with Semi-Markov Switching

We will now use the expressions defined above in further calculations below. Suppose that $x_n$ is a Markov chain with two states $i=\{0,1\}$ and a transition matrix $\mathbf{P}=\left(\begin{array}{cc}0.7& 0.3\\ 0.4& 0.6\end{array}\right)$.

Then, the stationary probabilities are ${\displaystyle \overrightarrow{\pi }=\left(\begin{array}{c}\pi \left(0\right)\\ \pi \left(1\right)\end{array}\right)=\left(\begin{array}{c}{\displaystyle 0.571}\\ {\displaystyle 0.429}\end{array}\right),}$ which follow from the solution of the following equation: $\overrightarrow{\pi }\mathbf{P}=\overrightarrow{\pi }.$

Thus, we consider the case of a two-state semi-Markov chain $x_n$ with an arbitrary distribution $G_i\left(x\right),i=0,1,$ for ${\tau }_n.$ Let us take the Weibull distribution (see [33]) $G_i\left(x\right)$ for ${\tau }_n$ with a probability density function $f_i\left(x\right):=d{G}_i\left(x\right)/dx:$

$$f_i\left(x\right)=\left\{\begin{array}{c}\hfill {\displaystyle \lambda \left(i\right)K\left(i\right){\left(\lambda \left(i\right)x\right)}^{K\left(i\right)-1}\exp [-{\left(\lambda \left(i\right)x\right)}^{K\left(i\right)}],x\ge 0,}\\ \hfill {\displaystyle 0,x<0,}\end{array}\right.$$

where $i=0,1.$ Recall that $K\left(i\right)$ is the shape parameter and $\lambda \left(i\right)$ is the scale parameter. We note that if we take $K\left(i\right)=1,i=0,1,$ then we have the exponential distribution for $G_i\left(t\right)$ and the Markov case for the process $x\left(t\right).$

Suppose that $\lambda \left(0\right)=8$ and $\lambda \left(1\right)=10.$ We recall that the mean value for a random variable with a Weibull density distribution is $(1/\lambda (i\left)\right)\mathrm{\Gamma }(1+1/K(i\left)\right),$ where $\mathrm{\Gamma }(·)$ stands for the Gamma function. Of course, we could take other non-exponential distributions, such as Gamma or Beta.

We consider the cases $K\left(i\right)=2\left(K\right(i)>1),i=0,1,$ and the case $K\left(i\right)=1/2\left(K\right(i)<1),i=0,1,$ can be considered in a similar way. We note that the case $K\left(i\right)=1,i=0,1,$ refers to the exponential distribution $G_i\left(x\right)=1-e^{-\lambda \left(i\right)x.}.$

Thus, let us take $K\left(i\right)=2\left(K\right(i)>1),i=0,1.$

We can now calculate the parameters $m\left(0\right)$ and $m\left(1\right)$ (see Equation (11)):

$$\begin{array}{ccc}\hfill m\left(0\right)& =& 1/\lambda \left(0\right)\left)\mathrm{\Gamma }\right(1+1/K\left(0\right))=(1/8\left)\mathrm{\Gamma }\right(3/2)\approx 0.111\hfill \\ \hfill m\left(1\right)& =& 1/\lambda \left(1\right)\left)\mathrm{\Gamma }\right(1+1/K\left(1\right))=(1/10\left)\mathrm{\Gamma }\right(3/2)\approx 0.089.\hfill \end{array}$$

Then (see Equation (11)), we have

$$\begin{array}{ccc}\hfill m& =& \pi \left(0\right)m\left(0\right)+\pi \left(1\right)m\left(1\right)=0.571\times 0.111+0.429\times 0.089\hfill \\ & =& 0.063381+0.038181\hfill \\ & =& 0.101562\approx 0.102.\hfill \end{array}$$

Let the initial function $\varphi \left(t\right)=e^{at}$ (see Equation (20)), where a could be positive or negative. Then, the formula for $J(K,u_{\min })$ has the following form (see Equation (23)):

$$J(K,u_{\min })={\displaystyle \frac{K^2\left(0\right)}{2}}(\hat{A}+C^2\left(0\right))+{\displaystyle \frac{1-e^{-2a}}{2a}}.$$

Now, suppose that $C\left(0\right)=1,$ $K\left(0\right)=1.$ We note that $\varphi \left(0\right)=1=K\left(0\right).$ We also take $A\left(0\right)=1$ and $A\left(1\right)=2.$

Then, according to previous formulas, we have

$$\hat{A}={\displaystyle \frac{1}{m}}[A\left(0\right)m\left(0\right)\pi \left(0\right)+A\left(1\right)m\left(1\right)\pi \left(1\right)]\approx 1.3.$$

In addition, $J(K,u_{\min })={\displaystyle \frac{K^2\left(0\right)}{2}}(\hat{A}+C^2\left(0\right))+{\displaystyle \frac{1-e^{-2a}}{2a}}={\displaystyle \frac{1}{2}}[1.3+1]+{\displaystyle \frac{1-e^{-2a}}{2a}}=1.15+{\displaystyle \frac{1-e^{-2a}}{2a}}.$

If we take $a=1,$ then $J(K,u_{\min })=1.58$, and if we take $a=-1,$ then $J(K,u_{\min })=4.35.$

As long as $t\in [-1,0]$ for the initial function $\varphi \left(t\right)=e^{at},$ then a positive value of $a=1$ corresponds to decaying capital (“bad” economy), and a negative value of $a=-1$ corresponds to increasing capital (“good” economy), which are reflected by the cost functions $J(K,u_{\min })=1.58$ and $J(K,u_{\min })=4.35,$ respectively.