Linear Quadratic Optimal Control of Discrete-Time Stochastic Systems Driven by Homogeneous Markov Processes

Abstract

Random terms in many natural and social science systems have distinct Markovian characteristics, such as Markov jump-taking values in a finite or countable set, and Wiener process-taking values in a continuous set. In general, these systems can be seen as Markov-process-driven systems, which can be used to describe more complex phenomena. In this paper, a discrete-time stochastic linear system driven by a homogeneous Markov process is studied, and the corresponding linear quadratic (LQ) optimal control problem for this system is solved. Firstly, the relations between the well-posedness of LQ problems and some linear matrix inequality (LMI) conditions are established. Then, based on the equivalence between the solvability of the generalized difference Riccati equation (GDRE) and the LMI condition, it is proven that the solvability of the GDRE is sufficient and necessary for the well-posedness of the LQ problem. Moreover, the solvability of the GDRE and the feasibility of the LMI condition are established, and it is proven that the LQ problem is attainable through a certain feedback control when any of the four conditions is satisfied, and the optimal feedback control of the LQ problem is given using the properties of homogeneous Markov processes and the smoothness of the conditional expectation. Finally, a practical example is used to illustrate the validity of the theory.

1. Introduction

Linear quadratic (LQ) optimal control problems play an important role in control theory and practical applications [1,2]. The Riccati equation method is a well-known method for studying the LQ problem for deterministic systems described by ordinary differential equations [3,4,5], and this method can be extended to stochastic cases for Itô-type stochastic systems [6,7,8,9]. With the development of control theories, the LQ problem for discrete-time stochastic systems has also been studied by many scholars, achieving impressive results [10,11,12]. In these studies, the value functions and optimal controls for finite- or infinite-horizon indefinite LQ problems were obtained based on the solutions to some Riccati equations with forms of algebraic, differential, and difference Riccati equations [13,14,15,16,17,18,19]. In system controller design, some external disturbances or parameter uncertainties should be addressed to eliminate or reduce their impact on the system’s performance [20,21,22]. In some studies, the uncertain parts of the systems are treated as unknown disturbances, e.g., Ref. [22] presented the robust non-linear generalized predictive control method to improve the system’s robustness and [23] used the $H_{\infty }$ control design method. Many studies describe these complex systems as stochastic systems in which the random or fluctuating terms are usually modeled by white noise, Markov chains, or Wiener processes [24,25,26,27,28,29,30,31,32].

In practice, random terms in many natural and social science systems exhibit distinct Markovian characteristics. Markovian processes find applications in various fields, such as economy, finance, and engineering systems. For example, Markov jumps, which take values in a finite or countable set, are widely used to describe the jumping phenomenon of various systems [33,34,35]. In some situations, there are processes that possess Markovian properties but take values in a continuous set, such as Wiener processes and fractal Brownian motions, which are used to describe different types of noise in continuous- or discrete-time systems [36,37,38]. These systems are driven by a type of stochastic process with Markovian properties. So, in general, such systems are called Markov-process-driven systems, in which the Markovian processes are seen as an extension of white noise. This paper discusses the LQ problem for discrete-time linear systems driven by Markov processes. In more detail, the following LQ optimal control is considered. Minimize the cost function

$$J\left(u\right)=\mathbb{E}\sum _{k=0}^{N-1}[x_k^{\mathrm{T}}Q\left(k\right)x_k+2x_k^{\mathrm{T}}S\left(k\right)u_k+u_k^{\mathrm{T}}R\left(k\right)u_k]+\mathbb{E}\left[x_N^{\mathrm{T}}Q\left(N\right)x_N\right]$$

(1)

subject to

$$x_{k+1}=A\left(k\right)x_k+B\left(k\right)u_k+A_1\left(k\right)x_k{\omega }_k,x_0\in {\mathbb{R}}^n,k=0,\cdots ,N-1$$

(2)

where $A\left(k\right),A_1\left(k\right)\in {\mathbb{R}}^{n\times n}$, $B\left(k\right),S\left(k\right)\in {\mathbb{R}}^{n\times n_v}$, $Q\left(k\right)\in {\mathcal{S}}^n$, and $R\left(k\right)\in {\mathcal{S}}^{n_v}$. $x_k\in {\mathbb{R}}^n$ is the state variable, $u_k\in {\mathbb{R}}^{n_v}$ is the control, and ${\omega }_k$ is a Markovian process. Compared to the stochastic systems discussed in [10], the random disturbance ${\omega }_k$ in system (2) is no longer an independent process but rather a Markovian process, i.e., the probability distributions of ${\omega }_k$ are related to $x_{k-1}$. Compared to the system discussed in [39], ${\omega }_k$ in (2) is a Markovian process that can take values in a continuous set rather than a countable set. So, the novelties of this paper can be summarized as follows: (1) A discussion of more general systems driven by Markovian processes, extending beyond systems driven by white noise, Markov jumps, Wiener processes, etc. (2) A derivation of more general results with distribution forms to describe the probability distributions of Markov processes.

This paper is organized as follows. In Section 2, the basic theoretical knowledge involved in this paper is introduced. In Section 3, the equivalence of the well-posedness and attainability of the LQ problem in (1) and (2) is discussed, and the optimal controller and minimum value of the cost function are obtained. In Section 4, two examples are used to illustrate the feasibility of the theory.

The following notations are used in this paper. ${\mathbb{R}}^n$: the set of all real n-dimensional vectors; ${\mathbb{R}}^{m\times n}$: the set of $m\times n$ real matrices; $A^{\mathrm{T}}$, $x^{\mathrm{T}}$: transpose of a matrix A or vector x; $A^{-1}$: inverse of matrix A; $A>0(A\ge 0)$: A is a positive (positive semi-) definite matrix; ${\mathbb{I}}_D\left(x\right)$: indicator function of set D with ${\mathbb{I}}_D\left(x\right)=1$ when $x\in D$ and ${\mathbb{I}}_D\left(x\right)=0$ when $x\notin D$; $\mathbb{E}\left[X\right]$: the expectation of a random variable X; $\mathbb{E}[X\mid Y=y]$: the conditional expectation of X under the condition $Y=y$; and ${\mathcal{L}}^2(\mathsf{\Omega },\mathcal{F},\mathbb{P})$: the complete space of random variables with $\mathbb{E}\left[\right|X{|}^2]<\infty$, where $X\in {\mathcal{L}}^2(\mathsf{\Omega },\mathcal{F},\mathbb{P})$.

2. Preliminaries

Let $\{{\omega }_k,k=0,1,\cdots ,N\}$ be a homogeneous Markovian process defined on a complete probability space $(\mathsf{\Omega },\mathcal{F},\mathbb{P})$ with a one-step transition probability density function $p(\xi ,\eta )$, $\xi ,\eta \in \mathbb{R}$, and the initial distribution is $p_0\left(\xi \right)$. For every given $k=1,2,\cdots ,N$, the joint probability density function of ${\omega }_0,{\omega }_1,\cdots ,{\omega }_k$ is

$$f({\xi }_0,{\xi }_1,\cdots ,{\xi }_k)=f_0\left({\xi }_0\right)f_1\left({\xi }_1\right|{\xi }_0)f_2\left({\xi }_2\right|{\xi }_0,{\xi }_1)\cdots f_k\left({\xi }_k\right|{\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1})$$

where $f_k\left({\xi }_k\right|{\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1})$ denotes the conditional probability density function of ${\omega }_k$ under the conditions of ${\omega }_0={\xi }_0,{\omega }_1={\xi }_1,\cdots ,{\omega }_{k-1}={\xi }_{k-1}$, ${\xi }_i\in \mathbb{R},0\le i\le k$. Because ${\omega }_k,k=0,1,\cdots ,N$ is a Markovian process, the conditional probability density function $f_k\left({\xi }_k\right|{\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1})$ only depends on ${\omega }_{k-1}={\xi }_{k-1}$, i.e.,

$$f_k\left({\xi }_k\right|{\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1})=f_k\left({\xi }_k\right|{\xi }_{k-1})$$

and $f_k\left({\xi }_k\right|{\xi }_{k-1})$ is just the one-step transition probability density function. So,

$$f_k\left({\xi }_k\right|{\xi }_{k-1})=p({\xi }_{k-1},{\xi }_k).$$

The joint probability density function of ${\omega }_0,{\omega }_1,\cdots ,{\omega }_k$ can be represented by

$$f({\xi }_0,{\xi }_1,\cdots ,{\xi }_k)=p_0\left({\xi }_0\right)p({\xi }_0,{\xi }_1)p({\xi }_1,{\xi }_2)\cdots p({\xi }_{k-1},{\xi }_k).$$

Suppose that X is a random variable generated by ${\omega }_0,{\omega }_1,\cdots ,{\omega }_k$, i.e., $X=X({\omega }_0,{\omega }_1,\cdots ,{\omega }_k)$. The conditional expectation of X is

$$\begin{array}{ccc}& & \mathbb{E}\left[X({\omega }_0,{\omega }_1,\cdots ,{\omega }_{k-1},{\omega }_k)\right|{\omega }_0={\xi }_0,{\omega }_1={\xi }_1,\cdots ,{\omega }_{k-1}={\xi }_{k-1}]\hfill \\ & & ={\int }_{\mathbb{R}}X({\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1},\eta )f\left(\eta \right|{\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1})d\eta \hfill \\ & & ={\int }_{\mathbb{R}}X({\xi }_0,{\xi }_1,\cdots ,{\xi }_{k-1},\eta )p({\xi }_{k-1},\eta )d\eta \hfill \end{array}$$

In the following discussion, the condition expectation $\mathbb{E}\left[X({\omega }_0,{\omega }_1,\cdots ,{\omega }_{k-1},{\omega }_k)\right|{\omega }_0={\xi }_0,{\omega }_1={\xi }_1,\cdots ,{\omega }_{k-1}={\xi }_{k-1}]$ is shortened by $\mathbb{E}\left[X\right|{\omega }_0,{\omega }_1,\cdots ,{\omega }_{k-1}]$

The following lemma is used in this paper.

Lemma 1.

Given a series of random symmetric matrices $P_1\left({\omega }_0\right),\cdots ,P_N\left({\omega }_{N-1}\right)$, the following results hold

$$\begin{array}{cc}\hfill & \mathbb{E}[P_{k+1}\left({\omega }_k\right)\mid {\omega }_0,\cdots ,{\omega }_{k-1}]\hfill \\ \hfill & =\mathbb{E}[P_{k+1}\left({\omega }_k\right)\mid {\omega }_{k-1}=\xi ]={\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta ,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & \mathbb{E}[P_{k+1}\left({\omega }_k\right){\omega }_k\mid {\omega }_0,\cdots ,{\omega }_{k-1}]\hfill \\ \hfill & =\mathbb{E}[P_{k+1}\left({\omega }_k\right){\omega }_k\mid {\omega }_{k-1}=\xi ]={\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta ,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & \mathbb{E}[P_{k+1}\left({\omega }_k\right){\omega }_k^2\mid {\omega }_0,\cdots ,{\omega }_{k-1}]\hfill \\ \hfill & =\mathbb{E}[P_{k+1}\left({\omega }_k\right){\omega }_k^2\mid {\omega }_{k-1}=\xi ]={\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta .\hfill \end{array}$$

(5)

Proof. 

According to the definition of a Markovian process, the conditional probability density function of ${\omega }_k$ satisfies $f({\omega }_k\mid {\omega }_0,\cdots ,{\omega }_{k-1})=f({\omega }_k\mid {\omega }_{k-1})$, and according to the definition of the conditional expectation,

$$\begin{array}{cc}\hfill & \mathbb{E}[\phi \left({\omega }_k\right)\mid {\omega }_0,\cdots ,{\omega }_{k-1}]={\int }_{\mathbb{R}}\phi \left({\omega }_k\right)f({\omega }_k\mid {\omega }_0,\cdots ,{\omega }_{k-1})d\eta \hfill \\ \hfill & ={\int }_{\mathbb{R}}\phi \left({\omega }_k\right)f({\omega }_k\mid {\omega }_{k-1})d\eta =\mathbb{E}[\phi \left({\omega }_k\right)\mid {\omega }_{k-1}],\hfill \end{array}$$

We obtain

$$\mathbb{E}[P_{k+1}\left({\omega }_k\right)\mid {\omega }_{k-1}=\xi ]={\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta ,$$

$$\mathbb{E}[P_{k+1}\left({\omega }_k\right){\omega }_k\mid {\omega }_{k-1}=\xi ]={\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta ,$$

$$\mathbb{E}[P_{k+1}\left({\omega }_k\right){\omega }_k^2\mid {\omega }_{k-1}=\xi ]={\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta .$$

This ends the proof. □

Let ${\mathcal{F}}_k$ be the $\sigma -$field generated by ${\sigma }_0,{\sigma }_1,\cdots ,{\sigma }_{k-1}(k\ge 1)$ and ${\sigma }_0=\{\varnothing ,\mathsf{\Omega }\}$.

Definition 1.

The LQ problem in (1) and (2) is considered well-posed if

$$\underset{u}{inf}J\left(u\right)>-\infty .$$

The LQ problem in (1) and (2) is called attainable if there exists an admissible control $(u_0^{\ast },\cdots ,u_{N-1}^{\ast })$, such that

$$\underset{u}{inf}J\left(u\right)=J\left(u^{\ast }\right),$$

and $u^{\ast }$ is called an optimal control.

3. LQ Problem of the Discrete-Time Linear Stochastic Systems Driven by a Homogeneous Markovian Process

In this section, the well-posedness and attainability of the LQ problem are studied. We suppose that in the optimal control problem in (1) and (2), the admissible control set $U={\left\{U_k\right\}}_{k=0}^{N-1}$, $U_k=L^2(\mathsf{\Omega },{\mathcal{F}}_k;{\mathbb{R}}^{n_u})$, and ${\left\{{\omega }_k\right\}}_{k=0}^N$ are the homogeneous Markovian processes given in Section 2. We also assume that $u_k^{\ast }$ is the optimal control problem in (1) and (2), and the corresponding optimal trajectory is $x_k^{\ast }$. Under the premise that the optimal cost function is finite, the LQ problem is always attainable through optimal control. Next, we establish the relationship between the well-posedness of the LQ problem and an LMI condition and then prove that the LMI condition is equivalent to both the solvability of the GDRE and the attainability of the LQ problem. In other words, we establish the equivalent relationship between the well-posedness and attainability of the LQ problem, the solvability of the GDRE, and the feasibility of the LMI condition and find the optimal control and optimal value function. In the following discussion, we denote

$$2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)$$

as the corresponding symmetric matrix

$$A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)+A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right).$$

3.1. Well-Posedness

The following provides a connection between the well-posedness and the feasibility of an LMI involving unknown symmetric matrices [40].

Theorem 1.

The LQ problem in (1) and (2) is well-posed if there exist symmetric matrices $P_0,\dots ,P_N$ satisfying the following LMI’s condition

$$\begin{array}{c}\hfill \left[\begin{array}{cc}Q\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)& \\ +2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)& \ast \\ +A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{\eta }^2{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)-P_k\left(\xi \right)& \\ S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)& \\ +B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)& R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)\end{array}\right]\ge 0\end{array}$$

(6)

for $k=0,\cdots ,N-1$, and $P_N=Q\left(N\right)\ge 0$, where ∗ denotes the symmetric part.

Proof. 

Let $P_1,\cdots ,P_N$ satisfy (6). Then, by adding the following trivial equality,

$$\sum _{k=0}^{N-1}\mathbb{E}(x_{k+1}^{\mathrm{T}}{P}_{k+1}{x}_{k+1}-x_k^{\mathrm{T}}{P}_k{x}_k)=\mathbb{E}(x_N^{\mathrm{T}}{P}_N{x}_N-x_0^{\mathrm{T}}{P}_0{x}_0)$$

to the cost function,

$$J\left(u\right)=\mathbb{E}\sum _{k=0}^{N-1}[x_k^{\mathrm{T}}Q\left(k\right)x_k+2x_k^{\mathrm{T}}S\left(k\right)u_k+u_k^{\mathrm{T}}R\left(k\right)u_k]+\mathbb{E}\left[x_N^{\mathrm{T}}{Q}_N{x}_N\right],$$

and using the system in Equation (1), we can rewrite the cost function as follows:

$$\begin{array}{cc}\hfill J\left(u\right)& =\mathbb{E}\sum _{k=0}^{N-1}\left\{x_k^{\mathrm{T}}\right[Q\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)\hfill \\ \hfill & +2A^{\mathrm{T}}\left(k\right)A_1\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta +A_1^{\mathrm{T}}\left(k\right)A_1\left(k\right){\int }_{\mathbb{R}}{\eta }^2{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta -P_k\left(\xi \right)]x_k\hfill \\ \hfill & +2u_k^{\mathrm{T}}[S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+B^{\mathrm{T}}\left(k\right)A_1\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta ]x_k\hfill \\ \hfill & +u_k^{\mathrm{T}}[R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)]u_k\}+\mathbb{E}\left[x_N^{\mathrm{T}}(Q_N-P_N)x_N\right]+\mathbb{E}\left(x_0^{\mathrm{T}}{P}_0{x}_0\right)\hfill \\ \hfill & =\mathbb{E}\sum _{k=0}^{N-1}{\left[\begin{array}{c}{x}_k\\ u_k\end{array}\right]}^{\mathrm{T}}\left[\begin{array}{cc}Q\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)& \\ +2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)& \ast \\ +A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{\eta }^2{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)-P_k\left(\xi \right)& \\ S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)& \\ +B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}\eta P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A_1\left(k\right)& \mathsf{\Xi }\end{array}\right]\hfill \\ \hfill & \times \left[\begin{array}{c}{x}_k\\ u_k\end{array}\right]+\mathbb{E}\left[x_N^{\mathrm{T}}(Q_N-P_N)x_N\right]+\mathbb{E}\left(x_0^{\mathrm{T}}{P}_0{x}_0\right),\hfill \end{array}$$

where $\mathsf{\Xi }=R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)$. From the above equality, we can see that the cost function $J\left(u\right)$ is bounded from below by $\mathbb{E}\left(x_0^{\mathrm{T}}{P}_0{x}_0\right)$; hence, the LQ problem in (1) and (2) is well-posed. □

We have shown in the proof of Theorem 1 that the proposed LMI condition (6) is sufficient for the well-posedness of the LQ problem. Next, we show that the LMI condition is equivalent to the solvability of the GDRE, and then provide a connection between the well-posedness of the LQ problem and the solvability of the GDRE. Meanwhile, the necessity of the LMI condition (6) for the well-posedness of the LQ problem is also proven.

Lemma 2

(Extended Schur’s lemma [41]). Let $M=M^{\mathrm{T}}$ and C and $D=D^{\mathrm{T}}$ be given matrices of appropriate proper sizes. Then, the following conditions are equivalent:

(i) 

$M-C{D}^{-1}{C}^{\mathrm{T}}\ge 0,D\ge 0.$

(ii) 

$\left[\begin{array}{cc}M& C\\ C^{\mathrm{T}}& D\end{array}\right]\ge 0.$

(iii) 

$\left[\begin{array}{cc}D& C^{\mathrm{T}}\\ C& M\end{array}\right]\ge 0.$

Lemma 3.

Let $F=F^{\mathrm{T}}$ and H and $G=G^{\mathrm{T}}$ be given matrices of appropriate sizes. Consider the following quadratic form:

$$f(x,u)=\mathbb{E}[x^{\mathrm{T}}Fx+2x^{\mathrm{T}}Hu+u^{\mathrm{T}}Gu],$$

where x and u are random variables belonging to the space ${\mathcal{L}}^2(\mathsf{\Omega },\mathcal{F},\mathbb{P})$. Then, the following conditions are equivalent:

(i) 

$\underset{u}{inf}f(x,u)>-\infty$ for any random variable x.

(ii) 

There exists a symmetric matrix $Z=Z^{\mathrm{T}}$, such that $\underset{u}{inf}f(x,u)=\mathbb{E}\left[x^{\mathrm{T}}Zx\right]$ for any random variable x.

(iii) 

$G\ge 0$ and $ker\left(G\right)\subseteq ker\left(H\right)$.

(iv) 

There exists a symmetric matrix $W=W^{\mathrm{T}}$, such that

$$\left[\begin{array}{cc}F-W& H\\ H^{\mathrm{T}}& G\end{array}\right]\ge 0.$$

Moreover, if $G>0$ and any of the above conditions holds, then (ii) is satisfied by $Z=F-H{G}^{-1}{H}^{\mathrm{T}}$, and for any random variable x, the random variable $u^{\ast }=-G^{-1}{H}^{\mathrm{T}}x$ is optimal with the following optimal value:

$$f(x,u^{\ast })=\mathbb{E}\left[x^{\mathrm{T}}(F-H{G}^{-1}{H}^{\mathrm{T}})x\right].$$

Proof. 

$\left(i\right)\Rightarrow \left(iii\right)$: Suppose there is a v such that $v^{\mathrm{T}}Gv<0$. Then, for any scalar $\gamma >0$, we have $\underset{\gamma \to +\infty }{lim}f(x,\gamma v)=-\infty$. This leads to a contradiction through the assumption. So, G must be positive.

Suppose now $ker\left(G\right)⊈ker\left(H\right)$. That is, there exists u, such that $Gu=0$ and $Hu\ne 0$. Take any scalar $\gamma >0$. Then, we have $\underset{\gamma \to +\infty }{lim}f(Hu,-\gamma u)=-\infty$, which contradicts (i).

$\left(iii\right)\Rightarrow \left(ii\right)$: Through a simple calculation, we can obtain the following:

$$f(x,u)=\mathbb{E}[x^{\mathrm{T}}(F-H{G}^{-1}{H}^{\mathrm{T}})x+(u^{\mathrm{T}}+x^{\mathrm{T}}H{G}^{-1})G(u+G^{-1}{H}^{\mathrm{T}}x)].$$

Let $Z=F-H{G}^{-1}{H}^{\mathrm{T}}$. Then, it is immediate that $\underset{u}{inf}f(x,u)=\mathbb{E}\left[x^{\mathrm{T}}Zx\right]$ for any random variable x.

$\left(ii\right)\Rightarrow \left(iv\right)$: Since $f(x,u)=\mathbb{E}[x^{\mathrm{T}}Fx+2x^{\mathrm{T}}Hu+u^{\mathrm{T}}Gu]\ge \mathbb{E}\left[x^{\mathrm{T}}Zx\right],$ that is,

$$\mathbb{E}{\left[\begin{array}{c}x\\ u\end{array}\right]}^{\mathrm{T}}\left[\begin{array}{cc}F-Z& H\\ H^{\mathrm{T}}& G\end{array}\right]\left[\begin{array}{c}x\\ u\end{array}\right]\ge 0,$$

the condition (iv) holds with $W=Z$.

$\left(iv\right)\Rightarrow \left(i\right)$: Since

$$\left[\begin{array}{cc}F& H\\ H^T& G\end{array}\right]-\left[\begin{array}{cc}W& 0\\ 0& 0\end{array}\right]=\left[\begin{array}{cc}F-W& H\\ H^T& G\end{array}\right]\ge 0,$$

for every x and u, there exists

$$\mathbb{E}\left({\left[\begin{array}{c}x\\ u\end{array}\right]}^T\left[\begin{array}{cc}F& H\\ H^T& G\end{array}\right]\left[\begin{array}{c}x\\ u\end{array}\right]\right)-\mathbb{E}\left({\left[\begin{array}{c}x\\ u\end{array}\right]}^T\left[\begin{array}{cc}W& 0\\ 0& 0\end{array}\right]\left[\begin{array}{c}x\\ u\end{array}\right]\right)\ge 0,$$

i.e.,

$$f(x,u)-\mathbb{E}\left[x^{T}Wx\right]\ge 0.$$

So, for every $x,u\in {\mathcal{L}}^2(\mathsf{\Omega },\mathcal{F},\mathbb{P})$

$$f(x,u)\ge \mathbb{E}\left[x^{T}Wx\right]>-\infty ,$$

and

$$\underset{u}{inf}f(x,u)>-\infty ,$$

This proves that (i) is true.

Furthermore, if $G>0$, by applying the completing square method to $f(x,u)$ with respect to u, we obtain

$$f(x,u)=\mathbb{E}\left[x^T(F-H{G}^{-1}{H}^T)x+{(u+G^{-1}{H}^{T}x)}^{T}G(u+G^{-1}{H}^{T}x)\right].$$

Let $u=u^{\ast }=-G^{-1}{H}^{T}x$. We can directly obtain $f(x,u^{\ast })=\mathbb{E}\left[x^T(F-H{G}^{-1}{H}^T)x\right]$. This ends the proof. □

The following provides a connection between the well-posedness of the LQ problem and the solvability of the GDRE.

Theorem 2.

The LQ problem in (1) and (2) is well-posed if and only if there exist symmetric matrices $P_0,\cdots ,P_N$ satisfying the GDRE, where the randomness of $P_{k+1}(k=0,1,\cdots ,N-1)$ is generated by ${\omega }_0,{\omega }_1,\cdots ,{\omega }_k$. Furthermore, the optimal cost is given by

$$\underset{u_0,\cdots ,u_{N-1}}{inf}J\left(u\right)=\mathbb{E}\left[x_0^{\mathrm{T}}{P}_0{x}_0\right].$$

Proof. 

We prove that the solvability of the GDRE is necessary for the well-posedness of the LQ problem by induction. This needs to consider the cost function from l to N. Suppose that

$$V^l\left(x_l\right)=\underset{u_l,\cdots ,u_{N-1}}{inf}\mathbb{E}[\sum _{k=l}^{N-1}(x_k^{\mathrm{T}}Q\left(k\right)x_k+2x_k^{\mathrm{T}}S\left(k\right)u_k+u_k^{\mathrm{T}}R\left(k\right)u_k)+x_N^{\mathrm{T}}{Q}_N{x}_N].$$

(7)

Then $V^{l_2}\left(x_{l_2}\right)$ is also finite for any $l_1\le l_2$ when $V^{l_1}\left(x_{l_1}\right)$ is finite. This fact is used at each step of the induction: the LQ problem is assumed to be well-posed at the initial time, so the cost function $V^l\left(x_l\right)$ is finite at any stage $0\le l\le N-1$.

First, we consider the case of $l=N-1$, and let $P_N=Q_N\ge 0$. There exists

$$\begin{array}{cc}\hfill & V^{N-1}\left(x_{N-1}\right)=\underset{u_{N-1}}{inf}\mathbb{E}\left\{x_{N-1}^{\mathrm{T}}\right[Q_{N-1}+A^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta A(N-1)\hfill \\ \hfill & +2A^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1(N-1)\hfill \\ \hfill & +A_1^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1(N-1)]x_{N-1}\hfill \\ \hfill & +2x_{N-1}^{\mathrm{T}}[S_{N-1}+A^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta B(N-1)\hfill \\ \hfill & +A_1^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)\eta p(\xi ,\eta )d\eta B(N-1)]u_{N-1}\hfill \\ \hfill & +u_{N-1}^{\mathrm{T}}[R_{N-1}+B^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta B(N-1)]u_{N-1}\}.\hfill \end{array}$$

So,

$$V^{N-1}\left(x_{N-1}\right)=\mathbb{E}\left[x_{N-1}^{\mathrm{T}}{P}_{N-1}\left(\xi \right)x_{N-1}\right].$$

According to Lemma 3, we have the following conditions:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill P_{N-1}\left(\xi \right)& =Q_{N-1}+A^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta A(N-1)\hfill \\ & +A^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1(N-1)\hfill \\ & +A_1^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)\eta p(\xi ,\eta )d\eta A(N-1)\hfill \\ & +A_1^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1(N-1)\hfill \\ & -H_{N-1}^{\mathrm{T}}\left(\xi \right)G_{N-1}^{-1}\left(\xi \right)H_{N-1}\left(\xi \right),\hfill \end{array}\end{array}$$

with $G_{N-1}\left(\xi \right)=R_{N-1}+B^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta B(N-1)$, where

$$\begin{array}{cc}\hfill H_{N-1}\left(\xi \right)& =S_{N-1}^{\mathrm{T}}+B^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta A(N-1)\hfill \\ \hfill & +B^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1(N-1).\hfill \end{array}$$

Because $p(\xi ,\eta )$ is the transition probability density function, there exists $p(\xi ,\eta )\ge 0$. So, for every $\xi ,\eta$, we have $P_N\left(\eta \right)p(\xi ,\eta )\ge 0$, and the following results can be obtained:

$$B^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta B(N-1)\ge 0.$$

Reminding that $R_{N-1}\ge 0$, we have $G_{N-1}\left(\xi \right)=R_{N-1}+B^{\mathrm{T}}(N-1){\int }_{\mathbb{R}}{P}_N\left(\eta \right)p(\xi ,\eta )d\eta B(N-1)\ge 0$. So, Equation (8) satisfies the GDRE for $k=N-1$.

Next, suppose that we have found a sequence of symmetric matrices $P_l,\dots ,P_{N-1}$, which solves the GDRE for $i=l,\dots ,N-1$, and satisfies

$$V^l\left(x_l\right)=\mathbb{E}\left[x_l^{\mathrm{T}}{P}_l\left(\xi \right)x_l\right].$$

Then, the following results are derived:

$$\begin{array}{cc}\hfill & V^{l-1}\left(x_{l-1}\right)=\underset{u_{l-1}}{inf}\mathbb{E}[x_{l-1}^{\mathrm{T}}{Q}_{l-1}{x}_{l-1}+2x_{l-1}^{\mathrm{T}}{S}_{l-1}{u}_{l-1}+u_{l-1}^{\mathrm{T}}{R}_{l-1}{u}_{l-1}+V^l\left(x_l\right)]\hfill \\ \hfill & =\underset{u_{l-1}}{inf}\mathbb{E}[x_{l-1}^{\mathrm{T}}{Q}_{l-1}{x}_{l-1}+2x_{l-1}^{\mathrm{T}}{S}_{l-1}{u}_{l-1}+u_{l-1}^{\mathrm{T}}{R}_{l-1}{u}_{l-1}+x_l^{\mathrm{T}}{P}_l{x}_l]\hfill \\ \hfill & =\underset{u_{l-1}}{inf}\mathbb{E}\left\{x_{l-1}^{\mathrm{T}}\right[Q_{l-1}+A^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)p(\xi ,\eta )d\eta A(l-1)\hfill \\ \hfill & +2A^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1(l-1)\hfill \\ \hfill & +A_1^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1(l-1)]x_{l-1}\hfill \\ \hfill & +2x_{l-1}^{\mathrm{T}}[S_{l-1}+A^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)p(\xi ,\eta )d\eta B(l-1)\hfill \\ \hfill & +A_1^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)\eta p(\xi ,\eta )d\eta B(l-1)]u_{l-1}\hfill \\ \hfill & +u_{l-1}^{\mathrm{T}}[R_{l-1}+B^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)p(\xi ,\eta )d\eta B(l-1)]u_{l-1}\}.\hfill \end{array}$$

Since $V^{l-1}\left(x_{l-1}\right)$ is finite, according to Lemma 3, we have

$$\left\{\begin{array}{cc}\hfill P_{l-1}\left(\xi \right)& =Q_{l-1}+A^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)p(\xi ,\eta )d\eta A(l-1)\hfill \\ \hfill & +A^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1(l-1)\hfill \\ \hfill & +A_1^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)\eta p(\xi ,\eta )d\eta A(l-1)\hfill \\ \hfill & +A_1^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1(l-1)\hfill \\ \hfill & -H_{l-1}^{\mathrm{T}}\left(\xi \right)G_{l-1}^{-1}\left(\xi \right)H_{l-1}\left(\xi \right),\hfill \\ \hfill G_{l-1}\left(\xi \right)& =R_{l-1}+B^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)p(\xi ,\eta )d\eta B(l-1)\ge 0,\hfill \\ \hfill H_{l-1}\left(\xi \right)& =S_{l-1}^{\mathrm{T}}+B^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)p(\xi ,\eta )d\eta A(l-1)\hfill \\ \hfill & +B^{\mathrm{T}}(l-1){\int }_{\mathbb{R}}{P}_l\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1(l-1).\hfill \end{array}\right.$$

In addition,

$$V^{l-1}\left(x_{l-1}\right)=\mathbb{E}\left[x_{l-1}^{\mathrm{T}}{P}_{l-1}\left(\xi \right)x_{l-1}\right].$$

By the recursion method, the necessity of the GDRE for the well-posedness of the LQ problem has been proven.

According to Lemma 2, we deduce that the solution of the GDRE also satisfies the LMI condition (6), which, according to Theorem 2, implies the well-posedness of the LQ problem. □

Remark 1.

The following constrained difference equation is called a generalized difference Riccati equation (GDRE):

$$\left\{\begin{array}{cc}\hfill P_k\left(\xi \right)& =Q\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)\hfill \\ \hfill & +A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A\left(k\right)+A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1\left(k\right)\hfill \\ \hfill & -H_k^{\mathrm{T}}\left(\xi \right)G_k^{-1}\left(\xi \right)H_k\left(\xi \right),\hfill \\ \hfill P_N& =Q_N,k=0,\dots ,N-1,\hfill \end{array}\right.$$

(8)

where

$$\begin{array}{ccc}\hfill & & H_k\left(\xi \right)=S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+B^{\mathrm{T}}\left(k\right)A_1\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta ,\hfill \\ & & G_k\left(\xi \right)=R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)\ge 0,k=0,\dots ,N-1.\hfill \end{array}$$

Compared to the Riccati equations obtained in [7], the forms of (9) are more complex, and the probability distributions of the Markov processes are necessary. Specifically, if ${\omega }_k$ and ${\omega }_{k-1}$ are independent with identical distributions and with the expectation $\mathbb{E}\left[{\omega }_k\right]=0$ and the variance $\mathbb{E}\left[{\omega }_k^2\right]=1$, we have $p(\xi ,\eta )=\tilde{p}\left(\eta \right)$ for all $\xi \in \mathbb{R}$, which implies independence between ${\omega }_k$ and ${\omega }_{k-1}$, and ${\int }_{\mathbb{R}}\eta p(\xi ,\eta )d\eta ={\int }_{\mathbb{R}}\eta \tilde{p}\left(\eta \right)d\eta =0$ and ${\int }_{\mathbb{R}}{\eta }^{2}p(\xi ,\eta )d\eta ={\int }_{\mathbb{R}}{\eta }^2\tilde{p}\left(\eta \right)d\eta =1$ for all $\xi \in \mathbb{R}$. In this case, we can take $P_k$ as deterministic matrices that are not dependent on ${\omega }_{k-1}$ or ξ. The forms of the Riccati equations are given more simply:

$$\left\{\begin{array}{cc}\hfill P_k& =Q\left(k\right)+A^{\mathrm{T}}\left(k\right)P_{k+1}A\left(k\right)+A_1^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)-H_k^{\mathrm{T}}{G}_k^{-1}{H}_k,\hfill \\ \hfill P_N& =Q_N,k=0,\cdots ,N-1,\hfill \end{array}\right.$$

(9)

where

$$\begin{array}{ccc}\hfill & & H_k=S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right)P_{k+1}A\left(k\right)+B^{\mathrm{T}}\left(k\right)A_1\left(k\right)P_{k+1},\hfill \\ & & G_k=R\left(k\right)+B^{\mathrm{T}}\left(k\right)P_{k+1}B\left(k\right)>0,k=0,\dots ,N-1,\hfill \end{array}$$

which is the form of the GDRE provided in [10]. Because the systems discussed in this paper are only for finite-time cases, the system’s stability is not discussed. However, the stability of such Markov-process-driven systems is also a topic worthy of further research, and excellent relevant results can be found in [42,43].

Remark 2.

From Lemma 2 and Theorem 2, it is obvious that the LMI condition (6) is also necessary for the well-posedness of the LQ problem. So, the LMI condition (8) is a sufficient and necessary condition for the well-posedness of the LQ problem.

3.2. Attainability

The following result shows the equivalent relationship between the well-posedness and attainability of the LQ problem, the solvability of the GDRE, and the feasibility of the LMI condition and provides the optimal control by which the LQ problem is attainable, as well as the optimal value function.

Theorem 3.

The following are equivalent:

(i) 

The LQ problem in (1) and (2) is well-posed.

(ii) 

The LQ problem in (1) and (2) is attainable.

(iii) 

The LMI condition (6) is feasible.

(iv) 

The GDRE (9) is solvable.

In addition, when any of the above conditions are satisfied, the LQ problem in (1) and (2) is attainable through

$$u_k^{\ast }=-G_k^{-1}\left(\xi \right)H_k\left(\xi \right)x_k,k=0,\dots ,N-1,$$

(10)

and the optimal cost function

$$J\left(u_k^{\ast }\right)=\underset{u_0,\dots ,u_{N-1}}{inf}J\left(u\right)=\mathbb{E}\left[x_0^{\mathrm{T}}{P}_0{x}_0\right],$$

(11)

where

$$\begin{array}{cc}\hfill G_k\left(\xi \right)& =R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right),\hfill \\ \hfill H_k\left(\xi \right)& =S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+B^{\mathrm{T}}\left(k\right)A_1\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta ,\hfill \end{array}$$

$P_0,\dots ,P_N$ are solutions to the GDRE (9).

Proof. 

According to Theorem 1 and Theorem 2, the equivalences $\left(i\right)\iff \left(iii\right)\iff \left(iv\right)$ are straightforward. Next, we prove that the LQ problem is attainable through the feedback control law (10). To this end, by introducing the symmetric matrices $P_k,P_{k+1}$, $k=0,\dots ,N-1$, where the randomness of $P_{k+1}$ is generated by ${\omega }_0,{\omega }_1,\dots ,{\omega }_k$, and the randomness of $P_k$ is generated by ${\omega }_0,{\omega }_1,\dots ,{\omega }_{k-1}$, we have

$$\begin{array}{ccc}& & \mathbb{E}(x_{k+1}^{\mathrm{T}}{P}_{k+1}{x}_{k+1}-x_k^{\mathrm{T}}{P}_k{x}_k)=\mathbb{E}\{x_k^{\mathrm{T}}[A^{\mathrm{T}}\left(k\right)P_{k+1}A\left(k\right)-P_k]x_k+2u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right)P_{k+1}A\left(k\right)x_k\hfill \\ & & +u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right)P_{k+1}B\left(k\right)u_k+2x_k^{\mathrm{T}}{A}^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)x_k{\omega }_k\hfill \\ & & +2u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)x_k{\omega }_k+{\omega }_k^{\mathrm{T}}{x}_k^{\mathrm{T}}{A}_1^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)x_k{\omega }_k\}.\hfill \end{array}$$

Through the smoothing property of the conditional expectation, we have

$$\begin{array}{ccc}& & \mathbb{E}(x_{k+1}^{\mathrm{T}}{P}_{k+1}{x}_{k+1}-x_k^{\mathrm{T}}{P}_k{x}_k)=\mathbb{E}\left\{\mathbb{E}\right[x_k^{\mathrm{T}}(A^{\mathrm{T}}\left(k\right)P_{k+1}A\left(k\right)-P_k)x_k+2u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right)P_{k+1}A\left(k\right)x_k\hfill \\ & & +u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right)P_{k+1}B\left(k\right)u_k+2x_k^{\mathrm{T}}{A}^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)x_k{\omega }_k\hfill \\ & & +2u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)x_k{\omega }_k+{\omega }_k^{\mathrm{T}}{x}_k^{\mathrm{T}}{A}_1^{\mathrm{T}}\left(k\right)P_{k+1}{A}_1\left(k\right)x_k{\omega }_k\mid {\omega }_0,\dots ,{\omega }_{k-1}\left]\right\},\hfill \end{array}$$

since the randomness of $x_k$, $u_k$, and $P_k$ is generated by ${\omega }_0,{\omega }_1,\dots ,{\omega }_{k-1}$, but the randomness of $P_{k+1}$ is generated by ${\omega }_0,{\omega }_1,\dots ,{\omega }_k$. According to Lemma 1, we can obtain

$$\begin{array}{ccc}& & \mathbb{E}(x_{k+1}^{\mathrm{T}}{P}_{k+1}{x}_{k+1}-x_k^{\mathrm{T}}{P}_k{x}_k)=\mathbb{E}\left\{x_k^{\mathrm{T}}\right[A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)-P_k\left(\xi \right)\hfill \\ & & +2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)+A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1\left(k\right)]x_k\hfill \\ & & +2u_k^{\mathrm{T}}[B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)]x_k\hfill \\ & & +u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)u_k\}.\hfill \end{array}$$

Denote

$$\begin{array}{ccc}& & \varphi =x_k^{\mathrm{T}}[A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)-P_k\left(\xi \right)\hfill \\ & & +2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)+A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1\left(k\right)]x_k\hfill \\ & & +2u_k^{\mathrm{T}}[B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)]x_k\hfill \\ & & +u_k^{\mathrm{T}}{B}^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)u_k,\hfill \end{array}$$

and then

$$\begin{array}{c}\hfill \mathbb{E}(x_{k+1}^{\mathrm{T}}{P}_{k+1}{x}_{k+1}-x_k^{\mathrm{T}}{P}_k{x}_k)=\mathbb{E}\left(\varphi \right)\end{array}$$

(12)

Summing k from 0 to $N-1$ on both sides of Equation (12) at the same time, we can obtain

$$\begin{array}{c}\hfill \mathbb{E}(x_N^{\mathrm{T}}{P}_N{x}_N-x_0^{\mathrm{T}}{P}_0{x}_0)=\mathbb{E}\sum _{k=0}^{N-1}\left(\varphi \right),\end{array}$$

i.e.,

$$\begin{array}{c}\hfill \mathbb{E}\sum _{k=0}^{N-1}\left(\varphi \right)-\mathbb{E}(x_N^{\mathrm{T}}{P}_N{x}_N-x_0^{\mathrm{T}}{P}_0{x}_0)=0.\end{array}$$

So,

$$\begin{array}{cc}\hfill J\left(u\right)& =\mathbb{E}\sum _{k=0}^{N-1}[x_k^{\mathrm{T}}Q\left(k\right)x_k+2x_k^{\mathrm{T}}S\left(k\right)u_k+u_k^{\mathrm{T}}R\left(k\right)u_k]+\mathbb{E}\left[x_N^{\mathrm{T}}{Q}_N{x}_N\right]\hfill \\ \hfill & +\mathbb{E}\sum _{k=0}^{N-1}\left(\varphi \right)-\mathbb{E}(x_N^{\mathrm{T}}{P}_N{x}_N-x_0^{\mathrm{T}}{P}_0{x}_0)\hfill \\ \hfill & =\mathbb{E}\sum _{k=0}^{N-1}\left\{x_k^{\mathrm{T}}[Q\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)\right.\hfill \\ \hfill & -P_k\left(\xi \right)+2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)\hfill \\ \hfill & +A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1\left(k\right)]x_k\hfill \\ \hfill & +2u_k^{\mathrm{T}}{[S\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)\hfill \\ \hfill & +B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right)]x_k\hfill \\ \hfill & \left.+u_k^{\mathrm{T}}[R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right)]u_k\right\}\hfill \\ \hfill & +\mathbb{E}\left[x_N^{\mathrm{T}}(Q_N-P_N)x_N\right]+\mathbb{E}\left[x_0^{\mathrm{T}}{P}_0{x}_0\right].\hfill \end{array}$$

Completing the square for $u_k$, we have

$$\begin{array}{ccc}& & J\left(u\right)=\mathbb{E}\sum _{k=0}^{N-1}\{x_k^{\mathrm{T}}\mathsf{\Phi }{x}_k+{(u_k-u_k^{\ast })}^{\mathrm{T}}[R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B_k](u_k-u_k^{\ast })\}\hfill \\ & & +\mathbb{E}\left[x_N^{\mathrm{T}}(Q_N-P_N)x_N\right]+\mathbb{E}\left(x_0^{\mathrm{T}}{P}_0{x}_0\right),\hfill \end{array}$$

where

$$\begin{array}{ccc}& & u_k^{\ast }=-G_k^{-1}\left(\xi \right)H_k\left(\xi \right)x_k,\hfill \\ & & \mathsf{\Phi }=Q\left(k\right)+A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)-P_k\left(\xi \right)+2A^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta \hfill \\ & & \times A_1\left(k\right)+A_1^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1\left(k\right)-H_k^{\mathrm{T}}\left(\xi \right)G_k^{-1}\left(\xi \right)H_k\left(\xi \right),\hfill \\ & & H_k\left(\xi \right)=S{\left(k\right)}^{\mathrm{T}}+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta A\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A_1\left(k\right),\hfill \\ & & G_k\left(\xi \right)=R\left(k\right)+B^{\mathrm{T}}\left(k\right){\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta B\left(k\right).\hfill \end{array}$$

Let $P_0,\dots ,P_N$ solve the GDRE (9), and then when $u_k=u_k^{\ast }$, we can obtain the optimal cost function

$$J\left(u_k^{\ast }\right)=\underset{u_0,\dots ,u_{N-1}}{inf}J\left(u\right)=\mathbb{E}\left[x_0^{\mathrm{T}}{P}_0{x}_0\right].$$

This shows that the optimal value equals $\mathbb{E}\left[x_0^{\mathrm{T}}{P}_0{x}_0\right]$ and the LQ problem is attainable through the feedback control $u_k^{\ast }=-G_k^{-1}\left(\xi \right)H_k\left(\xi \right)x_k.$ □

4. Examples

Example 1.

Consider the following one-dimensional system:

$$\left\{\begin{array}{c}{x}_{k+1}=a{x}_k+b{u}_k+a_1{x}_k{\omega }_k\hfill \\ x_0\in \mathbb{R},k=0,1,\cdots ,N-1\hfill \end{array}\right.$$

(13)

with the cost function

$$J\left(u\right)=\sum _{k=0}^{N-1}[q{x}_k^2+2s{x}_k{u}_k+r{u}_k^2]+x_N^2$$

(14)

where the coefficients of $a,b,a_1,q,s$, and r take values in $\mathbb{R}$ where $q>0,r>0$, $x_k\in \mathbb{R}$, and $u_k\in \mathbb{R}$ are the state and control, respectively, ${\omega }_k$ is a Markovian process with transition probability density $p(\xi ,\eta )=\frac{3}{4}[1-{(\eta -\xi )}^2]1_D$ with initial probability density $p_0\left(\xi \right)=\frac{3}{4}(1-{\xi }^2)1_{D_0}$, where the set $D=\left\{\right(\xi ,\eta \left)\right||\eta -\xi |<1,\xi ,\eta \in \mathbb{R}\}$ and $D_0=\left\{\xi \right|\left|\xi \right|<1,\xi \in \mathbb{R}\}$. According to Theorem 2, The Riccati equation of the LQ problem in (13) and (14) is

$$\begin{array}{ccc}\hfill & & P_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta +2a{a}_1{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta +a_1^2{\int }_{\mathbb{R}}{\eta }^{2}p(\xi ,\eta )d\eta \hfill \\ & & -\frac{{\left(s_k^2+ab{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta +a_{1}b{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta \right)}^2}{r_k+b^2{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta }\hfill \end{array}$$

(15)

and the optimal control is given by

$$u_k^{\ast }=-\frac{s_k^2+ab{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta +a_{1}b{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta }{r_k+b^2{\int }_{\mathbb{R}}{P}_{k+1}\left(\eta \right)p(\xi ,\eta )d\eta }{x}_k,k=0,1,\cdots ,N-1$$

Figure 1 shows the profile of the Markovian process ${\omega }_k$ and the trajectories of $P_k$ which is the solution of the Riccati Equation (15) with coefficients $a=0.97$, $b=0.55$, $a_1=0.2$, $q=0.2$, $s=0.5$, and $r=0.6$. Figure 2 shows the corresponding trajectories of the optimal control $u_k^{\ast }$ and the state $x_k$.

Figure 1. Profiles of Markov process ${\omega }_k$ and solutions of $P_k$ for the LQ problem in (13) and (14).

Figure 2. Trajectories of $u_k$ and $x_k$ for the LQ problem in (13) and (14).

Example 2.

In automobile manufacturing industrial design, automobile suspension is an important piece of equipment. Figure 3 shows the hybrid active suspension studied in [44].

Figure 3. Schematic diagram of hybrid active suspension mechanics.

$m_1$ represents the non-spring-loaded mass; $m_2$ represents the spring-loaded mass; $K_1$ is the equivalent stiffness of the tire; $K_2$ is the suspension stiffness; $C_2$ is the suspension damping; m, k, c, and u are, respectively, the mass block, spring stiffness, damper damping coefficient, and electromagnetic driving force of the electromagnetic reaction force actuator. When the electromagnetic coil is energized, the electromagnetic driving force u is generated. The force u drives the mass block m to vibrate so that the electromagnetic actuator as a whole generates a reaction force $F_t$. q, $X_1$, $X_2$, and $X_3$ represent the displacement of the road surface, wheel, body, and electromagnetic actuator mass block, respectively. The external force $F_t$ of the electromagnetic actuator acts only on the non-spring-loaded mass $m_1$ as an active control force. Therefore, as long as the magnitude and direction of the current of the electromagnetic actuator are controlled, the corresponding active control force can be generated to adjust the vibration of the entire vibration system. According to Newton’s second law, the differential Equation (16) of motion of the hybrid active suspension shown in Figure 1 can be obtained

$$\left\{\begin{array}{cc}\hfill m_1{\ddot{X}}_1& =K_2(X_2-X_1)+C_2({\dot{X}}_2-{\dot{X}}_1)-K_1(X_1-q)-F_t\hfill \\ \hfill m_2{\ddot{X}}_2& =-K_2(X_2-X_1)-C_2({\dot{X}}_2-{\dot{X}}_1)\hfill \end{array}\right.$$

(16)

where $q=q\left(t\right)$ is the displacement in the vertical direction of the road surface, and here, it is assumed that $q\left(t\right)$ satisfies the following model [45]:

$$\dot{q}\left(t\right)=\sqrt{2\pi G_0{U}_0}{x}_3\left(t\right)\omega \left(t\right),$$

(17)

where $G_0$ is the road roughness coefficient and $U_0$ is the speed of the vehicle. By selecting $x_1=X_2-X_1$, $x_2={\dot{X}}_2$, $x_4={\dot{X}}_1$, and $x_3=X_1-q$ as the state variables and taking $x={\left[x_1{x}_2{x}_3{x}_4\right]}^T$ as the notation, the system is discretized as

$$J\left(u\right)=\mathbb{E}\sum _{t=1}^N({\left|x\right(t\left)\right|}^2+{\left|u\right(t\left)\right|}^2)$$

(18)

$$x(t+1)=Ax\left(t\right)+Bu\left(t\right)+A_{1}x\left(t\right)\omega \left(t\right)$$

(19)

where the coefficient matrices A, B, and $A_1$ are obtained as follows:

$$A=\left[\begin{array}{cccc}1& \mathsf{\Delta }t& 0& -\mathsf{\Delta }t\\ -\frac{K_2\mathsf{\Delta }t}{m_2}& \frac{m_2-C_2\mathsf{\Delta }t}{m_2}& 0& \frac{C_2\mathsf{\Delta }t}{m_2}\\ 0& 0& 1& \mathsf{\Delta }t\\ \frac{K_2\mathsf{\Delta }t}{m_1}& \frac{C_2\mathsf{\Delta }t}{m_1}& -\frac{K_1\mathsf{\Delta }t}{m_1}& \frac{m_1-C_2\mathsf{\Delta }t}{m_1}\end{array}\right],$$

$$B=\left[\begin{array}{c}0\\ 0\\ 0\\ -\frac{\mathsf{\Delta }t}{m_1}\end{array}\right],A_1=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& -\mathsf{\Delta }t\sqrt{2\pi G_0{U}_0}& 0\\ 0& 0& 0& 0\end{array}\right],$$

and $\mathsf{\Delta }t$ is the time difference between this moment and the previous moment; $u\left(t\right)=F_t$ is the active control force as the control input; $\omega \left(t\right)$ represents the uncertainty input, generally white noise; $\omega \left(t\right)$ is a Markovian process; and its transition probability density function is $p(\xi ,\eta )=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(\eta -\xi )}^2}{2{\sigma }^2}}$.

$J\left(u\right)$ represents the output of the LQ metric, and our goal is to find the optimal control $u^{\ast }\left(t\right)$ that minimizes $J\left(u\right)$ under the premise that $x\left(t\right)$ is as small as possible.

According to Theorem 3, we can deduce that the optimal control of the system is

$$\begin{array}{cc}\hfill u^{\ast }\left(t\right)& =-{(I+B^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)p(\xi ,\eta )d\eta B)}^{-1}\hfill \\ \hfill & \times [B^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)p(\xi ,\eta )d\eta A+B^{\mathrm{T}}{A}_1{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta ]x\left(t\right)\hfill \end{array}$$

(20)

where

$$\left\{\begin{array}{cc}\hfill P_t\left(\xi \right)& =I+A^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)p(\xi ,\eta )d\eta A+A^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)\eta A_1\hfill \\ \hfill & +A_1^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta A+A_1^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right){\eta }^{2}p(\xi ,\eta )d\eta A_1\hfill \\ \hfill & -[A^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)p(\xi ,\eta )d\eta B+A_1^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta B]\hfill \\ \hfill & \times {(I+B^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)p(\xi ,\eta )d\eta B)}^{-1}\hfill \\ \hfill & \times [B^{\mathrm{T}}{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)p(\xi ,\eta )d\eta A+B^{\mathrm{T}}{A}_1{\int }_{\mathbb{R}}{P}_{t+1}\left(\eta \right)\eta p(\xi ,\eta )d\eta ],\hfill \\ \hfill P_N& =0,t=0,\dots ,N-1\hfill \end{array}\right.$$

(21)

So, we can obtain

$$\underset{u}{inf}J\left(u\right)=\mathbb{E}\left[x^{\mathrm{T}}\left(0\right)P_{0}x\left(0\right)\right].$$

(22)

According to [44,45], the coefficients of system (19) take the following values: $\mathsf{\Delta }t=0.1$, $k_1=30$, $k_2=159$, $c_1=0.1$, $c_2=1.1$, $m_1=24$, $m_2=400$, $G_0=2.2\times {10}^{-4}$, and $U_0=60$. By solving the Riccati Equation (21), the matrix $P_0$ is obtained as follows:

$$P_0=\left[\begin{array}{cccc}\hfill 250.2033& \hfill 44.4047& \hfill 2.2024& \hfill -45.9939\\ \hfill 44.4047& \hfill 93.3853& \hfill 5.9328& \hfill -40.0286\\ \hfill 2.2024& \hfill 5.9328& \hfill 2.7048& \hfill -6.5274\\ \hfill -45.9939& \hfill -40.0286& \hfill -6.5274& \hfill 45.3183\end{array}\right].$$

Figure 4 illustrates the profiles of the Markovian process with the transport probability density function $p(\xi ,\eta )$ and the trajectories of the optimal control $u^{\ast }\left(t\right)$ in the optimal control problem in (18) and (19), with the optimal cost value $J\left(u^{\ast }\right)=3.079$.

Figure 4. Profiles of $\omega \left(t\right)$ and the trajectories of the optimal control $u^{\ast }\left(t\right)$ of system (19).

Figure 5 illustrates the trajectories of the optimal control $u^{\ast }\left(t\right)$ and the trajectories of $x_1^{\ast }\left(t\right)$, $x_2^{\ast }\left(t\right)$, $x_3^{\ast }\left(t\right)$, and $x_4^{\ast }\left(t\right)$, which are the components of the optimal states $x^{\ast }\left(t\right)$ of the optimal control problem in (18) and (19), with the initial state $x\left(0\right)={[0.1,0.1,0.1,0.1]}^T$. This shows that under the effect of the optimal control, the fluctuations of the system’s states change in a small range, and less energy from the optimal controller is needed.

Figure 5. Trajectories of the control $u\left(t\right)$ of system (19) and the corresponding components of state $x\left(t\right)=[x_1\left(t\right),x_2\left(t\right),x_3\left(t\right),x_4\left(t\right)]$.

In the above two examples, Example 1 covers one-dimensional systems and Example 2 covers four-dimensional systems. These two examples show that the main results of this paper can be used in different dimensional systems. However, in Example 2, the computational simulations focus on a primitive model of a suspension, where the connections with Markov processes are completely artificial. The more demanding applications of these systems need further exploration and research.

5. Conclusions

A type of discrete-time stochastic system driven by general Markov processes, known as Markov-process-driven systems, is proposed to describe more complex noises. By using the properties of the probability distribution of Markov processes, the LQ problem of discrete-time stochastic systems driven by homogeneous Markovian processes is studied. The equivalent relationship between the well-posedness and attainability of the LQ problem, the solvability of the GDRE, and the feasibility of the LMI condition is established. By applying the completing square method to these linear systems, the relationship between the well-posedness of the LQ problem and the LMI condition is obtained: if there exists a series of positive definite matrices satisfying the LMI condition, the LQ problem is well-posed. These results extend the GDRE to the general forms in which the probability distributions of Markov processes are needed to describe the impacts of Markovian properties. In addition, the equivalent relationship between the well-posedness of the LQ problem and the solvability of the GDRE is also obtained, and the necessity of the LMI condition for the well-posedness is also proven. Moreover, by using the properties of Markov processes and the method of completing squares, we have proven that such LQ problems are attainable and optimal state-feedback control can be obtained. Finally, a numerical example and a practical example are used to illustrate the effectiveness and validity of the theory.