Optimization of Constrained Stochastic Linear-Quadratic Control on an Infinite Horizon: A Direct-Comparison Based Approach

Abstract

In this paper we study the optimization of the discrete-time stochastic linear-quadratic (LQ) control problem with conic control constraints on an infinite horizon, considering multiplicative noises. Stochastic control systems can be formulated as Markov Decision Problems (MDPs) with continuous state spaces and therefore we can apply the direct-comparison based optimization approach to solve the problem. We first derive the performance difference formula for the LQ problem by utilizing the state separation property of the system structure. Based on this, we successfully derive the optimality conditions and the stationary optimal feedback control. By introducing the optimization, we establish a general framework for infinite horizon stochastic control problems. The direct-comparison based approach is applicable to both linear and nonlinear systems. Our work provides a new perspective in LQ control problems; based on this approach, learning based algorithms can be developed without identifying all of the system parameters.

1. Introduction

In this paper we study the discrete-time stochastic linear-quadratic (LQ) control optimal problem with conic control constraints and multiplicative noises on an infinite horizon. There exist in the literature various studies on the estimation and control problems of systems with a multiplicative noise [1,2]. As for the LQ type of stochastic optimal control problems with multiplicative noise, investigations have been focused on the LQ formulation with indefinite penalty matrices on control and state variables for both continuous-time and discrete-time models (see, e.g., [3,4]).

In an LQ optimal problem, the system dynamics are both linear in state and control variables, and the cost function is quadratic in these two variables [5]. One important attractive quality of the LQ type of optimal control models is its explicit control policy which can be derived by solving the corresponding Riccati equation. Due to the elegant structure, the LQ problem has always been a hot issue in optimal control research. Since the fundamental research on deterministic LQ problems by Kalman [6], there have been a great number of studies on it; see [5,7,8]. In the past few years, stochastic LQ problems have drawn more and more attention on this topic, due to the promising applications in different fields, including dynamic portfolio management, financial derivative pricing, population models, and nuclear heat transfer problems; see [9,10,11].

This paper is motivated by two recent developments: LQ optimal control and Markov decision problems (MDPs). First, the constrained LQ problem is significant in both theory and applications. Due to the constraints on state and control variables, it is hard to obtain the explicit control policy by solving the Riccati equation [5]. Recently, there have been studies regarding the constrained LQ optimal control problems, such as [12,13,14]. Meanwhile, in real applications, considering some practical limits, such as the risk or the economic regulations, we have to take some constraints on the control variables into the consideration. In the LQ control problems, including the positivity constraint for the control, some literature, [15,16], propose the optimality conditions and some numerical methods to characterize the optimal control policy. In this paper, we characterise the limits as the conic control constraints considering the real applications.

Work by Cao [17] and Puterman [18] demonstrate that stochastic control problems can be viewed as Markov decision problems. Therefore, the constrained stochastic LQ control problem can be formulated as an MDP, such as [19]. A direct-comparison based approach (or relative optimization), which originated in the area of discrete event systems, has been developed in the past years for the optimization of MDPs [17].

With this approach, optimization is based on the comparison of the performance measures of the system under any two policies. It is intuitively clear, and it can provide new insights, leading to new results to many problems, such as [20,21,22,23,24,25,26]. This approach is very convenient and suitable to the performance optimization problems, leading to results including the property of under-selectivity in time-nonhomogeneous Markov processes [24]. In this paper, we show that the special features of the constrained stochastic LQ optimal control make it possible to be solved by the direct-comparison based approach, leading to some new insights for the problem.

In our work, we consider the stochastic LQ control problem through an MDP formulation in the infinite horizon. Through the direct-comparison based approach [17], we first derive the performance potentials for the LQ problem by utilizing the state separation property of the system structure. Based on this, we successfully derive the optimality conditions and the stationary optimal feedback control. We show that the optimal control policy is a piece-wise affine function with respect to the state variables. In real applications, the proposed methodology can be used in many fields, such as system risk contagion [26] and power grid systems [27].

Our work provides a new perspective for LQ control problems. Compared with the former literature, such as [5,13], we still consider the multiplicative noises. We establish a general framework for studying infinite horizon stochastic control problems. With the direct-comparison based approach, which is applicable to both linear and nonlinear systems, we propose more results for the performance optimization problems, and the results can be extended easily. In addition, without identifying all the system parameters, this approach can be implemented on-line, and learning based algorithms can be developed.

The paper is organized as follows. Section 2 introduces an MDP formulation of the constrained stochastic LQ problem with multiplicative noises; some preliminary knowledge on MDP and the state separation property is also provided. In Section 3, we derive the performance difference formula, which is the foundation of the performance optimization; based on it, the Poisson equation and Dynkin’s formula can be obtained. Then we derive the optimality condition and the optimal policy through the performance difference formula. In Section 4, we illustrate the results by numerical examples. Finally, we conclude the paper in Section 5.

2. Problem Formulation

2.1. Problem with Infinite Time Horizon

In this section, we study the infinite horizon discrete-time stochastic LQ optimal control problem, in which the conic control constraints are also considered; see [5,14]. For simplicity of the parameters, we consider a one dimensional dynamic system with a multiplicative noise described by

$$\begin{array}{cc}\hfill x_{l+1}=A{x}_l+& \mathbf{B}{\mathbf{u}}_l\left(x_l\right)+[A{x}_l+\mathbf{B}{\mathbf{u}}_l\left(x_l\right)]{\xi }_l,\hfill \end{array}$$

(1)

for time $l=0,1,\cdots$. By denoting $\mathbb{R}\left({\mathbb{R}}_{+}\right)$ as the set of real (nonnegative real) numbers, in this system, $A\in \mathbb{R}$ and $\mathbf{B}\in {\mathbb{R}}^{1\times m}$ are deterministic values; $x_l\in \mathbb{R}$ is the state with $x_0$ being given; and ${\mathbf{u}}_l\in {\mathbb{R}}^m$ is a feedback control law at time l. For each l, ${\xi }_l$ denotes an independent identically distributed one-dimensional multiplicative noise, satisfying a normal distribution with mean 0 and variance ${\sigma }^2,\sigma \ge 0$. For each l, ${\xi }_l$ denotes a one-dimensional noise. ${\xi }_l$ and ${\xi }_k$ are independent for every $l\ne k$.

Now, we consider the conic control constraint sets (cf. [5])

$$\begin{array}{c}\hfill {\mathcal{C}}_l:=\left\{{\mathbf{u}}_l\right|{\mathbf{u}}_l\in {\mathcal{F}}_l,\mathbf{H}{\mathbf{u}}_l\in {\mathbb{R}}_{+}^n\},\end{array}$$

(2)

for $l=0,1,\cdots$, where $\mathbf{H}\in {\mathbb{R}}^{n\times m}$ is a deterministic matrix; and ${\mathcal{F}}_l$ is the filtration of the information available at time l. Let ${\mathcal{C}}_l\subset R^m$ be a given closed cone; i.e., $\alpha {\mathbf{u}}_l\in {\mathcal{C}}_l$ whenever ${\mathbf{u}}_l\in {\mathcal{C}}_l$ and $\alpha \ge 0$; and ${\mathbf{u}}_l+{\mathbf{v}}_l\in {\mathcal{C}}_l$ whenever ${\mathbf{u}}_l,{\mathbf{v}}_l\in \mathcal{C}$.

The goal of optimization is to minimize the total reward performance measure in a quadratic form:

$$\begin{array}{c}\hfill \left({\mathcal{P}}_A\right)\underset{\left\{{\mathbf{u}}_l\right\}{|}_{l=0}^{\infty }}{min}{\eta }^{\mathbf{u}}=\underset{L\to \infty }{lim}\mathrm{E}[\sum _{l=0}^{L-1}(Q{x}_l^2+{\mathbf{u}}_l^{\prime }\mathbf{R}{\mathbf{u}}_l)|x_0]\\ \hfill (\mathrm{s}.\mathrm{t}.)\{x_t,{\mathbf{u}}_t\} \mathrm{satisfies} \left(1\right) \mathrm{and} \left(2\right) \mathrm{for} l=0,1,\cdots ,\end{array}$$

(3)

where $Q\in {\mathbb{R}}_{+}$ and $\mathbf{R}\in {\mathbb{R}}_{+}^{m\times m}$ are deterministic. Here we denote the transpose operation by a prime in the superscript, such as ${\mathbf{u}}_l^{\prime }$. $\left\{{\mathbf{u}}_l\right\}$ denotes the control sequence $\{{\mathbf{u}}_0,{\mathbf{u}}_1,\cdots \}$. We also assume that (3) exists.

Therefore, the performance function of (3) is

$$\begin{array}{c}\hfill f^{\mathbf{u}}\left(x\right)=Q{x}^2+{\mathbf{u}}^{\prime }\mathbf{R}\mathbf{u}.\end{array}$$

(4)

In this paper, we will show that the direct-comparison based approach leads to more new results for the total rewards problem [7], and that the results can be easily extended.

2.2. MDPs with Continuous State Spaces

For a stationary control law ${\mathbf{u}}_l=\mathbf{u}\left(x\right)$, at time $l=0,1,\cdots$, the constraint (2) can be written as

$$\begin{array}{c}\hfill \mathcal{C}:=\left\{\mathbf{u}\right|\mathbf{u}\in {\mathbb{R}}^m,\mathbf{H}\mathbf{u}\in {\mathbb{R}}_{+}^n\}.\end{array}$$

Then the above stochastic control problem can be viewed as an MDP with continuous state spaces. More precisely, $\mathbf{u}\left(x\right)$ plays a similar role of actions in MDPs, and then the control law $\mathbf{u}$ is the same as a policy.

Consider a discrete-time Markov chain $\mathbf{X}:={\left\{x_l\right\}}_{l=0}^{\infty }$ with a continuous state space on $\mathbb{R}$. The transition probability can be described by a transition operatorP as

$$\begin{array}{c}\hfill \left(Ph\right)\left(x\right):={\int }_{\mathbb{R}}h\left(y\right)P\left(dy\right|x),\end{array}$$

(5)

where $P\left(dy\right|x)$ is the transition probability function, with $x,y\in \mathbb{R}$; and $h\left(y\right)$ is any measurable function on $\mathbb{R}$. As ${\xi }_l$ is independent Gaussian noises, given the current state $x_l=x$, under the stationary control $\mathbf{u}\left(x\right),y=x_{l+1}$ satisfies a normal distribution with mean ${\mu }_y=Ax+\mathbf{B}\mathbf{u}\left(x\right)$ and variance ${\sigma }_y^2={[Ax+\mathbf{B}\mathbf{u}\left(x\right)]}^2{\sigma }^2$. Then we have the transition function of this system as follows,

$$\begin{array}{c}\hfill P^{\mathbf{u}}\left(dy\right|x)=\frac{1}{\sqrt{2\pi }{\sigma }_y}exp\{-\frac{{(y-{\mu }_y)}^2}{2{\sigma }_y^2}\}dy.\end{array}$$

(6)

Let $\mathbb{B}$ be the $\sigma$-field of $\mathbb{R}$ containing all the (Lebesgue) measurable sets. For any set $B\in \mathbb{B}$, we can define the identity transition function $I\left(B\right|x)$. $I\left(B\right|x)=1$ if $x\in B$; $I\left(B\right|x)=0$ otherwise. For any function h and $x\in \mathbb{R}$, we have $\left(Ih\right)\left(x\right)=h\left(x\right)$.

The product of two transition functions $P_1\left(B\right|x)$ and $P_2\left(B\right|x)$ is defined as a transition function $\left(P_1{P}_2\right)\left(B\right|x)$:

$$\begin{array}{c}\hfill \left(P_1{P}_2\right)\left(B\right|x):={\int }_{\mathbb{R}}{P}_2\left(B\right|y)P_1\left(dy\right|x),\end{array}$$

where $x,y\in \mathbb{R},B\in \mathbb{B}$.

For any transition function P, we can define the kth power, $k=0,1,\cdots$, as $P^0=I,P^1=P$, and $P^k=P{P}^{k-1},k=2,\cdots$. Suppose that the Markov chain $\mathbf{X}$ is time-homogeneous with transition function $P\left(B\right|x),x\in \mathbb{R},B\in \mathbb{B}$. Then the k-step transition probability functions, denoted as $P^{\left(k\right)}\left(B\right|x),k=1,2,\cdots$, are given by the 1-step transition function defined as $P^{\left(1\right)}\left(B\right|x)=P\left(B\right|x)$ and

$$\begin{array}{c}\hfill P^{\left(k\right)}\left(B\right|x):={\int }_{\mathbb{R}}P\left(dy\right|x)P^{k-1}\left(B\right|y),k\ge 2.\end{array}$$

For any function $h\left(x\right)$, we have

$$\begin{array}{c}\hfill \left(P^{\left(k\right)}h\right)\left(x\right)={\int }_{\mathbb{R}}h\left(y\right)P^{\left(k\right)}\left(dy\right|x)=P\left(P^{(k-1)}\right)h\left(x\right).\end{array}$$

That is, as an operator, we have $P^{\left(k\right)}=P\left(P^{(k-1)}\right)$. Recursively, we can prove that $P^{\left(k\right)}=P^k$.

Suppose that a Markov chain $\mathbf{X}$ with a continuous state space on $\mathbb{R}$ has a steady-state distribution $\pi$ satisfying $\pi =\pi P$. Define function $e\left(x\right)=1$ for all $x\in \mathbb{R}$. We denote the performance potential g as a function which satisfies the Poisson equation (cf. [17])

$$\begin{array}{c}\hfill (I-P)g\left(x\right)+\eta \left(x\right)=f\left(x\right),\end{array}$$

(7)

where I and P are two transition functions, and $\eta \left(x\right)=\left(\pi f\right)e\left(x\right)=\eta e\left(x\right)$. Then if g is a solution to (7), so is $g+ce$, with any constant c. We define

$$\begin{array}{c}\hfill g_K:=\{I+\sum _{k=1}^K(P^k-e\pi )\}f,\end{array}$$

(8)

and assume the limit $g\left(x\right):={lim}_{K\to \infty }{g}_K\left(x\right)$ exists for $x\in \mathbb{R}$. Then we have the following lemma,

Lemma 1

(Solution to Poisson Equations [17]). For any transition function P and performance function $f\left(x\right)$, if

$$\begin{array}{cc}\hfill & \underset{k\to \infty }{lim}{P}^{k}f=\left(e\pi \right)f=\eta e,\hfill \\ \hfill & \underset{K\to \infty }{lim}{g}_K=g,\mathrm{and}\underset{K\to \infty }{lim}P{g}_K=Pg,\hfill \end{array}$$

hold for every $x\in \mathbb{R}$, then

$$\begin{array}{c}\hfill g=\{I+\sum _{k=1}^{\infty }(P^k-e\pi )\}f,\end{array}$$

(9)

is a solution to the Poisson Equation (7).

2.3. State Separation Property

In order to derive the explicit solution of the stochastic LQ control problem with conic constraints, Reference [14] gives the following lemma for the state separation property of the LQ problem,

Lemma 2

(State Separation [14]). In the system (1), for any $x\in \mathbb{R}$, the optimal solution for problem (3) at time l is a piecewise linear feedback policy

$$\begin{array}{c}\hfill {\mathbf{u}}^{*}\left(x_l\right)=\left\{\begin{array}{cc}{\widehat{\mathbf{K}}}^{*}{x}_l,\hfill & if{x}_l\ge 0,\hfill \\ -{\overline{\mathbf{K}}}^{*}{x}_l,\hfill & if{x}_l<0,\hfill \end{array}\right.\end{array}$$

(10)

for $l=0,1,\cdots$, where $\mathcal{K}:=\{\mathbf{K}\in {\mathbb{R}}^m|\mathbf{H}\mathbf{K}\in {\mathbb{R}}_{+}^n\}$ associated with the control constraint sets ${\mathcal{C}}_l$; ${\widehat{\mathbf{K}}}^{*},{\overline{\mathbf{K}}}^{*}\in \mathcal{K}$, are the optimal values of two correspondent auxiliary optimization problems, and the superscript “*” corresponds to the optimal control.

Based on (10) in Lemma 2, the stationary control can be written as $\mathbf{u}\left(x\right)=\widehat{\mathbf{K}}x{\mathbf{1}}_{x\ge 0}-\overline{\mathbf{K}}x{\mathbf{1}}_{x<0}$, where ${\mathbf{1}}_B$ is an indicator function, such that ${\mathbf{1}}_B=1$, if the condition B holds true and ${\mathbf{1}}_B=0$ otherwise; and $\widehat{\mathbf{K}},\overline{\mathbf{K}}\in \mathcal{K}$. Applying this control, the system dynamics (1) becomes

$$\begin{array}{cc}\hfill x_{l+1}=& \widehat{C}{x}_l{\mathbf{1}}_{x_l\ge 0}+\overline{C}{x}_l{\mathbf{1}}_{x_l<0}\hfill \\ \hfill & +[\widehat{C}{x}_l{\mathbf{1}}_{x_l\ge 0}+\overline{C}{x}_l{\mathbf{1}}_{x_l<0}]{\xi }_l,\hfill \end{array}$$

(11)

for $l=0,1,\cdots$, where

$$\begin{array}{c}\hfill \widehat{C}=A+\mathbf{B}\widehat{\mathbf{K}},\overline{C}=A-\mathbf{B}\overline{\mathbf{K}}.\end{array}$$

(12)

Moreover, the performance measure (3) becomes

$$\begin{array}{c}\hfill {\eta }^{\mathbf{u}}\left(x\right)=\underset{L\to \infty }{lim}\mathrm{E}[\sum _{l=0}^{L-1}\widehat{W}{x}_l^2{\mathbf{1}}_{x_l\ge 0}+\overline{W}{x}_l^2{\mathbf{1}}_{x_l<0}|x_0=x],\end{array}$$

where $\widehat{W}=Q+{\widehat{\mathbf{K}}}^T\mathbf{R}\widehat{\mathbf{K}}$ and $\overline{W}=Q+{\overline{\mathbf{K}}}^T\mathbf{R}\overline{\mathbf{K}}$. Therefore, the performance function (4) becomes

$$\begin{array}{c}\hfill f\left(x\right)=\widehat{W}{x}^2{\mathbf{1}}_{x\ge 0}+\overline{W}{x}^2{\mathbf{1}}_{x<0}.\end{array}$$

(13)

It is easy to verify that $\widehat{W}$ and $\overline{W}$ are positive semi-definite. We assume that this one-dimensional state system is stable, and then the spectral radiuses of $\widehat{C}$ and $\overline{C}$ are less than 1, i.e., $C^{max}=max(\widehat{C},\overline{C})<1$. In the next section, we will derive the performance potentials for the LQ problem, which is the foundation of the performance optimization. Based on this, the Poisson equation and the Dynkin’s formula can be derived. The direct-comparison based approach provides a new perspective for this problem, and the results can be extended easily.

3. Performance Optimization

In this section, utilizing the state separation property, we derive the performance difference formula, which compares the performance measures of any tow policies, and then derive the optimality condition and the optimal policy with the direct-comparison based approach.

3.1. Performance Difference Formula

We denote ${\widehat{W}}_0=\widehat{W}$ and ${\overline{W}}_0=\overline{W}$. Then we have the performance function as $f\left(x\right)={\widehat{W}}_0{x}^2{\mathbf{1}}_{x\ge 0}+{\overline{W}}_0{x}^2{\mathbf{1}}_{x<0}$. With the initial condition $x_0=x$, by (5), (6), (11), and (13), the performance operator is

$$\begin{array}{c}\hfill \left(Pf\right)\left(x\right)={\widehat{W}}_1{x}^2{\mathbf{1}}_{x\ge 0}+{\overline{W}}_1{x}^2{\mathbf{1}}_{x<0},\end{array}$$

(14)

where

$$\begin{array}{c}\hfill {\widehat{W}}_1=(a_1{\widehat{W}}_0+a_2{\overline{W}}_0){\widehat{C}}^2,{\overline{W}}_1=(a_1{\widehat{W}}_0+a_2{\overline{W}}_0){\overline{C}}^2,\end{array}$$

and

$$\begin{array}{cc}\hfill a_1& =\sigma \varphi \left(\frac{1}{\sigma }\right)+(1+{\sigma }^2)\mathsf{\Phi }\left(\frac{1}{\sigma }\right),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill a_2& =-\sigma \varphi (-\frac{1}{\sigma })+(1+{\sigma }^2)\mathsf{\Phi }(-\frac{1}{\sigma }),\hfill \end{array}$$

(16)

with $\varphi (·)$ as the probability density function of a standard normal distribution. We can verify that $a_1$ and $a_2$ are both nonnegative constants, with $a_1+a_2=1+{\sigma }^2$.

As $P^{2}f=P\left(Pf\right)$, continuing this process, we obtain

$$\begin{array}{c}\hfill \left(P^{k}f\right)\left(x\right)={\widehat{W}}_k{x}^2{\mathbf{1}}_{x\ge 0}+{\overline{W}}_k{x}^2{\mathbf{1}}_{x<0},\end{array}$$

(17)

where

$$\begin{array}{cc}\hfill {\widehat{W}}_k& =(a_1{\widehat{W}}_{k-1}+a_2{\overline{W}}_{k-1}){\widehat{C}}^2,\hfill \\ \hfill {\overline{W}}_k& =(a_1{\widehat{W}}_{k-1}+a_2{\overline{W}}_{k-1}){\overline{C}}^2.\hfill \end{array}$$

We set $W_0^{*}=max(\widehat{W_0},\overline{W_0}).$

In order to ensure the stability of the system, Reference [14] gives some assumptions. Here we assume $max({\widehat{C}}^2,{\overline{C}}^2)<1/(1+{\sigma }^2)\le 1$. Then we have

$$\begin{array}{c}\hfill {\widehat{W}}_k\le {(1+{\sigma }^2)}^k{\left({\widehat{C}}^2\right)}^k{W}_0^{*},{\overline{W}}_k\le {(1+{\sigma }^2)}^k{\left({\overline{C}}^2\right)}^k{W}_0^{*}.\end{array}$$

Therefore, we have

$$\begin{array}{c}\hfill \underset{k\to +\infty }{lim}{\widehat{W}}_k=\underset{k\to +\infty }{lim}{\overline{W}}_k=0.\end{array}$$

(18)

We denote ${\widehat{\mathbf{G}}}_k:={\sum }_{i=0}^k{\widehat{W}}_i$ and ${\overline{\mathbf{G}}}_k:={\sum }_{i=0}^k{\overline{W}}_i$. Based on the above claims, we obtain that ${\widehat{\mathbf{G}}}_k$ and ${\overline{\mathbf{G}}}_k$ would converge when $k\to +\infty$. Thus we denote

$$\begin{array}{c}\hfill \widehat{\mathbf{G}}:=\underset{K\to +\infty }{lim}{\widehat{\mathbf{G}}}_K=\sum _{k=0}^{+\infty }{\widehat{W}}_k,\overline{\mathbf{G}}:=\underset{K\to +\infty }{lim}{\overline{\mathbf{G}}}_K=\sum _{k=0}^{+\infty }{\overline{W}}_k.\end{array}$$

Based on the definition of total rewards (3), we have

$$\begin{array}{c}\hfill \eta \left(x\right)=\widehat{\mathbf{G}}{x}^2{\mathbf{1}}_{x\ge 0}+\overline{\mathbf{G}}{x}^2{\mathbf{1}}_{x<0}.\end{array}$$

(19)

By (17) and (18), we have

$$\begin{array}{c}\hfill \underset{k\to +\infty }{lim}\left(P^{k}f\right)\left(x\right)=0.\end{array}$$

Then we have proved that the closed-loop system (11) is $L^2$-asymptotically stable, i.e., ${lim}_{l\to \infty }E[{\left(x_l\right)}^2]=0.$ Therefore, the total rewards $\eta \left(x\right)$ exists, that is, a piecewise quadratic function with positive semi-definite matrices $\widehat{\mathbf{G}}$ and $\overline{\mathbf{G}}$.

Now, we define the discrete version of generator, $\mathcal{A}$ for any function $h\left(x\right),x\in \mathbb{R}$, such that

$$\begin{array}{c}\hfill \mathcal{A}h\left(x\right):=\left(Ph\right)\left(x\right)-h\left(x\right).\end{array}$$

(20)

Taking $h\left(x\right)$ as $\eta \left(x\right)$, and by the definition of $\eta \left(x\right)$ in (3), we have the Poisson equation as follows,

$$\begin{array}{c}\hfill \mathcal{A}\eta \left(x\right)+f\left(x\right)=0.\end{array}$$

(21)

By (5) and (20), we obtain the discrete version of Dynkin’s formula as

$$\begin{array}{c}\hfill E\{\sum _{k=0}^{K-1}\left[\mathcal{A}h\left(x_k\right)\right]|x_0\}=E\{h\left(x_K\right)|x_0\}-h\left(x_0\right).\end{array}$$

(22)

and if the limit $K\to \infty$ exists, then

$$\begin{array}{c}\hfill E\{\sum _{k=0}^{\infty }\left[\mathcal{A}h\left(x_k\right)\right]|x_0\}=\underset{K\to \infty }{lim}E\left\{h\left(x_K\right)\right|x_0\}-h\left(x_0\right).\end{array}$$

Now, we consider two policies $\mathbf{u},{\mathbf{u}}^{\prime }\in {\mathcal{U}}_0$, resulting in two independent Markov chains $\mathbf{X}$ and ${\mathbf{X}}^{\prime }$ in the same state space $\mathbb{R}$, with $P,f,\eta ,\mathcal{A},E$, and $P^{\prime },f^{\prime },{\eta }^{\prime },{\mathcal{A}}^{\prime },E^{\prime }$, respectively. Let $x_0^{\prime }=x_0$. Applying the Dynkin’s Formula (22) on ${\mathbf{X}}^{\prime }$ with $h\left(x\right)=\eta \left(x\right)$ yields

$$\begin{array}{c}\hfill E^{\prime }\{\sum _{k=0}^{K-1}[{\mathcal{A}}^{\prime }\eta \left(x_k^{\prime }\right)|x_0\}=E^{\prime }\left\{\eta \left(x_K\right)\right]|x_0\}-\eta \left(x_0\right).\end{array}$$

(23)

Noting that ${\eta }^{\prime }\left(x_0\right)={lim}_{K\to \infty }{\sum }_{k=0}^{K-1}\left\{E^{\prime }\left[f^{\prime }\left(x_k\right)\right]\right|x_0\}$, and ${lim}_{K\to \infty }{E}^{\prime }\left\{\eta \left(x_K\right)\right|x_0\}=0$ due to asymptotical stability. Then by (23), we obtain the performance difference formula:

$$\begin{array}{cc}\hfill {\eta }^{\prime }\left(x_0\right)-\eta \left(x_0\right)=& \underset{K\to \infty }{lim}\sum _{k=0}^{K-1}{E}^{\prime }\left\{({\mathcal{A}}^{\prime }\eta +f^{\prime })\left(x_k^{\prime }\right)\right|x_0\}.\hfill \end{array}$$

(24)

3.2. Optimal Policy

Based on the performance difference Formula (24), we have the following optimality condition.

Theorem 1

(Optimality Condition). A policy ${\mathbf{u}}^{*}$ in $\mathcal{C}$ is optimal if, and only if,

$$\begin{array}{c}\hfill {\mathcal{A}}^{\mathbf{u}}{\eta }^{{\mathbf{u}}^{*}}+f^{\mathbf{u}}\ge 0={\mathcal{A}}^{{\mathbf{u}}^{*}}{\eta }^{{\mathbf{u}}^{*}}+f^{{\mathbf{u}}^{*}},\forall \mathbf{u}\in \mathcal{C}.\end{array}$$

(25)

From (25), the optimality equation is:

$$\begin{array}{c}\hfill \underset{\mathbf{u}\in \mathcal{C}}{min}\{{\mathcal{A}}^{\mathbf{u}}{\eta }^{{\mathbf{u}}^{*}}+f^{\mathbf{u}}\}=0.\end{array}$$

(26)

Proof. 

First, the “if” part follows from the performance difference Formula (24) and the Poisson Equation (21).

Next, we prove the “only if” part: Let ${\mathbf{u}}^{*}$ be an optimal policy. We need to prove that (25) holds. Suppose that this is not true. Then, there must exist one policy, denoted as ${\mathbf{u}}^{\prime }$, such that (25) does not hold. That is, there must be at least one state, denoted as y, such that

$$\begin{array}{c}\hfill P^{{\mathbf{u}}^{*}}{\eta }^{{\mathbf{u}}^{*}}\left(y\right)+f^{{\mathbf{u}}^{*}}\left(y\right)>P^{{\mathbf{u}}^{\prime }}{\eta }^{{\mathbf{u}}^{*}}\left(y\right)+f^{{\mathbf{u}}^{\prime }}\left(y\right).\end{array}$$

Then we can create a policy $\tilde{\mathbf{u}}$ by setting $\tilde{\mathbf{u}}={\mathbf{u}}^{\prime }$ when $x=y$, and $\tilde{\mathbf{u}}={\mathbf{u}}^{*}$ when $x\ne y$. We have ${\eta }^{{\mathbf{u}}^{*}}>{\eta }^{{\mathbf{u}}^{\prime }}$. This contradicts to the fact that ${\mathbf{u}}^{*}$ is an optimal policy.  □

Based on the optimality condition, the optimal control ${\mathbf{u}}^{*}$ can be obtained by developing policy iteration algorithms. Roughly speaking, we start with any policy ${\mathbf{u}}_0$. At the kth step, $k=0,1,\cdots$, given a piecewise linear policy ${\mathbf{u}}_k\left(x\right)=\widehat{\mathbf{K}}x{\mathbf{1}}_{x\ge 0}-\overline{\mathbf{K}}x{\mathbf{1}}_{x<0}$, where $\widehat{\mathbf{K}},\overline{\mathbf{K}}\in \mathcal{K}$, we want to find a better policy by (26). We consider any policy $\mathbf{u}\left(x\right)$. Setting $h\left(x\right)={\eta }^{{\mathbf{u}}_k}\left(x\right)=\widehat{\mathbf{G}}{x}^2{\mathbf{1}}_{x\ge 0}+\overline{\mathbf{G}}{x}^2{\mathbf{1}}_{x<0}$, by $\left(5\right)$, $\left(12\right)$, and $\left(14\right)$, we have

$$\begin{array}{cc}\hfill \left(P^{\mathbf{u}}{\eta }^{{\mathbf{u}}_k}\right)\left(x\right)=& (a_1\widehat{\mathbf{G}}+a_2\overline{\mathbf{G}}){(A+\mathbf{B}\widehat{\mathbf{K}})}^2{x}^2{\mathbf{1}}_{x\ge 0}\hfill \\ \hfill & +(a_1\widehat{\mathbf{G}}+a_2\overline{\mathbf{G}}){(A-\mathbf{B}\overline{\mathbf{K}})}^2{x}^2{\mathbf{1}}_{x<0}.\hfill \end{array}$$

(27)

where $a_1$ and $a_2$ satisfy Equations (15) and (16), respectively.

Then, from (4) and (27), we have

$$\begin{array}{cc}\hfill {\mathbf{u}}_{k+1}\left(x\right)& =arg\{\underset{\mathbf{u}\in \mathcal{C}}{min}[\left(P^{\mathbf{u}}{\eta }^{{\mathbf{u}}_k}\right)\left(x\right)+f^{\mathbf{u}}\left(x\right)]\}\hfill \\ \hfill & ={\widehat{\mathbf{K}}}_{k+1}x{\mathbf{1}}_{x\ge 0}-{\overline{\mathbf{K}}}_{k+1}x{\mathbf{1}}_{x<0},\hfill \end{array}$$

with

$$\begin{array}{c}\hfill {\widehat{\mathbf{K}}}_{k+1}=arg\underset{\mathbf{K}\in \mathcal{K}}{min}[a_1{\widehat{C}}^2\widehat{\mathbf{G}}+a_2{\widehat{C}}^2\overline{\mathbf{G}}+Q+{\mathbf{K}}^T\mathbf{R}\mathbf{K}],\\ \hfill {\overline{\mathbf{K}}}_{k+1}=arg\underset{\mathbf{K}\in \mathcal{K}}{min}[a_1{\overline{C}}^2\widehat{\mathbf{G}}+a_2{\overline{C}}^2\overline{\mathbf{G}}+Q+{\mathbf{K}}^T\mathbf{R}\mathbf{K}],\end{array}$$

where $\widehat{C}=A+\mathbf{B}\mathbf{K},$ and $\overline{C}=A-\mathbf{B}\mathbf{K}.$

It can be seen that if the policy ${\mathbf{u}}_k\left(x\right)$ is a piecewise linear control, then we can find an improved policy ${\mathbf{u}}_{k+1}\left(x\right)$, which is also piecewise linear. Moreover, if ${\widehat{\mathbf{K}}}_{k+1}=\widehat{\mathbf{K}}$ and ${\overline{\mathbf{K}}}_{k+1}=\overline{\mathbf{K}}$, i.e., ${\mathbf{u}}_{k+1}={\mathbf{u}}_k$, then the iteration stops. The policy ${\mathbf{u}}_k$ satisfies the optimal condition (26) in Theorem 1, and therefore is an optimal control.

Therefore, we can obtain the optimal policy as follows,

$$\begin{array}{c}\hfill {\mathbf{u}}^{*}\left(x\right)={\widehat{\mathbf{K}}}^{*}x{\mathbf{1}}_{x\ge 0}-{\overline{\mathbf{K}}}^{*}x{\mathbf{1}}_{x<0},\end{array}$$

(28)

where

$$\begin{array}{cc}\hfill {\widehat{\mathbf{K}}}^{*}& =arg\underset{\mathbf{K}\in \mathcal{K}}{min}[a_1{\widehat{C}}^2{\widehat{\mathbf{G}}}^{*}+a_2{\widehat{C}}^2{\overline{\mathbf{G}}}^{*}+Q+{\mathbf{K}}^T\mathbf{R}\mathbf{K}],\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill {\overline{\mathbf{K}}}^{*}& =arg\underset{\mathbf{K}\in \mathcal{K}}{min}[a_1{\overline{C}}^2{\widehat{\mathbf{G}}}^{*}+a_2{\overline{C}}^2{\overline{\mathbf{G}}}^{*}+Q+{\mathbf{K}}^T\mathbf{R}\mathbf{K}].\hfill \end{array}$$

(30)

Moreover,

$$\begin{array}{cc}\hfill {\widehat{\mathbf{G}}}^{*}& =\underset{\mathbf{K}\in \mathcal{K}}{min}\{a_1{\widehat{C}}^2{\widehat{\mathbf{G}}}^{*}+a_2{\widehat{C}}^2{\overline{\mathbf{G}}}^{*}+Q+{\mathbf{K}}^T\mathbf{R}\mathbf{K}\},\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\overline{\mathbf{G}}}^{*}& =\underset{\mathbf{K}\in \mathcal{K}}{min}\{a_1{\overline{C}}^2{\widehat{\mathbf{G}}}^{*}+a_2{\overline{C}}^2{\overline{\mathbf{G}}}^{*}+Q+{\mathbf{K}}^T\mathbf{R}\mathbf{K}\}.\hfill \end{array}$$

(32)

The original problem (3) is transformed to two auxiliary optimization problems (29) and (30). Under the optimal control ${\mathbf{u}}^{*}$ in (28), the closed-loop system (11) is $L^2$-asymptotically stable. From (19), with the initial condition $x_0=x$, we know the optimal total reward performance is

$$\begin{array}{c}\hfill {\eta }^{*}\left(x\right)={\widehat{\mathbf{G}}}^{*}{x}^2{\mathbf{1}}_{x\ge 0}+{\overline{\mathbf{G}}}^{*}{x}^2{\mathbf{1}}_{x<0},\end{array}$$

(33)

where ${\widehat{\mathbf{G}}}^{*}$ and ${\overline{\mathbf{G}}}^{*}$ satisfy (31) and (32), respectively.

Policy iteration can also be implemented on-line, the performance (potential) can be learned on a sample path without knowing all the transition probabilities. In on-line algorithms, the computation of policy evaluation is $O\left(n\right)$, where n is the length of a sample path. Additionally, Reference [14] also provides some algorithms for calculating the optimal policy.

4. Simulation Examples

In this section, we use two numerical examples to illustrate the optimal policy for the constrained LQ control problem (3).

Example 1.

We consider a stochastic LQ system with $x_0=10$, $m=3$, $A=0.8$, and $B={(-0.35,0.18,0.25)}^{\prime }$. The cost matrix is

$$\begin{array}{c}\hfill \mathbf{R}=\left(\begin{array}{ccc}1.2& 0.6& 0.4\\ 0.6& 1.8& 0.2\\ 0.4& 0.2& 2.4\end{array}\right),\mathrm{and}Q=1.2.\end{array}$$

For time $l=0,1,\cdots$, the variance of the 0-mean i.i.d. Gaussian noise ${\xi }_l$ is ${\sigma }^2=0.25$. We consider the conic constraint $\mathbf{u}\ge 0$. By applying Theorem 1, the stationary optimal control is ${\mathbf{u}}_l^{*}\left(x_l\right)={\widehat{\mathbf{K}}}^{*}{x}_l{\mathbf{1}}_{x_l\ge 0}-{\overline{\mathbf{K}}}^{*}{x}_l{\mathbf{1}}_{x_l<0}$, for $l=0,1,\cdots$, where ${\widehat{\mathbf{K}}}^{*}={(0.574,0,0)}^{\prime },{\overline{\mathbf{K}}}^{*}={(0,0.250,0.270)}^{\prime },{\widehat{\mathbf{G}}}^{*}=2.773$ and ${\overline{\mathbf{G}}}^{*}=3.473$. Furthermore, the optimal reward performance is ${\eta }^{*}\left(x_0\right)={\widehat{\mathbf{G}}}^{*}{x}_0^2{\mathbf{1}}_{x_0\ge 0}+{\overline{\mathbf{G}}}^{*}{x}_0^2{\mathbf{1}}_{x_0<0}=623.987$.

As shown in Figure 1a plots the outputs ${\overline{\mathbf{G}}}^{*}$ and ${\widehat{\mathbf{G}}}^{*}$ with respect to iteration time K; Figure 1b plots the state trajectories of 50 samples by setting $x_0=10$ and implementing the stationary optimal control ${\mathbf{u}}^{*}$. It can be observed that $x_l^{*}$ converges to 0 after time $l=20$ and this closed loop system is asymptotically stable.

Example 2.

In the second case, we assume $x_0=10,A$ and B, following the identical discrete distribution with five cases. We assume $A\in (-0.7,-0.6,0.9,1,1.1)$, and

$$\begin{array}{c}\hfill B\in \left\{\left(\begin{array}{c}0.18\\ -0.05\\ -0.140\end{array}\right),\left(\begin{array}{c}0.03\\ -0.12\\ -0.03\end{array}\right),\left(\begin{array}{c}-0.05\\ 0.05\\ 0.05\end{array}\right),\left(\begin{array}{c}-0.01\\ 0.05\\ 0.01\end{array}\right),\left(\begin{array}{c}-0.05\\ 0.01\\ 0.06\end{array}\right)\right\},\end{array}$$

each of which has the same probability 0.2. The cost matrix is

$$\begin{array}{c}\hfill \mathbf{R}=\left(\begin{array}{ccc}1.5& 0.6& 0.4\\ 0.6& 1.5& 0.2\\ 0.4& 0.2& 2.5\end{array}\right),\mathrm{and}Q=1.5.\end{array}$$

For time $l=0,1,\cdots$, the variance of the 0-mean i.i.d. Gaussian noise ${\xi }_l$ is ${\sigma }^2=0.25$. We consider the conic constraint $\mathbf{u}\ge 0$. By applying Theorem 1, the stationary optimal control is ${\mathbf{u}}_l^{*}\left(x_l\right)={\widehat{\mathbf{K}}}^{*}{x}_l{\mathbf{1}}_{x_l\ge 0}-{\overline{\mathbf{K}}}^{*}{x}_l{\mathbf{1}}_{x_l<0}$, for $l=0,1,\cdots$, where ${\widehat{\mathbf{K}}}^{*}$ and ${\overline{\mathbf{K}}}^{*}$ are identified as follows, ${\widehat{\mathbf{K}}}^{*}={(0.259,0.100,0.130)}^{\prime },{\overline{\mathbf{K}}}^{*}={(0.100,0.500,0.100)}^{\prime },{\widehat{\mathbf{G}}}^{*}=4.111$ and ${\overline{\mathbf{G}}}^{*}=3.859$. Furthermore, the optimal reward performance is ${\eta }^{*}\left(x_0\right)={\widehat{\mathbf{G}}}^{*}{x}_0^2{\mathbf{1}}_{x_0\ge 0}+{\overline{\mathbf{G}}}^{*}{x}_0^2{\mathbf{1}}_{x_0<0}=489.23$.

As shown in Figure 2a plots the outputs ${\overline{\mathbf{G}}}^{*}$ and ${\widehat{\mathbf{G}}}^{*}$ with respect to iteration times K; Figure 2b plots the state trajectories of 50 samples by setting $x_0=10$ and implementing the stationary optimal control ${\mathbf{u}}^{*}$. It can be observed that $x_l^{*}$ converges to 0 after time $l=35$ and this closed loop system is asymptotically stable.

5. Conclusions

In this paper, we apply the direct-comparison based optimization approach to study the rewards optimization of the discrete-time stochastic linear-quadratic control problem with conic constraints on an infinite horizon. We derive the performance difference formula by utilizing the state separation property of the system structure. Based on this, the optimality condition and the stationary optimal feedback control can be obtained. The direct-comparison based approach is applicable to both linear and nonlinear systems. By introducing the the LQ optimization problem, we establish a general framework for studying infinite horizon control problems with total rewards. We verify that the proposed optimal approach can solve the LQ problems. Then we illustrate our results by two simulation examples.

The results can easily be extended to the cases of non-Gaussian noises and average rewards. Most significantly, our methodology can deal with a very general class of linear constraints on state and control variables, which includes the cone constraints, positivity and negativity constraints, and the state-dependent upper and lower bound constraints as a special case. In addition to the problem with the infinite control horizon, our results still fit problems with a finite horizon. In addition, without identifying all the system structure parameters, this approach can also be implemented on-line, and learning based algorithms can be developed.

Finally, this work focuses on the discrete-time stochastic LQ control problem. Our next step is to investigate continuous cases. As the constrained LQ problem has a wide range of applications, we hope to apply our approach in more areas, such as dynamic portfolio management, security optimization of cyber-physical systems, and financial derivative pricing, in our future research.