Memory-Limited Partially Observable Stochastic Control and Its Mean-Field Control Approach

Abstract

Control problems with incomplete information and memory limitation appear in many practical situations. Although partially observable stochastic control (POSC) is a conventional theoretical framework that considers the optimal control problem with incomplete information, it cannot consider memory limitation. Furthermore, POSC cannot be solved in practice except in special cases. In order to address these issues, we propose an alternative theoretical framework, memory-limited POSC (ML-POSC). ML-POSC directly considers memory limitation as well as incomplete information, and it can be solved in practice by employing the technique of mean-field control theory. ML-POSC can generalize the linear-quadratic-Gaussian (LQG) problem to include memory limitation. Because estimation and control are not clearly separated in the LQG problem with memory limitation, the Riccati equation is modified to the partially observable Riccati equation, which improves estimation as well as control. Furthermore, we demonstrate the effectiveness of ML-POSC for a non-LQG problem by comparing it with the local LQG approximation.

1. Introduction

Control problems of systems with incomplete information and memory limitation appear in many practical situations. These constraints become especially predominant when designing the control of small devices [1,2], and are important for understanding the control mechanisms of biological systems [3,4,5,6,7,8] because their sensors are extremely noisy and their controllers can only have severely limited memories.

Partially observable stochastic control (POSC) is a conventional theoretical framework that considers the optimal control problem with one of these constraints, namely, the incomplete information of the system state (Figure 1b) [9]. Because the POSC controller cannot completely observe the state of the system, it determines the control based on the noisy observation history of the state. POSC can be solved in principle [10,11,12] by converting it to a completely observable stochastic control (COSC) of the posterior probability of the state, as the posterior probability represents the sufficient statistics of the observation history. The posterior probability and the optimal control are obtained by solving the Zakai equation and the Bellman equation, respectively.

Figure 1. Schematic diagram of (a) completely observable stochastic control (COSC), (b) partially observable stochastic control (POSC), and (c) memory-limited partially observable stochastic control (ML-POSC). The top and bottom figures represent the system and controller, respectively; $x_t\in {\mathbb{R}}^{d_x}$ is the state of the system; $y_t\in {\mathbb{R}}^{d_y}$, $z_t\in {\mathbb{R}}^{d_z}$, and $u_t\in {\mathbb{R}}^{d_u}$ are the observation, memory, and control of the controller, respectively. (a) In COSC, the controller can completely observe the state $x_t$, and determines the control $u_t$ based on the state $x_t$, i.e., $u_t=u(t,x_t)$. Only finite-dimensional memory is required to store the state $x_t$, and the optimal control $u_t^{*}$ is obtained by solving the Hamilton–Jacobi–Bellman (HJB) equation, which is a partial differential equation. (b) In POSC, the controller cannot completely observe the state $x_t$; instead, it obtains the noisy observation $y_t$ of the state $x_t$. The control $u_t$ is determined based on the observation history $y_{0:t}:=\left\{y_{\tau }\right|\tau \in [0,t]\}$, i.e., $u_t=u(t,y_{0:t})$. An infinite-dimensional memory is implicitly assumed to store the observation history $y_{0:t}$. Furthermore, to obtain the optimal control $u_t^{*}$, the Bellman equation (a functional differential equation) needs to be solved, which is generally intractable, even numerically. (c) In ML-POSC, the controller is only accessible to the noisy observation $y_t$ of the state $x_t$, as in POSC. In addition, it has only finite-dimensional memory $z_t$, which cannot completely memorize the the observation history $y_{0:t}$. The controller of ML-POSC compresses the observation history $y_{0:t}$ into the finite-dimensional memory $z_t$, then determines the control $u_t$ based on the memory $z_t$, i.e., $u_t=u(t,z_t)$. The optimal control $u_t^{*}$ is obtained by solving the HJB equation (a partial differential equation), as in COSC.

However, POSC has three practical problems with respect to the implementation of the controller which originate from the ignorance of the other constraint, namely, the memory limitation of the controller [1,2]. First, a controller designed by POSC should ideally have an infinite-dimensional memory to store and compute the posterior probability from the observation history. Second, the memory of the controller cannot have intrinsic stochasticity other than the observation noise to accurately compute the posterior probability via the Zakai equation. Third, POSC does not consider the cost originating from the memory update, which can be regarded as a cost of estimation. In light of the dualistic roles played by estimation and control, considering only control cost by ignoring estimation cost is asymmetric. As a result, POSC is not practical for control problems where the memory size, noise, and cost are non-negligible. Therefore, we need an alternative theoretical framework considering memory limitation to circumvent these three problems.

Furthermore, POSC has another crucial problem in obtaining the optimal state control by solving the Bellman equation [3,4]. Because the posterior probability of the state is infinite-dimensional, POSC corresponds to an infinite-dimensional COSC. In the infinite-dimensional COSC, the Bellman equation becomes a functional differential equation, which needs to be solved in order to obtain the optimal state control. However, solving a functional differential equation is generally intractable, even numerically.

In this work, we propose an alternative theoretical framework to the conventional POSC which can address the above-mentioned two issues. We call it memory-limited POSC (ML-POSC), in which memory limitation as well as incomplete information are directly accounted (Figure 1c). The conventional POSC derives the Zakai equation without considering memory limitations. Then, the optimal state control is supposed to be derived by solving the Bellman equation, even though we do not have any practical way to do this. In contrast, ML-POSC first postulates the finite-dimensional and stochastic memory dynamics explicitly by taking the memory limitation into account and then jointly optimizes the memory dynamics and state control by considering the memory and control costs. As a result, unlike the conventional POSC, ML-POSC finds both the optimal state control and the optimal memory dynamics with given memory limitations. Furthermore, we show that the Bellman equation of ML-POSC can be reduced to the Hamilton–Jacobi–Bellman (HJB) equation by employing a trick from the mean-field control theory [13,14,15]. While the Bellman equation is a functional differential equation, the HJB equation is a partial differential equation. As a result, ML-POSC can be solved, at least numerically.

The idea behind ML-POSC is closely related to that of the finite-state controller [16,17,18,19,20,21,22]. Finite-state controllers have been studied using the partially observable Markov decision process (POMDP), that is, the discrete time and state POSC. The finite-dimensional memory of ML-POSC can be regarded as an extension of the finite-state controller of POMDP to the continuous time and state setting. Nonetheless, the algorithms of the finite-state controller cannot be directly extended to the continuous setting, as they strongly depend on the discreteness. Although Fox and Tishby extended the finite-state controller to the continuous setting, their algorithm is restricted to the special case [1,2]. ML-POSC resolves this problem by employing the technique of the mean-field control theory.

In the linear-quadratic-Gaussian (LQG) problem of the conventional POSC, the Zakai equation and the Bellman equation are reduced to the Kalman filter and the Riccati equation, respectively [9,23]. Because the infinite-dimensional Zakai equation is reduced to the finite-dimensional Kalman filter, the LQG problem of the conventional POSC can be discussed in terms of ML-POSC. We show that the Kalman filter corresponds to the optimal memory dynamics of ML-POSC. Moreover, ML-POSC can generalize the LQG problem to include memory limitations such as the memory noise and cost. Because estimation and control are not clearly separated in the LQG problem with memory limitation, the Riccati equation for control is modified to include estimation, which in this paper is called the partially observable Riccati equation. We demonstrate that the partially observable Riccati equation is superior to the conventional Riccati equation as concerns the LQG problem with memory limitation.

Then, we investigate the potential effectiveness of ML-POSC for a non-LQG problem by comparing it with the local LQG approximation of the conventional POSC [3,4]. In the local LQG approximation, the Zakai equation and the Bellman equation are locally approximated by the Kalman filter and the Riccati equation, respectively. Because the Bellman equation (a functional differential equation) is reduced to the Riccati equation (an ordinary differential equation), the local LQG approximation can be solved numerically. However, the performance of the local LQG approximation may be poor in a highly non-LQG problem, as the local LQG approximation ignores non-LQG information. In contrast, ML-POSC reduces the Bellman equation to the HJB equation while maintaining non-LQG information. We demonstrate that ML-POSC can provide a better result than the local LQG approximation in a non-LQG problem.

This paper is organized as follows: In Section 2, we briefly review the conventional POSC. In Section 3, we formulate ML-POSC. In Section 4, we propose the mean-field control approach to ML-POSC. In Section 5, we investigate the LQG problem of the conventional POSC based on ML-POSC. In Section 6, we generalize the LQG problem to include memory limitation. In Section 7, we show numerical experiments involving a LQG problem with memory limitation and a non-LQG problem. Finally, in Section 8, we discuss our work.

2. Review of Partially Observable Stochastic Control

In this section, we briefly review the conventional POSC [11,15].

2.1. Problem Formulation

In this subsection, we formulate the conventional POSC [11,15]. The state $x_t\in {\mathbb{R}}^{d_x}$ and the observation $y_t\in {\mathbb{R}}^{d_y}$ at time $t\in [0,T]$ evolve by the following stochastic differential equations (SDEs):

$$\begin{array}{cc}\hfill d{x}_t& =b(t,x_t,u_t)dt+\sigma (t,x_t,u_t)d{\omega }_t,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill d{y}_t& =h(t,x_t)dt+\gamma \left(t\right)d{\nu }_t,\hfill \end{array}$$

(2)

where $x_0$ and $y_0$ obey $p_0\left(x_0\right)$ and $p_0\left(y_0\right)$, respectively, ${\omega }_t\in {\mathbb{R}}^{d_{\omega }}$ and ${\nu }_t\in {\mathbb{R}}^{d_{\nu }}$ are independent standard Wiener processes, and $u_t\in {\mathbb{R}}^{d_u}$ is the control. Here, $\gamma \left(t\right){\gamma }^{\top }\left(t\right)$ is assumed to be invertible. In POSC, because the controller cannot completely observe the state $x_t$, the control $u_t$ is determined based on the observation history $y_{0:t}:=\left\{y_{\tau }\right|\tau \in [0,t]\}$, as follows:

$$\begin{array}{c}\hfill u_t=u(t,y_{0:t}).\end{array}$$

(3)

The objective function of POSC is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J\left[u\right]:={\mathbb{E}}_{p(x_{0:T},y_{0:T};u)}\left[{\int }_0^{T}f(t,x_t,u_t)dt+g\left(x_T\right)\right],\end{array}$$

(4)

where f is the cost function, g is the terminal cost function, $p(x_{0:T},y_{0:T};u)$ is the probability of $x_{0:T}$ and $y_{0:T}$ given u as a parameter, and ${\mathbb{E}}_p\left[·\right]$ is the expectation with respect to probability p. Throughout this paper, the time horizon T is assumed to be finite.

POSC is the problem of finding the optimal control function $u^{*}$ that minimizes the objective function $J\left[u\right]$ as follows:

$$\begin{array}{c}\hfill u^{*}:=\underset{u}{\mathrm{argmin}}J\left[u\right].\end{array}$$

(5)

2.2. Derivation of Optimal Control Function

In this subsection, we briefly review the derivation of the optimal control function of the conventional POSC [11,15]. We first define the unnormalized posterior probability density function $q_t\left(x\right):=p(x_t=x,y_{0:t})$. We omit $y_{0:t}$ for notational simplicity. Here, $q_t\left(x\right)$ obeys the following Zakai equation:

$$\begin{array}{c}\hfill d{q}_t\left(x\right)={\mathcal{L}}^{†}{q}_t\left(x\right)dt+q_t\left(x\right)h^{\top }(t,x){\left(\gamma \left(t\right){\gamma }^{\top }\left(t\right)\right)}^{-1}d{y}_t,\end{array}$$

(6)

where $q_0\left(x\right)=p_0\left(x\right)p_0\left(y\right)$ and ${\mathcal{L}}^{†}$ is the forward diffusion operator, which is defined by

$$\begin{array}{cc}\hfill {\mathcal{L}}^{†}q\left(x\right)& :=-\sum _{i=1}^{d_x}\frac{\partial \left(b_i(t,x,u)q\left(x\right)\right)}{\partial x_i}+\frac{1}{2}\sum _{i,j=1}^{d_x}\frac{{\partial }^2\left(D_{ij}(t,x,u)q\left(x\right)\right)}{\partial x_i\partial x_j},\hfill \end{array}$$

(7)

where $D(t,x,u):=\sigma (t,x,u){\sigma }^{\top }(t,x,u)$. Then, the objective function (4) can be calculated as follows:

$$\begin{array}{c}\hfill J\left[u\right]={\mathbb{E}}_{p(q_{0:T};u)}\left[{\int }_0^T\overline{f}(t,q_t,u_t)dt+\overline{g}\left(q_T\right)\right],\end{array}$$

(8)

where $\overline{f}(t,q,u):={\mathbb{E}}_{q\left(x\right)}\left[f(t,x,u)\right]$ and $\overline{g}\left(q\right):={\mathbb{E}}_{q\left(x\right)}\left[g\left(x\right)\right]$. From (6) and (8), POSC is converted into a COSC of $q_t$. As a result, POSC can be approached in the similar way as COSC, and the optimal control function is provided by the following proposition.

Proposition 1 

([11,15]). The optimal control function of POSC is provided by

$$\begin{array}{c}\hfill u^{*}(t,q)=\underset{u}{\mathrm{argmin}}{\mathbb{E}}_{q\left(x\right)}\left[\mathcal{H}\left(t,x,u,\frac{\delta V(t,q)}{\delta q}\left(x\right)\right)\right],\end{array}$$

(9)

where $\mathcal{H}$ is the Hamiltonian, which is defined by

$$\begin{array}{c}\hfill \mathcal{H}\left(t,x,u,\frac{\delta V(t,q)}{\delta q}\left(x\right)\right):=f(t,x,u)+\mathcal{L}\frac{\delta V(t,q)}{\delta q}\left(x\right).\end{array}$$

(10)

$\mathcal{L}$ is the backward diffusion operator, which is defined by

$$\begin{array}{cc}\hfill \mathcal{L}q\left(x\right)& :=\sum _{i=1}^{d_x}{b}_i(t,x,u)\frac{\partial q\left(x\right)}{\partial x_i}+\frac{1}{2}\sum _{i,j=1}^{d_x}{D}_{ij}(t,x,u)\frac{{\partial }^{2}q\left(x\right)}{\partial x_i\partial x_j}.\hfill \end{array}$$

(11)

We note that $\mathcal{L}$ is the conjugate of ${\mathcal{L}}^{†}$; furthermore, $V(t,q)$ is the value function, which is the solution of the following Bellman equation:

$$\begin{array}{cc}\hfill & -\frac{\partial V(t,q)}{\partial t}={\mathbb{E}}_{q\left(x\right)}\left[\mathcal{H}\left(t,x,u^{*},\frac{\delta V(t,q)}{\delta q}\left(x\right)\right)\right]\hfill \\ \hfill & +\frac{1}{2}{\mathbb{E}}_{q\left(x\right)q\left(x^{\prime }\right)}\left[\frac{\delta }{\delta q}\frac{\delta V(t,q)}{\delta q}(x,x^{\prime })h^{\top }(t,x){\left(\gamma \left(t\right){\gamma }^{\top }\left(t\right)\right)}^{-1}h(t,x^{\prime })\right],\hfill \end{array}$$

(12)

where $V(T,q)={\mathbb{E}}_{q\left(x\right)}\left[g\left(x\right)\right]$.

Proof. 

The proof is shown in [11,15]. □

The optimal control function $u^{*}(t,q)$ is obtained by solving the Bellman Equation (12). The controller determines the optimal control $u_t^{*}=u^{*}(t,q_t)$ based on the posterior probability $q_t$. The posterior probability $q_t$ is obtained by solving the Zakai Equation (6). As a result, POSC can be solved in principle.

However, POSC has three practical problems with respect to the memory of the controller. First, the controller should have an infinite-dimensional memory to store and compute the posterior probability $q_t$ from the observation history $y_{0:t}$. Second, the memory of the controller cannot have intrinsic stochasticity other than the observation $d{y}_t$ to accurately compute the posterior probability $q_t$ via the Zakai Equation (6). Third, POSC does not consider the cost originating from the memory update, which can be regarded as a cost of estimation. In light of the dualistic roles played by estimation and control, considering only control cost by ignoring estimation cost is asymmetric. As a result, POCS is not practical for control problems where the memory size, noise, and cost are non-negligible.

Furthermore, POSC has another crucial problem in obtaining the optimal control function $u^{*}(t,q)$ by solving the Bellman Equation (12). Because the posterior probability q is infinite-dimensional, the associated Bellman Equation (12) becomes a functional differential equation. However, solving a functional differential equation is generally intractable even numerically. As a result, POCS cannot be solved in practice.

3. Memory-Limited Partially Observable Stochastic Control

In order to address the above-mentioned problems, we propose an alternative theoretical framework to the conventional POSC called ML-POSC. In this section, we formulate ML-POSC.

3.1. Problem Formulation

In this subsection, we formulate ML-POSC. ML-POSC determines the control $u_t$ based on the finite-dimensional memory $z_t\in {\mathbb{R}}^{d_z}$ as follows:

$$\begin{array}{c}\hfill u_t=u(t,z_t).\end{array}$$

(13)

The memory dimension $d_z$ is determined not by the optimization but by the prescribed memory limitation of the controller to be used. Comparing (3) and (13), the memory $z_t$ can be interpreted as the compression of the observation history $y_{0:t}$. While the conventional POSC compresses the observation history $y_{0:t}$ into the infinite-dimensional posterior probability $q_t$, ML-POSC compresses it into the finite-dimensional memory $z_t$.

ML-POSC formulates the memory dynamics with the following SDE:

$$\begin{array}{c}\hfill d{z}_t=c(t,z_t,v_t)dt+\kappa (t,z_t,v_t)d{y}_t+\eta (t,z_t,v_t)d{\xi }_t,\end{array}$$

(14)

where $z_0$ obeys $p_0\left(z_0\right)$, ${\xi }_t\in {\mathbb{R}}^{d_{\xi }}$ is the standard Wiener process, and $v_t=v(t,z_t)\in {\mathbb{R}}^{d_v}$ is the control for the memory dynamics. This memory dynamics has three important properties: (i) because it depends on the observation $d{y}_t$, the memory $z_t$ can be interpreted as the compression of the observation history $y_{0:t}$; (ii) because it depends on the standard Wiener process $d{\xi }_t$, ML-POSC can consider the memory noise explicitly; (iii) because it depends on the control $v_t$, it can be optimized through the control $v_t$.

The objective function of ML-POSC is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{p(x_{0:T},y_{0:T},z_{0:T};u,v)}\left[{\int }_0^{T}f(t,x_t,u_t,v_t)dt+g\left(x_T\right)\right].\end{array}$$

(15)

Because the cost function f depends on the memory control $v_t$ as well as the state control $u_t$, ML-POSC can consider the memory control cost (state estimation cost) as well as the state control cost explicitly.

ML-POSC optimizes the state control function u and the memory control function v based on the objective function $J[u,v]$, as follows:

$$\begin{array}{c}\hfill u^{*},v^{*}:=\underset{u,v}{\mathrm{argmin}}J[u,v].\end{array}$$

(16)

ML-POSC first postulates the finite-dimensional and stochastic memory dynamics explicitly, then jointly optimizes the state and memory control function by considering the state and memory control cost. As a result, unlike the conventional POSC, ML-POSC can consider memory limitation as well as incomplete information.

3.2. Problem Reformulation

Although the formulation of ML-POSC in the previous subsection clarifies its relationship with that of the conventional POSC, it is inconvenient for further mathematical investigations. In order to resolve this problem, we reformulate ML-POSC in this subsection. The formulation in this subsection is simpler and more general than that in the previous subsection.

We first define the extended state $s_t$ as follows:

$$\begin{array}{c}\hfill s_t:=\left(\begin{array}{c}{x}_t\\ z_t\end{array}\right)\in {\mathbb{R}}^{d_s},\end{array}$$

(17)

where $d_s=d_x+d_z$. The extended state $s_t$ evolves by the following SDE:

$$\begin{array}{c}\hfill d{s}_t=\tilde{b}(t,s_t,{\tilde{u}}_t)dt+\tilde{\sigma }(t,s_t,{\tilde{u}}_t)d{\tilde{\omega }}_t,\end{array}$$

(18)

where $s_0$ obeys $p_0\left(s_0\right)$, ${\tilde{\omega }}_t\in {\mathbb{R}}^{d_{\tilde{\omega }}}$ is the standard Wiener process, and ${\tilde{u}}_t\in {\mathbb{R}}^{d_{\tilde{u}}}$ is the control. ML-POSC determines the control ${\tilde{u}}_t\in {\mathbb{R}}^{d_{\tilde{u}}}$ based solely on the memory $z_t$, as follows:

$$\begin{array}{c}\hfill {\tilde{u}}_t=\tilde{u}(t,z_t).\end{array}$$

(19)

The extended state SDE (18) includes the previous state, observation, and memory SDEs (1), (2) and (14) as a special case; they can be represented as follows:

$$\begin{gathered} d{s}_t=\left(\begin{array}{c}b(t,x_t,u_t)\\ c(t,z_t,v_t)+\kappa (t,z_t,v_t)h(t,x_t)\end{array}\right)dt \\ +\left(\begin{array}{ccc}\sigma (t,x_t,u_t)& O& O\\ O& \kappa (t,z_t,v_t)\gamma \left(t\right)& \eta (t,z_t,v_t)\end{array}\right)\left(\begin{array}{c}d{\omega }_t\\ d{\nu }_t\\ d{\xi }_t\end{array}\right), \end{gathered}$$

(20)

where $p_0\left(s_0\right)=p_0\left(x_0\right)p_0\left(z_0\right)$.

The objective function of ML-POSC is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^T\tilde{f}(t,s_t,{\tilde{u}}_t)dt+\tilde{g}\left(s_T\right)\right],\end{array}$$

(21)

where $\tilde{f}$ is the cost function and $\tilde{g}$ is the terminal cost function. It is obvious that this objective function (21) is more general than the previous one (15).

ML-POSC is the problem of finding the optimal control function ${\tilde{u}}^{*}$ that minimizes the objective function $J\left[\tilde{u}\right]$ as follows:

$$\begin{array}{c}\hfill {\tilde{u}}^{*}:=\underset{\tilde{u}}{\mathrm{argmin}}J\left[\tilde{u}\right].\end{array}$$

(22)

In the following section, we mainly consider the formulation in this subsection rather than that of the previous subsection, as it is simpler and more general. Moreover, we omit $\widetilde{·}$ for the notational simplicity.

4. Mean-Field Control Approach

If the control $u_t$ is determined based on the extended state $s_t$, i.e., $u_t=u(t,s_t)$, ML-POSC is the same as COSC of the extended state $s_t$, and can be solved by the conventional COSC approach [10]. However, because ML-POSC determines the control $u_t$ based solely on the memory $z_t$, i.e., $u_t=u(t,z_t)$, ML-POSC cannot be solved in a similar way as COSC. In order to solve ML-POSC, we propose the mean-field control approach in this section. Because the mean-field control approach is more general than the COSC approach, it can solve COSC and ML-POSC in a unified way.

4.1. Derivation of Optimal Control Function

In this subsection, we propose the mean-field control approach to ML-POSC. We first show that ML-POSC can be converted into a deterministic control of the probability density function, which is similar to the conventional POSC [11,15]. This approach is used in the mean-field control as well [13,14,24,25]. The extended state SDE (18) can be converted into the following Fokker–Planck (FP) equation:

$$\begin{array}{c}\hfill \frac{\partial p_t\left(s\right)}{\partial t}={\mathcal{L}}^{†}{p}_t\left(s\right),\end{array}$$

(23)

where the initial condition is provided by $p_0\left(s\right)$ and the forward diffusion operator ${\mathcal{L}}^{†}$ is defined by (7). The objective function of ML-POSC (21) can be calculated as follows:

$$\begin{array}{c}\hfill J\left[u\right]={\int }_0^T\overline{f}(t,p_t,u_t)dt+\overline{g}\left(p_T\right),\end{array}$$

(24)

where $\overline{f}(t,p,u):={\mathbb{E}}_{p\left(s\right)}\left[f(t,s,u)\right]$ and $\overline{g}\left(p\right):={\mathbb{E}}_{p\left(s\right)}\left[g\left(s\right)\right]$. From (23) and (24), ML-POSC is converted into a deterministic control of $p_t$. As a result, ML-POSC can be approached in a similar way as the deterministic control, and the optimal control function is provided by the following lemma.

Lemma 1. 

The optimal control function of ML-POSC is provided by

$$\begin{array}{c}\hfill u^{*}(t,z)=\underset{u}{\mathrm{argmin}}{\mathbb{E}}_{p_t\left(x\right|z)}\left[\mathcal{H}\left(t,s,u,\frac{\delta V(t,p_t)}{\delta p}\left(s\right)\right)\right],\end{array}$$

(25)

where $\mathcal{H}$ is the Hamiltonian (10), $p_t\left(x\right|z)=p_t\left(s\right)/\int p_t\left(s\right)dx$ is the conditional probability density function of a state x given memory z, $p_t\left(s\right)$ is the solution of the FP Equation (23), and $V(t,p)$ is the solution of the following Bellman equation:

$$\begin{array}{c}\hfill -\frac{\partial V(t,p)}{\partial t}={\mathbb{E}}_{p\left(s\right)}\left[\mathcal{H}\left(t,s,u^{*},\frac{\delta V(t,p)}{\delta p}\left(s\right)\right)\right],\end{array}$$

(26)

where $V(T,p)={\mathbb{E}}_{p\left(s\right)}\left[g\left(s\right)\right]$.

Proof. 

The proof is shown in Appendix A. □

The controller of ML-POSC determines the optimal control $u_t^{*}=u^{*}(t,z_t)$ based on the memory $z_t$, not the posterior probability $q_t$. Therefore, ML-POSC can consider memory limitation as well as incomplete information.

However, because the Bellman Equation (26) is a functional differential equation, it cannot be solved, even numerically, which is the same problem as the conventional POSC. We resolve this problem by employing the technique of the mean-field control theory [13,14] as follows.

Theorem 1. 

The optimal control function of ML-POSC is provided by

$$\begin{array}{c}\hfill u^{*}(t,z)=\underset{u}{\mathrm{argmin}}{\mathbb{E}}_{p_t\left(x\right|z)}\left[\mathcal{H}\left(t,s,u,w(t,s)\right)\right],\end{array}$$

(27)

where $\mathcal{H}$ is the Hamiltonian (10), $p_t\left(x\right|z)=p_t\left(s\right)/\int p_t\left(s\right)dx$ is the conditional probability density function of a state x given memory z, $p_t\left(s\right)$ is the solution of the FP Equation (23), and $w(t,s)$ is the solution of the following Hamilton–Jacobi–Bellman (HJB) equation:

$$\begin{array}{c}\hfill -\frac{\partial w(t,s)}{\partial t}=\mathcal{H}\left(t,s,u^{*},w(t,s)\right),\end{array}$$

(28)

where $w(T,s)=g\left(s\right)$.

Proof. 

The proof is shown in Appendix B. □

While the Bellman Equation (26) is a functional differential equation, the HJB Equation (28) is a partial differential equation. As a result, unlike the conventional POSC, ML-POSC can be solved in practice.

We note that the mean-field control technique is applicable to the conventional POSC as well, and we obtain the HJB equation of the conventional POSC [15]. However, the HJB equation of the conventional POSC is not closed by a partial differential equation due to the last term of the Bellman Equation (12). As a result, the mean-field control technique is not effective with the conventional POSC except in a special case [15].

In the conventional POSC, the state estimation (memory control) and the state control are clearly separated. As a result, the state estimation and the state control are optimized by the Zakai Equation (6) and the Bellman Equation (12), respectively. In contrast, because ML-POSC considers memory limitation as well as incomplete information, the state estimation and the state control are not clearly separated. As a result, ML-POSC jointly optimizes the state estimation and the state control based on the FP Equation (23) and the HJB Equation (28).

4.2. Comparison with Completely Observable Stochastic Control

In this subsection, we show the similarities and differences between ML-POSC and COSC of the extended state. While ML-POSC determines the control $u_t$ based solely on the memory $z_t$, i.e., $u_t=u(t,z_t)$, COSC of the extended state determines the control $u_t$ based on the extended state $s_t$, i.e., $u_t=u(t,s_t)$. The optimal control function of COSC of the extended state is provided by the following proposition.

Proposition 2 

([10]). The optimal control function of COSC of the extended state is provided by

$$\begin{array}{c}\hfill u^{*}(t,s)=\underset{u}{\mathrm{argmin}}\mathcal{H}\left(t,s,u,w(t,s)\right),\end{array}$$

(29)

where $\mathcal{H}$ is the Hamiltonian (10) and $w(t,s)$ is the solution of the HJB Equation (28).

Proof. 

The conventional proof is shown in [10]. We note that it can be proven in a similar way as ML-POSC, which is shown in Appendix C. □

Although the HJB Equation (28) is the same between ML-POSC and COSC, the optimal control function is different. While the optimal control function of COSC is provided by the minimization of the Hamiltonian (29), that of ML-POSC is provided by the minimization of the conditional expectation of the Hamiltonian (27). This is reasonable, as the controller of ML-POSC needs to estimate the state from the memory.

4.3. Numerical Algorithm

In this subsection, we briefly explain a numerical algorithm to obtain the optimal control function of ML-POSC (27). Because the optimal control function of COSC (29) depends only on the backward HJB Equation (28), it can be obtained by solving the HJB equation backwards from the terminal condition [10,26,27]. In contrast, because the optimal control function of ML-POSC (27) depends on the forward FP Equation (23) as well as the backward HJB Equation (28), it cannot be obtained in a similar way as COSC. Because the backward HJB equation depends on the forward FP equation through the optimal control function of ML-POSC, the HJB equation cannot be solved backwards from the terminal condition. As a result, ML-POSC needs to solve the system of HJB-FP equations.

The system of HJB-FP equations appears in the mean-field game and control [28,29,30], and many numerical algorithms have been developed [31,32,33]. Therefore, unlike the conventional POSC, ML-POSC can be solved in practice using these algorithms. Furthermore, unlike the mean-field game and control, the coupling of HJB-FP equations is limited to the optimal control function in ML-POSC. By exploiting this property, more efficient algorithms may be proposed for ML-POSC [34].

In this paper, we use the forward–backward sweep method (the fixed-point iteration method) to obtain the optimal control function of ML-POSC [33,34,35,36,37], which is one of the most basic algorithms for the system of HJB-FP equations. The forward–backward sweep method computes the forward FP Equation (23) and the backward HJB Equation (28) alternately. In the mean-field game and control, the convergence of the forward–backward sweep method is not guaranteed. In contrast, it is guaranteed in ML-POSC because the coupling of HJB-FP equations is limited to the optimal control function [34].

5. Linear-Quadratic-Gaussian Problem without Memory Limitation

In the LQG problem of the conventional POSC, the Zakai Equation (6) and the Bellman Equation (12) are reduced to the Kalman filter and the Riccati equation, respectively [9,23]. Because the infinite-dimensional Zakai equation is reduced to the finite-dimensional Kalman filter, the LQG problem of the conventional POSC can be discussed in terms of ML-POSC. In this section, we briefly review the LQG problem of the conventional POSC, then reproduce the Kalman filter and the Riccati equation from the viewpoint of ML-POSC. The LQG problem of the conventional POSC corresponds to the LQG problem without memory limitation, as it does not consider the memory noise and cost.

5.1. Review of Partially Observable Stochastic Control

In this subsection, we briefly review the LQG problem of the conventional POSC [9,23]. The state $x_t\in {\mathbb{R}}^{d_x}$ and the observation $y_t\in {\mathbb{R}}^{d_y}$ at time $t\in [0,T]$ evolve by the following SDEs:

$$\begin{array}{cc}\hfill d{x}_t& =\left(A\left(t\right)x_t+B\left(t\right)u_t\right)dt+\sigma \left(t\right)d{\omega }_t,\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill d{y}_t& =H\left(t\right)x_{t}dt+\gamma \left(t\right)d{\nu }_t,\hfill \end{array}$$

(31)

where $x_0$ obeys the Gaussian distribution $p_0\left(x_0\right)=\mathcal{N}\left(x_0\left|{\mu }_{x,0},{\Sigma }_{xx,0}\right.\right)$, $y_0$ is an arbitrary real vector, ${\omega }_t\in {\mathbb{R}}^{d_{\omega }}$ and ${\nu }_t\in {\mathbb{R}}^{d_{\nu }}$ are independent standard Wiener processes, and $u_t=u(t,y_{0:t})\in {\mathbb{R}}^{d_u}$ is the control. Here, $\gamma \left(t\right){\gamma }^{\top }\left(t\right)$ is assumed to be invertible. The objective function is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J\left[u\right]:={\mathbb{E}}_{p(x_{0:T},y_{0:T};u)}\left[{\int }_0^T\left(x_t^{\top }Q\left(t\right)x_t+u_t^{\top }R\left(t\right)u_t\right)dt+x_T^{\top }P{x}_T\right],\end{array}$$

(32)

where $Q\left(t\right)⪰O$, $R\left(t\right)\succ O$, and $P⪰O$. The LQG problem of the conventional POSC is to find the optimal control function $u^{*}$ that minimizes the objective function $J\left[u\right]$, as follows:

$$\begin{array}{c}\hfill u^{*}:=\underset{u}{\mathrm{argmin}}J\left[u\right].\end{array}$$

(33)

In the LQG problem of the conventional POSC, the posterior probability is provided by the Gaussian distribution $p\left(x_t\right|y_{0:t})=\mathcal{N}\left(x_t\right|\check{\mu }\left(t\right),\check{\Sigma }\left(t\right))$, and $u_t=u(t,y_{0:t})$ is reduced to $u_t=u(t,{\check{\mu }}_t)$ without loss of performance.

Proposition 3 

([9,23]). In the LQG problem without memory limitation, the optimal control function of POSC (33) is provided by

$$\begin{array}{c}\hfill u^{*}(t,\check{\mu })=-R^{-1}{B}^{\top }\Psi \check{\mu },\end{array}$$

(34)

where $\check{\mu }\left(t\right)$ and $\check{\Sigma }\left(t\right)$ are the solutions of the following Kalman filter:

$$\begin{array}{cc}\hfill & d\check{\mu }=\left(A-B{R}^{-1}{B}^{\top }\Psi \right)\check{\mu }dt+\check{\Sigma }{H}^{\top }{\left(\gamma {\gamma }^{\top }\right)}^{-1}\left(d{y}_t-H\check{\mu }dt\right),\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & \frac{d\check{\Sigma }}{dt}=\sigma {\sigma }^{\top }+A\check{\Sigma }+\check{\Sigma }{A}^{\top }-\check{\Sigma }{H}^{\top }{\left(\gamma {\gamma }^{\top }\right)}^{-1}H\check{\Sigma },\hfill \end{array}$$

(36)

and where $\check{\mu }\left(0\right)={\mu }_{x,0}$ and $\check{\Sigma }\left(0\right)={\Sigma }_{xx,0}$. $\Psi \left(t\right)$ is the solution of the following Riccati equation:

$$\begin{array}{cc}\hfill -\frac{d\Psi }{dt}& =Q+A^{\top }\Psi +\Psi A-\Psi B{R}^{-1}{B}^{\top }\Psi ,\hfill \end{array}$$

(37)

where $\Psi \left(T\right)=P$.

Proof. 

The proof is shown in [9,23]. □

In the LQG problem of the conventional POSC, the Zakai Equation (6) and the Bellman Equation (12) are reduced to the Kalman filter (35) and (36) and the Riccati Equation (37), respectively.

5.2. Memory-Limited Partially Observable Stochastic Control

Because the infinite-dimensional Zakai Equation (6) is reduced to the finite-dimensional Kalman filter (35) and (36), the LQG problem of the conventional POSC can be discussed in terms of ML-POSC. In this subsection, we reproduce the Kalman filter (35) and (36) and the Riccati Equation (37) from the viewpoint of ML-POSC.

ML-POSC defines the finite-dimensional memory $z_t\in {\mathbb{R}}^{d_z}$. In the LQG problem of the conventional POSC, the memory dimension $d_z$ is the same as the state dimension $d_x$. The controller of ML-POSC determines the control $u_t$ based on the memory $z_t$, i.e., $u_t=u(t,z_t)$. The memory $z_t$ is assumed to evolve by the following SDE:

$$\begin{array}{c}\hfill d{z}_t=v_{t}dt+{\kappa }_{t}d{y}_t,\end{array}$$

(38)

where $z_0={\mu }_{0,xx}$, while $v_t=v(t,z_t)\in {\mathbb{R}}^{d_z}$ and ${\kappa }_t=\kappa (t,z_t)\in {\mathbb{R}}^{d_z\times d_y}$ are the memory controls. We note that the LQG problem of the conventional POSC does not consider the memory noise. The objective function of ML-POSC is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J[u,v,\kappa ]:={\mathbb{E}}_{p(x_{0:T},y_{0:T},z_{0:T};u,v,\kappa )}\left[{\int }_0^T\left(x_t^{\top }Q\left(t\right)x_t+u_t^{\top }R\left(t\right)u_t\right)dt+x_T^{\top }P{x}_T\right].\end{array}$$

(39)

We note that the LQG problem of the conventional POSC does not consider the memory control cost. ML-POSC optimizes u, v, and $\kappa$ based on $J[u,v,\kappa ]$, as follows:

$$\begin{array}{c}\hfill u^{*},v^{*},{\kappa }^{*}:=\underset{u,v,\kappa }{\mathrm{argmin}}J[u,v,\kappa ].\end{array}$$

(40)

In the LQG problem of the conventional POSC, the probability of the extended state $s_t$ (17) is provided by the Gaussian distribution $p_t\left(s_t\right)=\mathcal{N}\left(s_t\right|\mu \left(t\right),\Sigma \left(t\right))$. The posterior probability of the state $x_t$ given the memory $z_t$ is provided by the Gaussian distribution $p_t\left(x_t\right|z_t)=\mathcal{N}\left(x_t\right|{\mu }_{x|z}(t,z_t),{\Sigma }_{x|z}\left(t\right))$, where ${\mu }_{x|z}(t,z_t)$ and ${\Sigma }_{x|z}\left(t\right)$ are provided as follows:

$$\begin{array}{cc}\hfill {\mu }_{x|z}(t,z_t)& ={\mu }_x\left(t\right)+{\Sigma }_{xz}\left(t\right){\Sigma }_{zz}^{-1}\left(t\right)(z_t-{\mu }_z\left(t\right)),\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill {\Sigma }_{x|z}\left(t\right)& ={\Sigma }_{xx}\left(t\right)-{\Sigma }_{xz}\left(t\right){\Sigma }_{zz}^{-1}\left(t\right){\Sigma }_{zx}\left(t\right).\hfill \end{array}$$

(42)

Theorem 2. 

In the LQG problem without memory limitation, the optimal control functions of ML-POSC (40) are provided by

$$\begin{array}{cc}\hfill u^{*}(t,z)& =-R^{-1}{B}^{\top }\Psi z,\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill v^{*}(t,z)& =\left(A-B{R}^{-1}{B}^{\top }\Psi -{\Sigma }_{x|z}{H}^{\top }{\left(\gamma {\gamma }^{\top }\right)}^{-1}H\right)z,\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill {\kappa }^{*}(t,z)& ={\Sigma }_{x|z}{H}^{\top }{\left(\gamma {\gamma }^{\top }\right)}^{-1}.\hfill \end{array}$$

(45)

From $v^{*}(t,z)$ and ${\kappa }^{*}(t,z)$, $z_t$ and ${\Sigma }_{x|z}\left(t\right)$ obey the following equations:

$$\begin{array}{cc}\hfill & d{z}_t=\left(A-B{R}^{-1}{B}^{\top }\Psi \right)z_{t}dt+{\Sigma }_{x|z}{H}^{\top }{\left(\gamma {\gamma }^{\top }\right)}^{-1}\left(d{y}_t-H{z}_{t}dt\right),\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill & \frac{d{\Sigma }_{x|z}}{dt}=\sigma {\sigma }^{\top }+A{\Sigma }_{x|z}+{\Sigma }_{x|z}{A}^{\top }-{\Sigma }_{x|z}{H}^{\top }{\left(\gamma {\gamma }^{\top }\right)}^{-1}H{\Sigma }_{x|z},\hfill \end{array}$$

(47)

where $z_0={\mu }_{x,0}$ and ${\Sigma }_{x|z}\left(0\right)={\Sigma }_{xx,0}$. Furthermore, ${\mu }_{x|z}(t,z_t)=z_t$ holds in this problem. $\Psi \left(t\right)$ is the solution of the Riccati Equation (37).

Proof. 

The proof is shown in Appendix D. □

In the LQG problem of the conventional POSC, the optimal memory dynamics of ML-POSC (46) and (47) corresponds to the Kalman filter (35) and (36). Furthermore, ML-POSC reproduces the Riccati Equation (37).

6. Linear-Quadratic-Gaussian Problem with Memory Limitation

The LQG problem of the conventional POSC does not consider memory limitation because it does not consider the memory noise and cost. Furthermore, because the memory dimension is restricted to the state dimension, the memory dimension cannot be determined according to a given controller. ML-POSC can generalize the LQG problem to include the memory limitation. In this section, we discuss the LQG problem with memory limitation based on ML-POSC.

6.1. Problem Formulation

In this subsection, we formulate the LQG problem with memory limitation. The state and observation SDEs are the same as in the previous section, which are provided by (30) and (31), respectively. The controller of ML-POSC determines the control $u_t\in {\mathbb{R}}^{d_u}$ based on the memory $z_t\in {\mathbb{R}}^{d_z}$, i.e., $u_t=u(t,z_t)$. Unlike the LQG problem of the conventional POSC, the memory dimension $d_z$ is not necessarily the same as the state dimension $d_x$.

The memory $z_t$ is assumed to evolve according to the following SDE:

$$\begin{array}{cc}\hfill d{z}_t& =v_{t}dt+\kappa \left(t\right)d{y}_t+\eta \left(t\right)d{\xi }_t,\hfill \end{array}$$

(48)

where $z_0$ obeys the Gaussian distribution $p_0\left(z_0\right)=\mathcal{N}\left(z_0\left|{\mu }_{z,0},{\Sigma }_{zz,0}\right.\right)$, ${\xi }_t\in {\mathbb{R}}^{d_{\xi }}$ is the standard Wiener process, and $v_t=v(t,z_t)\in {\mathbb{R}}^{d_v}$ is the control. Because the initial condition $z_0$ is stochastic and the memory SDE (48) includes the intrinsic stochasticity $d{\xi }_t$, the LQG problem of ML-POSC can consider the memory noise explicitly. We note that $\kappa \left(t\right)$ is independent of the memory $z_t$. If $\kappa \left(t\right)$ depends on the memory $z_t$, the memory SDE (48) becomes non-linear and non-Gaussian. As a result, the optimal control functions cannot be derived explicitly in this case. In order to keep the memory SDE (48) linear and Gaussian for obtaining the optimal control functions explicitly, we restrict $\kappa \left(t\right)$ being independent of the memory $z_t$ in the LQG problem with memory limitation. The LQG problem without memory limitation is the special case in which the optimal control ${\kappa }_t^{*}={\kappa }^{*}(t,z_t)$ in (45) does not depend on the memory $z_t$.

The objective function is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{p(x_{0:T},y_{0:T},z_{0:T};u,v)}\left[{\int }_0^T\left(x_t^{\top }Q\left(t\right)x_t+u_t^{\top }R\left(t\right)u_t+v_t^{\top }M\left(t\right)v_t\right)dt+x_T^{\top }P{x}_T\right],\end{array}$$

(49)

where $Q\left(t\right)⪰O$, $R\left(t\right)\succ O$, $M\left(t\right)\succ O$, and $P⪰O$. Because the cost function includes $v_t^{\top }M\left(t\right)v_t$, the LQG problem of ML-POSC can consider the memory control cost explicitly. ML-POSC optimizes the state control function u and the memory control function v based on the objective function $J[u,v]$, as follows:

$$\begin{array}{c}\hfill u^{*},v^{*}:=\underset{u,v}{\mathrm{argmin}}J[u,v].\end{array}$$

(50)

For the sake of simplicity, we do not optimize $\kappa \left(t\right)$, although this can be accomplished by considering unobservable stochastic control.

6.2. Problem Reformulation

Although the formulation of the LQG problem with memory limitation in the previous subsection clarifies its relationship with that of the LQG problem without memory limitation, it is inconvenient for further mathematical investigations. In order to resolve this problem, we reformulate the LQG problem with memory limitation based on the extended state $s_t$ (17). The formulation in this subsection is simpler and more general than that in the previous subsection.

In the LQG problem with memory limitation, the extended state SDE (18) is provided as follows:

$$\begin{array}{c}\hfill d{s}_t=\left(\tilde{A}\left(t\right)s_t+\tilde{B}\left(t\right){\tilde{u}}_t\right)dt+\tilde{\sigma }\left(t\right)d{\tilde{\omega }}_t,\end{array}$$

(51)

where $s_0$ obeys the Gaussian distribution $p_0\left(s_0\right):=\mathcal{N}\left(s_0\left|{\mu }_0,{\Sigma }_0\right.\right)$, ${\tilde{\omega }}_t\in {\mathbb{R}}^{d_{\tilde{\omega }}}$ is the standard Wiener process, and ${\tilde{u}}_t=\tilde{u}(t,z_t)\in {\mathbb{R}}^{d_{\tilde{u}}}$ is the control. The extended state SDE (51) includes the previous state, observation, and memory SDEs (30), (31) and (48) as a special case because they can be represented as follows:

$$d{s}_t=\left(\left(\begin{array}{cc}A& O\\ \kappa H& O\end{array}\right)s_t+\left(\begin{array}{cc}B& O\\ O& I\end{array}\right){\tilde{u}}_t\right)dt+\left(\begin{array}{ccc}\sigma & O& O\\ O& \kappa \gamma & \eta \end{array}\right)\left(\begin{array}{c}d{\omega }_t\\ d{\nu }_t\\ d{\xi }_t\end{array}\right),$$

(52)

where $p_0\left(s_0\right)=p_0\left(x_0\right)p_0\left(z_0\right)$.

The objective function (21) is provided by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^T\left(s_t^{\top }\tilde{Q}\left(t\right)s_t+{\tilde{u}}_t^{\top }\tilde{R}\left(t\right){\tilde{u}}_t\right)dt+s_T^{\top }\tilde{P}{s}_T\right],\end{array}$$

(53)

where $\tilde{Q}\left(t\right)⪰O$, $\tilde{R}\left(t\right)\succ O$, and $\tilde{P}⪰O$. This objective function (53) includes the previous objective function (49) as a special case because it can be represented as follows:

$$J\left[\tilde{u}\right]={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^T\left(s_t^{\top }\left(\begin{array}{cc}Q& O\\ O& O\end{array}\right)s_t+{\tilde{u}}_t^{\top }\left(\begin{array}{cc}R& O\\ O& M\end{array}\right){\tilde{u}}_t\right)dt+s_T^{\top }\left(\begin{array}{cc}P& O\\ O& O\end{array}\right)s_T\right].$$

(54)

The objective of the LQG problem with memory limitation is to find the optimal control function ${\tilde{u}}^{*}$ that minimizes the objective function $J\left[\tilde{u}\right]$, as follows:

$$\begin{array}{c}\hfill {\tilde{u}}^{*}:=\underset{\tilde{u}}{\mathrm{argmin}}J\left[\tilde{u}\right].\end{array}$$

(55)

In the following subsection, we mainly consider the formulation of this subsection rather than that of the previous subsection because it is simpler and more general. Moreover, we omit $\widetilde{·}$ for notational simplicity.

6.3. Derivation of Optimal Control Function

In this subsection, we derive the optimal control function of the LQG problem with memory limitation by applying Theorem 1. In the LQG problem with memory limitation, the probability of the extended state s at time t is provided by the Gaussian distribution $p_t\left(s\right)=\mathcal{N}\left(s\left|\mu \right(t),\Sigma (t)\right)$. By defining the stochastic extended state $\widehat{s}:=s-\mu$, ${\mathbb{E}}_{p_t\left(x\right|z)}\left[s\right]$ is provided as follows:

$$\begin{array}{c}\hfill {\mathbb{E}}_{p_t\left(x\right|z)}\left[s\right]=K\left(t\right)\widehat{s}+\mu \left(t\right),\end{array}$$

(56)

where $K\left(t\right)$ is defined by

$$\begin{array}{c}\hfill K\left(t\right):=\left(\begin{array}{cc}O& {\Sigma }_{xz}\left(t\right){\Sigma }_{zz}^{-1}\left(t\right)\\ O& I\end{array}\right).\end{array}$$

(57)

By applying Theorem 1 to the LQG problem with memory limitation, we obtain the following theorem:

Theorem 3. 

In the LQG problem with memory limitation, the optimal control function of ML-POSC is provided by

$$\begin{array}{c}\hfill u^{*}(t,z)=-R^{-1}{B}^{\top }\left(\Pi K\widehat{s}+\Psi \mu \right),\end{array}$$

(58)

where $K\left(t\right)$ (57) depends on $\Sigma \left(t\right)$, and $\mu \left(t\right)$ and $\Sigma \left(t\right)$ are the solutions of the following ordinary differential equations:

$$\begin{array}{cc}\hfill \frac{d\mu }{dt}& =\left(A-B{R}^{-1}{B}^{\top }\Psi \right)\mu ,\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill \frac{d\Sigma }{dt}& =\sigma {\sigma }^{\top }+\left(A-B{R}^{-1}{B}^{\top }\Pi K\right)\Sigma +\Sigma {\left(A-B{R}^{-1}{B}^{\top }\Pi K\right)}^{\top },\hfill \end{array}$$

(60)

where $\mu \left(0\right)={\mu }_0$ and $\Sigma \left(0\right)={\Sigma }_0$, while $\Psi \left(t\right)$ and $\Pi \left(t\right)$ are the solutions of the following ordinary differential equations:

$$\begin{array}{cc}\hfill -\frac{d\Psi }{dt}& =Q+A^{\top }\Psi +\Psi A-\Psi B{R}^{-1}{B}^{\top }\Psi ,\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill -\frac{d\Pi }{dt}& =Q+A^{\top }\Pi +\Pi A-\Pi B{R}^{-1}{B}^{\top }\Pi +{(I-K)}^{\top }\Pi B{R}^{-1}{B}^{\top }\Pi (I-K),\hfill \end{array}$$

(62)

where $\Psi \left(T\right)=\Pi \left(T\right)=P$.

Proof. 

The proof is shown in Appendix E. □

Here, (61) is the Riccati equation [9,10,23], which appears in the LQG problem without memory limitation as well (37). In contrast, (62) is a new equation of the LQG problem with memory limitation, which in this paper we call the partially observable Riccati equation. Because estimation and control are not clearly separated in the LQG problem with memory limitation, the Riccati Equation (61) for control is modified to include estimation, which corresponds to the partially observable Riccati Equation (62). As a result, the partially observable Riccati Equation (62) is able to improve estimation as well as control.

In order to support this interpretation, we analyze the partially observable Riccati Equation (62) by comparing it with the Riccati Equation (61). Because only the last term of (62) is different from (61), we denote it as follows:

$$\begin{array}{c}\hfill \mathcal{Q}:={(I-K)}^{\top }\Pi B{R}^{-1}{B}^{\top }\Pi (I-K).\end{array}$$

(63)

$\mathcal{Q}$ can be calculated as follows:

$$\begin{array}{c}\hfill \mathcal{Q}=\left(\begin{array}{cc}{\mathcal{P}}_{xx}& -{\mathcal{P}}_{xx}{\Sigma }_{xz}{\Sigma }_{zz}^{-1}\\ -{\Sigma }_{zz}^{-1}{\Sigma }_{zx}{\mathcal{P}}_{xx}& {\Sigma }_{zz}^{-1}{\Sigma }_{zx}{\mathcal{P}}_{xx}{\Sigma }_{xz}{\Sigma }_{zz}^{-1}\end{array}\right),\end{array}$$

(64)

where ${\mathcal{P}}_{xx}:={(\Pi B{R}^{-1}{B}^{\top }\Pi )}_{xx}$. Because ${\mathcal{P}}_{xx}⪰O$ and ${\Sigma }_{zz}^{-1}{\Sigma }_{zx}{\mathcal{P}}_{xx}{\Sigma }_{xz}{\Sigma }_{zz}^{-1}⪰O$, ${\Pi }_{xx}$ and ${\Pi }_{zz}$ may be larger than ${\Psi }_{xx}$ and ${\Psi }_{zz}$, respectively. Because ${\Pi }_{xx}$ and ${\Pi }_{zz}$ are the negative feedback gains of the state x and the memory z, respectively, $\mathcal{Q}$ may decrease ${\Sigma }_{xx}$ and ${\Sigma }_{zz}$. Moreover, when ${\Sigma }_{xz}$ is positive/negative, ${\Pi }_{xz}$ may be smaller/larger than ${\Psi }_{xz}$, which may increase/decrease ${\Sigma }_{xz}$. A similar discussion is possible for ${\Sigma }_{zx}$, ${\Pi }_{zx}$, and ${\Psi }_{zx}$, as $\Sigma$, $\Pi$, and $\Psi$ are symmetric matrices. As a result, $\mathcal{Q}$ may decrease the following conditional covariance matrix:

$$\begin{array}{c}\hfill {\Sigma }_{x|z}:={\Sigma }_{xx}-{\Sigma }_{xz}{\Sigma }_{zz}^{-1}{\Sigma }_{zx},\end{array}$$

(65)

which corresponds to the estimation error of the state from the memory. Therefore, the partially observable Riccati Equation (62) may improve estimation as well as control, which is different from the Riccati Equation (61).

Because the problem in Section 6.1 is specialized more than that in Section 6.2, we can carry out a more specific discussion. In the problem in Section 6.1, ${\Psi }_{xx}$ is the same as the solution of the Riccati equation of the conventional POSC (37), and ${\Psi }_{xz}=O$, ${\Psi }_{zx}=O$, and ${\Psi }_{zz}=O$ are satisfied. As a result, the memory control does not appear in the Riccati equation of ML-POSC (61). In contrast, because of the last term of the partially observable Riccati Equation (62), ${\Pi }_{xx}$ is not the solution of the Riccati Equation (37), and ${\Pi }_{xz}\ne O$, ${\Pi }_{zx}\ne O$, and ${\Pi }_{zz}\ne O$ are satisfied. As a result, the memory control appears in the partially observable Riccati Equation (62), which may improve the state estimation.

6.4. Comparison with Completely Observable Stochastic Control

In this subsection, we compare ML-POSC with COSC of the extended state. By applying Proposition 2 in the LQG problem, the optimal control function of COSC of the extended state can be obtained as follows:

Proposition 4 

([10,23]). In the LQG problem, the optimal control function of COSC of the extended state is provided by

$$\begin{array}{c}\hfill u^{*}(t,s)=-R^{-1}{B}^{\top }\Psi s=-R^{-1}{B}^{\top }\left(\Psi \widehat{s}+\Psi \mu \right),\end{array}$$

(66)

where $\Psi \left(t\right)$ is the solution of the Riccati Equation (61).

Proof. 

The proof is shown in [10,23]. □

The optimal control function of COSC of the extended state (66) can be derived intuitively from that of ML-POSC (58). In ML-POSC, $K\widehat{s}={\mathbb{E}}_{p_t\left(x\right|z)}\left[\widehat{s}\right]$ is the estimator of the stochastic extended state. In COSC of the extended state, because the stochastic extended state is completely observable, its estimator is provided by $\widehat{s}$, which corresponds to $K=I$. By changing the definition of K from (57) to $K=I$, the partially observable Riccati Equation (62) is reduced to the Riccati Equation (61), and the optimal control function of ML-POSC (58) is reduced to that of COSC (66). As a result, the optimal control function of ML-POSC (58) can be interpreted as the generalization of that of COSC (66).

While the second term is the same between (58) and (66), the first term is different. The second term is the control of the expected extended state $\mu$, which does not depend on the realization. In contrast, the first term is the control of the stochastic extended state $\widehat{s}$, which depends on the realization. The first term has two different points: (i) The estimators of the stochastic extended state in COSC and ML-POSC are provided by $\widehat{s}$ and $K\widehat{s}={\mathbb{E}}_{p_t\left(x\right|z)}\left[\widehat{s}\right]$, respectively, which is reasonable because ML-POSC needs to estimate the state from the memory; and (ii) The control gains of the stochastic extended state in COSC and ML-POSC are provided by $\Psi$ and $\Pi$, respectively. While $\Psi$ improves only control, $\Pi$ improves estimation as well as control.

6.5. Numerical Algorithm

In the LQG problem, the partial differential equations are reduced to the ordinary differential equations. The FP Equation (23) is reduced to (59) and (60), and the HJB Equation (28) is reduced to (61) and (62). As a result, the optimal control function (58) can be obtained more easily in the LQG problem.

The Riccati Equation (61) can be solved backwards from the terminal condition. In contrast, the partially observable Riccati Equation (62) cannot be solved in the same way as the Riccati Equation (61), as it depends on the forward equation of $\Sigma$ (60) through K (57). Because the forward equation of $\Sigma$ (60) depends on the backward equation of $\Pi$ (62) as well, they must be solved simultaneously.

A similar problem appears in the mean-field game and control, and numerous numerical methods have been developed to deal with it [33]. In this paper, we solve the system of (60) and (62) using the forward–backward sweep method, which computes (60) and (62) alternately [33,34]. In ML-POSC, the convergence of the forward–backward sweep method is guaranteed [34].

7. Numerical Experiments

In this section, we demonstrate the effectiveness of ML-POSC using numerical experiments on the LQG problem with memory limitation as well as on the non-LQG problem.

7.1. LQG Problem with Memory Limitation

In this subsection, we show the significance of the partially observable Riccati Equation (62) by a numerical experiment of the LQG problem with memory limitation. We consider the state $x_t\in \mathbb{R}$, the observation $y_t\in \mathbb{R}$, and the memory $z_t\in \mathbb{R}$, which evolve by the following SDEs:

$$\begin{array}{cc}\hfill d{x}_t& =\left(x_t+u_t\right)dt+d{\omega }_t,\hfill \end{array}$$

(67)

$$\begin{array}{cc}\hfill d{y}_t& =x_{t}dt+d{\nu }_t,\hfill \end{array}$$

(68)

$$\begin{array}{cc}\hfill d{z}_t& =v_{t}dt+d{y}_t,\hfill \end{array}$$

(69)

where $x_0$ and $z_0$ obey standard Gaussian distributions, $y_0$ is an arbitrary real number, ${\omega }_t\in \mathbb{R}$ and ${\nu }_t\in \mathbb{R}$ are independent standard Wiener processes, and $u_t=u(t,z_t)\in \mathbb{R}$ and $v_t=v(t,z_t)\in \mathbb{R}$ are the controls. The objective function to be minimized is provided as follows:

$$\begin{array}{c}\hfill J[u,v]:=\mathbb{E}\left[{\int }_0^{10}\left(x_t^2+u_t^2+v_t^2\right)dt\right].\end{array}$$

(70)

Therefore, the objective of this problem is to minimize the state variance by the small state and memory controls. Because this problem includes the memory control cost, it corresponds to the LQG problem with memory limitation.

Figure 2a–c shows the trajectories of $\Psi$ and $\Pi$; ${\Pi }_{xx}$ and ${\Pi }_{zz}$ are larger than ${\Psi }_{xx}$ and ${\Psi }_{zz}$, respectively, and ${\Pi }_{xz}$ is smaller than ${\Psi }_{xz}$, which is consistent with our discussion in Section 6.3. Therefore, the partially observable Riccati equation may reduce the estimation error of the state from the memory. Moreover, while the memory control does not appear in the Riccati equation (${\Psi }_{xz}={\Psi }_{zz}=0$), it appears in the partially observable Riccati equation (${\Pi }_{xz}\ne 0$, ${\Pi }_{zz}\ne 0$), which is consistent with our discussion in Section 6.3. As a result, the memory control plays an important role in estimating the state from the memory.

Figure 2. Numerical simulation of the LQG problem with memory limitation. (a–c) Trajectories of the elements of $\Psi \left(t\right)\in {\mathbb{R}}^{2\times 2}$ and $\Pi \left(t\right)\in {\mathbb{R}}^{2\times 2}$. Because ${\Psi }_{zx}\left(t\right)={\Psi }_{xz}\left(t\right)$ and ${\Pi }_{zx}\left(t\right)={\Pi }_{xz}\left(t\right)$, ${\Psi }_{zx}\left(t\right)$ and ${\Pi }_{zx}\left(t\right)$ are not visualized. (d–f) Stochastic behaviors of the state $x_t$ (d), the memory $z_t$ (e), and the cumulative cost (f) for 100 samples. The expectation of the cumulative cost at $t=10$ corresponds to the objective function (70). Blue and orange curves are controlled by (71) and (58), respectively.

In order to clarify the significance of the partially observable Riccati Equation (62), we compare the performance of the optimal control function (58) with that of the following control function:

$$\begin{array}{c}\hfill u^{\Psi }(t,z)=-R^{-1}{B}^{\top }\left(\Psi K\widehat{s}+\Psi \mu \right),\end{array}$$

(71)

in which $\Pi$ is replaced with $\Psi$. This result is shown in Figure 2d–f. In the control function (71), the distributions of the state and the memory are unstable, and the cumulative cost diverges. By contrast, in the optimal control function (58), the distributions of the state and memory are stable, and the cumulative cost is smaller. This result indicates that the partially observable Riccati Equation (62) plays an important role in the LQG problem with memory limitation.

7.2. Non-LQG Problem

In this subsection, we investigate the potential effectiveness of ML-POSC for a non-LQG problem by comparing it with the local LQG approximation of the conventional POSC [3,4]. We consider the state $x_t\in \mathbb{R}$ and the observation $y_t\in \mathbb{R}$, which evolve according to the following SDEs:

$$\begin{array}{cc}\hfill d{x}_t& =u_{t}dt+d{\omega }_t,\hfill \end{array}$$

(72)

$$\begin{array}{cc}\hfill d{y}_t& =x_{t}dt+d{\nu }_t,\hfill \end{array}$$

(73)

where $x_0$ obeys the Gaussian distribution $p_0\left(x_0\right)=\mathcal{N}\left(x_0\right|0,0.01)$, $y_0$ is an arbitrary real number, ${\omega }_t\in \mathbb{R}$ and ${\nu }_t\in \mathbb{R}$ are independent standard Wiener processes, and $u_t=u(t,y_{0:t})\in \mathbb{R}$ is the control. The objective function to be minimized is provided as follows:

$$\begin{array}{c}\hfill J\left[u\right]:=\mathbb{E}\left[{\int }_0^1\left(Q(t,x_t)+u_t^2\right)dt+10x_1^2\right],\end{array}$$

(74)

where

$$\begin{array}{c}\hfill Q(t,x):=\left\{\begin{array}{cc}1000\hfill & (0.3\le t\le 0.6,0.1\le |x|\le 2.0),\hfill \\ 0\hfill & \left(others\right).\hfill \end{array}\right.\end{array}$$

(75)

The cost function is high on the black rectangles in Figure 3a, which represent the obstacles. In addition, the terminal cost function is the lowest on the black cross in Figure 3a, which represents the desirable goal. Therefore, the system should avoid the obstacles and reach the goal with the small control. Because the cost function is non-quadratic, it is a non-LQG problem, which cannot be solved exactly by the conventional POSC.

Figure 3. Numerical simulation of the non-LQG problem for the local LQG approximation (blue) and ML-POSC (orange). (a) Stochastic behaviors of state $x_t$ for 100 samples. The black rectangles and cross represent the obstacles and goal, respectively. (b) The objective function (74), computed from 100 samples.

In the local LQG approximation of the conventional POSC [3,4], the Zakai equation and the Bellman equation are locally approximated by the Kalman filter and the Riccati equation, respectively. Because the Bellman equation is reduced to the Riccati equation, the local LQG approximation can be solved numerically even in the non-LQG problem.

ML-POSC determines the control $u_t\in \mathbb{R}$ based on the memory $z_t\in \mathbb{R}$, i.e., $u_t=u(t,z_t)$. The memory dynamics is formulated with the following SDE:

$$\begin{array}{cc}\hfill d{z}_t& =d{y}_t,\hfill \end{array}$$

(76)

where $p_0\left(z_0\right)=\mathcal{N}\left(z_0\right|0,0.01)$. For the sake of simplicity, the memory control is not considered.

Figure 3 is the numerical result comparing the local LQG approximation and ML-POSC. Because the local LQG approximation reduces the Bellman equation to the Riccati equation by ignoring non-LQG information, it cannot avoid the obstacles, which results in a higher objective function. In contrast, because ML-POSC reduces the Bellman equation to the HJB equation while maintaining non-LQG information, it can avoid the obstacles, which results in a lower objective function. Therefore, our numerical experiment shows that ML-POSC can be superior to local LQG approximation.

8. Discussion

In this work, we propose ML-POSC, which is an alternative theoretical framework to the conventional POSC. ML-POSC first formulates the finite-dimensional and stochastic memory dynamics explicitly, then optimizes the memory dynamics considering the memory cost. As a result, unlike the conventional POSC, ML-POSC can consider memory limitation as well as incomplete information. Furthermore, because the optimal control function of ML-POSC is obtained by solving the system of HJB-FP equations, ML-POSC can be solved in practice even in non-LQG problems. ML-POSC can generalize the LQG problem to include memory limitation. Because estimation and control are not clearly separated in the LQG problem with memory limitation, the Riccati equation can be modified to the partially observable Riccati equation, which improves estimation as well as control. Furthermore, ML-POSC can provide a better result than the local LQG approximation in a non-LQG problem, as ML-POSC reduces the Bellman equation while maintaining non-LQG information.

ML-POSC is effective for the state estimation problem as well, which is a part of the POSC problem. Although the state estimation problem can be solved in principle by the Zakai equation [38,39,40], it cannot be solved directly, as the Zakai equation is infinite-dimensional. In order to resolve this problem, a particle filter is often used to approximate the infinite-dimensional Zakai equation as a finite number of particles [38,39,40]. However, because the performance of the particle filter is guaranteed only in the limit of a large number of particles, a particle filter may not be practical in cases where the available memory size is severely limited. Furthermore, a particle filter cannot take the memory noise and cost into account. ML-POSC resolves these problems, as it can optimize the state estimation under memory limitation.

ML-POSC may be extended from a single-agent system to a multi-agent system. POSC of a multi-agent system is called decentralized stochastic control (DSC) [41,42,43], which consists of a system and multiple controllers. In DSC, each controller needs to estimate the controls of the other controllers as well as the state of the system, which is essentially different from the conventional POSC. Because the estimation among the controllers is generally intractable, the conventional POSC approach cannot be straightforwardly extended to DSC. In contrast, ML-POSC compresses the observation history into the finite-dimensional memory, which simplifies estimation among the controllers. Therefore, ML-POSC may provide an effective approach to DSC. Actually, the finite-state controller, the idea of which is similar with ML-POSC, plays a key role in extending POMDP from a single-agent system to a multi-agent system [22,44,45,46,47,48]. ML-POSC may be extended to a multi-agent system in a similar way as a finite-state controller.

ML-POSC can be naturally extended to the mean-field control setting [28,29,30] because ML-POSC is solved based on the mean-field control theory. Therefore, ML-POSC can be applied to an infinite number of homogeneous agents. Furthermore, ML-POSC can be extended to a risk-sensitive setting, as this is a special case of the mean-field control setting [28,29,30]. Therefore, ML-POSC can consider the variance of the cost as well as its expectation.

Nonetheless, more efficient algorithms are needed in order to solve ML-POSC with a high-dimensional state and memory. In the mean-field game and control, neural network-based algorithms have recently been proposed which can solve high-dimensional problems efficiently [49,50]. By extending these algorithms, it might be possible to solve high-dimensional ML-POSC efficiently. Furthermore, unlike the mean-field game and control, the coupling of HJB-FP equations is limited to the optimal control function in ML-POSC. By exploiting this property, more efficient algorithms for ML-POSC may be proposed [34].