Decentralized Stochastic Control with Finite-Dimensional Memories: A Memory Limitation Approach

Abstract

Decentralized stochastic control (DSC) is a stochastic optimal control problem consisting of multiple controllers. DSC assumes that each controller is unable to accurately observe the target system and the other controllers. This setup results in two difficulties in DSC; one is that each controller has to memorize the infinite-dimensional observation history, which is not practical, because the memory of the actual controllers is limited. The other is that the reduction of infinite-dimensional sequential Bayesian estimation to finite-dimensional Kalman filter is impossible in general DSC, even for linear-quadratic-Gaussian (LQG) problems. In order to address these issues, we propose an alternative theoretical framework to DSC—memory-limited DSC (ML-DSC). ML-DSC explicitly formulates the finite-dimensional memories of the controllers. Each controller is jointly optimized to compress the infinite-dimensional observation history into the prescribed finite-dimensional memory and to determine the control based on it. Therefore, ML-DSC can be a practical formulation for actual memory-limited controllers. We demonstrate how ML-DSC works in the LQG problem. The conventional DSC cannot be solved except in the special LQG problems where the information the controllers have is independent or partially nested. We show that ML-DSC can be solved in more general LQG problems where the interaction among the controllers is not restricted.

1. Introduction

Optimal control problems of a stochastic dynamical system by decentralized multiple controllers appear in various practical applications, including real-time communication [1,2], decentralized hypothesis testing [3], and networked control [4]. Such problems have been extensively studied in stochastic optimal control theory as decentralized stochastic control (DSC) [5,6,7,8,9,10]. DSC consists of a target system and multiple controllers (Figure 1a) and assumes that each controller cannot completely observe the state of the system and the controls of the other controllers. The information of the target system and the other controllers was only obtained via their noisy observations. Thus, each controller should be optimized to determine its control solely from its own observation history. Even for a pair of finite-dimensional state and observation, the observation history is infinite-dimensional. As a result, ideally, the optimal controller obtained theoretically should possess infinite-dimensional memory. In practical applications, however, the available memory size of each controller is finite-dimensional and often severely limited. Thus, we have to obtain the solutions based on finite-dimensional memory by employing approximation methods heuristically, which may impair the optimality of the ideal solution, especially when the available memory size is not sufficient.

Moreover, another difficulty arises in DSC due to the decentralized setting. If the number of controllers is one, or if all controllers share their observation histories, DSC is reduced to partially observable stochastic control (POSC), in which the observation histories of all controllers can be summarized optimally as the posterior probability of the state by sequential Bayesian estimation [11,12,13,14,15,16]. The posterior probability of the state is also infinite-dimensional, and thus the same problem as DSC still survives, even for POSC. Nevertheless, in POSC, this difficulty can be circumvented by focusing only on the linear-quadratic-Gaussian (LQG) setting under which the posterior probability of the state can be represented by the finite-dimensional mean vector and covariance matrix of Gaussian distribution and the sequential Bayesian estimation can be computed by the Kalman filter [11,12,14]. Therefore, POSC is practically solved, at least in the LQG problem. The difficulty in DCS is that this nice property of LQG is not retained.

In DCS, each controller cannot access the observation histories of the other controllers, as well as the state of the system, which causes each controller to estimate all the others from their own observation history. This hampers the Bayesian estimation of the posterior to be computed sequentially and prevents the infinite-dimensional observation history from being compressed into the finite-dimensional sufficient statistics, even for the LQG problem. Some theoretical studies have addressed this issue by restricting the interaction among the controllers. If the information the controllers have is independent [8,9,10] or partially nested [17,18,19,20,21,22], DSC can enjoy the nice property of the LQG problem and be solved explicitly and optimally with finite-dimensional memory. However, the LQG problem with more general interactions as well as non-LQG problems are still an open problem in DSC.

In order to address these issues, we propose an alternative theoretical framework to DSC, memory-limited DSC (ML-DSC), which is the decentralized version of memory-limited POSC (ML-POSC) [23,24]. The two major difficulties in DSC originate from the ignorance of constraints over controllers when we derive the optimal estimation and control solution. Unlike conventional DSC, ML-DSC explicitly formulates finite-dimensional memories of the controllers and their capacities (Figure 1b). In ML-DSC, each controller is optimized to compress the infinite-dimensional observation history into the prescribed finite-dimensional memory and to determine the control based on it. In other words, each controller controls not only the dynamics of the target system but also the dynamics of its own memory. The formulation of ML-DSC enables us to evade the difficulties mentioned above.

Furthermore, we provide a way to solve the optimization problem associated with the ML-DSC formulation. Specifically, we address the optimization problem by converting ML-DSC in the state space into the deterministic optimal control problem in the probability density function space. This technique has recently been used in mean-field stochastic control [25,26] and ML-POSC [23,24], and is also effective for ML-DSC. Following that, we can solve ML-DSC in a similar way to the deterministic optimal control problem on the probability density function space; the optimal control function of ML-DSC was obtained by jointly solving the Hamilton–Jacobi–Bellman (HJB) equation and the Fokker–Planck (FP) equation. HJB–FP equations also appear in mean-field stochastic game and control [25,26,27,28,29] and ML-POSC [23,24], and numerous numerical algorithms have been proposed [24,30,31,32]. Using these numerical algorithms, ML-DSC may be solved effectively, even in general problems. It should be noted that a similar idea to ML-DSC was also employed in a decentralized partially observable Markov decision process (DEC-POMDP) with the finite-state controller for more than a decade [33,34,35,36,37,38,39]. However, the algorithms of the finite-state controller of DEC-POMDP strongly depend on the discreteness, and thus they are not applicable to ML-DSC where the continuous time and state are considered.

We applied ML-DSC and our algorithm to the LQG problem. The conventional DSC can only be solved for special LQG problems where the information of the controllers is independent [8,9,10] or partially nested [17,18,19,20,21,22]. In contrast, ML-DSC can be solved even in LQG problems with more general interactions among the controllers. In LQG problems of POSC, estimation and control are clearly separated, and are optimized by the Kalman filter and the Riccati equation, respectively [11,14]. In the LQG problems of ML-DSC, estimation and control are not clearly separated and are jointly optimized by the modified Riccati equation, which is called the decentralized Riccati equation in this paper. We noted that this coupling of estimation and control also appears in the conventional DSC [17,18,19,20,21,22] and ML-POSC [23,24]. Therefore, it may be induced by decentralized structure and memory limitation. Finally, we conducted two numerical experiments for the LQG problems of ML-DSC. One controls one-dimensional divergent state dynamics, and the other controls two-dimensional oscillatory state dynamics. These numerical experiments demonstrate that the decentralized Riccati equation is superior to the Riccati equation in the LQG problems of ML-DSC.

The rest of this paper is organized as follows. In Section 2, we briefly review the conventional DSC. In Section 3, we formulate ML-DSC. In Section 4, we solve ML-DSC. In Section 5, we apply ML-DSC to the LQG problem. In Section 6, we conduct the numerical experiments of two LQG problems in ML-DSC. In Section 7, we discuss this paper.

2. Review of Decentralized Stochastic Control

In this section, we briefly review the conventional DSC (Figure 1a) [8,9,10]. DSC consists of a target system and N controllers. Vector $x_t\in {\mathbb{R}}^{d_x}$ is the state of the system at time $t\in [0,T]$, which evolves by the following stochastic differential equation (SDE):

$$\begin{array}{cc}\hfill d{x}_t& =b(t,x_t,u_t)dt+\sigma (t,x_t,u_t)d{\omega }_t,\hfill \end{array}$$

(1)

where $x_0$ obeys $p_0\left(x_0\right)$, ${\omega }_t\in {\mathbb{R}}^{d_{\omega }}$ is the standard Wiener process, $u_t^i\in {\mathbb{R}}^{d_u^i}$ is the control of controller i, and $u_t:=(u_t^1,u_t^2,\dots ,u_t^N)\in {\mathbb{R}}^{d_u}$ is the joint control of all controllers. We noted that $d_u:={\sum }_{i=1}^N{d}_u^i$. DSC often assumes that the system is composed of N agents and that the state of the system is decomposed into $x_t:=(x_t^1,x_t^2,\dots ,x_t^N)\in {\mathbb{R}}^{d_x}$ where $x_t^i\in {\mathbb{R}}^{d_x^i}$ is the state of agent i. In this paper, we do not assume such a situation, because our formulation of the state of the system includes it as a special case.

In DSC, controller i cannot completely observe the state of the system $x_t$ and the joint control of all controllers $u_t$. It can only obtain the noisy observation $y_t^i\in {\mathbb{R}}^{d_y^i}$, which evolves by the following SDE:

$$\begin{array}{cc}\hfill d{y}_t^i& =h^i(t,x_t,u_t)dt+{\gamma }^i(t,x_t,u_t)d{\nu }_t^i,\hfill \end{array}$$

(2)

where $y_0^i$ obeys $p_0^i\left(y_0^i\right)$, and ${\nu }_t^i\in {\mathbb{R}}^{d_{\nu }^i}$ is the standard Wiener process. Controller i’s observation $y_t^i$ is controlled by the other controllers through the joint control $u_t$, which expresses the communication among the controllers. Controller i determines its control $u_t^i$ based on the observation history $y_{0:t}^i:=\left\{y_{\tau }^i\right|\tau \in [0,t]\}$ as follows:

$$\begin{array}{c}\hfill u_t^i=u^i(t,y_{0:t}^i).\end{array}$$

(3)

The objective function of DSC is given by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J\left[u\right]:={\mathbb{E}}_{p(x_{0:T},y_{0:T};u)}\left[{\int }_0^{T}f(t,x_t,u_t)dt+g\left(x_T\right)\right],\end{array}$$

(4)

where f is the running cost function, g is the terminal cost function, $p(x_{0:T},y_{0:T};u)$ is the joint probability of $x_{0:T}$ and $y_{0:T}$ given that u is a parameter, and ${\mathbb{E}}_p\left[·\right]$ is the expectation with respect to p. DSC is the problem to find the optimal joint control function $u^{\ast }$ that minimizes the objective function $J\left[u\right]$:

$$\begin{array}{c}\hfill u^{\ast }:=\arg \underset{u}{\min }J\left[u\right].\end{array}$$

(5)

In DSC, controller i needs to memorize the infinite-dimensional observation history $y_{0:t}^i$ to determine the optimal control $u_t^{i\ast }=u^{i\ast }(t,y_{0:t}^i)$. This is one of the major obstacles in DSC for implementing controllers with finite and limited memory.

3. Memory-Limited Decentralized Stochastic Control

In this section, we formulate ML-DSC, which can circumvent the difficulty in DSC by explicitly formulating finite-dimensional memory of the controllers.

3.1. Problem Formulation

In this subsection, we formulate ML-DSC (Figure 1b). ML-DSC explicitly formulates the finite-dimensional memory of controller i by $z_t^i\in {\mathbb{R}}^{d_z^i}$. The memory dimension $d_z^i$ is prescribed by the available memory size of the controller i. The controller i compresses the infinite-dimensional observation history $y_{0:t}^i$ into the finite-dimensional memory $z_t^i$ by the following SDE:

$$\begin{array}{c}\hfill d{z}_t^i=c^i(t,z_t^i,v_t^i)dt+{\kappa }^i(t,z_t^i,v_t^i)d{y}_t^i+{\eta }^i(t,z_t^i,v_t^i)d{\xi }_t^i,\end{array}$$

(6)

where $z_0^i$ obeys $p_0^i\left(z_0^i\right)$, ${\xi }_t^i\in {\mathbb{R}}^{d_{\xi }^i}$ is the standard Wiener process, and $v_t^i\in {\mathbb{R}}^{d_v^i}$ is the control of the memory. Unlike the conventional DSC, ML-DSC can take into account the intrinsic stochasticity of the memory, which is modeled by the standard Wiener process $d{\xi }_t^i$ in the memory dynamics (6). In addition, the compression of the infinite-dimensional observation history $y_{0:t}^i$ into the finite-dimensional memory $z_t^i$ is optimized by the memory control $v_t^i$. In ML-DSC, the controller i determines the state control $u_t^i$ and the memory control $v_t^i$ based on the finite-dimensional memory $z_t^i$ as follows:

$$\begin{array}{c}\hfill u_t^i=u^i(t,z_t^i),v_t^i=v^i(t,z_t^i).\end{array}$$

(7)

The objective function of ML-DSC is given by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{p(x_{0:T},y_{0:T},z_{0:T};u,v)}\left[{\int }_0^{T}f(t,x_t,u_t,v_t)dt+g\left(x_T\right)\right],\end{array}$$

(8)

where f is the running cost function, g is the terminal cost function, $p(x_{0:T},y_{0:T},z_{0:T};u,v)$ is the joint probability of $x_{0:T}$, $y_{0:T}$ and $z_{0:T}$ given u and v as parameters, and ${\mathbb{E}}_p\left[·\right]$ is the expectation with respect to p. Unlike the cost function f of DSC in Equation (4), the cost function f of ML-DSC in Equation (8) depends on the memory control $v_t$ as well as the state control $u_t$. From a practical viewpoint, it should be natural to consider both costs of control and memory. ML-DSC optimizes the state control function u and the memory control function v based on the objective function $J[u,v]$:

$$\begin{array}{c}\hfill u^{\ast },v^{\ast }:=\arg \underset{u,v}{\min }J[u,v].\end{array}$$

(9)

The optimal memory control function $v^{\ast }:=(v^{1\ast },v^{2\ast },\dots ,v^{N\ast })$ optimizes the memory dynamics (6), which can be interpreted as the optimization of the compression of the observation history into the finite-dimensional memory. In the LQG problem of POSC, the optimal memory control function $v^{\ast }$ makes the memory dynamics into the Kalman filter, which is the optimal compression of the observations history in this problem [23]. We expect that the optimal memory control function $v^{\ast }$ is also effective for more general problems of ML-DSC.

In ML-DSC, controller i determines the optimal control functions $u_t^{i\ast }$ and $v_t^{i\ast }$ based only on the finite-dimensional memory $z_t^i$. In addition, ML-DSC can take into account the intrinsic stochasticity and the control cost of the memory. Thus, ML-DSC can explicitly accommodate various realistic constrains of the controllers such as memory size, noise in the controllers, and cost for updating memory, none of which can be explicitly addressed in the conventional DSC.

It should be noted that here we consider memory size only for storing continuous time-series with finite-dimensional vectors. While memory size also matters when we consider quantization and storing of real valued observations, these topics are out of the scope of this work.

3.2. Extended State

In this subsection, we generalize the formulation of ML-DSC based on the extended state. This generalization is useful for mathematical investigations by simplifying the notation of ML-DSC. Furthermore, it clarifies the difference between ML-DSC and the conventional stochastic optimal control problems.

We define the extended state $s_t\in {\mathbb{R}}^{d_s}$, the extended control ${\tilde{u}}_t^i\in {\mathbb{R}}^{d_{\tilde{u}}^i}$, the extended joint control ${\tilde{u}}_t\in {\mathbb{R}}^{d_{\tilde{u}}}$, and the extended standard Wiener process ${\tilde{\omega }}_t\in {\mathbb{R}}^{d_{\tilde{\omega }}}$ as follows:

$$s_t:=\left(\begin{array}{c}{x}_t\\ z_t^1\\ ⋮\\ z_t^N\end{array}\right),{\tilde{u}}_t^i:=\left(\begin{array}{c}{u}_t^i\\ v_t^i\end{array}\right),{\tilde{u}}_t:=\left(\begin{array}{c}{\tilde{u}}_t^1\\ ⋮\\ {\tilde{u}}_t^N\end{array}\right),{\tilde{\omega }}_t:=\left(\begin{array}{c}{\omega }_t\\ {\nu }_t^1\\ ⋮\\ {\nu }_t^N\\ {\xi }_t^1\\ ⋮\\ {\xi }_t^N\end{array}\right),$$

(10)

where $d_s:=d_x+{\sum }_{i=1}^N{d}_z^i$, $d_{\tilde{u}}^i:=d_u^i+d_v^i$, $d_{\tilde{u}}:={\sum }_{i=1}^N{d}_{\tilde{u}}^i$, and $d_{\tilde{\omega }}:=d_{\omega }+{\sum }_{i=1}^N{d}_{\nu }^i+{\sum }_{i=1}^N{d}_{\xi }^i$.

Based on the extended state $s_t$, the extended joint control ${\tilde{u}}_t$, and the extended standard Wiener process ${\tilde{\omega }}_t$, the state, observation, and memory SDEs, i.e., Equations (1), (2), and (6), are summarized as follows:

$$d{s}_t=\underset{=:\tilde{b}(t,s_t,{\tilde{u}}_t)}{\underbrace{\left(\begin{array}{c}b\\ c^1+{\kappa }^1{h}^1\\ ⋮\\ c^N+{\kappa }^N{h}^N\end{array}\right)}}dt+\underset{=:\tilde{\sigma }(t,s_t,{\tilde{u}}_t)}{\underbrace{\left(\begin{array}{ccccccc}\sigma & O& \dots & O& O& \dots & O\\ O& {\kappa }^1{\gamma }^1& \dots & O& {\eta }^1& \dots & O\\ ⋮& ⋮& \ddots & ⋮& ⋮& \ddots & ⋮\\ O& O& \dots & {\kappa }^N{\gamma }^N& O& \dots & {\eta }^N\end{array}\right)}}d{\tilde{\omega }}_t,$$

(11)

where $p_0\left(s_0\right)=p_0\left(x_0\right){\prod }_{i=1}^N{p}_0^i\left(z_0^i\right)$. Thus, the SDE of ML-DSC can be generalized as follows:

$$\begin{array}{c}\hfill d{s}_t=\tilde{b}(t,s_t,{\tilde{u}}_t)dt+\tilde{\sigma }(t,s_t,{\tilde{u}}_t)d{\tilde{\omega }}_t,\end{array}$$

(12)

where $s_0$ obeys $p_0\left(s_0\right)$. We note that the structures of $\tilde{b}(t,s_t,{\tilde{u}}_t)$ and $\tilde{\sigma }(t,s_t,{\tilde{u}}_t)$ in Equation (12) are not necessarily restricted to those in Equation (11). Importantly, in ML-DSC, controller i determines the extended control ${\tilde{u}}_t^i$ based solely on the memory $z_t^i$ as follows:

$$\begin{array}{c}\hfill {\tilde{u}}_t^i={\tilde{u}}^i(t,z_t^i).\end{array}$$

(13)

The objective function of ML-DSC (8) is generalized as follows:

$$\begin{array}{c}\hfill J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^T\tilde{f}(t,s_t,{\tilde{u}}_t)dt+\tilde{g}\left(s_T\right)\right],\end{array}$$

(14)

where $\tilde{f}$ is the running cost function and $\tilde{g}$ is the terminal cost function. Therefore, the generalized ML-DSC is the problem to find the optimal extended joint control function ${\tilde{u}}^{\ast }$ that minimizes the objective function $J\left[\tilde{u}\right]$:

$$\begin{array}{c}\hfill {\tilde{u}}^{\ast }:=\arg \underset{\tilde{u}}{\min }J\left[\tilde{u}\right]\end{array}$$

(15)

under the constraint of Equation (13).

This generalization (12)–(15) clarifies the difference between ML-DSC and the conventional stochastic optimal control problems. If controller i determines the extended control ${\tilde{u}}_t^i$ based on the whole extended state $s_t:=(x_t,z_t^1,\dots ,z_t^N)$ as ${\tilde{u}}_t^i={\tilde{u}}^i(t,s_t)$, this problem becomes equivalent to completely observable stochastic control (COSC), which is the most basic stochastic optimal control problem (Figure 2a) [13,14,40]. Furthermore, if controller i determines the extended control ${\tilde{u}}_t^i$ based on the joint memory $z_t:=(z_t^1,\dots ,z_t^N)$ as ${\tilde{u}}_t^i={\tilde{u}}^i(t,z_t)$, this problem is reduced to ML-POSC in which all controllers share their information (Figure 2b) [23,24]. ML-DSC determines the extended control ${\tilde{u}}_t^i$ based solely on its own memory $z_t^i$ as ${\tilde{u}}_t^i={\tilde{u}}^i(t,z_t^i)$ (13), which is different from COSC and ML-POSC (Figure 2c). While ML-DSC cannot be solved in a similar way as COSC [14,40,41], as is shown in the next section, it can be solved in a similar way as ML-POSC [23,24] because the method of ML-POSC is more general than that of COSC.

In the following section, we mainly consider the formulation of this subsection rather than that of Section 3.1 because it is simpler and more general. Moreover, we omit $\widetilde{·}$ for the notational simplicity.

4. Derivation of Optimal Control Function

In this section, we solve ML-DSC by employing the technique in mean-field stochastic control [25,26] and ML-POSC [23,24].

4.1. Derivation of Optimal Control Function

In this subsection, we derive the optimal control function of ML-DSC. In ML-DSC, each controller cannot directly access the information about the state of the system and the memories of the other controllers. This constraint makes ML-DSC unable to be solved by the conventional methods of COSC, such as Bellman’s dynamic programming principle on the extended state space [13,14,40]. In order to address this issue, we converted ML-DSC on the extended state space into the deterministic optimal control problem on the probability density function space. The similar technique has also been used in mean-field stochastic control [25,26] and ML-POSC [23,24], and it is more effective for a broader class of stochastic optimal control problems than the conventional methods of COSC.

The extended state SDE (12) can be converted into the following Fokker–Planck (FP) equation:

$$\begin{array}{c}\hfill \frac{\partial p_t\left(s\right)}{\partial t}={\mathcal{L}}_u^{†}{p}_t\left(s\right),\end{array}$$

(16)

where the initial condition is given by $p_0\left(s\right)$, and ${\mathcal{L}}_u^{†}$ is the forward diffusion operator, which is defined by

$$\begin{array}{cc}\hfill {\mathcal{L}}_u^{†}{p}_t\left(s\right)& :=-\sum _{i=1}^{d_s}\frac{\partial \left(b_i(t,s,u)p_t\left(s\right)\right)}{\partial s_i}+\frac{1}{2}\sum _{i,j=1}^{d_s}\frac{{\partial }^2\left(D_{ij}(t,s,u)p_t\left(s\right)\right)}{\partial s_i\partial s_j},\hfill \end{array}$$

(17)

where $D(t,s,u):=\sigma (t,s,u){\sigma }^{\top }(t,s,u)$. The objective function (14) can be calculated as follows:

$$\begin{array}{c}\hfill J\left[u\right]={\int }_0^T\overline{f}(t,p_t,u_t)dt+\overline{g}\left(p_T\right),\end{array}$$

(18)

where $\overline{f}(t,p,u):={\mathbb{E}}_{p\left(s\right)}\left[f(t,s,u)\right]$ and $\overline{g}\left(p\right):={\mathbb{E}}_{p\left(s\right)}\left[g\left(s\right)\right]$. We note that $\widetilde{·}$ is omitted for the notational simplicity. From Equations (16) and (18), ML-DSC on the extended state space is converted into the deterministic optimal control problem on the probability density function space.

If being represented by the extended state, each controller cannot completely access the information of the extended state in ML-DSC, which hampers the conventional methods of COSC. By lifting the state variable from the extended state to its probability density function, such a difficulty can be avoided, because any controllers can completely access the probability density function from its deterministic nature. As a result, the optimal condition of ML-DSC was obtained by a similar way to the deterministic optimal control problem, i.e., Pontryagin’s minimum principle on the probability density function, which can be interpreted as the generalization of Bellman’s dynamic programming principle on the extended state space [23,24]:

Theorem 1.

The optimal control function of ML-DSC satisfies the following equation:

$$\begin{array}{c}\hfill u^{i\ast }(t,z^i)=\arg \underset{u^i}{\min }{\mathbb{E}}_{p_t\left(s^{-i}\right|z^i)}\left[\mathcal{H}\left(t,s,(u^{-i\ast },u^i),w\right)\right],{}^{\forall }i\in \{1,2,\dots ,N\},\end{array}$$

(19)

where $\mathcal{H}$ is the Hamiltonian, which is defined as follows:

$$\begin{array}{c}\hfill \mathcal{H}\left(t,s,u,w\right):=f(t,s,u)+{\mathcal{L}}_{u}w(t,s),\end{array}$$

(20)

where ${\mathcal{L}}_u$ is the backward diffusion operator, which is defined as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{u}w(t,s)& :=\sum _{i=1}^{d_s}{b}_i(t,s,u)\frac{\partial w(t,s)}{\partial s_i}+\frac{1}{2}\sum _{i,j=1}^{d_s}{D}_{ij}(t,s,u)\frac{{\partial }^{2}w(t,s)}{\partial s_i\partial s_j},\hfill \end{array}$$

(21)

which is the conjugate of ${\mathcal{L}}_u^{†}$ as follows:

$$\begin{array}{c}\hfill \int w(t,s){\mathcal{L}}_u^{†}p(t,s)ds=\int p(t,s){\mathcal{L}}_{u}w(t,s)ds.\end{array}$$

(22)

Variables $s^{-i}\in {\mathbb{R}}^{d_s^{-i}}$, $u^{-i\ast }\in {\mathbb{R}}^{d_u^{-i}}$, and $(u^{-i\ast },u^i)\in {\mathbb{R}}^{d_u}$ are defined as follows:

$$s^{-i}:=\left(\begin{array}{c}x\\ z^1\\ ⋮\\ z^{i-1}\\ z^{i+1}\\ ⋮\\ z^N\end{array}\right),u^{-i\ast }:=\left(\begin{array}{c}{u}^{1\ast }\\ ⋮\\ u^{i-1\ast }\\ u^{i+1\ast }\\ ⋮\\ u^{N\ast }\end{array}\right),(u^{-i\ast },u^i):=\left(\begin{array}{c}{u}^{1\ast }\\ ⋮\\ u^{i-1\ast }\\ u^i\\ u^{i+1\ast }\\ ⋮\\ u^{N\ast }\end{array}\right),$$

(23)

where $d_s^{-i}:=d_s-d_z^i$ and $d_u^{-i}:=d_u-d_u^i$. Function $w(t,s)$ is the solution of the following Hamilton–Jacobi–Bellman (HJB) equation:

$$\begin{array}{c}\hfill -\frac{\partial w(t,s)}{\partial t}=\mathcal{H}\left(t,s,u^{\ast },w\right),\end{array}$$

(24)

where $w(T,s)=g\left(s\right)$. Function $p_t\left(s^{-i}\right|z^i):=p_t\left(s\right)/\int p_t\left(s\right)d{s}^{-i}$ is the conditional probability density function of $s^{-i}$ given $z^i$, and $p_t\left(s\right)$ is the solution of FP Equation (16) driven by $u^{\ast }$. We note that $\widetilde{·}$ is omitted for the notational simplicity.

Proof. 

The proof is shown in Appendix A. □

We note that the optimality condition (19) is a necessary condition of the optimal control function of ML-DSC, not a sufficient condition. The optimality condition (19) becomes a necessary and sufficient condition when the expected Hamiltonian $\overline{\mathcal{H}}\left(t,p,u,w\right):={\mathbb{E}}_{p\left(s\right)}\left[\mathcal{H}\left(t,s,u,w\right)\right]$ is convex with respect to p and u. This proof is almost the same with Reference [24]. In the following, the control function of ML-DSC that satisfies the optimality condition (19) is called the optimal control function of ML-DSC.

4.2. Numerical Algorithm

The optimal control function of ML-DSC (19) is obtained by jointly solving FP Equation (16) and HJB Equation (24). HJB-FP equations also appear in mean-field stochastic game and control [25,26,27,28,29] and ML-POSC [23,24], and numerous numerical algorithms have been developed [24,32]. As a result, ML-DSC may be solved practically by employing these numerical algorithms.

One of the most basic numerical algorithms to solve HJB-FP equations is the forward-backward sweep method (the fixed-point iteration method) [24,32,42,43,44], which computes the forward FP Equation (16) and the backward HJB Equation (24) alternately. While the convergence of the forward-backward sweep method is not guaranteed in mean-field stochastic game and control [32,42,43,44], it is guaranteed in ML-POSC because the coupling of HJB-FP equations is limited to the optimal control function in ML-POSC [24]. The convergence of the forward-backward sweep method is also guaranteed in ML-DSC for the same reason as ML-POSC. This proof is almost the same as Reference [24].

4.3. Comparison with Completely Observable Stochastic Control or Memory-Limited Partially Observable Stochastic Control

COSC and ML-POSC can be solved in a similar way to ML-DSC [23,24]. In COSC, controller i can completely observe the state of the target system $x_t$ and the memories of the other controllers $z_t^j(j\ne i)$, as well as its own memory $z_t^i$ (Figure 2a) [13,14,40]. As a result, the control $u_t^i$ is determined based on the whole extended state $s_t:=(x_t,z_t^1,\dots ,z_t^N)$ as $u_t^i=u^i(t,s_t)$. From Pontryagin’s minimum principle on the probability density function space, the optimal control function of COSC is given by the following equation:

$$\begin{array}{c}\hfill u^{i\ast }(t,s)=\arg \underset{u^i}{\min }\mathcal{H}\left(t,s,(u^{-i\ast },u^i),w\right),\end{array}$$

(25)

where $w(t,s)$ is the solution of HJB Equation (24). This result is the same with Bellman’s dynamic programming principle on the state space. Thus, Pontryagin’s minimum principle on the probability density function space can be interpreted as the generalization of Bellman’s dynamic programming principle on the state space.

In ML-POSC, controller i can observe the memories of the other controllers $z_t^j(j\ne i)$ as well as its own memory $z_t^i$ (Figure 2b) [23,24]. As a result, the control $u_t^i$ is determined based on the joint memory $z_t:=(z_t^1,\dots ,z_t^N)$ as $u_t^i=u^i(t,z_t)$. From Pontryagin’s minimum principle on the probability density function space, the optimal control function of ML-POSC is given by the following equation:

$$\begin{array}{c}\hfill u^{i\ast }(t,z)=\arg \underset{u^i}{\min }{\mathbb{E}}_{p_t\left(x\right|z)}\left[H\left(t,s,(u^{-i\ast },u^i),w\right)\right],\end{array}$$

(26)

where $w(t,s)$ is the solution of HJB Equation (24). $p_t\left(x\right|z):=p_t\left(s\right)/\int p_t\left(s\right)dx$ is the conditional probability density function of x given z, and $p_t\left(s\right)$ is the solution of FP Equation (16) driven by $u^{\ast }$.

Although HJB Equation (24) is the same for COSC, ML-POSC, and ML-DSC, the optimal control function is different. Notably, the optimal control functions of ML-POSC and ML-DSC depend on FP Equation (16) because they need to estimate unobservables from observables. ML-POSC needs to estimate the state of the system $x_t$ from the joint memory of all controllers $z_t$. ML-DSC needs to estimate the memories of the other controllers $z_t^j(j\ne i)$ as well as the state of the system $x_t$ from its own memory $z_t^i$.

In COSC, the optimal control function depends only on HJB Equation (24). As a result, it can be obtained by solving the HJB Equation (24) backward in time from the terminal condition, which is called the value iteration method [45,46,47]. By contrast, in ML-POSC and ML-DSC, the optimal control function cannot be obtained by the value iteration method because it depends on FP Equation (16) as well as HJB Equation (24). Instead, it can be obtained by the forward-backward sweep method, which computes the forward FP Equation (16) and the backward HJB Equation (24) alternately [24].

5. Linear-Quadratic-Gaussian Problem

In this section, we demonstrate how ML-DSC works by applying it to the LQG problem. In the conventional DSC, the LQG problem can be solved only when the information of the controllers is independent [8,9,10] or partially nested [17,18,19,20,21,22]. By contrast, in ML-DSC, the LQG problem can be solved without restricting the interaction among the controllers.

5.1. Problem Formulation

In this subsection, we formulate the LQG problem of ML-DSC. In this problem, the extended state SDE (12) is given as follows:

$$\begin{array}{cc}\hfill d{s}_t& =\left(A\left(t\right)s_t+B\left(t\right)u_t\right)dt+\sigma \left(t\right)d{\omega }_t\hfill \\ \hfill & =\left(A\left(t\right)s_t+\sum _{i=1}^N{B}_i\left(t\right)u_t^i\right)dt+\sigma \left(t\right)d{\omega }_t,\hfill \end{array}$$

(27)

where the initial condition is given by the Gaussian distribution $p_0\left(s\right):=\mathcal{N}\left(s\left|{\mu }_0,{\sum }_0\right.\right)$. Furthermore, we note that $\widetilde{·}$ is omitted for the notational simplicity. In ML-DSC, controller i determines the control $u_t^i$ based on the memory $z_t^i$ as follows:

$$\begin{array}{c}\hfill u_t^i=u^i(t,z_t^i).\end{array}$$

(28)

In the extended state SDE (27), the interaction among the controllers is not restricted. For example, each controller is allowed to control the memories of the other controllers from its own memory through the state or the observations. In this case, the memories of the controllers do not become independent or partially nested. It becomes obvious in the numerical experiments in Section 6.

The objective function (14) is given as follows:

$$\begin{array}{cc}\hfill J\left[u\right]& :={\mathbb{E}}_{p(s_{0:T};u)}\left[{\int }_0^T\left(s_t^{\top }Q\left(t\right)s_t+u_t^{\top }R\left(t\right)u_t\right)dt+s_T^{\top }P{s}_T\right]\hfill \\ \hfill & ={\mathbb{E}}_{p(s_{0:T};u)}\left[{\int }_0^T\left(s_t^{\top }Q\left(t\right)s_t+\sum _{i=1}^N\sum _{j=1}^N{\left(u_t^i\right)}^{\top }{R}_{ij}\left(t\right)u_t^j\right)dt+s_T^{\top }P{s}_T\right],\hfill \end{array}$$

(29)

where $Q\left(t\right)⪰O$, $R\left(t\right)\succ O$, and $P⪰O$. The objective of this problem is to find the optimal control function $u^{\ast }$ that minimizes the objective function $J\left[u\right]$:

$$\begin{array}{c}\hfill u^{\ast }:=\arg \underset{u}{\min }J\left[u\right].\end{array}$$

(30)

In this paper, we assume that $R\left(t\right)$ is the block diagonal matrix, i.e., the costs of different controllers are independent as follows:

$$\begin{array}{c}\hfill R\left(t\right)=\left(\begin{array}{cccc}{R}_{11}\left(t\right)& O& \dots & O\\ O& R_{22}\left(t\right)& \dots & O\\ ⋮& ⋮& \ddots & ⋮\\ O& O& \dots & R_{NN}\left(t\right)\end{array}\right),\end{array}$$

(31)

where $R_{ii}\left(t\right)\succ O$. In this case, the objective function (29) can be calculated as follows:

$$\begin{array}{cc}\hfill J\left[u\right]& ={\mathbb{E}}_{p(s_{0:T};u)}\left[{\int }_0^T\left(s_t^{\top }Q\left(t\right)s_t+\sum _{i=1}^N{\left(u_t^i\right)}^{\top }{R}_{ii}\left(t\right)u_t^i\right)dt+s_T^{\top }P{s}_T\right].\hfill \end{array}$$

(32)

If this assumption does not hold, the optimal control function cannot be derived explicitly. This problem is similar with the Witsenhausen’s counterexample, which demonstrates the difficulty of DSC and is historically recognized as an important problem [5,48,49]. However, this assumption is not critical in many applications as the control cost matrix is usually diagonal.

5.2. Derivation of Optimal Control Function

In this subsection, we derive the optimal control function of the LQG problem of ML-DSC by applying Theorem 1. In this problem, the probability density function of the extended state s at time t is given by the Gaussian distribution $p_t\left(s\right):=\mathcal{N}\left(s\left|\mu \right(t),\sum (t)\right)$, in which $\mu \left(t\right)$ is the mean vector and $\sum \left(t\right)$ is the covariance matrix. Defining the stochastic extended state $\widehat{s}:=s-\mu$, the conditional expectation ${\mathbb{E}}_{p_t\left(s^{-i}\right|z^i)}\left[s\right]$ can be calculated as follows:

$$\begin{array}{c}\hfill {\mathbb{E}}_{p_t\left(s^{-i}\right|z^i)}\left[s\right]=\mu \left(t\right)+K_i\left(t\right)\widehat{s},\end{array}$$

(33)

where $K_i\left(t\right)\in {\mathbb{R}}^{d_s\times d_s}$ is defined as follows:

$$\begin{array}{c}\hfill K_i\left(t\right):=\left(\begin{array}{ccccccc}O& \dots & O& {\sum }_{x{z}^i}\left(t\right){\sum }_{z^i{z}^i}^{-1}\left(t\right)& O& \dots & O\\ O& \dots & O& {\sum }_{z^1{z}^i}\left(t\right){\sum }_{z^i{z}^i}^{-1}\left(t\right)& O& \dots & O\\ ⋮& \ddots & ⋮& ⋮& ⋮& \ddots & ⋮\\ O& \dots & O& {\sum }_{z^{i-1}{z}^i}\left(t\right){\sum }_{z^i{z}^i}^{-1}\left(t\right)& O& \dots & O\\ O& \dots & O& I& O& \dots & O\\ O& \dots & O& {\sum }_{z^{i+1}{z}^i}\left(t\right){\sum }_{z^i{z}^i}^{-1}\left(t\right)& O& \dots & O\\ ⋮& \ddots & ⋮& ⋮& ⋮& \ddots & ⋮\\ O& \dots & O& {\sum }_{z^N{z}^i}\left(t\right){\sum }_{z^i{z}^i}^{-1}\left(t\right)& O& \dots & O\end{array}\right).\end{array}$$

(34)

Function $K_i\left(t\right)$ is the zero matrix except for the columns corresponding to $z^i$. By applying Theorem 1 to the LQG problem of ML-DSC, we obtained the following theorem:

Theorem 2.

In the LQG problem, the optimal control function of ML-DSC satisfies the following equation:

$$\begin{array}{c}\hfill u^{i\ast }(t,z^i)=-R_{ii}^{-1}{B}_i^{\top }\left(\mathrm{\Psi }\mu +\mathrm{\Phi }{K}_i\widehat{s}\right),\end{array}$$

(35)

where $K_i\left(t\right)$ is defined by Equation (34), which depends on $\sum \left(t\right)$. Functions $\mu \left(t\right)$ and $\sum \left(t\right)$ are the solutions of the following ordinary differential equations (ODEs):

$$\begin{array}{cc}\hfill \frac{d\mu }{dt}& =\left(A-B{R}^{-1}{B}^{\top }\mathrm{\Psi }\right)\mu ,\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill \frac{d\sum }{dt}& =\sigma {\sigma }^{\top }+\left(A-\sum _{i=1}^N{B}_i{R}_{ii}^{-1}{B}_i^{\top }\mathrm{\Phi }{K}_i\right)\sum +\sum {\left(A-\sum _{i=1}^N{B}_i{R}_{ii}^{-1}{B}_i^{\top }\mathrm{\Phi }{K}_i\right)}^{\top },\hfill \end{array}$$

(37)

where $\mu \left(0\right)={\mu }_0$ and $\sum \left(0\right)={\sum }_0$. Functions $\mathrm{\Psi }\left(t\right)$ and $\mathrm{\Phi }\left(t\right)$ are the solutions of the following ODEs:

$$\begin{array}{cc}\hfill -\frac{d\mathrm{\Psi }}{dt}& =Q+A^{\top }\mathrm{\Psi }+\mathrm{\Psi }A-\mathrm{\Psi }B{R}^{-1}{B}^{\top }\mathrm{\Psi },\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill -\frac{d\mathrm{\Phi }}{dt}& =Q+A^{\top }\mathrm{\Phi }+\mathrm{\Phi }A-\mathrm{\Phi }B{R}^{-1}{B}^{\top }\mathrm{\Phi }+\sum _{i=1}^N{(I-K_i)}^{\top }\mathrm{\Phi }{B}_i{R}_{ii}^{-1}{B}_i^{\top }\mathrm{\Phi }(I-K_i),\hfill \end{array}$$

(39)

where $\mathrm{\Psi }\left(T\right)=\mathrm{\Phi }\left(T\right)=P$.

Proof. 

The proof is shown in Appendix B. □

In the LQG problem of ML-DSC, FP Equation (16) is reduced to Equations (36) and (37), and HJB Equation (24) is reduced to Equations (38) and (39). The optimal control function (35) is decomposed into the deterministic part and the stochastic part, which correspond to the first term and the second term, respectively. The first term of the optimal control function (35) also appears in the linear-quadratic (LQ) problem of deterministic control, and Equation (38) is called the Riccati equation [14,40]. In contrast, the second term of the optimal control function (35) is new in the LQG problem of ML-DSC, and Equation (39) is called the decentralized Riccati equation in this paper.

5.3. Comparison with Completely Observable Stochastic Control or Memory-Limited Partially Observable Stochastic Control

In the LQG problem, the optimal control function of COSC is given as follows [14,40]:

$$\begin{array}{c}\hfill u^{i\ast }(t,s)=-R_{ii}^{-1}{B}_i^{\top }\left(\mathrm{\Psi }\mu +\mathrm{\Psi }\widehat{s}\right),\end{array}$$

(40)

where $\mathrm{\Psi }\left(t\right)$ is the solution of the Riccati Equation (38). Function $\mathrm{\Psi }\left(t\right)$ appears in the second term and the first term in COSC. In addition, the optimal control function of ML-POSC is given as follows [23,24]:

$$\begin{array}{c}\hfill u^{i\ast }(t,z)=-R_{ii}^{-1}{B}_i^{\top }\left(\mathrm{\Psi }\mu +\Pi K\widehat{s}\right),\end{array}$$

(41)

where $\Pi \left(t\right)$ is the solution of the following partially observable Riccati equation:

$$\begin{array}{cc}\hfill -\frac{d\Pi }{dt}& =Q+A^{\top }\Pi +\Pi A-\Pi B{R}^{-1}{B}^{\top }\Pi +{(I-K)}^{\top }\Pi B{R}^{-1}{B}^{\top }\Pi (I-K),\hfill \end{array}$$

(42)

where $\Pi \left(T\right)=P$ and $K\left(t\right)$ is defined by

$$\begin{array}{c}\hfill K\left(t\right):=\left(\begin{array}{cc}O& {\sum }_{xz}\left(t\right){\sum }_{zz}^{-1}\left(t\right)\\ O& I\end{array}\right).\end{array}$$

(43)

Thus, the decentralized Riccati Equation (39) is a natural extension of the partially observable Riccati Equation (42) from a centralized problem to a decentralized problem.

While the first deterministic term of the optimal control function is the same for COSC (40), ML-POSC (41), and ML-DSC (35), the second stochastic term is different, reflecting the fact that the observable part of the stochastic extended state is different.

5.4. Decentralized Riccati Equation

In this subsection, we analyze the decentralized Riccati Equation (39) by comparing it with the Riccati Equation (38). The Riccati Equation (38) is the control gain matrix of deterministic control and COSC. While the Riccati Equation (38) optimizes control, it does not improve estimation, because estimation is not needed in deterministic control and COSC. In contrast, the decentralized Riccati Equation (39) may improve estimation as well as control because the controllers of ML-DSC need to estimate the state of the system and the memories of the other controllers from their own memories.

In order to support this discussion, we analyze the last term of the decentralized Riccati Equation (39), which is denoted as follows:

$$\begin{array}{c}\hfill {\mathcal{Q}}_i:={(I-K_i)}^{\top }\mathrm{\Phi }{B}_i{R}_{ii}^{-1}{B}_i^{\top }\mathrm{\Phi }(I-K_i).\end{array}$$

(44)

This term is the main difference between the Riccati Equation (38) and the decentralized Riccati Equation (39), and thus accounts for the contribution of estimation in ML-DSC. For the sake of simplicity, we focused on ${\mathcal{Q}}_N$. A similar discussion is possible for ${\mathcal{Q}}_i$ by the permutation of the controllers’ indices. We also denoted $a:=s^{-N}$ and $b:=z^N$ for notational simplicity. a is unobservable and b is observable for the controller N. ${\mathcal{Q}}_N$ can be calculated as follows:

$$\begin{array}{c}\hfill {\mathcal{Q}}_N=\left(\begin{array}{cc}{\mathcal{P}}_{aa}& -{\mathcal{P}}_{aa}{\sum }_{ab}{\sum }_{bb}^{-1}\\ -{\sum }_{bb}^{-1}{\sum }_{ba}{\mathcal{P}}_{aa}& {\sum }_{bb}^{-1}{\sum }_{ba}{\mathcal{P}}_{aa}{\sum }_{ab}{\sum }_{bb}^{-1}\end{array}\right),\end{array}$$

(45)

where ${\mathcal{P}}_{aa}:={\left(\mathrm{\Phi }{B}_N{R}_{NN}^{-1}{B}_N^{\top }\mathrm{\Phi }\right)}_{aa}$. Because ${\mathcal{P}}_{aa}⪰O$ and ${\sum }_{bb}^{-1}{\sum }_{ba}{\mathcal{P}}_{aa}{\sum }_{ab}{\sum }_{bb}^{-1}⪰O$, ${\mathrm{\Phi }}_{aa}$ and ${\mathrm{\Phi }}_{bb}$ may be larger than ${\mathrm{\Psi }}_{aa}$ and ${\mathrm{\Psi }}_{bb}$, respectively. Because ${\mathrm{\Phi }}_{aa}$ and ${\mathrm{\Phi }}_{bb}$ are the negative feedback gains of a and b, respectively, ${\mathcal{Q}}_N$ may contribute to decreasing ${\sum }_{aa}$ and ${\sum }_{bb}$. Moreover, when ${\sum }_{ab}$ is positive/negative, ${\mathrm{\Phi }}_{ab}$ may be smaller/larger than ${\mathrm{\Psi }}_{ab}$, which may increase/decrease ${\sum }_{ab}$. The similar discussion is possible for ${\sum }_{ba}$, ${\mathrm{\Phi }}_{ba}$, and ${\mathrm{\Psi }}_{ba}$ because $\sum$, $\mathrm{\Phi }$, and $\mathrm{\Psi }$ are symmetric matrices. As a result, ${\mathcal{Q}}_N$ may contribute to decreasing the following conditional covariance matrix:

$$\begin{array}{c}\hfill {\sum }_{a|b}:={\sum }_{aa}-{\sum }_{ab}{\sum }_{bb}^{-1}{\sum }_{ba},\end{array}$$

(46)

which corresponds to the estimation error of the unobservable a from the observable b. It indicates that the decentralized Riccati Equation (39) may improve estimation as well as control.

It should be noted that estimation and control are not clearly separated in the LQG problem of ML-DSC. In the LQG problem of POSC, estimation and control are clearly separated, and they are optimized by the Kalman filter and the Riccati equation, respectively [11,14]. By contrast, in the LQG problem of ML-DSC, both estimation and control are optimized by the decentralized Riccati equation. This coupling of estimation and control also appears in the conventional DSC [17,18,19,20,21,22] and ML-POSC [23,24], which indicates that it may be caused by a decentralized structure and memory limitation.

6. Numerical Experiments

In this section, we demonstrate the significance of the decentralized Riccati Equation (39) by conducting the numerical experiments of two LQG problems in ML-DSC. One is the one-dimensional state case, and the other is the two-dimensional state case.

6.1. One-Dimensional State Case

In this subsection, we conduct a numerical experiment in one-dimensional state case (Figure 3a). In this case, we consider the state $x_t\in \mathbb{R}$, the observation $y_t^i\in \mathbb{R}$ and the memory $z_t^i\in \mathbb{R}$ of the controller $i\in \{1,2\}$, which evolve by the following SDEs:

$$\begin{array}{cc}\hfill d{x}_t& =\left(x_t+u_{x,t}^1+u_{x,t}^2\right)dt+d{\omega }_t,\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill d{y}_t^1& =\left(x_t+u_{y,t}^2\right)dt+d{\nu }_t^1,\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill d{y}_t^2& =\left(x_t+u_{y,t}^1\right)dt+d{\nu }_t^2,\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill d{z}_t^1& =v_t^{1}dt+d{y}_t^1,\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill d{z}_t^2& =v_t^{2}dt+d{y}_t^2.\hfill \end{array}$$

(51)

The initial conditions are given by the standard Gaussian distributions. ${\omega }_t\in \mathbb{R}$, ${\nu }_t^1\in \mathbb{R}$, and ${\nu }_t^2\in \mathbb{R}$ are the independent standard Wiener processes. $u_t^i:=(u_{x,t}^i,u_{y,t}^i)=u^i(t,z_t^i)\in {\mathbb{R}}^2$ and $v_t^i:=v^i(t,z_t^i)\in \mathbb{R}$ are the controls of controller i. Each controller can control the memory of the other controller through $u_{y,t}^i$, which can be interpreted as the communication among the controllers. The objective function to be minimized is given as follows:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{p(x_{0:T},y_{0:T},z_{0:T};u,v)}\left[{\int }_0^{10}\left(x_t^2+\sum _{i=1}^2\left({\left(u_{x,t}^i\right)}^2+{\left(u_{y,t}^i\right)}^2+{\left(v_t^i\right)}^2\right)\right)dt\right],\end{array}$$

(52)

where $u:=(u^1,u^2)$ and $v:=(v^1,v^2)$. Therefore, the objective of this problem is to minimize the state variance by small controls (Figure 3b).

In this problem, the information of the controllers is neither independent nor partially nested. The information of the controller 1’s memory $z_t^1$ propagates to the controller 2’s memory $z_t^2$ through the state control $u_{x,t}^1$ and the observation control $u_{y,t}^1$, and vice versa (Figure 3a). While such a problem cannot be solved by the conventional DSC, ML-DSC can address it.

Figure 3. Schematic diagram of the LQG problem in Section 6.1. (a) The state of the system $x_t$ is one-dimensional. (b) The two controllers control the state of the system to be close to 0.

Figure 3. Schematic diagram of the LQG problem in Section 6.1. (a) The state of the system $x_t$ is one-dimensional. (b) The two controllers control the state of the system to be close to 0.

In the representation using the extended state $s_t:=(x_t,z_t^1,z_t^2)\in {\mathbb{R}}^3$, the extended control ${\tilde{u}}_t^i:=(u_{x,t}^i,u_{y,t}^i,v_t^i)={\tilde{u}}^i(t,z_t^i)\in {\mathbb{R}}^3$, and the extended standard Wiener process ${\tilde{\omega }}_t:=({\omega }_t,{\nu }_t^1,{\nu }_t^2)\in {\mathbb{R}}^3$ as in Equations (27) and (32), the SDEs defined by Equations (47)–(51) can be described as follows:

$$d{s}_t=\left(\left(\begin{array}{ccc}1& 0& 0\\ 1& 0& 0\\ 1& 0& 0\end{array}\right)s_t+\left(\begin{array}{ccc}1& 0& 0\\ 0& 0& 1\\ 0& 1& 0\end{array}\right){\tilde{u}}_t^1+\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right){\tilde{u}}_t^2\right)dt+d{\tilde{\omega }}_t,$$

(53)

which corresponds to Equation (27). The objective function (52) can be rewritten as follows:

$$J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^{10}\left(s_t^{\top }\left(\begin{array}{ccc}1& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right)s_t+\sum _{i=1}^2{\left({\tilde{u}}_t^i\right)}^{\top }\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right){\tilde{u}}_t^i\right)dt\right],$$

(54)

which corresponds to Equation (32). In addition, it satisfies the block diagonal matrix assumption of $R\left(t\right)$ (31).

The Riccati Equation (38) can be solved backwards in time from the terminal condition. In contrast, the decentralized Riccati Equation (39) cannot be solved in a similar way to the Riccati Equation (38) because it depends on the covariance matrix $\sum \left(t\right)$ via the estimation gain matrix $K_i\left(t\right)$, which is the solution of the time-forward ODE (37). In order to solve the decentralized Riccati Equation (39), which is the time-backward ODE of $\mathrm{\Phi }\left(t\right)$, we use the forward-backward sweep method, which computes the time-forward ODE of $\sum \left(t\right)$ (37) and the time-backward ODE of $\mathrm{\Phi }\left(t\right)$ (39) alternately [24]. We note that the partially observable Riccati Equation (42) can also be solved by the forward-backward sweep method [24].

Figure 4 shows the trajectories of $\mathrm{\Psi }\left(t\right)$, $\Pi \left(t\right)$, and $\mathrm{\Phi }\left(t\right)$, which are the optimal control gain matrices of COSC, ML-POSC, and ML-DSC, respectively. Functions ${\mathrm{\Psi }}_{ab}\left(t\right)$, ${\Pi }_{ab}\left(t\right)$, and ${\mathrm{\Phi }}_{ab}\left(t\right)$ are the negative control gains from b to a. We noted that ${\mathrm{\Psi }}_{ab}\left(t\right)$, ${\Pi }_{ab}\left(t\right)$, and ${\mathrm{\Phi }}_{ab}\left(t\right)$ are also the negative control gains from a to b because $\mathrm{\Psi }\left(t\right)$, $\Pi \left(t\right)$, and $\mathrm{\Phi }\left(t\right)$ are symmetric matrices. While the elements of $\mathrm{\Psi }\left(t\right)$ related with the memories $z_1$ and $z_2$ are always 0, those of $\Pi \left(t\right)$ and $\mathrm{\Phi }\left(t\right)$ are not (Figure 4b–f). Thus, the controls of the memories do not appear in COSC, but appear in ML-POSC and ML-DSC. This result indicates that the controls of the memories play an important role in estimation.

We first compares $\mathrm{\Phi }$ with $\mathrm{\Psi }$ in more detail to investigate $\mathrm{\Phi }$. ${\mathrm{\Phi }}_{xx}$ and ${\mathrm{\Phi }}_{z^i{z}^i}$ are larger than ${\mathrm{\Psi }}_{xx}$ and ${\mathrm{\Psi }}_{z^i{z}^i}$ (Figure 4a,d,f), which may decrease ${\sum }_{xx}$ and ${\sum }_{z^i{z}^i}$. Moreover, ${\mathrm{\Phi }}_{x{z}^i}$ is smaller than ${\mathrm{\Psi }}_{x{z}^i}$ (Figure 4b,c), which may strengthen the positive correlation between x and $z^i$. Therefore, ${\mathrm{\Phi }}_{xx}$, ${\mathrm{\Phi }}_{z^i{z}^i}$, and ${\mathrm{\Phi }}_{x{z}^i}$ may improve estimation, which is consistent with our discussion in Section 5.4. However, ${\mathrm{\Phi }}_{z^1{z}^2}$ is larger than ${\mathrm{\Psi }}_{z^1{z}^2}$ (Figure 4e), which may weaken the positive correlation between $z^1$ and $z^2$. It seems to be contrary to our discussion because it may worsen estimation.

In order to understand ${\mathrm{\Phi }}_{z^1{z}^2}$, we also compared $\mathrm{\Phi }$ with $\Pi$. Estimation in ML-DSC is more challenging than that in ML-POSC because the controllers cannot completely share their information. Thus, except for ${\mathrm{\Phi }}_{z^1{z}^2}$, the absolute values of $\mathrm{\Phi }$ are larger than those of $\Pi$ (Figure 4a–d,f) for the same reason as the comparison with $\mathrm{\Psi }$. The problem is only ${\mathrm{\Phi }}_{z^1{z}^2}$ (Figure 4e). In ML-POSC, the estimation between $z^1$ and $z^2$ is not needed because the controllers share their information. As a result, ${\Pi }_{z^1{z}^2}$ is determined only from a control perspective, not an estimation perspective. ${\Pi }_{z^1{z}^2}$ is almost the same with ${\Pi }_{z^i{z}^i}$ (Figure 4d–f), presumably because the cooperative control by the controllers 1 and 2 is more efficient than the independent control. By contrast, in ML-DSC, the estimation between $z^1$ and $z^2$ is necessary because each controller cannot directly access the other controller. ${\mathrm{\Phi }}_{z^1{z}^2}$ is smaller than ${\Pi }_{z^1{z}^2}$ (Figure 4e), which may strengthen the positive correlation between $z^1$ and $z^2$. Therefore, ${\mathrm{\Phi }}_{z^1{z}^2}$ may be determined by a trade-off between control and estimation.

In order to clarify the significance of the decentralized Riccati Equation (39), we compared the performance of the optimal control function of ML-DSC (35) with that of the following control functions:

$$\begin{array}{cc}\hfill & u^{i,\mathrm{\Psi }}(t,z^i)=-R_{ii}^{-1}{B}_i^{\top }\left(\mathrm{\Psi }\mu +\mathrm{\Psi }{K}_i\widehat{s}\right),\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill & u^{i,\Pi }(t,z^i)=-R_{ii}^{-1}{B}_i^{\top }\left(\mathrm{\Psi }\mu +\Pi K_i\widehat{s}\right),\hfill \end{array}$$

(56)

which replaced $\mathrm{\Phi }$ with $\mathrm{\Psi }$ and $\Pi$, respectively. We noted that the first terms are not important because $\mu \left(t\right)=0$ is satisfied in this set-up. The result is shown in Figure 5. The variances of the state and the memories are $u^{\mathrm{\Psi }}$ > $u^{\Pi }$ > $u^{\ast }$ (Figure 5a–c). Similarly, the expected cumulative costs are $u^{\mathrm{\Psi }}$ > $u^{\Pi }$ > $u^{\ast }$ (Figure 5d). These orders are consistent with the extent of the optimization of estimation. $u^{\mathrm{\Psi }}$ does not take into account estimation at all, and its performance is the worst. $u^{\Pi }$ takes into account only the state estimation, and it performs better than $u^{\mathrm{\Psi }}$, but not optimally. $u^{\ast }$ takes into account the estimation of the other memories and the state, and its performance is optimal.

6.2. Two-Dimensional State Case

In this subsection, we conduct a numerical experiment in two-dimensional state case (Figure 6a). In this case, we formulated the target state $x_t^{\mathrm{tar}}:=(x_t^{\mathrm{tar},1},x_t^{\mathrm{tar},2})\in {\mathbb{R}}^2$, the actual state $x_t^{\mathrm{act}}:=(x_t^{\mathrm{act},1},x_t^{\mathrm{act},2})\in {\mathbb{R}}^2$, the observation $y_t:=(y_t^1,y_t^2)\in {\mathbb{R}}^2$, and the memory $z_t:=(z_t^1,z_t^2)\in {\mathbb{R}}^2$ as follows (Figure 6b):

$$\begin{array}{cc}\hfill d{x}_t^{\mathrm{tar},1}& =-2\pi x_t^{\mathrm{tar},2}dt,\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill d{x}_t^{\mathrm{tar},2}& =2\pi x_t^{\mathrm{tar},1}dt,\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill d{x}_t^{\mathrm{act},1}& =\left(-2\pi x_t^{\mathrm{act},2}+u_{x,t}^1\right)dt+d{\omega }_t^1,\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill d{x}_t^{\mathrm{act},2}& =\left(2\pi x_t^{\mathrm{act},1}+u_{x,t}^2\right)dt+d{\omega }_t^2,\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill d{y}_t^1& =\left(x_t^{\mathrm{act},1}-x_t^{\mathrm{tar},1}+u_{y,t}^2\right)dt+d{\nu }_t^1,\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill d{y}_t^2& =\left(x_t^{\mathrm{act},2}-x_t^{\mathrm{tar},2}+u_{y,t}^1\right)dt+d{\nu }_t^2,\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill d{z}_t^1& =v_t^{1}dt+d{y}_t^1,\hfill \end{array}$$

(63)

$$\begin{array}{cc}\hfill d{z}_t^2& =v_t^{2}dt+d{y}_t^2,\hfill \end{array}$$

(64)

where $x_0^{\mathrm{tar}}=(10,0)$, $x_0^{\mathrm{act}}\sim \mathcal{N}\left(x_0^{\mathrm{act}}\right|(10,0),I)$, $y_0\sim \mathcal{N}\left(y_0\right|0,I)$, and $z_0\sim \mathcal{N}\left(z_0\right|0,I)$. ${\omega }_t^i\in \mathbb{R}$ and ${\nu }_t^i\in \mathbb{R}$ are the independent standard Wiener processes. $u_t^i:=(u_{x,t}^i,u_{y,t}^i)=u^i(t,z_t^i)\in {\mathbb{R}}^2$ and $v_t^i:=v^i(t,z_t^i)\in \mathbb{R}$ are the controls of controller i. The objective function to be minimized is given as follows:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{u,v}\left[{\int }_0^{10}\sum _{i=1}^2\left(10{\left(x_t^{\mathrm{act},i}-x_t^{\mathrm{tar},i}\right)}^2+{\left(u_{x,t}^i\right)}^2+{\left(u_{y,t}^i\right)}^2+{\left(v_t^i\right)}^2\right)dt\right].\end{array}$$

(65)

The objective of this problem is to minimize the distance of the actual state $x_t^{\mathrm{act}}$ from the rotating target state $x_t^{\mathrm{tar}}$ by small controls. The solution of Equations (57) and (58) is given by $x_t^{\mathrm{tar}}=(x_t^{\mathrm{tar},1},x_t^{\mathrm{tar},2})=(10cos(2\pi t),10sin(2\pi t))$. If Equations (59) and (60) do not have the standard Wiener processes $d{\omega }_t^1$ and $d{\omega }_t^2$, respectively, $u_{x,t}^1=0$ and $u_{x,t}^2=0$ are optimal because $x_t^{\mathrm{tar}}=x_t^{\mathrm{act}}$ is satisfied. In practice, however, the actual state $x_t^{\mathrm{act}}$ does not coincide with the target state $x_t^{\mathrm{tar}}$ without the control $u_{x,t}$ due to the state noise $d{\omega }_t$, and needs to be controlled. Controller i observes and controls the actual state $x_t^{\mathrm{act}}$ only $x_t^{\mathrm{act},i}$-axis direction. As a result, the communication between the controllers is more important in this problem.

Figure 6. Schematic diagram of the LQG problem in Section 6.2. (a) The state of the system $x_t=(x_t^1,x_t^2)$ is two-dimensional. (b) The two controllers control the actual state $x_t^{\mathrm{act}}$ to be close to the target state $x_t^{\mathrm{tar}}=(10cos\left(2\pi t\right),10sin\left(2\pi t\right))$. Controller i observes and controls the actual state $x_t^{\mathrm{act}}$ only $x_t^{\mathrm{act},i}$-axis direction.

Figure 6. Schematic diagram of the LQG problem in Section 6.2. (a) The state of the system $x_t=(x_t^1,x_t^2)$ is two-dimensional. (b) The two controllers control the actual state $x_t^{\mathrm{act}}$ to be close to the target state $x_t^{\mathrm{tar}}=(10cos\left(2\pi t\right),10sin\left(2\pi t\right))$. Controller i observes and controls the actual state $x_t^{\mathrm{act}}$ only $x_t^{\mathrm{act},i}$-axis direction.

By defining the state $x_t:=x_t^{\mathrm{act}}-x_t^{\mathrm{tar}}$, Equations (57)–(64) are converted as follows:

$$\begin{array}{cc}\hfill d{x}_t^1& =\left(-2\pi x_t^2+u_{x,t}^1\right)dt+d{\omega }_t^1,\hfill \end{array}$$

(66)

$$\begin{array}{cc}\hfill d{x}_t^2& =\left(2\pi x_t^1+u_{x,t}^2\right)dt+d{\omega }_t^2,\hfill \end{array}$$

(67)

$$\begin{array}{cc}\hfill d{y}_t^1& =\left(x_t^1+u_{y,t}^2\right)dt+d{\nu }_t^1,\hfill \end{array}$$

(68)

$$\begin{array}{cc}\hfill d{y}_t^2& =\left(x_t^2+u_{y,t}^1\right)dt+d{\nu }_t^2,\hfill \end{array}$$

(69)

$$\begin{array}{cc}\hfill d{z}_t^1& =v_t^{1}dt+d{y}_t^1,\hfill \end{array}$$

(70)

$$\begin{array}{cc}\hfill d{z}_t^2& =v_t^{2}dt+d{y}_t^2,\hfill \end{array}$$

(71)

where the initial conditions are given by the standard Gaussian distributions. Furthermore, the objective function (65) is converted as follows:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{u,v}\left[{\int }_0^{10}\sum _{i=1}^2\left(10{\left(x_t^i\right)}^2+{\left(u_{x,t}^i\right)}^2+{\left(u_{y,t}^i\right)}^2+{\left(v_t^i\right)}^2\right)dt\right].\end{array}$$

(72)

As a result, the problem of controlling the actual state $x_t^{\mathrm{act}}$ to be close to the target state $x_t^{\mathrm{tar}}$ is equivalent to the problem of controlling the state $x_t$ to be close to 0.

In the representation using the extended state $s_t:=(x_t^1,x_t^2,z_t^1,z_t^2)\in {\mathbb{R}}^4$, the extended control ${\tilde{u}}_t^i:=(u_{x,t}^i,u_{y,t}^i,v_t^i)={\tilde{u}}^i(t,z_t^i)\in {\mathbb{R}}^3$, and the extended standard Wiener process ${\tilde{\omega }}_t:=({\omega }_t^1,{\omega }_t^2,{\nu }_t^1,{\nu }_t^2)\in {\mathbb{R}}^4$ as in Equations (27) and (32), the SDEs defined by Equations (66)–(71) can be described as follows:

$$d{s}_t=\left(\left(\begin{array}{cccc}0& -2\pi & 0& 0\\ 2\pi & 0& 0& 0\\ 1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right)s_t+\left(\begin{array}{ccc}1& 0& 0\\ 0& 0& 0\\ 0& 0& 1\\ 0& 1& 0\end{array}\right){\tilde{u}}_t^1+\left(\begin{array}{ccc}0& 0& 0\\ 1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right){\tilde{u}}_t^2\right)dt+d{\tilde{\omega }}_t,$$

(73)

which corresponds to Equation (27). The objective function (72) can be rewritten as follows:

$$J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^{10}\left(s_t^{\top }\left(\begin{array}{cccc}10& 0& 0& 0\\ 0& 10& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right)s_t+\sum _{i=1}^2{\left({\tilde{u}}_t^i\right)}^{\top }\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right){\tilde{u}}_t^i\right)dt\right],$$

(74)

which corresponds to Equation (32). In addition, it satisfies the block diagonal matrix assumption of $R\left(t\right)$ (31).

Figure 7 shows the trajectories of $\mathrm{\Psi }\left(t\right)$, $\Pi \left(t\right)$, and $\mathrm{\Phi }\left(t\right)$. Unlike $\mathrm{\Psi }\left(t\right)$, the elements of $\Pi \left(t\right)$ and $\mathrm{\Phi }\left(t\right)$ related with the memories $z_1$ and $z_2$ are not always 0 (Figure 7e–j), which indicates that the controls of the memories appear in ML-POSC and ML-DSC. The elements of $\mathrm{\Phi }\left(t\right)$ only largely deviate from those of $\Pi \left(t\right)$ in Figure 7g–j. Figure 7g,h show the negative feedback control gain of the memory $z^i$. ${\mathrm{\Phi }}_{z^i{z}^i}\left(t\right)$ is larger than ${\Pi }_{z^i{z}^i}\left(t\right)$, which indicates that the memories in ML-DSC are controlled more strongly than those in ML-POSC. Figure 7i shows the control gain between the state $x_t^1$ and the memory $z_t^2$. While the controller 1 can control the state $x_t^1$ based on the memory $z_t^2$ in ML-POSC, it cannot in ML-DSC, because the controllers do not share their memories in ML-DSC. Furthermore, while controller 1 does not need to send the information of the state $x_t^1$ to controller 2’s memory $z_t^2$, this is required in ML-DSC. As a result, ${\mathrm{\Phi }}_{x^1{z}^2}\left(t\right)$ differs greatly from ${\Pi }_{x^1{z}^2}\left(t\right)$. A similar discussion is possible for Figure 7j.

In order to clarify the significance of the decentralized Riccati Equation (39), we compared the performance of the optimal control function $u^{\ast }$ (35) with that of the control functions $u^{\mathrm{\Psi }}$ (55) and $u^{\Pi }$ (56). The result is shown in Figure 8. The actual state $x^{\mathrm{act}}$ faithfully tracks the target state $x^{\mathrm{tar}}$ under the optimal control function $u^{\ast }$ (Figure 8a–c (green)). Similarly, the memory z is stably controlled in the optimal control function $u^{\ast }$ (Figure 8d,e (green)). As a result, the performance of the optimal control function $u^{\ast }$ is optimal (Figure 8f (green)).

7. Discussion

In this paper, we proposed ML-DSC, which explicitly formulates the finite-dimensional memories of the controllers. In ML-DSC, each controller is designed to compress the infinite-dimensional observation histories appropriately into the finite-dimensional memory, and determines the optimal control based on it. As a result, ML-DSC can handle the difficulty in the conventional DSC that arises from the finiteness of actual memory of controllers. We demonstrated the effectiveness of ML-DSC in the LQG problem. While the conventional DSC needs to restrict the interaction among the controllers to solve the LQG problem, ML-DSC is free from such a restriction. We found that estimation and control are optimized by the decentralized Riccati equation in the LQG problem of ML-DSC. Our numerical experiments showed that the decentralized Riccati equation is superior to the Riccati equation and the partially observable Riccati equation in this problem.

ML-DSC can also address non-LQG problems. In DSC, the non-LQG problem cannot be solved numerically even if the number of the controllers is one, which corresponds to POSC [12,13]. This is because a functional differential equation needs to be solved in the non-LQG problem of POSC, which is generally intractable, even numerically. ML-POSC and ML-DSC are more tractable than the conventional POSC and DSC because only HJB-FP equations need to be solved, which are partial differential equations. The previous work showed that ML-POSC is more effective than the conventional POSC in a non-LQG problem [23]. Therefore, unlike the conventional DSC, ML-DSC may also be effective to non-LQG problems.

In order to solve ML-DSC with a large number of controllers, more efficient numerical algorithms are needed because HJB-FP equations become high-dimensional partial differential equations. In order to solve high-dimensional HJB-FP equations, neural network-based algorithms have been proposed in mean-field stochastic game and control [50,51]. Therefore, by exploiting these neural network-based algorithms, we may efficiently solve ML-DSC with a large number of controllers.