Next Article in Journal
Reversible Data Hiding in Encrypted Image Using Multiple Data-Hiders Sharing Algorithm
Previous Article in Journal
A Modified Multiplicative Thinning-Based INARCH Model: Properties, Saddlepoint Maximum Likelihood Estimation, and Application
Previous Article in Special Issue
H State-Feedback Control of Multi-Agent Systems with Data Packet Dropout in the Communication Channels: A Markovian Approach
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control

by
Takehiro Tottori
1,* and
Tetsuya J. Kobayashi
1,2,3,4
1
Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8654, Japan
2
Institute of Industrial Science, The University of Tokyo, Tokyo 153-8505, Japan
3
Department of Electrical Engineering and Information Systems, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8654, Japan
4
Universal Biology Institute, The University of Tokyo, Tokyo 113-8654, Japan
*
Author to whom correspondence should be addressed.
Entropy 2023, 25(2), 208; https://doi.org/10.3390/e25020208
Submission received: 5 November 2022 / Revised: 9 January 2023 / Accepted: 16 January 2023 / Published: 21 January 2023
(This article belongs to the Special Issue Information Theory in Control Systems)

Abstract

:
Memory-limited partially observable stochastic control (ML-POSC) is the stochastic optimal control problem under incomplete information and memory limitation. To obtain the optimal control function of ML-POSC, a system of the forward Fokker–Planck (FP) equation and the backward Hamilton–Jacobi–Bellman (HJB) equation needs to be solved. In this work, we first show that the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space. Based on this interpretation, we then propose the forward-backward sweep method (FBSM) for ML-POSC. FBSM is one of the most basic algorithms for Pontryagin’s minimum principle, which alternately computes the forward FP equation and the backward HJB equation in ML-POSC. Although the convergence of FBSM is generally not guaranteed in deterministic control and mean-field stochastic control, it is guaranteed in ML-POSC because the coupling of the HJB-FP equations is limited to the optimal control function in ML-POSC.

1. Introduction

In many practical applications of the stochastic optimal control theory, several constraints need to be considered. In the cases of small devices [1,2] and biological systems [3,4,5,6,7,8], for example, incomplete information and memory limitation become predominant because their sensors are extremely noisy and their memory resources are severely limited. To take into account one of these constraints, incomplete information, partially observable stochastic control (POSC) has been extensively studied in the stochastic optimal control theory [9,10,11,12,13]. However, because POSC cannot take into account the other constraint, memory limitation, it is not practical enough for designing memory-limited controllers for small devices and biological systems. To resolve this problem, memory-limited POSC (ML-POSC) has recently been proposed [14]. Because ML-POSC formulates noisy observation and limited memory explicitly, ML-POSC can take into account both incomplete information and memory limitation in the stochastic optimal control problem.
However, ML-POSC cannot be solved in a similar way as completely observable stochastic control (COSC), which is the most basic stochastic optimal control problem [15,16,17,18]. In COSC, the optimal control function depends only on the Hamilton–Jacobi–Bellman (HJB) equation, which is a time-backward partial differential equation given a terminal condition (Figure 1a) [15,16,17,18]. Therefore, the optimal control function of COSC can be obtained by solving the HJB equation backward in time from the terminal condition, which is called the value iteration method [19,20,21]. In contrast, the optimal control function of ML-POSC depends not only on the HJB equation but also on the Fokker–Planck (FP) equation, which is a time-forward partial differential equation given an initial condition (Figure 1b) [14]. Because the HJB equation and the FP equation interact with each other through the optimal control function in ML-POSC, the optimal control function of ML-POSC cannot be obtained by the value iteration method.
To propose an algorithm to solve ML-POSC, we first show that the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space. Pontryagin’s minimum principle is one of the most representative approaches to the deterministic optimal control problem, which converts it into the two-point boundary value problem of the forward state equation and the backward adjoint equation [22,23,24,25]. We formally show that the system of HJB-FP equations is an extension of the system of adjoint and state equations from the deterministic optimal control problem to the stochastic optimal control problem.
The system of HJB-FP equations also appears in the mean-field stochastic control (MFSC) [26,27,28]. Although the relationship between the system of HJB-FP equations and Pontryagin’s minimum principle has been briefly mentioned in MFSC [29,30,31], its details have not yet been investigated. In this work, we investigate it in more detail by deriving the system of HJB-FP equations in a similar way to Pontryagin’s minimum principle. We note that our derivations are formal, not analytical, and more mathematically rigorous proofs remain future challenges. However, our results are consistent with many conventional results and also provide a useful perspective in proposing an algorithm.
We then propose the forward-backward sweep method (FBSM) for ML-POSC. FBSM is an algorithm to compute the forward FP equation and the backward HJB equation alternately, which can be interpreted as an extension of the value iteration method. FBSM has been proposed in Pontryagin’s minimum principle of the deterministic optimal control problem, which computes the forward state equation and the backward adjoint equation alternately [32,33,34]. Because FBSM is easy to implement, it has been used in many applications [35,36]. However, the convergence of FBSM is not guaranteed in deterministic control except for special cases [37,38] because the coupling of adjoint and state equations is not limited to the optimal control function (Figure 1c). In contrast, we show that the convergence of FBSM is generally guaranteed in ML-POSC because the coupling of the HJB-FP equations is limited only to the optimal control function (Figure 1b).
FBSM is called the fixed-point iteration method in MFSC [39,40,41,42]. Although the fixed-point iteration method is the most basic algorithm to solve MFSC, its convergence is not guaranteed for the same reason as deterministic control (Figure 1d). Therefore, ML-POSC is a special and nice class of optimal control problems where FBSM or the fixed-point iteration method is guaranteed to converge.
This paper is organized as follows: In Section 2, we formulate ML-POSC. In Section 3, we derive the system of HJB-FP equations of ML-POSC from the viewpoint of Pontryagin’s minimum principle. In Section 4, we propose FBSM for ML-POSC and prove its convergence. In Section 5, we apply FBSM to the linear-quadratic-Gaussian (LQG) problem. In Section 6, we verify the convergence of FBSM by numerical experiments. In Section 7, we discuss our work. In Appendix A, we briefly review Pontryagin’s minimum principle of deterministic control. In Appendix B, we derive the system of HJB-FP equations of MFSC from the viewpoint of Pontryagin’s minimum principle. In Appendix C, we show the detailed derivations of our results.

2. Memory-Limited Partially Observable Stochastic Control

In this section, we briefly review the formulation of ML-POSC [14], which is the stochastic optimal control problem under incomplete information and memory limitation.

2.1. Problem Formulation

This subsection outlines the formulation of ML-POSC [14]. The state of the system x t R d x at time t [ 0 , T ] evolves by the following stochastic differential equation (SDE):
d x t = b ( t , x t , u t ) d t + σ ( t , x t , u t ) d ω t ,
where x 0 obeys p 0 ( x 0 ) , u t R d u is the control, and ω t R d ω is the standard Wiener process. In COSC [15,16,17,18], because the controller can completely observe the state x t , it determines the control u t based on the state x t as u t = u ( t , x t ) . By contrast, in POSC [9,10,11,12,13] and ML-POSC [14], the controller cannot directly observe the state x t and instead obtains the observation y t R d y , which evolves by the following SDE:
d y t = h ( t , x t ) d t + γ ( t ) d ν t ,
where y 0 obeys p 0 ( y 0 ) , and ν t R d ν is the standard Wiener process. In POSC [9,10,11,12,13], because the controller can completely memorize the observation history y 0 : t : = { y τ | τ [ 0 , t ] } , it determines the control u t based on the observation history y 0 : t as u t = u ( t , y 0 : t ) . In ML-POSC [14], by contrast, because the controller cannot completely memorize the observation history y 0 : t , it compresses the observation history y 0 : t into the finite-dimensional memory z t R d z , which evolves by the following SDE:
d z t = c ( t , z t , v t ) d t + κ ( t , z t , v t ) d y t + η ( t , z t , v t ) d ξ t ,
where z 0 obeys p 0 ( z 0 ) , v t R d v is the control, and ξ t R d ξ is the standard Wiener process. The memory dimension d z is determined by the available memory size of the controller. In addition, the memory noise ξ t represents the intrinsic stochasticity of the memory to be used. Therefore, unlike the conventional POSC, ML-POSC can explicitly take into account the memory size and noise of the controller. Furthermore, because the memory dynamics (3) depends on the memory control v t , it can be optimized through the memory control v t , which is expected to realize the optimal compression of the observation history y 0 : t into the limited memory z t . In ML-POSC [14], the controller determines the state control u t and the memory control v t based on the memory z t as follows:
u t = u ( t , z t ) , v t = v ( t , z t ) .
The objective function of ML-POSC is given by the following expected cumulative cost function:
J [ u , v ] : = E p ( x 0 : T , y 0 : T , z 0 : T ; u , v ) 0 T f ( t , x t , u t , v t ) d t + g ( x T ) ,
where f is the cost function, g is the terminal cost function, p ( x 0 : T , y 0 : T , z 0 : T ; u , v ) is the probability of x 0 : T , y 0 : T , and z 0 : T given u and v as parameters, and E p [ · ] is the expectation with respect to the probability p. Because the cost function f depends on the memory control v t , ML-POSC can explicitly take into account the memory control cost, which is also impossible with the conventional POSC.
ML-POSC is the problem of finding the optimal state control function u * and the optimal memory control function v * that minimize the expected cumulative cost function J [ u , v ] as follows:
u * , v * : = arg min u , v J [ u , v ] .
ML-POSC first formulates the finite-dimensional and stochastic memory dynamics explicitly, then optimizes the memory control by considering the memory control cost. As a result, unlike the conventional POSC, ML-POSC is a practical framework for memory-limited controllers where the memory size, noise, and cost are imposed and non-negligible.
The previous work [14] has shown the validity and effectiveness of ML-POSC. In the LQG problem of conventional POSC, the observation history y 0 : T can be compressed into the Kalman filter without a loss of performance [10,18,43]. Because the Kalman filter is finite-dimensional, it can be interpreted as the finite-dimensional memory z t and discussed in terms of ML-POSC. The previous work [14] has proven that the optimal memory dynamics of ML-POSC become the Kalman filter in this problem, which indicates that ML-POSC is a consistent framework with the conventional POSC. Furthermore, the previous work [14] has demonstrated the effectiveness of ML-POSC in the LQG problem with memory limitation and in the non-LQG problem by numerical experiments.

2.2. Problem Reformulation

Although the formulation of ML-POSC in the previous subsection is intuitive, it is inconvenient for further mathematical investigations. To address this problem, we reformulate ML-POSC in this subsection. The formulation in this subsection is simpler and more general than that in the previous subsection.
First, we define an extended state s t as follows:
s t : = x t z t R d s ,
where d s = d x + d z . The extended state s t evolves by the following SDE:
d s t = b ˜ ( t , s t , u ˜ t ) d t + σ ˜ ( t , s t , u ˜ t ) d ω ˜ t ,
where s 0 obeys p 0 ( s 0 ) , u ˜ t R d u ˜ is the control, and ω ˜ t R d ω ˜ is the standard Wiener process. ML-POSC determines the control u ˜ t R d u ˜ based on the memory z t as follows:
u ˜ t = u ˜ ( t , z t ) .
The extended state SDE (8) includes the previous SDEs (1)–(3) as a special case because they can be represented as follows:
d s t = b ( t , x t , u t ) c ( t , z t , v t ) + κ ( t , z t , v t ) h ( t , x t ) d t + σ ( t , x t , u t ) O O O κ ( t , z t , v t ) γ ( t ) η ( t , z t , v t ) d ω t d ν t d ξ t ,
where p 0 ( s 0 ) = p 0 ( x 0 ) p 0 ( z 0 ) .
The objective function of ML-POSC is given by the following expected cumulative cost function:
J [ u ˜ ] : = E p ( s 0 : T ; u ˜ ) 0 T f ˜ ( t , s t , u ˜ t ) d t + g ˜ ( s T ) .
where f ˜ is the cost function and g ˜ is the terminal cost function. It is obvious that this objective function (11) is more general than that in the previous subsection (5).
ML-POSC is the problem of finding the optimal control function u ˜ * that minimizes the expected cumulative cost function J [ u ˜ ] as follows:
u ˜ * : = arg min u ˜ J [ u ˜ ] .
In the following sections, we mainly consider the formulation of this subsection because it is simpler and more general than that in the previous subsection. Moreover, we omit · ˜ for simplicity of notation.

3. Pontryagin’s Minimum Principle

If the control u t is determined based on the extended state s t as u t = u ( t , s t ) , ML-POSC is the same problem with COSC of the extended state, and its optimality conditions can be obtained in the conventional way [15,16,17,18]. In reality, however, because ML-POSC determines the control u t based only on the memory z t as u t = u ( t , z t ) , its optimality conditions cannot be obtained in a similar way as COSC. In the previous work [14], the optimality conditions of ML-POSC were obtained by employing a mathematical technique of MFSC [30,31].
In this section, we obtain the optimality conditions of ML-POSC by employing Pontryagin’s minimum principle [22,23,24,25] on the probability density function space (Figure 2 (bottom right)). The conventional approach in ML-POSC [14] and MFSC [30,31] can be interpreted as a conversion from Bellman’s dynamic programming principle (Figure 2 (top right)) to Pontryagin’s minimum principle (Figure 2 (bottom right)) on the probability density function space.
In Appendix A, we briefly review Pontryagin’s minimum principle in deterministic control (Figure 2 (left)). In this section, we obtain the optimality conditions of ML-POSC in a similar way as Appendix A (Figure 2 (right)). Furthermore, in Appendix B, we obtain the optimality conditions of MFSC in a similar way as Appendix A (Figure 2 (right)). MFSC is more general than ML-POSC except for the partial observability. In particular, the expected Hamiltonian is non-linear with respect to the probability density function in MFSC, while it is linear in ML-POSC.
Although our derivations are formal, not analytical, and more mathematically rigorous proofs remain future challenges, our results are consistent with the conventional results of COSC [15,16,17,18], ML-POSC [14], and MFSC [26,27,28,30,31], and also provide a useful perspective in proposing an algorithm.

3.1. Preliminary

In this subsection, we show a useful result in obtaining Pontryagin’s minimum principle. Given arbitrary control functions u and u , J [ u ] J [ u ] can be calculated as follows:
J [ u ] J [ u ] = 0 T E p ( t , s ) H ( t , s , u , w ) E p ( t , s ) H ( t , s , u , w ) d t ,
where H is the Hamiltonian, which is defined as follows:
H t , s , u , w : = f ( t , s , u ) + L u w ( t , s ) .
L u is the backward diffusion operator, which is defined as follows:
L u w ( t , s ) : = i = 1 d s b i ( t , s , u ) w ( t , s ) s i + 1 2 i , j = 1 d s D i j ( t , s , u ) 2 w ( t , s ) s i s j ,
where D ( t , s , u ) : = σ ( t , s , u ) σ ( t , s , u ) . w ( t , s ) is the solution of the following Hamilton–Jacobi–Bellman (HJB) equation driven by u :
w ( t , s ) t = H t , s , u , w ,
where w ( T , s ) = g ( s ) . p ( t , s ) is the solution of the following Fokker–Planck (FP) equation driven by u:
p ( t , s ) t = L u p ( t , s ) ,
where p ( 0 , s ) = p 0 ( s ) . L u is the forward diffusion operator, which is defined as follows:
L u p ( t , s ) : = i = 1 d s ( b i ( t , s , u ) p ( t , s ) ) s i + 1 2 i , j = 1 d s 2 ( D i j ( t , s , u ) p ( t , s ) ) s i s j .
L u is the conjugate of L u as follows:
w ( t , s ) L u p ( t , s ) d s = p ( t , s ) L u w ( t , s ) d s .
We derive Equation (13) in Appendix C.1.

3.2. Necessary Condition

In this subsection, we show the necessary condition of the optimal control function of ML-POSC. It corresponds to Pontryagin’s minimum principle on the probability density function space (Figure 2 (bottom right)). If u * is the optimal control function of ML-POSC (12), then the following equation is satisfied:
u * ( t , z ) = arg min u E p t * ( x | z ) H t , s , u , w * , a . s . t [ 0 , T ] , z R d z ,
where w * ( t , s ) is the solution of the following HJB equation driven by u * :
w * ( t , s ) t = H t , s , u * , w * ,
where w * ( T , s ) = g ( s ) . p t * ( x | z ) : = p * ( t , s ) / p * ( t , s ) d x is the conditional probability density function of state x given memory z, and p * ( t , s ) is the solution of the following FP equation driven by u * :
p * ( t , s ) t = L u * p * ( t , s ) ,
where p * ( 0 , s ) = p 0 ( s ) . We derive this result in Appendix C.2.
In deterministic control, Pontryagin’s minimum principle can be expressed by the derivatives of the Hamiltonian (Figure 2 (bottom left)). Similarly, the system of HJB-FP Equations (21) and (22) can be expressed by the variations of the expected Hamiltonian
H ¯ ( t , p , u , w ) : = E p ( s ) H t , s , u , w
as follows:
p * ( t , s ) t = δ H ¯ ( t , p * , u * , w * ) δ w ( s ) ,
w * ( t , s ) t = δ H ¯ ( t , p * , u * , w * ) δ p ( s ) ,
where p * ( 0 , s ) = p 0 ( s ) and w * ( T , s ) = g ( s ) (Figure 2 (bottom right)). Therefore, the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space.

3.3. Sufficient Condition

Pontryagin’s minimum principle (20) is only a necessary condition and generally not a sufficient condition. Pontryagin’s minimum principle (20) becomes a necessary and sufficient condition if the expected Hamiltonian H ¯ ( t , p , u , w ) is convex with respect to p and u. We obtain this result in Appendix C.3.

3.4. Relationship with Bellman’s Dynamic Programming Principle

From Bellman’s dynamic programming principle on the probability density function space (Figure 2 (top right)) [14], the optimal control function of ML-POSC is given by the following equation:
u * ( t , z , p ) = arg min u E p ( x | z ) H t , s , u , δ V * ( t , p ) δ p ( s ) ,
where V * ( t , p ) is the value function on the probability density function space, which is the solution of the following Bellman equation:
V * ( t , p ) t = E p ( s ) H t , s , u * , δ V * ( t , p ) δ p ( s ) ,
where V * ( T , p ) = E p ( s ) g ( s ) . More specifically, the optimal control function of ML-POSC is given by u * ( t , z ) = u * ( t , z , p * ) , where p * is the solution of the FP Equation (22).
Because the Bellman Equation (27) is a functional differential equation, it cannot be solved even numerically. To resolve this problem, the previous work [14] converted the Bellman Equation (27) into the HJB Equation (21) by defining
w * ( t , s ) : = δ V * ( t , p * ) δ p ( s ) ,
where p * is the solution of FP Equation (22). This approach can be interpreted as the conversion from Bellman’s dynamic programming principle (Figure 2 (top right)) to Pontryagin’s minimum principle (Figure 2 (bottom right)) on the probability density function space.

3.5. Relationship with Completely Observable Stochastic Control

In the COSC of the extended state, the control u t is determined based on the extended state s t as u t = u ( t , s t ) . Therefore, in the COSC of the extended state, Pontryagin’s minimum principle on the probability density function space is given by the following equation:
u * ( t , s ) = arg min u H t , s , u , w * , a . s . t [ 0 , T ] , s R d s ,
where w * ( t , s ) is the solution of the HJB Equation (21). Because this proof is almost identical to that of Section 3.2, it is omitted in this paper.
While the optimal control function of ML-POSC (20) depends on the FP equation and the HJB equation, the optimal control function of COSC (29) depends only on the HJB equation. From this nice property of COSC, Equation (29) is not only a necessary condition but also a sufficient condition without assuming the convexity of the expected Hamiltonian. We derive this result in Appendix C.4.
This result is consistent with the conventional result of COSC [15,16,17,18]. Unlike ML-POSC and MFSC, COSC can be solved by Bellman’s dynamic programming principle on the state space. In COSC, Pontryagin’s minimum principle on the probability density function space is equivalent to Bellman’s dynamic programming principle on the state space. Because Bellman’s dynamic programming principle on the state space is a necessary and sufficient condition, Pontryagin’s minimum principle on the probability density function space may also become a necessary and sufficient condition.

4. Forward-Backward Sweep Method

In this section, we propose FBSM for ML-POSC and then prove its convergence by employing the interpretation of the system of HJB-FP equations by Pontryagin’s minimum principle introduced in the previous section.

4.1. Forward-Backward Sweep Method

In this subsection, we propose FBSM for ML-POSC, which is summarized in Algorithm 1. FBSM is an algorithm to compute the forward FP equation and the backward HJB equation alternately. More specifically, in the initial step of FBSM, we initialize the control function u 0 : T d t 0 and obtain p 0 : T 0 by computing the FP equation forward in time from the initial condition. In the backward step, we obtain w 0 : T 1 by computing the HJB equation backward in time from the terminal condition and simultaneously update the control function from u 0 : T d t 0 to u 0 : T d t 1 by minimizing the conditional expected Hamiltonian. In the forward step, we obtain p 0 : T 2 by computing the FP equation forward in time from the initial condition and simultaneously update the control function from u 0 : T d t 1 to u 0 : T d t 2 by minimizing the conditional expected Hamiltonian. By iterating the backward and forward steps, the objective function of ML-POSC J [ u 0 : T d t k ] monotonically decreases and finally converges to the local minimum at which the control function of ML-POSC u 0 : T d t k satisfies Pontryagin’s minimum principle.
Pontryagin’s minimum principle is only a necessary condition of the optimal control function, not a sufficient condition. Therefore, the control function obtained by FBSM is not necessarily the global optimum except in the case where the expected Hamiltonian is convex. Nevertheless, the control function obtained by FBSM is expected to be superior to most control functions because it is locally optimal.
FBSM has been used in deterministic control [32,34,35,38] and MFSC [39,40,41,42]. However, the convergence of FBSM for these problems is not guaranteed because the backward dynamics depend on the forward dynamics even without the optimal control function (Figure 1c,d). In contrast, the convergence of FBSM is guaranteed in ML-POSC because the backward HJB equation does not depend on the forward FP equation without the optimal control function (Figure 1b). More specifically, in FBSM for ML-POSC, the objective function J [ u 0 : T d t k ] monotonically decreases and finally converges to Pontryagin’s minimum principle. In the following subsections, we prove this nice property of FBSM for ML-POSC.
Algorithm 1: Forward-Backward Sweep Method (FBSM)
  • //— Initial step —//
  • k 0
  • p 0 k ( s ) p 0 ( s )
  • for  t = 0 to T d t  do
  •    Initialize u t k ( z )
  •     p t + d t k ( s ) p t k ( s ) + L u t k p t k ( s ) d t
  • end for
  • while  J [ u 0 : T d t k ] do not converge do
  •    if k is even then
  •      //— Backward step —//
  •       w T k + 1 ( s ) g ( s )
  •      for  t = T d t to 0 do
  •          u t k + 1 ( z ) arg min u E p t k ( x | z ) H ( t , s , u , w t + d t k + 1 )
  •          w t k + 1 ( s ) w t + d t k + 1 ( s ) + H ( t , s , u t k + 1 , w t + d t k + 1 ) d t
  •      end for
  •    else
  •      //— Forward step —//
  •       p 0 k + 1 ( s ) p 0 ( s )
  •      for  t = 0 to T d t  do
  •          u t k + 1 ( z ) arg min u E p t k + 1 ( x | z ) H ( t , s , u , w t + d t k )
  •          p t + d t k + 1 ( s ) p t k + 1 ( s ) + L u t k + 1 p t k + 1 ( s ) d t
  •      end for
  •    end if
  •     k k + 1
  • end while
  • return  u 0 : T d t k

4.2. Preliminary

In this subsection, we show an important result in proving the convergence of FBSM for ML-POSC. We suppose that u 0 : t d t , t + d t : T d t : = { u 0 , . . . , u t d t , u t + d t , . . . , u T d t } is given and only u t is optimized as follows:
u t * : = arg min u t J [ u 0 : T d t ] .
In ML-POSC, u t * can be calculated as follows:
u t * ( z ) = arg min u t E p t ( x | z ) H t , s , u t , w t + d t , a . s . z R d z ,
where w t + d t ( s ) is the solution of the following time-discretized HJB equation driven by u t + d t : T d t :
w τ ( s ) = w τ + d t ( s ) + H τ , s , u τ , w τ + d t d t , τ { t + d t , . . . , T d t } ,
where w T ( s ) = g ( s ) . p t ( x | z ) : = p t ( s ) / p t ( s ) d x is the conditional probability density function of state x given memory z, and p t ( s ) is the solution of the following time-discretized FP equation driven by u 0 : t d t :
p τ + d t ( s ) = p τ ( s ) + L u τ p τ ( s ) d t , τ { 0 , . . . , t d t } ,
where p 0 ( s ) . Equation (31) is obtained by the similar way to Pontyragin’s minimum principle in Appendix C.5 and also by the time discretization method in Appendix C.6.
Importantly, w t + d t does not depend on u t in ML-POSC (Figure 3a) while λ t + d t and w t + d t depend on u t in deterministic control (Figure 3b) and MFSC (Figure 3c), respectively. Therefore, u t * can be obtained without modifying w t + d t in ML-POSC, which is essentially different from deterministic control and MFSC. From this nice property, the convergence of FBSM is guaranteed in ML-POSC.

4.3. Monotonicity

In FBSM for ML-POSC, the objective function is monotonically non-increasing with respect to the update of the control function at each time step. More specifically,
J [ u 0 : t d t k , u t : T d t k + 1 ] J [ u 0 : t k , u t + d t : T d t k + 1 ]
is satisfied in the backward step, and
J [ u 0 : t d t k + 1 , u t : T d t k ] J [ u 0 : t k + 1 , u t + d t : T d t k ]
is satisfied in the forward step. We prove this result in Appendix C.7. Furthermore, in FBSM for ML-POSC, the objective function is monotonically non-increasing with respect to the update of the control function at each iteration step as follows:
J [ u 0 : T d t k + 1 ] J [ u 0 : T d t k ] .
Equation (36) is obviously satisfied from Equations (34) and (35).

4.4. Convergence to Pontryagin’s Minimum Principle

We assume that J [ u 0 : T d t ] has a lower bound. From Equation (36), FBSM for ML-POSC is guaranteed to converge to the local minimum. Furthermore, we assume that if the candidate of u t k + 1 includes u t k , then set u t k + 1 at u t k . Under these assumptions, FBSM for ML-POSC converges to Pontryagin’s minimum principle (20). More specifically, if J [ u 0 : T d t k + 1 ] = J [ u 0 : T d t k ] holds, u 0 : T d t k + 1 satisfies Pontryagin’s minimum principle (20). We prove this result in Appendix C.8.
Therefore, unlike deterministic control and MFSC, in FBSM for ML-POSC, the objective function J [ u 0 : T d t k ] monotonically decreases and finally converges to the local minimum at which the control function u 0 : T d t k satisfies Pontryagin’s minimum principle (20).

5. Linear-Quadratic-Gaussian Problem

In this section, we apply FBSM to the LQG problem of ML-POSC [14]. In the LQG problem of ML-POSC, the system of HJB-FP equations is reduced from partial differential equations to ordinary differential equations.

5.1. Problem Formulation

In the LQG problem of ML-POSC, the extended state SDE (8) is given as follows [14]:
d s t = A ( t ) s t + B ( t ) u t d t + σ ( t ) d ω t ,
where s 0 obeys the Gaussian distribution p 0 ( s 0 ) : = N s 0 μ 0 , Λ 0 where μ 0 is the mean vector and Λ 0 is the precision matrix. The objective function (11) is given as follows:
J [ u ] : = E p ( s 0 : T ; u ) 0 T s t Q ( t ) s t + u t R ( t ) u t d t + s T P s T ,
where Q ( t ) O , R ( t ) O , and P O . The LQG problem of ML-POSC is the problem of finding the optimal control function u * that minimizes the objective function J [ u ] as follows:
u * : = arg min u J [ u ] .

5.2. Pontryagin’s Minimum Principle

In the LQG problem of ML-POSC, Pontryagin’s minimum principle (20) can be calculated as follows [14]:
u * ( t , z ) = R 1 B Π K ( Λ ) ( s μ ) + Ψ μ , a . s . t [ 0 , T ] , z R d z ,
where K ( Λ ) is defined as follows:
K ( Λ ) : = O Λ x x 1 Λ x z O I ,
where μ ( t ) and Λ ( t ) are the mean vector and the precision matrix of the extended state, respectively, which correspond to the solution of the FP Equation (22). We note that E p t ( z | x ) s = K ( Λ ) ( s μ ) + μ is satisfied. μ ( t ) and Λ ( t ) are the solutions of the following ordinary differential equations (ODEs):
μ ˙ = A B R 1 B Ψ μ ,
Λ ˙ = A B R 1 B Π K ( Λ ) Λ Λ A B R 1 B Π K ( Λ ) Λ σ σ Λ ,
where μ ( 0 ) = μ 0 and Λ ( 0 ) = Λ 0 . Ψ ( t ) and Π ( t ) are the control gain matrices of the deterministic and stochastic extended state, respectively, which correspond to the solution of the HJB Equation (21). Ψ ( t ) and Π ( t ) are the solutions of the following ODEs:
Ψ ˙ = Q + A Ψ + Ψ A Ψ B R 1 B Ψ ,
Π ˙ = Q + A Π + Π A Π B R 1 B Π + ( I K ( Λ ) ) Π B R 1 B Π ( I K ( Λ ) ) ,
where Ψ ( T ) = Π ( T ) = P . The ODE of Ψ (44) is the Riccati equation [16,17,18], which also appears in the LQG problem of COSC. In contrast, the ODE of Π (45) is the partially observable Riccati equation [14], which appears only in the LQG problem of ML-POSC. The above result is obtained in [14].
The ODE of Ψ (44) can be solved backward in time from the terminal condition. Using Ψ , the ODE of μ (42) can be solved forward in time from the initial condition. In contrast, the ODEs of Π (45) and Λ (43) cannot be solved in a similar way as the ODEs of Ψ (44) and μ (42) because they interact with each other, which is a similar problem to the system of HJB-FP equations.

5.3. Forward-Backward Sweep Method

In the LQG problem of ML-POSC, FBSM is reduced from Algorithm 1 to Algorithm 2. F ( Λ , Π ) and G ( Λ , Π ) are defined by the right-hand sides of the ODEs of Λ (43) and Π (45), respectively, as follows:
F ( Λ , Π ) : = A B R 1 B Π K ( Λ ) Λ Λ A B R 1 B Π K ( Λ ) Λ σ σ Λ , G ( Λ , Π ) : = Q + A Π + Π A Π B R 1 B Π + ( I K ( Λ ) ) Π B R 1 B Π ( I K ( Λ ) ) .
This result is obtained in Appendix C.9. Importantly, in the LQG problem of ML-POSC, FBSM computes the ODEs of Λ (43) and Π (45) instead of the FP Equation (22) and the HJB Equation (21).
Algorithm 2: Forward-Backward Sweep Method (FBSM) in the LQG problem
  • //— Initial step —//
  • k 0
  • Λ 0 k Λ 0
  • for  t = 0 to T d t  do
  •    Initialize Π t + d t k
  •     Λ t + d t k Λ t k + F ( Λ t k , Π t + d t k ) d t
  • end for
  • while  J [ u 0 : T d t k ] do not converge do
  •    if k is even then
  •      //— Backward step —//
  •       Π T k + 1 P
  •      for  t = T d t to 0 do
  •          Π t k + 1 Π t + d t k + 1 + G ( Λ t k , Π t + d t k + 1 ) d t
  •      end for
  •    else
  •      //— Forward step —//
  •       Λ 0 k + 1 Λ 0
  •      for  t = 0 to T d t  do
  •          Λ t + d t k + 1 Λ t k + 1 + F ( Λ t k + 1 , Π t + d t k ) d t
  •      end for
  •    end if
  •     k k + 1
  • end while
  • return  u 0 : T d t k

6. Numerical Experiments

In this section, we verify the convergence of FBSM in ML-POSC by performing numerical experiments on the LQG and non-LQG problems. The setting of the numerical experiments is the same as the previous work [14].

6.1. LQG Problem

In this subsection, we verify the convergence of FBSM for ML-POSC by conducting a numerical experiment on the LQG problem. We consider state x t R , observation y t R , and memory z t R , which evolve by the following SDEs:
d x t = x t + u t d t + d ω t ,
d y t = x t d t + d ν t ,
d z t = v t d t + d y t ,
where x 0 and z 0 obey the standard Gaussian distributions, y 0 is an arbitrary real number, ω t R and ν t R are independent standard Wiener processes, and u t = u ( t , z t ) R and v t = v ( t , z t ) R are the controls. The objective function to be minimized is given as follows:
J [ u , v ] : = E p ( x 0 : 10 , y 0 : 10 , z 0 : 10 ; u , v ) 0 10 x t 2 + u t 2 + v t 2 d t .
Therefore, the objective of this problem is to minimize the state variance with small state and memory controls.
This problem corresponds to the LQG problem, which is defined by (37) and (38). By defining s t : = ( x t , z t ) R 2 , u ˜ t : = ( u t , v t ) R 2 , and ω ˜ t : = ( ω t , ν t ) R 2 , the SDEs (46)–(48) can be rewritten as follows:
d s t = 1 0 1 0 s t + u ˜ t d t + d ω ˜ t ,
which corresponds to (37). Furthermore, the objective function (49) can be rewritten as follows:
J [ u ˜ ] : = E p ( s 0 : 10 ; u ˜ ) 0 10 s t 1 0 0 0 s t + u ˜ t u ˜ t d t ,
which corresponds to (38).
We apply the FBSM of the LQG problem (Algorithm 2) to this problem. Π 0 ( t ) is initialized by Π 0 ( t ) = O . To solve the ODEs of Π k ( t ) and Λ k ( t ) , we use the fourth-order Runge–Kutta method. Figure 4 shows the control gain matrix Π k ( t ) R 2 × 2 and the precision matrix Λ k ( t ) R 2 × 2 obtained by FBSM. The color of each curve represents the iteration k. The darkest curve corresponds to the first iteration k = 0 , and the brightest curve corresponds to the last iteration k = 50 . Importantly, Π k ( t ) and Λ k ( t ) converge with respect to the iteration k.
Figure 5a shows the objective function J [ u k ] with respect to iteration k. The objective function J [ u k ] monotonically decreases with respect to iteration k, which is consistent with Section 4.3. This monotonicity of FBSM is the nice property of ML-POSC that is not guaranteed in deterministic control and MFSC. The objective function J [ u k ] finally converges, and u k satisfies Pontryagin’s minimum principle from Section 4.4.
Figure 5b–d compare the performance of the control function u k at the first iteration k = 0 and the last iteration k = 50 by performing a stochastic simulation. At the first iteration k = 0 , the distributions of state and memory are unstable, and the cumulative cost diverges. In contrast, at the last iteration k = 50 , the distributions of state and memory are stabilized and the cumulative cost is smaller. This result indicates that FBSM improves the performance in ML-POSC.
Although Figure 5b–d look similar to Figure 2d–f in the previous work [14], they are comparing different things. While Figure 5b–d demonstrate the performance improvement by the FBSM iteration, the previous work [14] compares the performance of the partially observable Riccati Equation (45) with that of the conventional Riccati Equation (44).

6.2. Non-LQG Problem

In this subsection, we verify the convergence of FBSM in ML-POSC by conducting a numerical experiment on the non-LQG problem. We consider state x t R , observation y t R , and memory z t R , which evolve by the following SDEs:
d x t = u t d t + d ω t ,
d y t = x t d t + d ν t ,
d z t = d y t ,
where x 0 and z 0 obey the Gaussian distributions p 0 ( x 0 ) = N ( x 0 | 0 , 0.01 ) and p 0 ( z 0 ) = N ( z 0 | 0 , 0.01 ) , respectively. y 0 is an arbitrary real number, ω t R and ν t R are independent standard Wiener processes, and u t = u ( t , z t ) R is the control. For the sake of simplicity, memory control is not considered. The objective function to be minimized is given as follows:
J [ u ] : = E p ( x 0 : 1 , y 0 : 1 , z 0 : 1 ; u ) 0 1 Q ( t , x t ) + u t 2 d t + 10 x 1 2 ,
where
Q ( t , x ) : = 1000 ( 0.3 t 0.6 , 0.1 | x | 2.0 ) , 0 ( o t h e r s ) .
The cost function is high in 0.3 t 0.6 and 0.1 | x | 2.0 , which represents the obstacles. In addition, the terminal cost function is the lowest at x = 0 , which represents the desirable goal. Therefore, the system should avoid the obstacles and reach the goal with a small control. Because the cost function is non-quadratic, it is a non-LQG problem.
We apply the FBSM (Algorithm 1) to this problem. u 0 ( t , z ) is initialized by u 0 ( t , z ) = 0 . To solve the HJB equation and the FP equation, we use the finite-difference method. Figure 6 shows w k ( t , s ) and p k ( t , s ) obtained by FBSM at the first iteration k = 0 and at the last iteration k = 50 . From Appendix C.6, w k ( t , s ) is given as follows:
w k ( t , s ) = E p ( s t + d t : 1 | s t = s ; u k ) t 1 Q ( τ , x τ ) + ( u τ k ) 2 d τ + 10 x 1 2 .
Because u 0 ( t , z ) = 0 , w 0 ( t , s ) reflects the cost function corresponding to the obstacles and the goal (Figure 6a–e). In contrast, because u 50 ( t , z ) 0 , w 50 ( t , s ) becomes more complex (Figure 6f–j). In particular, while w 0 ( t , s ) does not depend on memory z, w 50 ( t , s ) depends on memory z, which indicates that the control function u 50 ( t , z ) is adjusted by the memory z. We note that w 0 ( 1 , s ) (Figure 6e) and w 50 ( 1 , s ) (Figure 6j) are the same because they are given by the terminal cost function as w 0 ( 1 , s ) = w 50 ( 1 , s ) = 10 x 2 . Furthermore, while p 0 ( t , s ) is a unimodal distribution (Figure 6k–o), p 50 ( t , s ) is a bimodal distribution (Figure 6p–t), which can avoid the obstacles.
Figure 7a shows the objective function J [ u k ] with respect to iteration k. The objective function J [ u k ] monotonically decreases with respect to iteration k, which is consistent with Section 4.3. This monotonicity of FBSM is the nice property of ML-POSC that is not guaranteed in deterministic control and MFSC. The objective function J [ u k ] finally converges, and its u k satisfies Pontryagin’s minimum principle from Section 4.4.
Figure 7b,c compare the performance of the control function u k at the first iteration k = 0 and the last iteration k = 50 by conducting the stochastic simulation. At the first iteration k = 0 , the obstacles cannot be avoided, which results in a higher objective function. In contrast, at the last iteration k = 50 , the obstacles can be avoided, which results in a lower objective function. This result indicates that FBSM improves the performance in ML-POSC.
Although Figure 7b,c look similar to Figure 3a,b in the previous work [14], they are comparing different things. While Figure 7b,c demonstrate the performance improvement by the FBSM iteration, the previous work [14] compares the performance of ML-POSC with the local LQG approximation of the conventional POSC.

7. Discussion

In this work, we first showed that the system of HJB-FP equations corresponds to Pontryagin’s minimum principle on the probability density function space. Although the relationship between the system of HJB-FP equations and Pontryagin’s minimum principle has been briefly mentioned in MFSC [29,30,31], its details have not yet been investigated. We addressed this problem by deriving the system of HJB-FP equations in a similar way to Pontryagin’s minimum principle. We then proposed FBSM to ML-POSC. Although the convergence of FBSM is generally not guaranteed in deterministic control [32,34,35,38] and MFSC [39,40,41,42], we proved the convergence in ML-POSC by noting the fact that the update of the current control function does not affect the future HJB equation in ML-POSC. Therefore, ML-POSC is a special and nice class where FBSM is guaranteed to converge.
Our derivation of Pontryagin’s minimum principle on the probability density function space is formal, not analytical. Therefore, more mathematically rigorous proofs should be pursued in future work. Nevertheless, because our results are consistent with the conventional results of COSC [15,16,17,18], ML-POSC [14], and MFSC [26,27,28,30,31], they would be reliable except for special cases. Furthermore, our results provide a unified perspective on FBSM in deterministic control [32,34,35,38] and the fixed-point iteration method in MFSC [39,40,41,42], which have been studied independently. It clarifies the different properties of ML-POSC from deterministic control and MFSC, which ensures the convergence of FBSM.
The regularized FBSM has recently been proposed in deterministic control, which is guaranteed to converge even in the general deterministic control [44,45]. Our work gives an intuitive reason why the regularized FBSM is guaranteed to converge. In the regularized FBSM, the Hamiltonian is regularized, which makes the update of the control function smaller. When the regularization is sufficiently strong, the effect of the current control function on the future backward dynamics would be negligible. Therefore, the regularized FBSM of deterministic control would be guaranteed to converge for a similar reason to the FBSM of ML-POSC. However, the convergence of the regularized FBSM is much slower because the stronger regularization makes the update of the control function smaller. The FBSM of ML-POSC does not suffer from such a problem because the future backward dynamics already do not depend on the current control function without regularization.
Our work gives a hint about a modification of the fixed-point iteration method to ensure convergence in MFSC. Although the fixed-point iteration method is the most basic algorithm in MFSC, its convergence is not guaranteed [39,40,41,42]. Our work showed that the fixed-point iteration method is equivalent to the FBSM on the probability density function space. Therefore, the idea of regularized FBSM may also be applied to the fixed-point iteration method. More specifically, the fixed-point iteration method may be guaranteed to converge by regularizing the expected Hamiltonian.
In FBSM, we solve the HJB equation and the FP equation using the finite-difference method. However, because the finite-difference method is prone to the curse of dimensionality, it is difficult to solve high-dimensional ML-POSC. To resolve this problem, two directions can be considered. One direction is the policy iteration method [21,46,47]. Although the policy iteration method is almost the same as FBSM, only the update of the control function is different. While FBSM updates the system of HJB-FP equations and the control function simultaneously, the policy iteration method updates them separately. In the policy iteration method, the system of HJB-FP equations becomes linear, which can be solved by the sampling method [48,49,50]. Because the sampling method is more tractable than the finite-difference method, the policy iteration method may allow high-dimensional ML-POSC to be solved. Furthermore, the policy iteration method has recently been studied in MFSC [51,52,53]. However, its convergence is not guaranteed except for special cases in MFSC. In a similar way to FBSM, the convergence of the policy iteration method may be guaranteed in ML-POSC.
The other direction is machine learning. Neural network-based algorithms have recently been proposed in MFSC, which can solve high-dimensional problems efficiently [54,55]. By extending these algorithms, high-dimensional ML-POSC may be solved efficiently. Furthermore, unlike MFSC, the coupling of the HJB-FP equations is limited only to the optimal control function in ML-POSC. By exploiting this nice property, more efficient algorithms may be devised for ML-POSC.

Author Contributions

Conceptualization, Formal analysis, Funding acquisition, Writing—original draft, T.T. and T.J.K.; Software, Visualization, T.T. All authors have read and agreed to the published version of the manuscript.

Funding

The first author received a JSPS Research Fellowship (Grant No. 21J20436). This work was supported by JSPS KAKENHI (Grant No. 19H05799) and JST CREST (Grant No. JPMJCR2011).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
COSCCompletely Observable Stochastic Control
POSCPartially Observable Stochastic Control
ML-POSCMemory-Limited Partially Observable Stochastic Control
MFSCMean-Field Stochastic Control
FBSMForward-Backward Sweep Method
HJBHamilton-Jacobi-Bellman
FPFokker-Planck
SDEStochastic Differential Equation
ODEOrdinary Differential Equation
LQGLinear-Quadratic-Gaussian

Appendix A. Deterministic Control

In this section, we briefly review Pontryagin’s minimum principle in deterministic control [22,23,24,25].

Appendix A.1. Problem Formulation

In this subsection, we formulate deterministic control [22,23,24,25]. The state of the system s t R d s at time t [ 0 , T ] evolves according to the following ordinary differential equation (ODE):
d s t d t = b ( t , s t , u t ) ,
where the initial state is s 0 , and the control is u t = u ( t ) R d u . The objective function is given by the following cumulative cost function:
J [ u ] : = 0 T f ( t , s t , u t ) d t + g ( s T ) ,
where f is the cost function and g is the terminal cost function. Deterministic control is the problem of finding the optimal control function u * that minimizes the cumulative cost function J [ u ] as follows:
u * : = arg min u J [ u ] .

Appendix A.2. Preliminary

In this subsection, we show a useful result in deriving Pontryagin’s minimum principle. Given arbitrary control functions u and u , J [ u ] J [ u ] can be calculated as follows [16]:
J [ u ] J [ u ] = 0 T H ( t , s t , u t , λ t ) H ( t , s t , u t , λ t ) H ( t , s t , u t , λ t ) s ( s t s t ) d t + g ( s T ) g ( s T ) g ( s T ) s ( s T s T ) ,
where H is the Hamiltonian, which is defined as follows:
H t , s , u , λ : = f ( t , s , u ) + λ b ( t , s , u ) .
λ t is the solution of the following adjoint equation driven by u :
d λ t d t = H t , s t , u t , λ t s ,
where λ T = g ( s T ) / s . s t and s t are the solutions of the state Equation (A1) driven by u and u , respectively.
In the following, we derive Equation (A4). J [ u ] J [ u ] can be calculated as follows:
J [ u ] J [ u ] = 0 T f ( t , s t , u t ) d t + g ( s T ) 0 T f ( t , s t , u t ) d t + g ( s T ) = 0 T H ( t , s t , u t , λ t ) ( λ t ) b ( t , s t , u t ) d t + g ( s T ) 0 T H ( t , s t , u t , λ t ) ( λ t ) b ( t , s t , u t ) d t + g ( s T ) = 0 T H ( t , s t , u t , λ t ) H ( t , s t , u t , λ t ) d t 0 T ( λ t ) b ( t , s t , u t ) b ( t , s t , u t ) d t + g ( s T ) g ( s T ) .
From the state Equation (A1),
J [ u ] J [ u ] = 0 T H ( t , s t , u t , λ t ) H ( t , s t , u t , λ t ) d t 0 T ( λ t ) d ( s t s t ) d t d t + g ( s T ) g ( s T ) .
From the integration by parts and s 0 s 0 = 0 ,
J [ u ] J [ u ] = 0 T H ( t , s t , u t , λ t ) H ( t , s t , u t , λ t ) d t + 0 T d λ t d t ( s t s t ) d t + g ( s T ) g ( s T ) ( λ T ) ( s T s T ) .
From the adjoint Equation (A6), Equation (A4) is obtained.

Appendix A.3. Necessary Condition

In this subsection, we show the necessary condition of the optimal control function of deterministic control. It corresponds to Pontryagin’s minimum principle on the state space (Figure 2 (bottom left)). If u * is the optimal control function of deterministic control (A3), then the following equation is satisfied [16]:
u * ( t ) = arg min u H t , s t * , u , λ t * , t [ 0 , T ] ,
where λ t * is the solution of the following adjoint equation driven by u * :
d λ t * d t = H ( t , s t * , u t * , λ t * ) s ,
where λ T * = g ( s T * ) / s . s t * is the solution of the following state equation driven by u * :
d s t * d t = H ( t , s t * , u t * , λ t * ) λ ,
where s 0 * = s 0 . Because H ( t , s t * , u t * , λ t * ) / λ = b ( t , s t * , u t * ) , Equation (A12) is consistent with Equation (A1).
In the following, we show that Equation (A10) is the necessary condition of the optimal control function of deterministic control. We define the control function:
u ε ( t ) : = u * ( t ) t [ 0 , T ] \ E ε , u ( t ) t E ε ,
where E ε : = [ t , t + ε ] [ 0 , T ] , and u : [ 0 , T ] R d u . From Equation (A4), J [ u ε ] J [ u * ] can be calculated as follows:
J [ u ε ] J [ u * ] = 0 T H ( t , s t ε , u t ε , λ t * ) H ( t , s t * , u t * , λ t * ) H ( t , s t * , u t * , λ t * ) s ( s t ε s t * ) d t + g ( s T ε ) g ( s T * ) g ( s T * ) s ( s T ε s T * ) = 0 T H ( t , s t ε , u t * , λ t * ) H ( t , s t * , u t * , λ t * ) H ( t , s t * , u t * , λ t * ) s ( s t ε s t * ) d t + g ( s T ε ) g ( s T * ) g ( s T * ) s ( s T ε s T * ) + E ε H ( t , s t ε , u t , λ t * ) H ( t , s t ε , u t * , λ t * ) d t .
Letting ε 0 ,
J [ u ε ] J [ u * ] = 0 T H ( t , s t * , u t * , λ t * ) s ( s t ε s t * ) H ( t , s t * , u t * , λ t * ) s ( s t ε s t * ) d t + g ( s T * ) s ( s T ε s T * ) g ( s T * ) s ( s T ε s T * ) + H ( t , s t * , u t , λ t * ) H ( t , s t * , u t * , λ t * ) d t = H ( t , s t * , u t , λ t * ) H ( t , s t * , u t * , λ t * ) d t .
Because u * is the optimal control function, the following inequality is satisfied:
0 J [ u ε ] J [ u * ] = H ( t , s t * , u t , λ t * ) H ( t , s t * , u t * , λ t * ) d t .
Therefore, Equation (A10) is the necessary condition of the optimal control function of deterministic control.

Appendix A.4. Sufficient Condition

Pontryagin’s minimum principle (A10) is a necessary condition and generally not a sufficient condition. Pontryagin’s minimum principle (A10) becomes a necessary and sufficient condition if the Hamiltonian H ( t , s , u , λ ) is convex with respect to s and u and the terminal cost function g ( s ) is convex with respect to s.
In the following, we show this result. We define the arbitrary control function u : [ 0 , T ] R d u . From Equation (A4), J [ u ] J [ u * ] is given by the following equation:
J [ u ] J [ u * ] = 0 T H ( t , s t , u t , λ t * ) H ( t , s t * , u t * , λ t * ) H ( t , s t * , u t * , λ t * ) s ( s t s t * ) d t + g ( s T ) g ( s T * ) g ( s T * ) s ( s T s T * ) .
Since H ( t , s , u , λ ) is convex with respect to s and u and g ( s ) is convex with respect to s, the following inequalities are satisfied:
H ( t , s t , u t , λ t * ) H ( t , s t * , u t * , λ t * ) + H ( t , s t * , u t * , λ t * ) s ( s t s t * )
+ H ( t , s t * , u t * , λ t * ) u ( u t u t * ) ,
g ( s T ) g ( s T * ) + g ( s T * ) s ( s T s T * ) .
Hence, the following inequality is satisfied:
J [ u ] J [ u * ] 0 T H ( t , s t * , u t * , λ t * ) u ( u t u t * ) d t .
Because u * satisfies (A10), the following stationary condition is satisfied:
H ( t , s t * , u t * , λ t * ) u = 0 .
Hence, the following inequality is satisfied:
J [ u ] J [ u * ] 0 .
Therefore, Equation (A10) is the sufficient condition of the optimal control function of deterministic control if H ( t , s , u , λ ) is convex with respect to s and u and g ( s ) is convex with respect to s.

Appendix A.5. Relationship with Bellman’s Dynamic Programming Principle

From Bellman’s dynamic programming principle on the state space (Figure 2 (top left)) [16], the optimal control function of deterministic control is given by the following equation:
u * ( t , s ) = arg min u H t , s , u , w * ( t , s ) s ,
where w * ( t , s ) is the value function on the state space, which is the solution of the following Hamilton-Jacobi-Bellman (HJB) equation:
w * ( t , s ) t = H t , s , u * , w * ( t , s ) s ,
where w * ( T , s ) = g ( s ) . More specifically, the optimal control function of deterministic control is given by u * ( t ) = u * ( t , s t * ) , where s t * is the solution of the state Equation (A12).
The HJB Equation (A24) can be converted into the adjoint Equation (A11) by defining
λ t * : = w * ( t , s t * ) s ,
where s t * is the solution of the state Equation (A12). This approach can be interpreted as the conversion from Bellman’s dynamic programming principle (Figure 2 (top left)) to Pontryagin’s minimum principle (Figure 2 (bottom left)) on the state space.
In the following, we obtain this result. First, we define
Λ * ( t , s ) : = w * ( t , s ) s .
By differentiating the HJB Equation (A24) with respect to s, the following equation is obtained:
Λ * ( t , s ) t = H t , s , u * , Λ * s + Λ * ( t , s ) s b ( t , s * , u * ) ,
where Λ * ( T , s ) = g ( s ) / s . Then the derivative of λ t * = Λ * ( t , s t * ) with respect to t can be calculated as follows:
d λ t * d t = Λ * ( t , s t * ) t + Λ * ( t , s t * ) s d s t * d t .
By substituting Equation (A27) into Equation (A28), the following equation is obtained:
d λ t * d t = H t , s , u * , λ * s Λ * ( t , s t * ) s d s t * d t b ( t , s * , u * ) ( * ) .
From the state Equation (A12), ( * ) = 0 is satisfied. Therefore, λ * ( t ) satisfies the adjoint Equation (A11).

Appendix B. Mean-Field Stochastic Control

In this section, we show that the system of HJB-FP equations in MFSC corresponds to Pontryagin’s minimum principle on the probability density function space. Although the relationship between the system of HJB-FP equations and Pontryagin’s minimum principle has been mentioned briefly in MFSC [29,30,31], its details have not yet been investigated. In this section, we address this problem by deriving the system of HJB-FP equations in the similar way as Appendix A. Although our derivations are formal, not analytical, our results are consistent with the conventional results of MFSC [26,27,28,30,31].

Appendix B.1. Problem Formulation

In this subsection, we formulate MFSC [26,27,28]. The state of the system s t R d s at time t [ 0 , T ] evolves by the following stochastic differential equation (SDE):
d s t = b ( t , s t , p t , u t ) d t + σ ( t , s t , p t , u t ) d ω t ,
where s 0 obeys p 0 ( s 0 ) , p t ( s ) : = p ( t , s ) is the probability density function of the state s, u t ( s ) : = u ( t , s ) R d u is the control, and ω t R d ω is the standard Wiener process. The objective function is given by the following expected cumulative cost function:
J [ u ] : = E p ( s 0 : T ; u ) 0 T f ( t , s t , p t , u t ) d t + g ( s T , p T ) ,
where f is the cost function, g is the terminal cost function, p ( s 0 : T ; u ) is the probability of s 0 : t : = { s τ | τ [ 0 , t ] } given u as a parameter, and E p [ · ] is the expectation with respect to probability p. MFSC is the problem of finding the optimal control function u * that minimizes the expected cumulative cost function J [ u ] as follows:
u * : = arg min u J [ u ] .

Appendix B.2. Preliminary

In this subsection, we show a useful result in deriving Pontryagin’s minimum principle. Given arbitrary control functions u and u , J [ u ] J [ u ] can be calculated as follows:
J [ u ] J [ u ] = 0 T H ¯ ( t , p , u , w ) H ¯ ( t , p , u , w ) δ H ¯ t , p , u , w δ p ( s ) p ( t , s ) p ( t , s ) d s δ H ¯ t , p , u , w δ p ( s ) p ( t , s ) p ( t , s ) d s d t + g ¯ ( p ) g ¯ ( p ) δ g ¯ ( p ) δ p ( s ) p ( T , s ) p ( T , s ) d s ,
where H ¯ and g ¯ are the expected Hamiltonian and terminal cost function, respectively, which are defined as follows:
H ¯ ( t , p , u , w ) : = E p ( s ) H t , s , p , u , w ,
g ¯ ( p ) : = E p ( s ) g ( s , p ) .
H is the Hamiltonian, which is defined as follows:
H t , s , p , u , w : = f ( t , s , p , u ) + L u w ( t , s ) .
L u is the backward diffusion operator, which is defined as follows:
L u w ( t , s ) : = i = 1 d s b i ( t , s , p , u ) w ( t , s ) s i + 1 2 i , j = 1 d s D i j ( t , s , p , u ) 2 w ( t , s ) s i s j ,
where D ( t , s , p , u ) : = σ ( t , s , p , u ) σ ( t , s , p , u ) . w is the solution of the following Hamilton-Jacobi-Bellman (HJB) equation driven by u :
w ( t , s ) t = δ H ¯ t , p , u , w δ p ( s ) ,
where w ( T , s ) = ( δ g ¯ ( p ) / δ p ) ( s ) . p is the solution of the following Fokker-Planck (FP) equation driven by u:
p ( t , s ) t = L u p ( t , s ) ,
where p ( 0 , s ) = p 0 ( s ) . p is the solution of the FP Equation (A39) driven by u . L u is the forward diffusion operator, which is defined as follows:
L u p ( t , s ) : = i = 1 d s ( b i ( t , s , p , u ) p ( t , s ) ) s i + 1 2 i , j = 1 d s 2 ( D i j ( t , s , p , u ) p ( t , s ) ) s i s j .
L u is the conjugate of L u as follows:
w ( t , s ) L u p ( t , s ) d s = p ( t , s ) L u w ( t , s ) d s .
In the following, we derive Equation (A33). J [ u ] J [ u ] can be calculated as follows:
J [ u ] J [ u ] = E p ( s 0 : T ) 0 T f ( t , s t , p t , u t ) d t + g ( s T , p T ) E p ( s 0 : T ) 0 T f ( t , s t , p t , u t ) d t + g ( s T , p T ) = E p ( s 0 : T ) 0 T H ( t , s t , p t , u t , w ) L u t w ( t , s t ) d t + g ( s T , p T ) E p ( s 0 : T ) 0 T H ( t , s t , p t , u t , w ) L u t w ( t , s t ) d t + g ( s T , p T ) = 0 T H ¯ ( t , p , u , w ) H ¯ ( t , p , u , w ) d t 0 T E p ( t , s ) L u w ( t , s ) E p ( t , s ) L u w ( t , s ) d t + g ¯ ( p ) g ¯ ( p ) .
Because L u t and L u t are the conjugates of L u t and L u t , respectively,
J [ u ] J [ u ] = 0 T H ¯ ( t , p , u , w ) H ¯ ( t , p , u , w ) d t 0 T L u p ( t , s ) L u p ( t , s ) w ( t , s ) d s d t + g ¯ ( p ) g ¯ ( p ) .
From the FP Equation (A39),
J [ u ] J [ u ] = 0 T H ¯ ( t , p , u , w ) H ¯ ( t , p , u , w ) d t 0 T p ( t , s ) p ( t , s ) t w ( t , s ) d s d t + g ¯ ( p ) g ¯ ( p ) .
From the integration by parts and p ( 0 , s ) p ( 0 , s ) = p 0 ( s ) p 0 ( s ) = 0 ,
J [ u ] J [ u ] = 0 T H ¯ ( t , p , u , w ) H ¯ ( t , p , u , w ) d t + 0 T p ( t , s ) p ( t , s ) w ( t , s ) t d s d t + g ¯ ( p ) g ¯ ( p ) p ( T , s ) p ( T , s ) w ( T , s ) d s .
From the HJB Equation (A38), Equation (A33) is obtained.

Appendix B.3. Necessary Condition

In this subsection, we show the necessary condition of the optimal control function of MFSC. It corresponds to Pontryagin’s minimum principle on the probability density function space (Figure 2 (bottom right)). If u * is the optimal control function of MFSC (A32), then the following equation is satisfied:
u * ( t , s ) = arg min u H t , s , p * , u , w * , a . s . t [ 0 , T ] , s R d s ,
where w * is the solution of the following HJB equation driven by u * :
w * ( t , s ) t = δ H ¯ ( t , p * , u * , w * ) δ p ( s ) ,
where w * ( T , s ) = ( δ g ¯ ( p * ) / δ p ) ( s ) . p * is the solution of the following FP equation driven by u * :
p * ( t , s ) t = δ H ¯ ( t , p * , u * , w * ) δ w ( s ) ,
where p * ( 0 , s ) = p 0 ( s ) .
In the following, we show that Equation (A46) is the necessary condition of the optimal control function of MFSC. We define the control function
u ε ( t , z ) : = u * ( t , s ) ( t , s ) ( [ 0 , T ] × R d s ) \ ( E ε 1 × F ε 2 ) , u ( t , s ) ( t , s ) E ε 1 × F ε 2 ,
where E ε 1 : = [ t , t + ε 1 ] [ 0 , T ] , F ε 2 : = [ s , s + ε 2 ] R d s , and u : [ 0 , T ] × R d s R d u . From Equation (A33), J [ u ε ] J [ u * ] can be calculated as follows:
J [ u ε ] J [ u * ] = 0 T H ¯ ( t , p ε , u ε , w * ) H ¯ ( t , p * , u * , w * ) δ H ¯ t , p * , u * , w * δ p ( s ) p ε ( t , s ) p * ( t , s ) d s δ H ¯ t , p * , u * , w * δ p ( s ) p ε ( t , s ) p * ( t , s ) d s d t + g ¯ ( p ε ) g ¯ ( p * ) δ g ¯ ( p * ) δ p ( s ) p ε ( T , s ) p * ( T , s ) d s = 0 T H ¯ ( t , p ε , u * , w * ) H ¯ ( t , p * , u * , w * ) δ H ¯ t , p * , u * , w * δ p ( s ) p ε ( t , s ) p * ( t , s ) d s δ H ¯ t , p * , u * , w * δ p ( s ) p ε ( t , s ) p * ( t , s ) d s d t + g ¯ ( p ε ) g ¯ ( p * ) δ g ¯ ( p * ) δ p ( s ) p ε ( T , s ) p * ( T , s ) d s + E ε 1 F ε 2 H ( t , s , p ε , u , w * ) H ( t , s , p ε , u * , w * ) p ε ( t , s ) d s d t .
Letting ε 1 0 and ε 2 0 ,
J [ u ε ] J [ u * ] = 0 T δ H ¯ t , p * , u * , w * δ p ( s ) p ε ( t , s ) p * ( t , s ) d s δ H ¯ t , p * , u * , w * δ p ( s ) p ε ( t , s ) p * ( t , s ) d s d t + δ g ¯ ( p * ) δ p ( s ) p ε ( T , s ) p * ( T , s ) d s δ g ¯ ( p * ) δ p ( s ) p ε ( T , s ) p * ( T , s ) d s + H ( t , s , p * , u , w * ) H ( t , s , p * , u * , w * ) p * ( t , s ) d s d t = H ( t , s , p * , u , w * ) H ( t , s , p * , u * , w * ) p * ( t , s ) d s d t .
Because u * is the optimal control function, the following inequality is satisfied:
0 J [ u ε ] J [ u * ] = H ( t , s , p * , u , w * ) H ( t , s , p * , u * , w * ) p * ( t , s ) d s d t .
Therefore, Equation (A46) is the necessary condition of the optimal control function of MFSC.

Appendix B.4. Sufficient Condition

Pontryagin’s minimum principle (A46) is a necessary condition and generally not a sufficient condition. Pontryagin’s minimum principle (A46) becomes a necessary and sufficient condition if the expected Hamiltonian H ¯ ( t , p , u , w ) is convex with respect to p and u and the expected terminal cost function g ¯ ( p ) is convex with respect to p.
In the following, we show this result. We define the arbitrary control function u : [ 0 , T ] × R d s R d u . From Equation (A33), J [ u ] J [ u * ] is given by the following equation:
J [ u ] J [ u * ] = 0 T H ¯ ( t , p , u , w * ) H ¯ ( t , p * , u * , w * ) δ H ¯ t , p * , u * , w * δ p ( s ) p ( t , s ) p * ( t , s ) d s δ H ¯ t , p * , u * , w * δ p ( s ) p ( t , s ) p * ( t , s ) d s d t + g ¯ ( p ) g ¯ ( p * ) δ g ¯ ( p * ) δ p ( s ) p ( T , s ) p * ( T , s ) d s .
Because H ¯ ( t , p , u , w ) is convex with respect to p and u and g ¯ ( p ) is convex with respect to p, the following inequalities are satisfied:
H ¯ ( t , p , u , w * ) H ¯ ( t , p * , u * , w * ) + δ H ¯ ( t , p * , u * , w * ) δ p ( s ) ( p ( t , s ) p * ( t , s ) ) d s
+ δ H ¯ ( t , p * , u * , w * ) δ u ( s ) ( u ( t , s ) u * ( t , s ) ) d s ,
g ¯ ( p ) g ¯ ( p * ) + δ g ¯ ( p * ) δ p ( s ) ( p ( T , s ) p * ( T , s ) ) d s .
Hence, the following inequality is satisfied:
J [ u ] J [ u * ] 0 T E p * ( t , s ) H ( t , s , p * , u * , w * ) u ( u ( t , s ) u * ( t , s ) ) d t .
Because u * satisfies Equation (A46), the following stationary condition is satisfied:
H ( t , s , p * , u * , w * ) u = 0 .
Hence, the following inequality is satisfied:
J [ u ] J [ u * ] 0
Therefore, Equation (A46) is the sufficient condition of the optimal control function of MFSC if the expected Hamiltonian H ¯ ( t , p , u , w ) is convex with respect to p and u and the expected terminal cost function g ¯ ( p ) is convex with respect to p.

Appendix B.5. Relationship with Bellman’s Dynamic Programming Principle

From Bellman’s dynamic programming principle on the probability density function space (Figure 2 (top right)) [56,57,58], the optimal control function of MFSC is given by the following equation:
u * ( t , s , p ) = arg min u H t , s , p , u , δ V * ( t , p ) δ p ( s ) ,
where V * ( t , p ) is the value function on the probability density function space, which is the solution of the following Bellman equation:
V * ( t , p ) t = E p ( s ) H t , s , p , u * , δ V * ( t , p ) δ p ( s ) ,
where V * ( T , p ) = E p ( s ) g ( s ) . More specifically, the optimal control function of MFSC is given by u * ( t , s ) = u * ( t , s , p * ) , where p * is the solution of the FP Equation (A48).
Because the Bellman Equation (A60) is a functional differential equation, it cannot be solved even numerically. To resolve this problem, the previous works [30,31] converted the Bellman Equation (A60) into the HJB Equation (A47) by defining
w * ( t , s ) : = δ V * ( t , p * ) δ p ( s ) ,
where p * is the solution of FP Equation (A48). This approach can be interpreted as the conversion from Bellman’s dynamic programming principle (Figure 2 (top right)) to Pontryagin’s minimum principle (Figure 2 (bottom right)) on the probability density function space.

Appendix C. Derivation of Main Results

Appendix C.1. Derivation of Result in Section 3.1

In this subsection, we derive Equation (13). J [ u ] J [ u ] can be calculated as follows:
J [ u ] J [ u ] = E p ( s 0 : T ) 0 T f ( t , s t , u t ) d t + g ( s T ) E p ( s 0 : T ) 0 T f ( t , s t , u t ) d t + g ( s T ) = E p ( s 0 : T ) 0 T H ( t , s t , u t , w ) L u t w ( t , s t ) d t + g ( s T ) E p ( s 0 : T ) 0 T H ( t , s t , u t , w ) L u t w ( t , s t ) d t + g ( s T ) = 0 T E p ( t , s ) H ( t , s , u , w ) E p ( t , s ) H ( t , s , u , w ) d t 0 T E p ( t , s ) L u w ( t , s ) E p ( t , s ) L u w ( t , s ) d t + E p ( T , s ) g ( s ) E p ( T , s ) g ( s ) .
Because L u t and L u t are the conjugates of L u t and L u t , respectively,
J [ u ] J [ u ] = 0 T E p ( t , s ) H ( t , s , u , w ) E p ( t , s ) H ( t , s , u , w ) d t 0 T L u p ( t , s ) L u p ( t , s ) w ( t , s ) d s d t + E p ( T , s ) g ( s ) E p ( T , s ) g ( s ) .
From the FP Equation (17),
J [ u ] J [ u ] = 0 T E p ( t , s ) H ( t , s , u , w ) E p ( t , s ) H ( t , s , u , w ) d t 0 T p ( t , s ) p ( t , s ) t w ( t , s ) d s d t + E p ( T , s ) g ( s ) E p ( T , s ) g ( s ) .
From the integration by parts and p ( 0 , s ) p ( 0 , s ) = p 0 ( s ) p 0 ( s ) = 0 ,
J [ u ] J [ u ] = 0 T E p ( t , s ) H ( t , s , u , w ) E p ( t , s ) H ( t , s , u , w ) d t + 0 T p ( t , s ) p ( t , s ) w ( t , s ) t d s d t + E p ( T , s ) g ( s ) E p ( T , s ) g ( s ) p ( T , s ) p ( T , s ) w ( T , s ) d s .
From the HJB Equation (16), Equation (13) is obtained.

Appendix C.2. Derivation of Result in Section 3.2

In this subsection, we show that Equation (20) is the necessary condition of the optimal control function of ML-POSC. It corresponds to Pontryagin’s minimum principle on the probability density function space. We define the control function
u ε ( t , z ) : = u * ( t , z ) ( t , z ) ( [ 0 , T ] × R d z ) \ ( E ε 1 × F ε 2 ) , u ( t , z ) ( t , z ) E ε 1 × F ε 2 ,
where E ε 1 : = [ t , t + ε 1 ] [ 0 , T ] , F ε 2 : = [ z , z + ε 2 ] R d z , and u : [ 0 , T ] × R d z R d u . From Equation (13), J [ u ε ] J [ u * ] can be calculated as follows:
J [ u ε ] J [ u * ] = 0 T E p ε ( t , s ) H ( t , s , u ε , w * ) E p ε ( t , s ) H ( t , s , u * , w * ) d t = E ε 1 F ε 2 E p t ε ( x | z ) H ( t , s , u , w * ) E p t ε ( x | z ) H ( t , s , u * , w * ) p t ε ( z ) d z d t .
Letting ε 1 0 and ε 2 0 ,
J [ u ε ] J [ u * ] = E p t * ( x | z ) H ( t , s , u , w * ) E p t * ( x | z ) H ( t , s , u * , w * ) p t * ( z ) d z d t .
Because u * is the optimal control function, the following inequality is satisfied:
0 J [ u ε ] J [ u * ] = E p t * ( x | z ) H ( t , s , u , w * ) E p t * ( x | z ) H ( t , s , u * , w * ) p t * ( z ) d z d t .
Therefore, Equation (20) is the necessary condition of the optimal control function of ML-POSC.

Appendix C.3. Derivation of Result in Section 3.3

In this subsection, we show that Equation (20) is the sufficient condition of the optimal control function of ML-POSC if the expected Hamiltonian H ¯ ( t , p , u , w ) is convex with respect to p and u. We define the arbitrary control function u : [ 0 , T ] × R d z R d u . From Equation (13), J [ u ] J [ u * ] is given by the following equation:
J [ u ] J [ u * ] = 0 T E p ( t , s ) H ( t , s , u , w * ) E p ( t , s ) H ( t , s , u * , w * ) d t .
Because H ¯ ( t , p , u , w ) is convex with respect to p and u, the following inequality is satisfied:
E p ( t , s ) H ( t , s , u , w * ) = H ¯ ( t , p , u , w * ) H ¯ ( t , p * , u * , w * ) + δ H ¯ ( t , p * , u * , w * ) δ p ( s ) ( p ( t , s ) p * ( t , s ) ) d s + δ H ¯ ( t , p * , u * , w * ) δ u ( z ) ( u ( t , z ) u * ( t , z ) ) d z .
Because
δ H ¯ ( t , p * , u * , w * ) δ p ( s ) = δ δ p p ( s ) H ( t , s , u * , w * ) d s p = p *
= H ( t , s , u * , w * ) , δ H ¯ ( t , p * , u * , w * ) δ u ( z ) = δ δ u p t * ( z ) E p t * ( x | z ) H ( t , s , u , w * ) d z u = u *
= p t * ( z ) E p t * ( x | z ) H ( t , s , u * , w * ) u ,
the above inequality can be calculated as follows:
E p ( t , s ) H ( t , s , u , w * ) p * ( t , s ) H ( t , s , u * , w * ) d s + H ( t , s , u * , w * ) ( p ( t , s ) p * ( t , s ) ) d s + p t * ( z ) E p t * ( x | z ) H ( t , s , u * , w * ) u ( u ( t , z ) u * ( t , z ) ) d z = E p ( t , s ) H ( t , s , u * , w * ) + E p t * ( z ) E p t * ( x | z ) H ( t , s , u * , w * ) u ( u ( t , z ) u * ( t , z ) ) .
Hence, the following inequality is satisfied:
J [ u ] J [ u * ] 0 T E p t * ( z ) E p t * ( x | z ) H ( t , s , u * , w * ) u ( u ( t , z ) u * ( t , z ) ) d t .
Because u * satisfies Equation (20), the following stationary condition is satisfied:
E p t * ( x | z ) H ( t , s , u * , w * ) u = 0 .
Hence, the following inequality is satisfied:
J [ u ] J [ u * ] 0
Therefore, Equation (20) is the sufficient condition of the optimal control function of ML-POSC if H ¯ ( t , p , u , w ) is convex with respect to p and u.

Appendix C.4. Derivation of Result in Section 3.5

In this subsection, we show that Equation (29) is the sufficient condition of the optimal control function of COSC without assuming the convexity of the expected Hamiltonian. We define the arbitrary control function u : [ 0 , T ] × R d s R d u . From Equation (13), J [ u ] J [ u * ] is given by the following equation:
J [ u ] J [ u * ] = 0 T E p ( t , s ) H ( t , s , u , w * ) E p ( t , s ) H ( t , s , u * , w * ) d t .
From (29), the following inequality is satisfied:
J [ u ] J [ u * ] 0 T E p ( t , s ) H ( t , s , u * , w * ) E p ( t , s ) H ( t , s , u * , w * ) d t = 0 .
Therefore, Equation (29) is the sufficient condition of the optimal control function of COSC.

Appendix C.5. Derivation of Result in Section 4.2 by the Similar Way as Pontyragin’s Minimum Principle

In this subsection, we derive Equation (31) from Equation (30) by the similar way as Pontyragin’s minimum principle. From Equation (13), the following equality is satisfied:
J [ u 0 : t d t , u t , u t + d t : T d t ] J [ u 0 : t d t , u t * , u t + d t : T d t ] = E p t ( s ) H ( t , s , u t , w t + d t ) E p t ( s ) H ( t , s , u t * , w t + d t ) d t = E p t ( z ) E p t ( x | z ) H ( t , s , u t , w t + d t ) E p t ( x | z ) H ( t , s , u t * , w t + d t ) d t .
Therefore, Equation (31) is equivalent with Equation (30).

Appendix C.6. Derivation of Result in Section 4.2 by the Time Discretized Method

In this subsection, we derive Equation (31) from Equation (30) by the time discretized method. Equation (30) can be calculated as follows:
u t * = arg min u t J [ u 0 : T d t ] = arg min u t E p ( s 0 : T ; u 0 : T d t ) 0 T f ( τ , s τ , u τ ) d τ + g ( s T ) = arg min u t E p ( s t : T ; u 0 : T d t ) t T f ( τ , s τ , u τ ) d τ + g ( s T ) = arg min u t E p ( s t : T ; u 0 : T d t ) f ( t , s t , u t ) d t + t + d t T f ( τ , s τ , u τ ) d τ + g ( s T ) = arg min u t E p t ( s t ) f ( t , s t , u t ) d t + E p ( s t + d t : T | s t ; u t : T d t ) t + d t T f ( τ , s τ , u τ ) d τ + g ( s T ) = arg min u t E p t ( s t ) f ( t , s t , u t ) d t + E p ( s t + d t | s t ; u t ) w t + d t ( s t + d t ) ,
where p t ( s ) is the solution of the FP Equation (33) driven by u 0 : t d t , and w t + d t ( s ) is defined as follows:
w t + d t ( s ) : = E p ( s t + 2 d t : T | s t + d t = s ; u t + d t : T d t ) t + d t T f ( τ , s τ , u τ ) d τ + g ( s T ) .
From Ito’s lemma,
u t * = arg min u t E p t ( s t ) f ( t , s t , u t ) d t + w t + d t ( s t ) + L u t w t + d t ( s t ) d t = arg min u t E p t ( s t ) f ( t , s t , u t ) d t + L u t w t + d t ( s t ) d t = arg min u t E p t ( s ) H ( t , s , u t , w t + d t ) .
Because control u t is a function of memory z in ML-POSC, the minimization by u t can be exchanged with the expectation by p t ( z ) as follows:
u t * ( z ) = arg min u t E p t ( x | z ) H t , s , u t , w t + d t .
Therefore, Equation (31) is derived from Equation (30). Finally, we prove that w t ( s ) is the solution of the HJB Equation (32) driven by u t + d t : T d t . w t ( s ) can be calculated as follows:
w t ( s ) = E p ( s t + d t : T | s t = s ; u t : T d t ) t T f ( τ , s τ , u τ ) d τ + g ( s T ) = f ( t , s , u t ) d t + E p ( s t + d t | s t = s ; u t ) w t + d t ( s t + d t ) = f ( t , s , u t ) d t + w t + d t ( s ) + L u t w t + d t ( s ) d t = w t + d t ( s ) + H ( t , s , u t , w t + d t ) d t ,
where w T ( s ) = g ( s ) . Therefore, w t ( s ) defined by Equation (A79) is the solution of the HJB Equation (32) driven by u t + d t : T d t .

Appendix C.7. Derivation of Result in Section 4.3

In this subsection, we mainly derive the inequality of the forward step (35). The inequality of the backward step (34) can be derived in a similar way. In the forward step, u 0 : t d t k + 1 and u t + d t : T d t k are given, and u t k + 1 is defined by
u t k + 1 ( z ) : = arg min u t E p t k + 1 ( x | z ) H t , s , u t , w t + d t k .
From the equivalence of Equations (30) and (31), the following equation is satisfied:
u t k + 1 = arg min u t J [ u 0 : t d t k + 1 , u t , u t + d t : T d t k ] .
Therefore, the inequality of the forward step (35) is satisfied.

Appendix C.8. Derivation of Result in Section 4.4

In this subsection, we show that FBSM for ML-POSC converges to Pontryagin’s minimum principle (20). More specifically, we prove that if J [ u 0 : T d t k + 1 ] = J [ u 0 : T d t k ] holds, u 0 : T d t k + 1 satisfies Pontryagin’s minimum principle (20). We mainly consider the forward step. We can make a similar discussion in the backward step. If J [ u 0 : T d t k + 1 ] = J [ u 0 : T d t k ] holds, then J [ u 0 : t k + 1 , u t + d t : T d t k ] = J [ u 0 : t d t k + 1 , u t : T d t k ] holds from Equation (35). Because J [ u 0 k + 1 , u d t : T d t k ] = J [ u 0 : T d t k ] holds, u 0 k + 1 = u 0 k holds. Then, because J [ u 0 k , u d t k + 1 , u 2 d t : T d t k ] = J [ u 0 : T d t k ] holds, u d t k + 1 = u d t k holds. Iterating this procedure from t = 0 to t = T d t , u 0 : T d t k + 1 = u 0 : T d t k holds. Therefore, because the HJB equation and the FP equation depend on the same control function u 0 : T d t k + 1 = u 0 : T d t k , u 0 : T d t k + 1 satisfies Pontryagin’s minimum principle (20).

Appendix C.9. Derivation of Result in Section 5.3

In this subsection, we show that FBSM is reduced from Algorithm 1 to Algorithm 2 in the LQG problem of ML-POSC.
We first consider the initial step. We assume that the control function is initialized by
u 0 ( t , z ) = R 1 B Π 0 K ( Λ 0 ) ( s μ ) + Ψ μ ,
where Π 0 is arbitrary and Λ 0 is the solution of Λ ˙ 0 = F ( Λ 0 , Π 0 ) given Λ 0 ( 0 ) = Λ 0 . When the control function is initialized by (A85), the solution of the FP equation is given by the Gaussian distribution p t 0 ( s ) : = N ( s | μ , Λ 0 ) , where μ is the solution of (42) and Λ 0 is the solution of Λ ˙ 0 = F ( Λ 0 , Π 0 ) given Λ 0 ( 0 ) = Λ 0 .
We then consider the backward step. When the solution of the FP equation is given by the Gaussian distribution p t k ( s ) : = N ( s | μ , Λ k ) , the solution of the HJB equation is given by the quadratic function w t k + 1 ( s ) = s Π k + 1 s + ( α k + 1 ) s + β k + 1 , where Π k + 1 , α k + 1 , and β k + 1 are the solutions of the following ODEs:
Π ˙ k + 1 = G ( Λ k , Π k + 1 ) ,
α ˙ k + 1 = ( A B R 1 B Π k + 1 ) α k + 1 2 ( I K ( Λ k ) ) Π k + 1 B R 1 B Π k + 1 ( I K ( Λ k ) ) μ ,
β ˙ k + 1 = Π k + 1 σ σ 1 4 ( α k + 1 ) B R 1 B α k + 1 + μ ( I K ( Λ k ) ) Π k + 1 B R 1 B Π k + 1 ( I K ( Λ k ) ) μ ,
where Π k + 1 ( T ) = P , α k + 1 ( T ) = 0 , and β k + 1 ( T ) = 0 .
We finally consider the forward step. When the solution of the HJB equation is given by the quadratic function w t k ( s ) = s Π k s + ( α k ) s + β k , the solution of the FP equation is given by the Gaussian distribution p t k + 1 ( s ) : = N ( s | μ , Λ k + 1 ) , where μ is the solution of (42) and Λ k + 1 is the solution of Λ ˙ k + 1 = F ( Λ k + 1 , Π k ) given Λ k + 1 ( 0 ) = Λ 0 . Therefore, FBSM is reduced from Algorithm 1 to Algorithm 2 in the LQG problem of ML-POSC. The details of these calculations are almost the same with [14].

References

  1. Fox, R.; Tishby, N. Minimum-information LQG control Part II: Retentive controllers. In Proceedings of the 2016 IEEE 55th Conference on Decision and Control (CDC), Las Vegas, NV, USA, 12–14 December 2016; pp. 5603–5609. [Google Scholar] [CrossRef] [Green Version]
  2. Fox, R.; Tishby, N. Minimum-information LQG control part I: Memoryless controllers. In Proceedings of the 2016 IEEE 55th Conference on Decision and Control (CDC), Las Vegas, NV, USA, 12–14 December 2016; pp. 5610–5616. [Google Scholar] [CrossRef] [Green Version]
  3. Li, W.; Todorov, E. An Iterative Optimal Control and Estimation Design for Nonlinear Stochastic System. In Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, CA, USA, 13–15 December 2006; pp. 3242–3247. [Google Scholar] [CrossRef]
  4. Li, W.; Todorov, E. Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic system. Int. J. Control. 2007, 80, 1439–1453. [Google Scholar] [CrossRef]
  5. Nakamura, K.; Kobayashi, T.J. Connection between the Bacterial Chemotactic Network and Optimal Filtering. Phys. Rev. Lett. 2021, 126, 128102. [Google Scholar] [CrossRef] [PubMed]
  6. Nakamura, K.; Kobayashi, T.J. Optimal sensing and control of run-and-tumble chemotaxis. Phys. Rev. Res. 2022, 4, 013120. [Google Scholar] [CrossRef]
  7. Pezzotta, A.; Adorisio, M.; Celani, A. Chemotaxis emerges as the optimal solution to cooperative search games. Phys. Rev. E 2018, 98, 042401. [Google Scholar] [CrossRef] [Green Version]
  8. Borra, F.; Cencini, M.; Celani, A. Optimal collision avoidance in swarms of active Brownian particles. J. Stat. Mech. Theory Exp. 2021, 2021, 083401. [Google Scholar] [CrossRef]
  9. Davis, M.H.A.; Varaiya, P. Dynamic Programming Conditions for Partially Observable Stochastic Systems. SIAM J. Control. 1973, 11, 226–261. [Google Scholar] [CrossRef]
  10. Bensoussan, A. Stochastic Control of Partially Observable Systems; Cambridge University Press: Cambridge, UK, 1992. [Google Scholar] [CrossRef]
  11. Fabbri, G.; Gozzi, F.; Święch, A. Stochastic Optimal Control in Infinite Dimension. In Probability Theory and Stochastic Modelling; Springer International Publishing: Cham, Switzerland, 2017; Volume 82. [Google Scholar] [CrossRef]
  12. Wang, G.; Wu, Z.; Xiong, J. An Introduction to Optimal Control of FBSDE with Incomplete Information; Springer Briefs in Mathematics; Springer International Publishing: Cham, Switzerland, 2018. [Google Scholar] [CrossRef]
  13. Bensoussan, A.; Yam, S.C.P. Mean field approach to stochastic control with partial information. ESAIM Control. Optim. Calc. Var. 2021, 27, 89. [Google Scholar] [CrossRef]
  14. Tottori, T.; Kobayashi, T.J. Memory-Limited Partially Observable Stochastic Control and Its Mean-Field Control Approach. Entropy 2022, 24, 1599. [Google Scholar] [CrossRef]
  15. Kushner, H. Optimal stochastic control. IRE Trans. Autom. Control. 1962, 7, 120–122. [Google Scholar] [CrossRef]
  16. Yong, J.; Zhou, X.Y. Stochastic Controls; Springer: New York, NY, USA, 1999. [Google Scholar] [CrossRef]
  17. Nisio, M. Stochastic Control Theory. In Probability Theory and Stochastic Modelling; Springer: Tokyo, Japan, 2015; Volume 72. [Google Scholar] [CrossRef]
  18. Bensoussan, A. Estimation and Control of Dynamical Systems. In Interdisciplinary Applied Mathematics; Springer International Publishing: Cham, Switzerland, 2018; Volume 48. [Google Scholar] [CrossRef]
  19. Kushner, H.J.; Dupuis, P.G. Numerical Methods for Stochastic Control Problems in Continuous Time; Springer: New York, NY, USA, 1992. [Google Scholar] [CrossRef]
  20. Fleming, W.H.; Soner, H.M. Controlled Markov Processes and Viscosity Solutions, 2nd ed.; Number 25 in Applications of Mathematics; Springer: New York, NY, USA, 2006. [Google Scholar] [CrossRef]
  21. Puterman, M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming; Wiley-Interscience: Hoboken, NJ, USA, 2014. [Google Scholar]
  22. Pontryagin, L.S. Mathematical Theory of Optimal Processes; CRC Press: Boca Raton, FL, USA, 1987. [Google Scholar]
  23. Vinter, R. Optimal Control; Birkhäuser Boston: Boston, MA, USA, 2010. [Google Scholar] [CrossRef]
  24. Lewis, F.L.; Vrabie, D.; Syrmos, V.L. Optimal Control; John Wiley & Sons: New York, NY, USA, 2012. [Google Scholar]
  25. Aschepkov, L.T.; Dolgy, D.V.; Kim, T.; Agarwal, R.P. Optimal Control; Springer International Publishing: Cham, Switzerland, 2016. [Google Scholar] [CrossRef]
  26. Bensoussan, A.; Frehse, J.; Yam, P. Mean Field Games and Mean Field Type Control Theory; Springer Briefs in Mathematics; Springer: New York, NY, USA, 2013. [Google Scholar] [CrossRef]
  27. Carmona, R.; Delarue, F. Probabilistic Theory of Mean Field Games with Applications I; Number Volume 83 in Probability Theory and Stochastic Modelling; Springer Nature: Cham, Switzerland, 2018. [Google Scholar] [CrossRef]
  28. Carmona, R.; Delarue, F. Probabilistic Theory of Mean Field Games with Applications II; Volume 84, Probability Theory and Stochastic Modelling; Springer International Publishing: Cham, Switzerland, 2018. [Google Scholar] [CrossRef]
  29. Carmona, R.; Delarue, F. The Master Equation for Large Population Equilibriums. In Stochastic Analysis and Applications 2014; Crisan, D., Hambly, B., Zariphopoulou, T., Eds.; Springer International Publishing: Cham, Switzerland, 2014; Volume 100, pp. 77–128. [Google Scholar] [CrossRef] [Green Version]
  30. Bensoussan, A.; Frehse, J.; Yam, S.C.P. The Master equation in mean field theory. J. Math. Pures Appl. 2015, 103, 1441–1474. [Google Scholar] [CrossRef]
  31. Bensoussan, A.; Frehse, J.; Yam, S.C.P. On the interpretation of the Master Equation. Stoch. Process. Their Appl. 2017, 127, 2093–2137. [Google Scholar] [CrossRef] [Green Version]
  32. Krylov, I.; Chernous’ko, F. On a method of successive approximations for the solution of problems of optimal control. USSR Comput. Math. Math. Phys. 1963, 2, 1371–1382. [Google Scholar] [CrossRef]
  33. Mitter, S.K. Successive approximation methods for the solution of optimal control problems. Automatica 1966, 3, 135–149. [Google Scholar] [CrossRef]
  34. Chernousko, F.L.; Lyubushin, A.A. Method of successive approximations for solution of optimal control problems. Optim. Control. Appl. Methods 1982, 3, 101–114. [Google Scholar] [CrossRef]
  35. Lenhart, S.; Workman, J.T. Optimal Control Applied to Biological Models; Chapman and Hall/CRC: New York, NY, USA, 2007. [Google Scholar] [CrossRef]
  36. Sharp, J.A.; Burrage, K.; Simpson, M.J. Implementation and acceleration of optimal control for systems biology. J. R. Soc. Interface 2021, 18, 20210241. [Google Scholar] [CrossRef]
  37. Hackbusch, W. A numerical method for solving parabolic equations with opposite orientations. Computing 1978, 20, 229–240. [Google Scholar] [CrossRef]
  38. McAsey, M.; Mou, L.; Han, W. Convergence of the forward-backward sweep method in optimal control. Comput. Optim. Appl. 2012, 53, 207–226. [Google Scholar] [CrossRef]
  39. Carlini, E.; Silva, F.J. Semi-Lagrangian schemes for mean field game models. In Proceedings of the 52nd IEEE Conference on Decision and Control, Firenze, Italy, 10–13 December 2013; pp. 3115–3120. [Google Scholar] [CrossRef] [Green Version]
  40. Carlini, E.; Silva, F.J. A Fully Discrete Semi-Lagrangian Scheme for a First Order Mean Field Game Problem. SIAM J. Numer. Anal. 2014, 52, 45–67. [Google Scholar] [CrossRef] [Green Version]
  41. Carlini, E.; Silva, F.J. A semi-Lagrangian scheme for a degenerate second order mean field game system. Discret. Contin. Dyn. Syst. 2015, 35, 4269. [Google Scholar] [CrossRef]
  42. Lauriere, M. Numerical Methods for Mean Field Games and Mean Field Type Control. arXiv 2021, arXiv:2106.06231. [Google Scholar]
  43. Wonham, W.M. On the Separation Theorem of Stochastic Control. SIAM J. Control. 1968, 6, 312–326. [Google Scholar] [CrossRef]
  44. Li, Q.; Chen, L.; Tai, C.; E, W. Maximum Principle Based Algorithms for Deep Learning. J. Mach. Learn. Res. 2018, 18, 1–29. [Google Scholar]
  45. Liu, X.; Frank, J. Symplectic Runge–Kutta discretization of a regularized forward–backward sweep iteration for optimal control problems. J. Comput. Appl. Math. 2021, 383, 113133. [Google Scholar] [CrossRef]
  46. Bellman, R. Dynamic Programming; Princeton University Press: Princeton, NJ, USA, 1957. [Google Scholar]
  47. Howard, R.A. Dynamic Programming and Markov Processes; John Wiley: Oxford, UK, 1960. [Google Scholar]
  48. Kappen, H.J. Linear Theory for Control of Nonlinear Stochastic Systems. Phys. Rev. Lett. 2005, 95, 200201. [Google Scholar] [CrossRef] [Green Version]
  49. Kappen, H.J. Path integrals and symmetry breaking for optimal control theory. J. Stat. Mech. Theory Exp. 2005, 2005, P11011. [Google Scholar] [CrossRef] [Green Version]
  50. Satoh, S.; Kappen, H.J.; Saeki, M. An Iterative Method for Nonlinear Stochastic Optimal Control Based on Path Integrals. IEEE Trans. Autom. Control. 2017, 62, 262–276. [Google Scholar] [CrossRef]
  51. Cacace, S.; Camilli, F.; Goffi, A. A policy iteration method for Mean Field Games. arXiv 2021, arXiv:2007.04818. [Google Scholar] [CrossRef]
  52. Laurière, M.; Song, J.; Tang, Q. Policy iteration method for time-dependent Mean Field Games systems with non-separable Hamiltonians. arXiv 2021, arXiv:2110.02552. [Google Scholar] [CrossRef]
  53. Camilli, F.; Tang, Q. Rates of convergence for the policy iteration method for Mean Field Games systems. arXiv 2022, arXiv:2108.00755. [Google Scholar] [CrossRef]
  54. Ruthotto, L.; Osher, S.J.; Li, W.; Nurbekyan, L.; Fung, S.W. A machine learning framework for solving high-dimensional mean field game and mean field control problems. Proc. Natl. Acad. Sci. USA 2020, 117, 9183–9193. [Google Scholar] [CrossRef] [Green Version]
  55. Lin, A.T.; Fung, S.W.; Li, W.; Nurbekyan, L.; Osher, S.J. Alternating the population and control neural networks to solve high-dimensional stochastic mean-field games. Proc. Natl. Acad. Sci. USA 2021, 118, e2024713118. [Google Scholar] [CrossRef]
  56. Laurière, M.; Pironneau, O. Dynamic programming for mean-field type control. C. R. Math. 2014, 352, 707–713. [Google Scholar] [CrossRef] [Green Version]
  57. Laurière, M.; Pironneau, O. Dynamic programming for mean-field type control. J. Optim. Theory Appl. 2016, 169, 902–924. [Google Scholar] [CrossRef]
  58. Pham, H.; Wei, X. Bellman equation and viscosity solutions for mean-field stochastic control problem. ESAIM Control. Optim. Calc. Var. 2018, 24, 437–461. [Google Scholar] [CrossRef]
Figure 1. Schematic diagram of the relationship between the backward dynamics, the optimal control function, and the forward dynamics in (a) COSC, (b) ML-POSC, (c) deterministic control, and (d) MFSC. w * , p * , λ * , and s * are the solutions of the HJB equation, the FP equation, the adjoint equation, and the state equation, respectively. u * is the optimal control function. The arrows indicate the dependence of variables. The variable at the head of an arrow depends on the variable at the tail of the arrow. (a) In COSC, because the optimal control function u * depends only on the HJB equation w * , it can be obtained by solving the HJB equation w * backward in time from the terminal condition, which is called the value iteration method. (b) In ML-POSC, because the optimal control function u * depends on the FP equation p * as well as the HJB equation w * (orange), it cannot be obtained by the value iteration method. In this paper, we propose FBSM for ML-POSC, which computes the HJB equation w * and the FP equation p * alternately. Because the coupling of the HJB equation w * and the FP equation p * is limited only to the optimal control function u * , the convergence of FBSM is guaranteed in ML-POSC. (c) In deterministic control, because the coupling of the adjoint equation λ * and the state equation s * is not limited to the optimal control function u * (green), the convergence of FBSM is not guaranteed. (d) In MFSC, because the coupling of the HJB equation w * and the FP equation p * is not limited to the optimal control function u * (green), the convergence of FBSM is not guaranteed.
Figure 1. Schematic diagram of the relationship between the backward dynamics, the optimal control function, and the forward dynamics in (a) COSC, (b) ML-POSC, (c) deterministic control, and (d) MFSC. w * , p * , λ * , and s * are the solutions of the HJB equation, the FP equation, the adjoint equation, and the state equation, respectively. u * is the optimal control function. The arrows indicate the dependence of variables. The variable at the head of an arrow depends on the variable at the tail of the arrow. (a) In COSC, because the optimal control function u * depends only on the HJB equation w * , it can be obtained by solving the HJB equation w * backward in time from the terminal condition, which is called the value iteration method. (b) In ML-POSC, because the optimal control function u * depends on the FP equation p * as well as the HJB equation w * (orange), it cannot be obtained by the value iteration method. In this paper, we propose FBSM for ML-POSC, which computes the HJB equation w * and the FP equation p * alternately. Because the coupling of the HJB equation w * and the FP equation p * is limited only to the optimal control function u * , the convergence of FBSM is guaranteed in ML-POSC. (c) In deterministic control, because the coupling of the adjoint equation λ * and the state equation s * is not limited to the optimal control function u * (green), the convergence of FBSM is not guaranteed. (d) In MFSC, because the coupling of the HJB equation w * and the FP equation p * is not limited to the optimal control function u * (green), the convergence of FBSM is not guaranteed.
Entropy 25 00208 g001
Figure 2. The relationship between Bellman’s dynamic programming principle (top) and Pontryagin’s minimum principle (bottom) on the state space (left) and on the probability density function space (right). The left-hand side corresponds to deterministic control, which is briefly reviewed in Appendix A. The right-hand side corresponds to ML-POSC and MFSC, which are shown in Section 3 and Appendix B, respectively. The conventional approach in ML-POSC [14] and MFSC [30,31] can be interpreted as the conversion from Bellman’s dynamic programming principle (top right) to Pontryagin’s minimum principle (bottom right) on the probability density function space.
Figure 2. The relationship between Bellman’s dynamic programming principle (top) and Pontryagin’s minimum principle (bottom) on the state space (left) and on the probability density function space (right). The left-hand side corresponds to deterministic control, which is briefly reviewed in Appendix A. The right-hand side corresponds to ML-POSC and MFSC, which are shown in Section 3 and Appendix B, respectively. The conventional approach in ML-POSC [14] and MFSC [30,31] can be interpreted as the conversion from Bellman’s dynamic programming principle (top right) to Pontryagin’s minimum principle (bottom right) on the probability density function space.
Entropy 25 00208 g002
Figure 3. Schematic diagram of the effect of updating the control function to the forward and backward dynamics in (a) ML-POSC, (b) deterministic control, and (c) MFSC. w 0 : T , p 0 : T , λ 0 : T , and s 0 : T are the solutions of the HJB equation, the FP equation, the adjoint equation, and the state equation, respectively. u 0 : T d t is a given control function. The arrows indicate the dependence of variables. The variable at the head of an arrow depends on the variable at the tail of the arrow. (a) In ML-POSC, while the update from u t to u t (yellow) changes w 0 : t and p t + d t : T to w 0 : t and p t + d t : T , respectively (red), it does not change p 0 : t and w t + d t : T (blue). From this property, the convergence of FBSM is guaranteed in ML-POSC. (b) In deterministic control, the update from u t to u t (yellow) changes λ t + d t : T to λ t + d t : T as well (red) because the adjoint equation depends on the state equation (green). Because FBSM does not take into account the change of λ t + d t : T , the convergence of FBSM is not guaranteed in deterministic control. (c) In MFSC, the update from u t to u t (yellow) changes w t + d t : T to w t + d t : T as well (red) because the HJB equation depends on the FP equation (green). Because FBSM does not take into account the change of w t + d t : T , the convergence of FBSM is not guaranteed in MFSC.
Figure 3. Schematic diagram of the effect of updating the control function to the forward and backward dynamics in (a) ML-POSC, (b) deterministic control, and (c) MFSC. w 0 : T , p 0 : T , λ 0 : T , and s 0 : T are the solutions of the HJB equation, the FP equation, the adjoint equation, and the state equation, respectively. u 0 : T d t is a given control function. The arrows indicate the dependence of variables. The variable at the head of an arrow depends on the variable at the tail of the arrow. (a) In ML-POSC, while the update from u t to u t (yellow) changes w 0 : t and p t + d t : T to w 0 : t and p t + d t : T , respectively (red), it does not change p 0 : t and w t + d t : T (blue). From this property, the convergence of FBSM is guaranteed in ML-POSC. (b) In deterministic control, the update from u t to u t (yellow) changes λ t + d t : T to λ t + d t : T as well (red) because the adjoint equation depends on the state equation (green). Because FBSM does not take into account the change of λ t + d t : T , the convergence of FBSM is not guaranteed in deterministic control. (c) In MFSC, the update from u t to u t (yellow) changes w t + d t : T to w t + d t : T as well (red) because the HJB equation depends on the FP equation (green). Because FBSM does not take into account the change of w t + d t : T , the convergence of FBSM is not guaranteed in MFSC.
Entropy 25 00208 g003
Figure 4. The elements of the control gain matrix Π k ( t ) R 2 × 2 (ac) and the precision matrix Λ k ( t ) R 2 × 2 (df) obtained by FBSM (Algorithm 2) in the numerical experiment of the LQG problem of ML-POSC. Because Π z x k ( t ) = Π x z k ( t ) and Λ z x k ( t ) = Λ x z k ( t ) , Π z x k ( t ) and Λ z x k ( t ) are not visualized. The darkest curve corresponds to the first iteration k = 0 , and the brightest curve corresponds to the last iteration k = 50 . Π 0 ( t ) is initialized by Π 0 ( t ) = O .
Figure 4. The elements of the control gain matrix Π k ( t ) R 2 × 2 (ac) and the precision matrix Λ k ( t ) R 2 × 2 (df) obtained by FBSM (Algorithm 2) in the numerical experiment of the LQG problem of ML-POSC. Because Π z x k ( t ) = Π x z k ( t ) and Λ z x k ( t ) = Λ x z k ( t ) , Π z x k ( t ) and Λ z x k ( t ) are not visualized. The darkest curve corresponds to the first iteration k = 0 , and the brightest curve corresponds to the last iteration k = 50 . Π 0 ( t ) is initialized by Π 0 ( t ) = O .
Entropy 25 00208 g004
Figure 5. Performance of FBSM in the numerical experiment of the LQG problem of ML-POSC. (a) The objective function J [ u k ] with respect to the iteration k. (bd) Stochastic simulation of state x t (b), memory z t (c), and the cumulative cost (d) for 100 samples. The expectation of the cumulative cost at t = 10 corresponds to the objective function (49). Blue and orange curves correspond to the first iteration k = 0 and the last iteration k = 50 , respectively.
Figure 5. Performance of FBSM in the numerical experiment of the LQG problem of ML-POSC. (a) The objective function J [ u k ] with respect to the iteration k. (bd) Stochastic simulation of state x t (b), memory z t (c), and the cumulative cost (d) for 100 samples. The expectation of the cumulative cost at t = 10 corresponds to the objective function (49). Blue and orange curves correspond to the first iteration k = 0 and the last iteration k = 50 , respectively.
Entropy 25 00208 g005
Figure 6. The solutions of the HJB equation w k ( t , s ) (aj) and the FP equation p k ( t , s ) (kt) at the first iteration k = 0 (ae,ko) and at the last iteration k = 50 (fj,pt) of FBSM (Algorithm 1) in the numerical experiment of the non-LQG problem of ML-POSC. u 0 ( t , z ) is initialized by u 0 ( t , z ) = 0 .
Figure 6. The solutions of the HJB equation w k ( t , s ) (aj) and the FP equation p k ( t , s ) (kt) at the first iteration k = 0 (ae,ko) and at the last iteration k = 50 (fj,pt) of FBSM (Algorithm 1) in the numerical experiment of the non-LQG problem of ML-POSC. u 0 ( t , z ) is initialized by u 0 ( t , z ) = 0 .
Entropy 25 00208 g006
Figure 7. Performance of FBSM in the numerical experiment of the non-LQG problem of ML-POSC. (a) The objective function J [ u k ] with respect to the iteration k. (b) Stochastic simulation of the state x t for 100 samples. The black rectangles and the cross represent the obstacles and the goal, respectively. Blue and orange curves correspond to the first iteration k = 0 and the last iteration k = 50 , respectively. (c) The objective function (55), which is computed from 100 samples.
Figure 7. Performance of FBSM in the numerical experiment of the non-LQG problem of ML-POSC. (a) The objective function J [ u k ] with respect to the iteration k. (b) Stochastic simulation of the state x t for 100 samples. The black rectangles and the cross represent the obstacles and the goal, respectively. Blue and orange curves correspond to the first iteration k = 0 and the last iteration k = 50 , respectively. (c) The objective function (55), which is computed from 100 samples.
Entropy 25 00208 g007
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Tottori, T.; Kobayashi, T.J. Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control. Entropy 2023, 25, 208. https://doi.org/10.3390/e25020208

AMA Style

Tottori T, Kobayashi TJ. Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control. Entropy. 2023; 25(2):208. https://doi.org/10.3390/e25020208

Chicago/Turabian Style

Tottori, Takehiro, and Tetsuya J. Kobayashi. 2023. "Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control" Entropy 25, no. 2: 208. https://doi.org/10.3390/e25020208

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics