Probabilistic Estimation and Control of Dynamical Systems Using Particle Filter with Adaptive Backward Sampling

Abstract

Estimating and controlling dynamical systems from observable time-series data are essential for understanding and manipulating nonlinear dynamics. This paper proposes a probabilistic framework for simultaneously estimating and controlling nonlinear dynamics under noisy observation conditions. Our proposed method utilizes the particle filter not only as a state estimator and a prior estimator for the dynamics but also as a controller. This approach allows us to handle the nonlinearity of the dynamics and uncertainty of the latent state. We apply two distinct dynamics to verify the effectiveness of our proposed framework: a chaotic system defined by the Lorenz equation and a nonlinear neuronal system defined by the Morris–Lecar neuron model. The results indicate that our proposed framework can simultaneously estimate and control complex nonlinear dynamical systems.

1. Introduction

Estimating dynamical systems from noisy time-series data is vital to understanding complex systems [1]. Furthermore, controlling dynamical systems has attracted significant attention in various fields for manipulating known complex systems. Controlling unknown complex systems requires simultaneously estimating and controlling dynamical systems from time-series data.

The state-space model has been utilized in various fields to estimate latent variables from noisy observational data [2,3,4,5,6,7,8,9,10]. The state-space model (SSM), also called the general hidden Markov model, is a probabilistic framework used to represent the dynamical system governing the latent variable and the probabilistic observation producing the observation variable. Under the assumption that the parameters governing the entire dynamics are known, the latent variable can be estimated efficiently using a sequential Bayesian filter such as a particle filter [11,12,13]. Nonetheless, in practice, it is unrealistic that the parameter values are completely known. If the parameters are unknown, it is necessary to estimate the latent variable and the parameters simultaneously. Therefore, many parameter estimation methods based on the particle filter have been proposed to estimate the latent variable and the parameters of the SSM (see [14] and the references therein). Some offline methods have been proposed, where every recursive parameter update requires a batch of observational data [15]. Other online methods have been proposed, where the parameter values can be estimated at the same time that new observational data become available [16]. However, previous online parameter estimation methods require multiple samples for each particle drawn from backward ancestor sampling every time, which leads to high computational costs.

In contrast, model predictive control has been applied in various fields to control nonlinear dynamical systems [17]. Model predictive control is a control strategy that predicts a known model’s behavior in a future interval and calculates an optimal control input; however, to apply model predictive control to a nonlinear model, it is necessary to formulate an optimal control problem, depicted by the calculus of variations or the Euler–Lagrange equation. Solving the optimal control problem, including nonlinearity, usually requires high computational cost and is not always feasible. In particular, particle filter-based model predictive control (PF-MPC) has been proposed [18]. PF-MPC can directly handle known nonlinear models by using a particle filter; however, in previous studies on control, the model dynamics are assumed to be known [18]. For considering real-world applications, the dynamics of complex systems are often unknown.

In this paper, we propose a framework for simultaneously estimating the latent variable and the parameters of the state-space model, as well as controlling the latent variable, fully based on the particle filter. The proposed method realizes a general method for estimating and controlling dynamical systems described by the state-space model by integrating and adapting the framework of the particle filter (PF), the online expectation-maximization (EM) algorithm, and model predictive control (MPC). With the PF, we estimate the latent variable of the nonlinear dynamics online. In particular, to fully realize the online algorithm for simultaneous estimation and control, we derive a novel online parameter estimation method for the state-space model by applying an adaptive smoothing (AdaSmooth) algorithm, which has recently been proposed for smoothing statistics based on the PF [19]. By combining AdaSmooth with the online EM algorithm, we efficiently approximate the sufficient statistics and estimate the parameter values based on the smoothed sufficient statistics. Furthermore, we consider combining PF-based model predictive control (PF-MPC) [18] with our AdaSmooth-based online EM algorithm to form the feedback control law. By introducing the PF not only as a state estimator and a prior estimator for the dynamics but also as a controller, we can directly handle the nonlinearity of the dynamics and uncertainty of the latent state.

This paper is organized as follows. Section 2 introduces the state-space model and the particle filter. We then propose the AdaSmooth-based online EM algorithm for estimating the parameters of the state-space model. Finally, by combining PF-MPC with the state estimation PF and the AdaSmooth-based online EM algorithm, we propose a framework for estimating and controlling dynamical systems. In Section 3, we verify the effectiveness of our proposed method using simulation environments. We apply our proposed framework to two distinct dynamical systems: a chaotic system defined by the Lorenz equation and a nonlinear neuronal model. In each subsection, we first formulate the state-space model for each dynamical system. We then derive the parameter estimation procedure based on the AdaSmooth-based online EM algorithm. Next, we derive the concrete control law based on PF-MPC before applying our proposed framework to each complex nonlinear dynamical system. Futhermore, we present the results for estimating and control the chaostic system and neural system. Finally, Section 4 presents our concluding remarks.

2. Methods

This study proposes a probabilistic framework for concurrent data assimilation-based control of dynamical systems. To handle a realistic situation, we assume that we can only observe noisy data via some observation model, i.e., we cannot directly obtain the true latent state. Figure 1 shows the overview of our proposed framework. Our proposed framework comprises three parts: (i) estimation of the posterior distribution of the latent state from noisy observational time-series data, (ii) online estimation of the model parameters governing the dynamical system, and (iii) solving the optimal control problem using a control PF with the estimated state and parameters.

2.1. Formulation of State-Space Model

First, we formulate the SSM. Figure 2 shows the overview of the SSM. The SSM includes the system model and the observation model. The system model represents the dynamics of the latent variable ${\mathit{x}}_t$. Conversely, the observation model represents how we obtain the observation variable ${\mathit{y}}_t$ under a realistic situation where some noise exists.

We consider the system model, including state ${\mathit{x}}_t$, control input ${\mathit{u}}_t$, and certain system noise ${\mathit{z}}_t$, with a Markov property:

$${\mathit{x}}_t=f({\mathit{x}}_{t-1},{\mathit{u}}_{t-1},{\mathit{z}}_{t-1};\theta ),$$

(1)

where t denotes a discrete time index, $\theta$ is an unknown parameter governing the entire dynamics, and f is a function determining the dynamics, which can include nonlinearity. Corresponding to Equation (1), we also define the probabilistic representation of the system model for the SSM as follows:

$${\mathit{x}}_t\sim p({\mathit{x}}_t\mid {\mathit{x}}_{t-1},{\mathit{u}}_{t-1};\theta ).$$

(2)

We then formulate the observation model for the SSM. To reflect a realistic scenario, we assume that we can only obtain the observation variable ${\mathit{y}}_t$ and not the latent variable ${\mathit{x}}_t$ directly at time t. Hence, we formulate the following observation model for the SSM:

$${\mathit{y}}_t=g({\mathit{x}}_t,{\mathsf{\eta }}_t;\theta ),$$

(3)

where g is a function determining the information we obtain via the observation and ${\mathsf{\eta }}_t$ represents certain noise. Corresponding to Equation (3), we also define the probabilistic representation of the observation model for the SSM:

$${\mathit{y}}_t\sim p({\mathit{y}}_t\mid {\mathit{x}}_t;\theta ).$$

(4)

For simplicity, we assume that the joint probability $p({\mathit{x}}_{1:t},{\mathit{y}}_{1:t};\theta )$ belongs to the exponential family, which is a common assumption. Here, ${\mathit{x}}_{1:t}$ and ${\mathit{y}}_{1:t}$ represent the time-series of the latent variable and observation variable ${\mathit{x}}_{1:t}=\{{\mathit{x}}_1,\dots ,{\mathit{x}}_t\}$ and ${\mathit{y}}_{1:t}=\{{\mathit{y}}_1,\dots ,{\mathit{y}}_t\}$, respectively. For example, when both the latent state and observation variable are one-dimensional real numbers and the model is linear Gaussian, the state-space model is described as follows:

$$\left\{\begin{array}{cc}\hfill x_t& =a{x}_{t-1}+b{z}_{t-1}\hfill \\ \hfill y_t& =c{x}_t+d{\eta }_t\hfill \end{array}\right.,$$

(5)

where $x_t$ and $y_t\in \mathbb{R}$ are the latent state and observation variable, respectively; $a,b,c,$ and $d\in \mathbb{R}$ are parameters; and $z_{t-1}$ and ${\eta }_t$ are noise terms, both of which obey standard Gaussian distributions $\mathcal{N}(0,1)$. In this linear Gaussian case, the probabilistic representation of the SSM is expressed as follows:

$$\begin{array}{c}{x}_t\sim \mathcal{N}(x_t\mid a{x}_{t-1},b^2),\end{array}$$

(6)

$$\begin{array}{c}{y}_t\sim \mathcal{N}(y_t\mid c{x}_t,d^2),\end{array}$$

(7)

where $\mathcal{N}(x\mid \mu ,{\sigma }^2)$ denotes the Gaussian distribution with mean $\mu$ and variance ${\sigma }^2$.

2.2. Estimation of the Hidden State

To estimate the latent state ${\mathit{x}}_t$ from noisy time-series observational data ${\mathit{y}}_{1:t}=\{{\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_t\}$ online, we apply the particle filter (PF) [11,12,13,20].

Here, we estimate the latent variable using the filtering distribution $p({\mathit{x}}_t\mid {\mathit{y}}_{1:t})$. In the PF, the filtering distribution is approximated online using particles $\{{\mathit{x}}_t^{(1)},\dots ,{\mathit{x}}_t^{(N)}\}$ and their associated weights $\{w_t^{(1)},\dots ,w_t^{(N)}\}$, where N is the number of particles. We calculate these particles by alternately performing two steps: a resampling step and a prediction step.

In the resampling step, we resample particles obtained previously based on their weights. Although many resampling schemes have been proposed, we use multinomial resampling [21,22]. Using a multinomial distribution $\mathcal{M}\left(w_{t-1}^{(1)},\dots ,w_{t-1}^{(N)}\right)$ with weights, the ancestor index $A_t^{(i)}$ for the i-th particle at time t is sampled as follows:

$$A_t^{(i)}\sim \mathcal{M}\left(w_{t-1}^{(1)},\dots ,w_{t-1}^{(N)}\right),$$

(8)

where $i\in \left\{1,2,\dots ,N\right\}$ is the index of the particle and $\mathcal{M}(w^{(1)},\dots ,w^{(N)})$ denotes the multinomial distribution supported on $\{1,2,\dots ,N\}$. Using the ancestor index $A_t^{(i)}$, the i-th particle ${\overline{\mathit{x}}}_t^{(i)}$ at time t is resampled as follows:

$${\overline{\mathit{x}}}_t^{(i)}={\mathit{x}}_t^{(A_t^{(i)})}.$$

(9)

We apply this resampling step only when the estimated effective sample size ${\widehat{N}}_{\mathrm{eff}}$ satisfies ${\widehat{N}}_{\mathrm{eff}}(t)\le \alpha N$, where $0\le \alpha \le 1$ is a hyperparameter that determines the threshold of the effective sample size [23]. Here, the estimated effective sample size ${\widehat{N}}_{\mathrm{eff}}$ is calculated as follows:

$${\widehat{N}}_{\mathrm{eff}}(t)={\displaystyle \frac{1}{{\sum }_{i=1}^N{(w_t^{(i)})}^2}}.$$

(10)

When ${\widehat{N}}_{\mathrm{eff}}(t)>\alpha N$, instead of probabilistic resampling, as described in Equation (8), we simply set $A_t^{(i)}=i$ for all particles.

In the prediction step, we sample particles that approximate the filtering distribution when new observational data ${\mathit{y}}_t$ are obtained. First, we sample each particle using the system model:

$${\mathit{x}}_t^{(i)}\sim p({\mathit{x}}_t\mid {\mathit{x}}_{t-1}^{(A_t^{(i)})}).$$

(11)

We then calculate the weight of each particle using the observation model.

$${\tilde{w}}_t^{(i)}=w_{t-1}^{(A_t^{(i)})}p({\mathit{y}}_t\mid {\mathit{x}}_t^{(A_t^{(i)})}),$$

(12)

where ${\tilde{w}}_t^{(i)}$ denotes an un-normalized weight, and we assume that the weights from the previous time step and the ancestors are accessible. Note that the weight update given by Equation (12) is derived from sequential importance sampling [20], where the importance distribution is assumed to be the same as in the system model. Finally, we normalize the weights as follows:

$$w_t^{(i)}={\displaystyle \frac{{\tilde{w}}_t^{(i)}}{{\sum }_{l=1}^N{\tilde{w}}_t^{(l)}}}.$$

(13)

Using the sampled particles and their weights ${\left\{({\mathit{x}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ allows us to obtain the filtering distribution of the latent variable online as follows:

$$p({\mathit{x}}_t\mid {\mathit{y}}_{1:t})=\sum _{i=1}^N{w}_t^{(i)}\delta ({\mathit{x}}_t-{\mathit{x}}_t^{(i)}),$$

(14)

where $\delta (·)$ is Dirac’s delta function. Here, the particles and their weights at time $t=0$ are generated as follows:

$${\mathit{x}}_0^{(i)}\sim p_0(\mathit{x}),w_0^{(i)}={\displaystyle \frac{1}{N}},$$

(15)

where $p_0(\mathit{x})$ is the initial distribution.

We estimate the latent state online by alternately applying the resampling and prediction steps as new observational data arrive. Although the PF can suffer from the path-degeneracy problem, it can be mitigated using the resampling step. In this study, we apply multinomial resampling; however, other resampling methods can be used in our proposed framework. The particles and their associated weights ${\left\{({\mathit{x}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ are used to estimate the latent state and the parameters, including a prior for the state in the controller, within our proposed framework.

2.3. Estimation of the Parameter

We utilize the particle filter-based online parameter estimation strategy to estimate the parameters governing the dynamics online. This subsection briefly introduces the original expectation-maximization (EM) algorithm [15] and particle-based online EM algorithm [14,16]. We then derive an efficient parameter estimation procedure using additive smoothing (AdaSmooth) [19].

2.3.1. EM Algorithm

The EM algorithm [15] is a widely used offline parameter estimation method. In the EM algorithm, when we have a batch of observational data $\{{\mathit{y}}_1,\dots ,{\mathit{y}}_T\}$, we estimate the parameter sequence $\{{\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_k,\dots \}$ by alternately conducting an E-step and an M-step, where T is the terminal time and k is the number of iterations.

In the E-step, we calculate the expectation of the joint probability $p({\mathit{x}}_{1:T},{\mathit{y}}_{1:T};\theta )$ given the previous iteration parameter ${\widehat{\theta }}_{k-1}$.

$${\Phi }_{1:T;{\widehat{\theta }}_{k-1}}\left[{\mathcal{L}}_{1:T}(\theta )\right]=E{\left[\log p({\mathit{x}}_{1:T},{\mathit{y}}_{1:T};\theta )\right]}_{p({\mathit{x}}_{1:T}\mid {\mathit{y}}_{1:T},{\widehat{\theta }}_{k-1})},$$

(16)

where $E{[·]}_{p({\mathit{x}}_{1:T}\mid {\mathit{y}}_{1:T},{\widehat{\theta }}_{k-1})}$ denotes the expectation of the joint filtering distribution with respect to ${\mathit{x}}_{1:T}$ given the observational data ${\mathit{y}}_{1:T}$ and the parameter ${\widehat{\theta }}_{k-1}$. ${\mathcal{L}}_{1:T}(\theta )$ corresponds to the log-likelihood of the joint probability $p({\mathit{x}}_{1:T},{\mathit{y}}_{1:T};\theta )$. In the M-step, we update the estimation parameter at this iteration ${\widehat{\theta }}_k$ as follows:

$$\begin{array}{c}\hfill {\widehat{\theta }}_k=arg\underset{\theta }{\max }{\Phi }_{1:T;{\widehat{\theta }}_{k-1}}\left[{\mathcal{L}}_{1:T}(\theta )\right],\end{array}$$

(17)

where $arg\underset{\theta }{\max }$ denotes the maximizer with respect to the parameter $\theta$.

The EM algorithm is an appealing parameter estimation method; however, we need a batch of observational data, and the computational cost of repeating two steps as the size of the observational data increases cannot be ignored. Furthermore, it is difficult to calculate the expectation in the E-step in typical cases. Moreover, an online parameter estimation algorithm must be utilized to realize a simultaneous method for estimating and controlling nonlinear dynamics.

2.3.2. AdaSmooth-Based Online EM Algorithm

In this subsection, we propose the AdaSmooth-based online EM algorithm for estimating the parameters of the SSM. To overcome the problems in the original EM algorithm, some online adaptations have been proposed [14,24,25]. In particular, a particle-based online EM algorithm for time-series state-space models has been proposed [16]. However, a previous parameter estimation method based on PaRIS [26] requires multiple samples for each particle drawn from backward ancestor sampling every time, which leads to high computational costs.

Figure 3 shows the flow of our proposed online EM algorithm for estimating the parameter ${\theta }_t$ online. We consider the parameter ${\theta }_t$ at time t rather than the iteration number k used in the offline EM algorithm. Note that the model parameter $\theta$ in the true dynamics is not time-varying, whereas the model parameter for estimating and controlling the dynamical system should be treated as a time-varying state parameter ${\widehat{\theta }}_t$ in an online manner. Furthermore, we approximate smoothing sufficient statistics ${\Phi }_{1:t;{\widehat{\theta }}_{t-1}}\left[S_t\right]$ using particles obtained online before and after time ${\left\{({\mathit{x}}_{t-1}^{(i)},w_{t-1}^{(i)})\right\}}_{i=1}^N,{\left\{({\mathit{x}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ via the PF [16]. To efficiently estimate the parameters, we apply AdaSmooth [19] to smooth the sufficient statistics. AdaSmooth requires only one sample for each particle drawn from backward ancestor sampling. In addition, AdaSmooth decides whether we should conduct backward ancestor sampling adaptively; hence, we can obtain approximated smoothing sufficient statistics ${\Phi }_{1:t;{\widehat{\theta }}_{t-1}}\left[S_t\right]$ and estimate the parameter ${\widehat{\theta }}_t$ online.

First, we rewrite the M-step in the EM algorithm. Assuming that the joint probability $p({\mathit{x}}_{1:t},{\mathit{y}}_{1:t};\theta )$ belongs to the exponential family, a suitable function $\Lambda (·)$ [14] exists as a maximizer of the log-likelihood in a specific form. That is, we can rewrite the M-step in the context of online estimation as follows:

$${\widehat{\theta }}_t=\Lambda \left({\displaystyle \frac{1}{t}}{\Phi }_{1:t;{\widehat{\theta }}_{t-1}}\left[S_t\right]\right),$$

(18)

where ${\Phi }_{1:t;{\widehat{\theta }}_{t-1}}$ denotes the expectation with respect to the joint probability $p({\mathit{x}}_{1:t}\mid {\mathit{y}}_{1:t},{\widehat{\theta }}_{t-1})$. The parameter index now becomes time t, contrasting with the iteration count k used in Equation (17) in the offline context. $S_t$ is the sufficient statistics defined as follows:

$$S_t({\mathit{x}}_{0:t})=\sum _{\tau =1}^t{\tilde{s}}_{\tau }({\mathit{x}}_{\tau -1:\tau }).$$

(19)

Here, ${\tilde{s}}_{\tau }({\mathit{x}}_{\tau -1:\tau })={\tilde{s}}_{\tau }({\mathit{x}}_{\tau -1},{\mathit{x}}_{\tau })$ is a statistic given by the states ${\mathit{x}}_{\tau -1}$ and ${\mathit{x}}_{\tau }$ before and after times $\tau -1,\tau$.

Next, we estimate ${\Phi }_{1:t;{\widehat{\theta }}_{t-1}}\left[S_t\right]$ online. At time t, we calculate the approximated smoothed sufficient statistics in the E-step [14,16] as follows:

$${\Phi }_{0:t;{\widehat{\theta }}_{t-1}}\left[S_t({\mathit{x}}_{0:t})\right]=\int {\kappa }_t({\mathit{x}}_t)p({\mathit{x}}_t\mid {\mathit{y}}_{1:t})d{\mathit{x}}_t,$$

(20)

where ${\kappa }_t({\mathit{x}}_t)$ is a statistic updated each time via the following procedure:

$${\kappa }_t({\mathit{x}}_t)=\int \{{\gamma }_t{\tilde{s}}_t({\mathit{x}}_{t-1},{\mathit{x}}_t)+(1-{\gamma }_t){\kappa }_{t-1}({\mathit{x}}_{t-1})\}p({\mathit{x}}_{t-1}\mid {\mathit{y}}_{1:t-1},{\widehat{\theta }}_{1:t-1})d{\mathit{x}}_{t-1},$$

(21)

where ${\kappa }_0({\mathit{x}}_0)=0$ and ${\gamma }_t$ is a decay rate that satisfies ${\sum }_{t=1}^{\infty }{\gamma }_t=\infty$ and ${\sum }_{t=1}^{\infty }{\gamma }_t^2<\infty$.

Here, we estimate ${\kappa }_t({\mathit{x}}_t)$ using particles. We prepare a new set of particles ${\kappa }_t({\mathit{x}}_t)$. Then, we derive the update dynamics of ${\kappa }_t({\mathit{x}}_t)$ based on AdaSmooth. Essentially, we update each particle ${\kappa }_t^{(i)}$ corresponding to Equation (21) as follows:

$${\kappa }_t^{(i)}=(1-{\gamma }_t){\kappa }_{t-1}^{(i)}+{\gamma }_t{\tilde{s}}_t({\mathit{x}}_{t-1}^{(A_t^{(i)})},{\mathit{x}}_t^{(i)}),$$

(22)

where $A_t^{(i)}$ is the ancestor index obtained in the PF resampling step. The first term in Equation (22) represents the value at a previous time. The second term in Equation (22) represents the sufficient statistics, including the information obtained at time t. Nonetheless, updating ${\kappa }_t^{(i)}$ using only Equation (22) can lead to the particle path-degeneracy phenomenon and usually leads to instability. We introduce adaptive ancestor backward resampling to avoid this problem [19]. During resampling in the PF, when the diversity of the ancestors ${\widehat{N}}_{\mathrm{anc}}(t)$ satisfies ${\widehat{N}}_{\mathrm{anc}}(t)\le \beta N$, we conduct backward ancestor resampling to sample $B_t^{(i)}$. We then update the dynamics using the following instead of Equation (22):

$${\kappa }_t^{(i)}={\displaystyle \frac{1}{2}}\{(1-{\gamma }_t){\kappa }_{t-1}^{(i)}+{\gamma }_t{\tilde{s}}_t({\mathit{x}}_{t-1}^{(A_t^{(i)})},{\mathit{x}}_t^{(i)})\}+{\displaystyle \frac{1}{2}}\{(1-{\gamma }_t){\kappa }_{t-1}^{(i)}+{\gamma }_t{\tilde{s}}_t({\mathit{x}}_{t-1}^{(B_t^{(i)})},{\mathit{x}}_t^{(i)})\}.$$

(23)

The first term in Equation (23) is the same as in Equation (22). The second term in Equation (23) represents the correction of the sufficient statistics using a new ancestor drawn from backward ancestor sampling. Here, $0\le \beta \le 1$ is a hyperparameter that determines the threshold of ancestor diversity. ${\widehat{N}}_{\mathrm{anc}}(t)$ is the number of unique Enoch indices $\left\{E_t^{(i)}\right\}$, defined as follows [19]:

$${\widehat{N}}_{\mathrm{anc}}(t)=\left|\{E_t^{(1)},\dots ,E_t^{(N)}\}\right|.$$

(24)

Here, the Enoch index $E_t^{(i)}$ is updated via the following procedure:

$$E_t^{(i)}=\left\{\begin{array}{cc}i\hfill & (t=t_0+1)\hfill \\ E_{t-1}^{(A_t^{(i)})}\hfill & (t>t_0+1)\hfill \end{array}\right.,$$

(25)

where $t_0$ represents the last time backward ancestor sampling was conducted. The definition of each Enoch index shows that it retains the index of the ancestor at the last time backward ancestor sampling was conducted. Hence, a decrease in ${\widehat{N}}_{\mathrm{anc}}(t)$ corresponds to a decrease in the diversity of the ancestors. The value of ${\widehat{N}}_{\mathrm{anc}}(t)$ is used to determine whether we should conduct backward ancestor sampling.

Here, we show the procedure used in backward ancestor sampling to obtain the backward ancestor index $B_t^{(i)}$. According to Bayes’ theorem, the backward ancestor index is sampled as follows:

$$B_t^{(i)}\sim \mathcal{M}\left({\left\{w_{t-1}^{(l)}p({\mathit{x}}_t^{(i)}\mid {\mathit{x}}_{t-1}^{(l)})\right\}}_{l=1}^N\right).$$

(26)

We implement this sampling scheme efficiently using rejection-acceptance sampling [19,26].

Once ${\kappa }_t(x_t)$ is approximated, the smoothing sufficient statistics are estimated as follows:

$${\Phi }_{0:t;{\widehat{\theta }}_{t-1}}\left[S_t(x_{0:t})\right]\simeq \sum _{i=1}^N{w}_t^{(i)}{\kappa }_t^{(i)}.$$

(27)

Finally, we estimate the parameter using Equation (18). The parameter ${\widehat{\theta }}_t$, estimated using our AdaSmooth-based online EM algorithm, is also used in the controller described in Section 2.4.

Here, we briefly compare our proposed AdaSmooth-based online EM algorithm with the offline EM algorithm and their variants without considering control contexts. The offline EM algorithm requires many iterations with a batch of data. As a result, it has high computational costs. On the other hand, our AdaSmooth-based online EM algorithm requires no iteration for each step. Hence, our proposed framework realizes efficient online estimation of model parameters.

2.4. Control Strategy for the Dynamics

To design an efficient controller for the general state-space model, PF-based model predictive control (PF-MPC) [18] is integrated with our dynamics estimator in the proposed method. Model predictive control (MPC) is a theoretical framework used for feedback control in dynamical systems. In conventional MPC, an optimal control problem is formulated at time t over a future interval called a horizon, using the model ${\mathit{x}}_{t+1}=f({\mathit{x}}_t,{\mathit{u}}_t)$. Next, the problem is numerically solved to obtain the optimal control input series ${\left\{{\mathit{u}}_{\tau }^{*}\right\}}_{\tau \in \mathrm{horizon}}$ over the horizon. Finally, the first control input from this series is adopted as the actual input. Specifically, in conventional MPC, the optimal control problem in the horizon is treated as solving the Euler–Lagrange equation numerically, which is computationally expensive because it includes the derivative of the model concerning the state ${\mathit{x}}_t$ and control input ${\mathit{u}}_t$. In contrast, in PF-MPC, the optimal control problem is treated as a filtering problem for the control input ${\mathit{u}}_t$ based on an augmented state-space model within the horizon.

Figure 4 shows the overview of PF-MPC. Our framework has two distinct PF modules: one (state PF) acts as a state estimator, as described in Section 2.2, and the other (control PF) serves as a solver for the control-filtering problem defined below. First, the latent state particles and their associated weights ${\left\{({\mathit{x}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ are obtained via the PF. Then, the control input particles $\left\{{\overline{\mathit{u}}}_t^{(i)}\right\}$ are sampled from the proposal distribution $p({\overline{\mathit{u}}}_t\mid {\mathit{u}}_{t-1})$. Subsequently, by regarding the input variable as part of the latent state, we form augmented state particles ${\left\{({\mathit{x}}_t^{(i)},{\overline{\mathit{u}}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ over the horizon. With the given reference trajectory $\{{\mathit{r}}_t,\dots ,{\mathit{r}}_{T+T_H}\}$, we solve the control-filtering problem by applying the control PF within the horizon and obtain the filtering distribution of the control input $p({\overline{\mathit{u}}}_t\mid {\mathit{r}}_t,\dots ,{\mathit{r}}_{T+T_H})$. Here, $T_H$ is the duration of the horizon. Finally, we adopt ${\mathit{u}}_t$ as the actual input based on the filtering distribution of the initial control input.

Here, we discuss the details of PF-MPC. To formulate the control-filtering problem, we define an augmented state-space model over the horizon. The augmented state-space model consists of the augmented system model and the augmented observation model.

First, we formulate the augmented system model over the horizon. Within this horizon, we consider the control input ${\mathit{u}}_t$ as part of the augmented state ${\zeta }_t=({\overline{\mathit{x}}}_t,{\overline{\mathit{u}}}_t,{\tilde{\mathit{u}}}_t)$, where ${\tilde{\mathit{u}}}_t$ serves as an auxiliary variable to preserve the initial input throughout the horizon. Here, ${\overline{\mathit{x}}}_t,{\overline{\mathit{u}}}_t$, respectively, denote the predictive state and control input within the horizon. Note that the ${\overline{\mathit{x}}}_t$ and ${\overline{\mathit{u}}}_t$ notations are required to distinguish the predicted state and control input transitions over the horizon $\{({\overline{\mathit{x}}}_t,{\overline{\mathit{u}}}_t),\dots ,({\overline{\mathit{x}}}_{t+T_H},{\overline{\mathit{u}}}_{t+T_H})\}$ from the actual state and input time series $\{({\mathit{x}}_1,{\mathit{u}}_1),\dots ,({\mathit{x}}_t,{\mathit{u}}_t),\dots \}$. Therefore, we define the following augmented system model for the augmented state ${\zeta }_t$ within the horizon:

$${\zeta }_{\tau +1}\sim p({\zeta }_{\tau +1}\mid {\zeta }_{\tau };{\widehat{\theta }}_t)=p({\overline{\mathit{x}}}_{\tau +1}\mid {\overline{\mathit{x}}}_{\tau },{\overline{\mathit{u}}}_{\tau };{\widehat{\theta }}_t)p({\overline{\mathit{u}}}_{\tau +1}\mid {\zeta }_t)\delta ({\tilde{\mathit{u}}}_{\tau +1}-{\tilde{\mathit{u}}}_{\tau }),$$

(28)

where $\tau \in \{t,t+1,\dots ,t+T_H\}$ is the time index within the horizon, $p({\overline{\mathit{x}}}_{\tau +1}\mid {\overline{\mathit{x}}}_{\tau },{\overline{\mathit{u}}}_{\tau };{\widehat{\theta }}_t)$ is defined using the original system model within the state-space model described in Section 2.1, $p({\overline{\mathit{u}}}_{\tau +1}\mid {\zeta }_{\tau })$ is the transition probability, and $\delta ({\tilde{\mathit{u}}}_{\tau +1}-{\tilde{\mathit{u}}}_{\tau })$ is the probability corresponding to the deterministic transition ${\tilde{\mathit{u}}}_{\tau +1}={\tilde{\mathit{u}}}_{\tau }$. The parameter ${\widehat{\theta }}_t$ is estimated using our AdaSmooth-based online EM algorithm and is fixed throughout the horizon.

Next, we formulate the augmented observation model. We consider controlling the latent state ${\mathit{x}}_t$ toward the given reference trajectory $\{{\mathit{r}}_1,\dots ,{\mathit{r}}_t,\dots \}$. We define the augmented observation model as follows:

$${\mathit{r}}_{\tau }\sim p({\mathit{r}}_{\tau }\mid {\zeta }_{\tau }).$$

(29)

For example, a Gaussian distribution can be used as the observation model:

$${\mathit{r}}_{\tau }\sim \mathcal{N}({\mathit{r}}_{\tau }\mid R_{\tau }{\zeta }_{\tau },{\Sigma }_r),$$

(30)

where $R_{\tau }$ is an appropriately sized matrix and ${\Sigma }_r$ is a diagonal covariance matrix that adjusts the error between the state and reference trajectory. Note that the reference value’s dimension does not have to be the same as that of the latent variable, i.e., $R_{\tau }$ can be a non-square matrix. Furthermore, the augmented observation model is very flexible, but its details are omitted here (see [18]).

We then solve the control-filtering problem given by the augmented state-space model [Equations (28) and (29)] and the given reference trajectory ${\left\{{\mathit{r}}_{\tau }\right\}}_{\tau =t}^{t+T_H}$. Using control PF in the horizon $\{t,t+1,\dots ,t+T_H\}$, we solve the control-filtering problem efficiently to estimate the augmented state ${\zeta }_t=\left({\overline{\mathit{x}}}_t,{\overline{\mathit{u}}}_t,{\tilde{\mathit{u}}}_t\right)$. Here, the initial augmented particles ${\{{\zeta }_t^{(i)}=({\overline{\mathit{x}}}_t^{(i)},{\overline{\mathit{u}}}_t^{(i)},{\tilde{\mathit{u}}}_t^{(i)})\}}_{i=1}^N$ and their associated weights ${\left\{{\overline{w}}_t^{(i)}\right\}}_{i=1}^N$ in the horizon are initialized for each particle using information obtained from the state PF as follows:

$$\begin{array}{cc}\hfill {\overline{\mathit{x}}}_t^{(i)}& ={\mathit{x}}_t^{(i)},\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\overline{\mathit{u}}}_t^{(i)}& \sim p({\overline{\mathit{u}}}_t^{(i)}\mid {\mathit{u}}_t^{(i)}),\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill {\tilde{\mathit{u}}}_t^{(i)}& ={\overline{\mathit{u}}}_t^{(i)},\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill {\overline{w}}_t^{(i)}& =w_t^{(i)},\hfill \end{array}$$

(34)

where ${\left\{({\mathit{x}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ are the particles obtained from the state PF for the latent state estimation and $p({\overline{\mathit{u}}}_t^{(i)}\mid {\mathit{u}}_t^{(i)})$ is the initial proposal distribution for the control input. To avoid confusion, we note here that the aim of the control PF is to estimate the optimal control input obtained using the initial control input via the proposal distribution. For filtering the initial control input ${\overline{\mathit{u}}}_t$, an auxiliary variable ${\tilde{\mathit{u}}}_t$ is defined to preserve the initial input ${\overline{\mathit{u}}}_t$. Although ${\tilde{\mathit{u}}}_t^{(i)}={\overline{\mathit{u}}}_t^{(i)}$ at time t is defined in Equation (31), the predictive control inputs ${\overline{\mathit{u}}}_{t+1}^{(i)},{\overline{\mathit{u}}}_{t+2}^{(i)},\dots$ in the horizon have values different from ${\tilde{\mathit{u}}}_t^{(i)}$. Therefore we need both notations ${\overline{\mathit{u}}}_t$ and ${\tilde{\mathit{u}}}_{\tau }$ for the control PF in the horizon.

Finally, we estimate the optimal control input based on the filtered particles. By extracting the auxiliary variables that retain the initial control input and their associated weights ${\left\{({\tilde{\mathit{u}}}_{t+T_H}^{(i)},{\overline{w}}_{t+T_H}^{(i)})\right\}}_{i=1}^N$, we compute the actual control input at time t via point estimation of the filtering distribution as follows:

$${\mathit{u}}_t=\sum _{i=1}^N{\overline{w}}_{t+T_H}^{(i)}{\tilde{\mathit{u}}}_{t+T_H}^{(i)}.$$

(35)

It should be noted that the auxiliary variable ${\tilde{\mathit{u}}}_{t+T_H}$ inherits the information at time $t,\dots ,t+T_H-1$. Hence, to obtain the estimated optimal control input, we focus solely on these particles at time $t,\dots ,t+T_H-1$ within the horizon.

Here, we briefly discuss the differences between conventional MPC and PF-MPC. MPC requires derivatives of the model concerning the state and the control input. Furthermore, exact point estimation of the latent state is required. Conversely, PF-MPC does not require model derivatives and only relies on particles representing the latent variable. The only requirement of PF-MPC is that the model can be simulated forward in time, which is satisfied in typical cases. Nonetheless, under certain assumptions, the control input obtained using PF-MPC is equivalent to that of conventional MPC [18]. Moreover, to clarify the advantage of our proposed method compared with previous work, we briefly touch on the differences between the previous work [27] and our proposed framework. Three significant differences exist between the previous work and our proposed framework. First, the previous work updates parameters at fixed intervals. In contrast, our proposed framework updates parameters continuously, with the controller referencing these updated parameters; hence, our proposed framework is more adaptive. Second, the previous work uses the offline EM algorithm for parameter estimation. Due to the structure of the offline EM algorithm, the previous framework maintains a history of particles within a window. In contrast, our proposed method is based on the online EM algorithm. Using the online EM algorithm as the fundamental parameter estimation strategy, we retain only the preceding and succeeding particle sets; therefore, our framework is more memory-efficient. Third, in the previous work, there are complex hyperparameters, e.g., the parameter update interval and window length. These complex hyperparameters require sensitive hand-tuning and domain-specific knowledge. Conversely, the hyperparameters of our proposed framework are defined in a simpler form. For further details, the typical values of the hyperparameters for AdaSmooth are discussed in [19]. Therefore, our proposed framework is easier to tune compared with the previous work.

We have proposed a fully PF-based framework for simultaneously estimating the latent state and the parameters governing the dynamics, and controlling the dynamics against a general state-space model. Although conventional MPC can handle the model’s nonlinearity directly and PF-MPC can address the model’s uncertainty, the latent variable and dynamics are assumed to be known. The previous method [27] attempts to resolve this problem; however, it requires difficult hyperparameter tuning and applies only to neuronal models. Our proposed method establishes a feedback control law for general state-space models by combining a PF and an AdaSmooth-based online EM algorithm.

3. Experiments

We verify the effectiveness of our proposed method in a simulation environment. We apply our proposed method to two complex systems: the Lorenz system and the Morris–Lecar neuron model. For each distinct model, we first formulate its complex dynamics as a general SSM. Next, we derive the sufficient statistics necessary for our AdaSmooth-based online EM algorithm to estimate the parameters. Then, we formulate the augmented SSM over the horizon to control the dynamics. Finally, the experimental settings and results for each experiment are shown.

3.1. Application to Chaotic Lorenz System

The Lorenz system, introduced in [28], is known as a chaotic system. Because chaotic behavior is not always desirable, eliminating it is essential for practical application. Delayed feedback control has been proposed to stabilize chaotic models [29,30]. In contrast, a control strategy for chaotic behavior with simultaneous parameter estimation using statistical machine learning has not been studied. By applying our proposed method to the Lorenz system, we aim to confirm its ability to estimate the latent state and parameters of the chaotic Lorenz system and to stabilize its chaotic behavior.

The Lorenz system is given as follows:

$$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{dx}{d{t}^{\prime }}}& =\sigma (y-x),\hfill \\ \hfill {\displaystyle \frac{dy}{d{t}^{\prime }}}& =rx-y-xz+u,\hfill \\ \hfill {\displaystyle \frac{dz}{d{t}^{\prime }}}& =xy-bz,\hfill \end{array}\right.$$

(36)

where $t^{\prime }$ represents continuous time; ${(x,y,z)}^{\top }\in {\mathbb{R}}^3$ represents the three-dimensional latent states; u is the control input; and $\sigma ,r,b$ are parameters. Based on a previous study on suppressing chaotic behavior [30], we assume the control input only affects one element, y. To avoid confusion, the variables used in the Lorenz system are shown in

Table 1. Input and output variables used in the Lorenz system.

	Variable	Description
Input	u	Control input
Output	$x,y,z$	Latent state
Output	$\sigma ,r,b$	Parameters

. When we fix $u=0$, the Lorenz system with $\sigma =10,b=8/3,r>r_H$ exhibits subcritical Hopf bifurcation, where $r_H\simeq 24.73684211$. We aim to control the y-element value toward the unstable fixed point $y_f=\sqrt{b(r-1)}$ under noisy environmental conditions of the latent state where only noisy observations of the latent state can be observed. To this end, we first derive the SSM for the Lorenz system. We then derive the E-step and the M-step of our online EM algorithm. Finally, we formulate the augmented SSM for the Lorenz system to control the latent state. With these Lorenz system formulations, we apply the proposed method to the Lorenz system.

3.1.1. Formulation of State-Space Model

Here, we derive the SSM for the Lorenz system. We first derive the system model of the SSM. We discretize the continuous-time model given by Equation (36) with a time step $\Delta t$.

$${\mathit{x}}_{t+1}={\mathit{x}}_t+f({\mathit{x}}_t,u_t;\theta )\Delta t,$$

(37)

where ${\mathit{x}}_t={[x_t,y_t,z_t]}^{\top }\in {\mathbb{R}}^3$ corresponds to the latent state at time $t\Delta t$, $u_t\in \mathbb{R}$ is the control input at time $t\Delta t$, and $\theta ={[\sigma ,r,b]}^{\top }\in {\mathbb{R}}^3$ corresponds to the unknown parameters of the system. $f({\mathit{x}}_t,u_t;\theta )$ is a function that describes the dynamics of the Lorenz system as follows:

$$f({\mathit{x}}_t,u_t;\theta )=\left[\begin{array}{c}\sigma (y_t-x_t)\\ r{x}_t-y_t-x_t{z}_t+u_t\\ x_t{y}_t-b{z}_t\end{array}\right].$$

(38)

Next, we introduce system noise governed by additive Gaussian noise with zero mean and covariance matrix ${\Sigma }_x=\mathrm{diag}({\sigma }_{x_1}^2,{\sigma }_{x_2}^2,{\sigma }_{x_3}^2)\in {\mathbb{R}}^{3\times 3}$. Here, $\mathrm{diag}(a_1,a_2,a_3)$ denotes the diagonal matrix with diagonal elements $a_1,a_2,a_3$. The system model of the SSM for the Lorenz system is formulated as follows:

$$p({\mathit{x}}_{t+1}\mid {\mathit{x}}_t,u_t,\theta )=\mathcal{N}({\mathit{x}}_{t+1}\mid {\mathit{x}}_t+f({\mathit{x}}_t,u_t;\theta )\Delta t,{\Sigma }_x\Delta t).$$

(39)

Next, we derive the observation model of the SSM. We assume that the latent state with additive Gaussian noise is obtained. Hence, we formulate the observation model as follows:

$${\mathit{y}}_t\sim \mathcal{N}({\mathit{y}}_t\mid {\mathit{x}}_t,{\Sigma }_y),$$

(40)

where ${\Sigma }_y\in {\mathbb{R}}^{3\times 3}$ is the covariance matrix corresponding to the observation noise.

3.1.2. Derivation of Online EM Algorithm

Here, we derive the sufficient statistics $S_t({\mathit{x}}_{0:t})$ and the update function $\Lambda (·)$ for the Lorenz SSM, using Equations (39) and (40), in our AdaSmooth-based online EM algorithm. After calculation, we obtain the maximizer of the expectation of log-likelihood $\Lambda (·)$ as follows:

$$\Lambda (S_t=(A_t,b_t))=A_t^{-1}{b}_t,$$

(41)

where the sufficient statistics $S_t=(A_t,b_t)$ comprise two additive-form statistics $A_t\in {\mathbb{R}}^{3\times 3}$ and $b_t\in {\mathbb{R}}^3$, given as follows:

$$\begin{array}{c}{A}_t({\mathit{x}}_{0:t})=\sum _{\tau =1}^t{\tilde{A}}_{\tau }({\mathit{x}}_{\tau -1:\tau }),\end{array}$$

(42)

$$\begin{array}{c}{b}_t({\mathit{x}}_{0:t})=\sum _{\tau =1}^t{\tilde{b}}_{\tau }({\mathit{x}}_{\tau -1:\tau }),\end{array}$$

(43)

where ${\tilde{A}}_{\tau }$ and ${\tilde{b}}_{\tau }$ are defined as follows:

$$\begin{array}{c}{\tilde{A}}_{\tau }({\mathit{x}}_{\tau -1:\tau })=\mathrm{diag}({\displaystyle \frac{{(y_{\tau -1}-x_{\tau -1})}^2}{{\sigma }_{x_1}^2}},{\displaystyle \frac{x_{\tau -1}^2}{{\sigma }_{x_2}^2}},{\displaystyle \frac{z_{\tau -1}^2}{{\sigma }_{x_3}^2}})\Delta t,\end{array}$$

(44)

$$\begin{array}{c}{\tilde{b}}_{\tau }({\mathit{x}}_{\tau -1:\tau })=\left[\begin{array}{c}{\displaystyle \frac{y_{\tau -1}-x_{\tau -1}}{{\sigma }_{x_1}^2}}(x_{\tau }-x_{\tau -1})\\ {\displaystyle \frac{x_{\tau -1}}{{\sigma }_{x_2}^2}}\{y_{\tau }-y_{\tau -1}+(y_{\tau -1}+x_{\tau -1}{z}_{\tau -1}-u_{\tau -1})\Delta t\}\\ -{\displaystyle \frac{z_{\tau -1}}{{\sigma }_{x_3}^2}}(z_{\tau }-z_{\tau -1}-x_{\tau -1}{y}_{\tau -1}\Delta t)\end{array}\right].\end{array}$$

(45)

3.1.3. Formulation of the Augmented State-Space Model for Control

We aim to suppress the chaotic behavior and control the y-element of the latent state toward the fixed point $y_f=\sqrt{b(r-1)}$ [30]. First, we define the control transition over the horizon. To consider realistic scenarios, we impose a constraint concerning the control input at every time t as follows:

$$-u_{\lim }\le u_t\le u_{\lim },$$

(46)

where $u_{\lim }>0$ is the limit of the control input. Therefore, we define the augmented system model for the control input transition as follows:

$${\overline{u}}_{\tau +1}={\mathrm{clamp}}_{u_{\lim }}(u_{\tau }+z_{\tau +1}),$$

(47)

where $z_{\tau +1}\sim \mathcal{N}(0,{\sigma }_u^2)$ is Gaussian noise and ${\mathrm{clamp}}_{u_{\lim }}(·)$ is a clamp function defined as follows:

$${\mathrm{clamp}}_{u_{\lim }}(u)={\displaystyle \frac{|u+u_{\lim }|-|u-u_{\lim }|}{2}}.$$

(48)

To initialize the augmented particles, we define the following initialization procedure corresponding to the initial proposal distribution $p({\overline{u}}_t^{(i)}\mid u_t^{(i)})$:

$${\overline{u}}_t^{(i)}={\mathrm{clamp}}_{u_{\lim }}(u_t+z_t^{(i)}),$$

(49)

where $z_t^{(i)}\sim \mathcal{N}(0,{\sigma }_{u0}^2)$ is Gaussian noise. Note that we do not need a probabilistic form of the augmented SSM.

We then define the augmented observation model of the augmented SSM over the horizon. We aim to control the y-element of the latent state toward the fixed point $y_f$; hence, we define the augmented observation model for the PF-MPC controller as follows:

$$r_{\tau }\sim \mathcal{N}(r_{\tau }\mid y_t,{\sigma }_r^2),$$

(50)

where ${\sigma }_r^2\in \mathbb{R}$ is a variance that adjusts for acceptable error against the reference trajectory. The given reference value is also always equivalent to $y_f$, i.e., $r_t=y_f$ for $t=1,2,\dots$.

3.1.4. Settings

The true dynamics are assumed to be the Lorenz system defined by Equation (36), and in the simulation, they are solved using the Runge–Kutta method. Here, the true parameters are $\sigma =10$ and $r=28,b=8/3$. We apply our proposed method to the Lorenz system to suppress its chaotic behavior toward the unstable fixed point $y_f=\sqrt{b(r-1)}$. Note that we can only observe the noisy time series of the latent variable, not the true value.

The hyperparameters are set as follows. For simulation, the time step is $\Delta t=0.01$. The variances of the system noise and the observation noise are ${\Sigma }_x=\mathrm{diag}({\sigma }_{x_1}^2,{\sigma }_{x_2}^2,{\sigma }_{x_3}^2)=\mathrm{diag}(1^2,1^2,1^2)$ and ${\Sigma }_y=\mathrm{diag}(1^2,1^2,1^2)$, respectively. For the PF, the number of particles is $N=1000$, and the threshold rate of resampling is $\alpha =0.8$. The initial particles ${\left\{({\mathit{x}}_t^{(i)},w_t^{(i)})\right\}}_{i=1}^N$ are sampled from the initial distribution, defined as follows:

$$p_0(\mathit{x})=\mathcal{N}(\mathit{x}\mid {\mathit{x}}_0,{\Sigma }_{x0}),$$

(51)

where ${\Sigma }_{x0}=\mathrm{diag}({10}^2,{10}^2,{10}^2)$. For parameter estimation, we use the threshold rate of backward resampling $\beta =0.7$ and the decay rate ${\gamma }_t=t^{-1}$. Following [16], we start updating the parameter after $T_{\mathrm{burnIn}}$ steps. In this experiment, we set $T_{\mathrm{burnIn}}=100$. To control the Lorenz system, we set the number of steps in the horizon to $T_H=10$. As a hyperparameter in the augmented SSM, the acceptable error variance and the control input limit are ${\sigma }_r^2=1^2$ and $u_{\lim }=10$, respectively. The variances of the augmented SSM’s control transitions are ${\sigma }_u^2=1^2$ and ${\sigma }_{u0}^2={10}^2$.

3.1.5. Results

Figure 5 shows the simulation results after applying our proposed framework to the Lorenz system. We confirmed that our method can simultaneously estimate and control the Lorenz dynamics. To verify whether our proposed method can suppress chaotic behavior, we did not adopt the control input until $t=5$.

The graphs on the left in Figure 5 show the $x,y,z$ elements of the latent state from top to bottom, respectively. The state estimated by the PF represents the filtering distribution of the latent state. Hence, to confirm that we can estimate the latent variable, the estimated value was calculated using the weighted mean of the particles as follows:

$${\mathit{x}}_{\mathrm{est},t}=\sum _{i=1}^N{w}_t^{(i)}{\mathit{x}}_t^{(i)}.$$

(52)

The three graphs on the left in Figure 5 show the estimated values $\left(x_{\mathrm{est}},y_{\mathrm{est}},z_{\mathrm{est}}\right)$ (blue solid lines) and the true values $\left(x_{\mathrm{true}},y_{\mathrm{true}},z_{\mathrm{true}}\right)$ (green dashed lines), respectively. We found that the estimated values tracked the true values over time. In particular, the middle graph in Figure 5 also shows the target unstable fixed point $y_f$ (blue dotted line). We also found that the estimated values $\left(x_{\mathrm{est}},y_{\mathrm{est}},z_{\mathrm{est}}\right)$ accurately reproduced the true values $\left(x_{\mathrm{true}},y_{\mathrm{true}},z_{\mathrm{true}}\right)$, and the state was successfully manipulated. Here, the red dash-dotted line represents the simulation results when no input was applied, as shown in the graphs on the left in Figure 5. Without our proposed method, the Lorenz system still exhibited chaotic behavior. Conversely, after applying our proposed method, the Lorenz system stabilized and converged to the fixed point $y_f$.

To confirm whether our proposed method could suppress chaotic behavior, we visualized the trajectory of the latent variable $(x_t,y_t,z_t)$ for the non-controlled and controlled Lorenz systems in a three-dimensional graph. Figure 6 shows the two trajectories of the non-controlled Lorenz system and the Lorenz system controlled by our proposed automatic control law. The left side of Figure 6 shows the simulation results of the true dynamics when the control input was fixed at zero: $u_t=0$. When we did not adopt any control input, the Lorenz system exhibited chaotic behavior around two fixed points. Conversely, by applying our proposed framework, as shown on the right in Figure 6, the Lorenz system stabilized around the fixed point determined in advance.

Next, we clarified whether our proposed method could estimate the unknown parameters. The graphs on the right in Figure 5 show the parameters governing the Lorenz system. Each estimated parameter value of three types $({\sigma }_t,r_t,b_t)$ of the Lorenz system converged to the true values over time, although the estimated value for ${\sigma }_t$ oscillated slightly. In particular, the bifurcation parameter $r_t$, which influences the dynamics of y, was accurately estimated by our AdaSmooth-based online EM algorithm. Additionally, $b_t$, governing the dynamics of z, was also estimated accurately by our proposed method.

Finally, we clarified whether our proposed method could estimate and control the Lorenz dynamics using the control input $u_t$ with a limited range of strengths. Figure 7 shows the control input (blue line) and the limit value (blue dash-dotted line). We maintained $u_t=0$ during the time period marked with a gray background. The control input $u_t$ always satisfied the norm constraint $|u_t|\le u_{\lim }$. Hence, we can control the Lorenz dynamics simultaneously, even if we impose input constraints.

3.2. Application to Morris–Lecar Neuron Model

It is important to estimate and control the nonlinear dynamics of neurons to understand nerve systems and brain functions [31,32]. Nonetheless, it is difficult to control neuronal dynamics due to their complexity and partial observability. Typically, only noisy membrane potentials are observable [33,34,35].

Following previous work [27], this section aims to control the time series of membrane potentials toward a given reference trajectory under realistic conditions, where only noisy time series of membrane potentials can be observed. To this end, we first derive the SSM for the Morris–Lecar neuron model. We then derive the sufficient statistics and the update function for our online AdaSmooth-based EM algorithm for online parameter estimation. Finally, we formulate the augmented SSM for the Morris–Lecar neuron model to control the membrane potential. We apply the proposed method to the Morris–Lecar neuron model based on these SSM formulations.

3.2.1. Formulation of State-Space Model

In the Morris–Lecar neuron model [36], the nonlinear dynamics of the membrane potential v and the channel variable n are described by the following continuous- time model:

$$\left\{\begin{array}{cc}\hfill C_m{\displaystyle \frac{dv}{d{t}^{\prime }}}& =-g_L(v-E_L)-g_{Ca}{m}_{\infty }(v)(v-E_{Ca})-g_{K}n(v-E_K)+I\hfill \\ \hfill \tau (v){\displaystyle \frac{dn}{d{t}^{\prime }}}& =-n+n_{\infty }(v),\hfill \end{array}\right.$$

(53)

where v is the membrane potential, n is the channel variable, and I is the external input current. Here, $\tau (v)={\displaystyle \frac{1}{\varphi }}{cosh}^{-1}({\displaystyle \frac{v-V_3}{2V_4}})$ is a function that determines the speed of the dynamics. $m_{\infty }(v)$ and $n_{\infty }(v)$ are nonlinear functions that describe the calcium and potassium dynamics; $E_L,E_{Ca},$ and $E_K$ are the reversal potentials; and $C_m$ is the membrane capacitance [27]. To avoid confusion, the variables used in the Morris–Lecar neuron model are shown in

Table 2. Input and output variables used in the Morris–Lecar neuron model.

	Variable	Description
Input	I	External input current
Output	$v,n$	Latent state
Output	$g_L,g_{Ca},g_K$	Parameters

.

First, we derive the system model for the SSM. The continuous-time model given by Equation (53) is discretized with a time step $\Delta t$.

$${\mathit{x}}_{t+1}={\mathit{x}}_t+f({\mathit{x}}_t,u_t;\theta )\Delta t,$$

(54)

where ${\mathit{x}}_t={[v_t,n_t]}^{\top }\in {\mathbb{R}}^2$ corresponds to the latent state at time $t\Delta t$, $u_t\in \mathbb{R}$ corresponds to the control input at time $t\Delta t$, and $\theta ={[g_L,g_{Ca},g_K]}^{\top }\in {\mathbb{R}}^3$ corresponds to the unknown parameters. Following previous work [27], we separate the net input $I_t$ into the controllable input $u_t$ and the known streaming currents from other neurons $I_t^{\mathrm{inj}}$, i.e., $I_t=u_t+I_t^{\mathrm{inj}}$. $f({\mathit{x}}_t,u_t;\theta )$ is a function representing the dynamics of the Morris–Lecar neuron model as follows:

$$f({\mathit{x}}_t,u_t;\theta )=\left[\begin{array}{c}-{\displaystyle \frac{1}{C_m}}\{g_L(v_t-E_L)+g_{Ca}{m}_{\infty }(v_t)(v_t-E_{Ca})+g_{K}n(v_t-E_K)\}+{\displaystyle \frac{1}{C_m}}{I}_t\\ -\varphi cosh\left({\displaystyle \frac{v_t-V_3}{2V_4}}\right)(n_t-n_{\infty }(v_t))\end{array}\right].$$

(55)

We then introduce system noise governed by white Gaussian noise with zero mean and covariance matrix ${\Sigma }_x=\mathrm{diag}({\sigma }_v^2,{\sigma }_n^2)\in {\mathbb{R}}^{2\times 2}$. The system model of the SSM for the neuron model is formulated as follows:

$$p({\mathit{x}}_{t+1}\mid {\mathit{x}}_t,u_t,\theta )=\mathcal{N}({\mathit{x}}_{t+1}\mid {\mathit{x}}_t+f({\mathit{x}}_t,u_t;\theta )\Delta t,{\Sigma }_x\Delta t).$$

(56)

Next, we formulate the observation model for the SSM. We consider that only noisy membrane potentials $y_t\in \mathbb{R}$ can be observed. Introducing additive Gaussian noise as observation noise, we formulate the observation model as follows:

$$y_t\sim \mathcal{N}(y_t\mid v_t,{\sigma }_y^2),$$

(57)

where ${\sigma }_y^2\in \mathbb{R}$ is the variance corresponding to the observation noise.

3.2.2. Derivation of Online EM Algorithm

Here, we derive the sufficient statistics $S_t({\mathit{x}}_{0:t})$ and the update function $\Lambda (·)$ for the neuron model described by Equations (56) and (57), which are used in our AdaSmooth-based online EM algorithm. After calculation, we obtain the maximizer of the expectation of log-likelihood $\Lambda (·)$ as follows:

$$\Lambda (S_t=(A_t,b_t))=-A_t^{-1}{b}_t,$$

(58)

where the sufficient statistics $S_t=(A_t,b_t)$ include two additive-form statistics $A_t({\mathit{x}}_{0:t})\in {\mathbb{R}}^{3\times 3}$ and $b_t({\mathit{x}}_{0:t})\in {\mathbb{R}}^3$, given as follows:

$$\begin{array}{c}{A}_t({\mathit{x}}_{0:t})=\sum _{\tau =1}^t{\left({\displaystyle \frac{\Delta t}{C_m}}\right)}^2{V}_{\mathrm{ion}}({\mathit{x}}_{\tau -1})V_{\mathrm{ion}}^{\top }({\mathit{x}}_{\tau -1}),\end{array}$$

(59)

$$\begin{array}{c}{b}_t({\mathit{x}}_{0:t})=\sum _{\tau =1}^t{\tilde{b}}_{\tau }({\mathit{x}}_{\tau -1:\tau })={\displaystyle \frac{\Delta t}{C_m}}(v_{\tau }-v_{\tau -1}-{\displaystyle \frac{\Delta t}{C_m}}{I}_{\tau -1})V_{\mathrm{ion}}({\mathit{x}}_{\tau -1}).\end{array}$$

(60)

Here, $V_{\mathrm{ion}}({\mathit{x}}_t)\in {\mathbb{R}}^3$ represents the potentials of each ion channel, given as follows:

$$V_{\mathrm{ion}}({\mathit{x}}_t)=\left[\begin{array}{c}{v}_t-E_L\\ m_{\infty }(v_t)(v_t-E_{Ca})\\ n_t(v_t-E_K)\end{array}\right].$$

(61)

3.2.3. Formulation of the Augmented State-Space Model for Control

Here, we formulate the augmented SSM over the horizon. First, we derive the control transition of the augmented system model $p({\overline{u}}_{\tau +1}\mid {\zeta }_{\tau })$. In realistic scenarios, it is essential to guarantee that the net input current $I_t=u_t+I_t^{\mathrm{inj}}$ is always within a specific range. Therefore, we impose a norm constraint on the control input at every time t as follows:

$$|u_t+I_t^{\mathrm{inj}}|\le I_{\lim },$$

(62)

where $I_t^{\mathrm{inj}}$ is the current injected from other neurons and $I_{\lim }$ is the limit current. To this end, the control transition within the horizon is described as follows:

$${\overline{u}}_{\tau +1}={\mathrm{clip}}_{I_{\lim }}({\overline{u}}_{\tau }+{\overline{I}}_{\tau +1}^{\mathrm{inj}}+z_{\tau +1})-{\overline{I}}_{\tau +1}^{\mathrm{inj}},$$

(63)

where $z_{\tau +1}\sim \mathcal{N}(0,{\sigma }_I^2)$ is Gaussian noise. The proposed framework fixes the prediction of the external currents ${\overline{I}}_{\tau +1}^{\mathrm{inj}}$ at the value observed at time t, i.e., ${\overline{I}}_{\tau +1}^{\mathrm{inj}}=I_t^{\mathrm{inj}}$ for $\tau \in \left\{t,t+1,\dots ,t+T_H-1\right\}$. Furthermore, the initialization for the control input particles is described as follows:

$${\overline{u}}_t^{(i)}={\mathrm{clip}}_{I_{\lim }}(u_{t-1}^{(i)}+{\overline{I}}_t^{\mathrm{inj}}+z_t^{(i)})-{\overline{I}}_t^{\mathrm{inj}},$$

(64)

where $z_t^{(i)}\sim \mathcal{N}(0,{\sigma }_{I0}^2)$ is Gaussian noise.

Next, we derive the augmented observation model for the augmented SSM over the horizon. We define the following augmented observation model to control the time series of membrane potentials toward a given reference trajectory $r_t\in \mathbb{R}$:

$$r_t\sim \mathcal{N}(r_t\mid v_t,{\sigma }_r^2),$$

(65)

where ${\sigma }_r^2\in \mathbb{R}$ is a variance that adjusts for acceptable error between the predictive state transition and the reference trajectory within the horizon.

3.2.4. Settings

The true dynamics are assumed to follow the Morris–Lecar neuron model defined by Equation (53), and in simulation, they are solved using the Euler method. Here, the true parameters are those for the Homoclinic [37]. We apply our proposed method to the Morris–Lecar neuron model to control the membrane potentials $v_t$ toward the desired reference membrane potentials $r_t$. The reference trajectory $r_t$ is generated by the true model in advance. Note that we can only observe noisy membrane potentials; that is, we assume a partial observation situation.

The hyperparameters are set as follows. For simulation, the time step is $\Delta t=0.1$. The variances of the system noise and observation noise are ${\Sigma }_x=\mathrm{diag}({\sigma }_v^2,{\sigma }_n^2)=\mathrm{diag}(0.1^2,0.{001}^2)$ and ${\sigma }_y^2=0.1^2$, respectively. For the PF, the number of particles is $N=1000$, and the threshold rate of resampling is $\alpha =0.8$. The initial distribution for the initial particles $p_0(\mathit{x})$ is $p_0(\mathit{x})=\mathcal{N}(v\mid {\displaystyle \frac{2}{3}}{E}_L,{10}^2)\mathcal{U}(n\mid 0,1)$, where $\mathcal{U}(n\mid 0,1)$ is a uniform distribution within $[0,1]$. For parameter estimation, we use the threshold rate of backward resampling $\beta =0.7$ and the decay rate ${\gamma }_t=t^{-1}$. We update the parameters after $T_{\mathrm{burnIn}}=1000$ steps. To control the Morris–Lecar neuron model, we set the number of steps in the horizon as $T_H=10$. As a hyperparameter in the augmented SSM, the acceptable error variance and the control input limit are set to ${\sigma }_r^2=0.1^2$ and $I_{\lim }=150$, respectively. The variances of the augmented SSM’s control transitions are ${\sigma }_I^2=1^2$ and ${\sigma }_{I0}^2={10}^2$.

3.2.5. Results

Figure 8 shows the simulation results after applying our proposed framework to the Morris–Lecar neuronal system. Here, we controlled the neuronal dynamics toward the reference trajectory, which represents the Homoclinic behavior during time intervals $0\le t^{\prime }\le 250$ and $500\le t^{\prime }\le 750$, and the resting state during intervals $250\le t^{\prime }\le 500$ and $750\le t^{\prime }\le 1000$. First, we found that our method could simultaneously estimate and control the Morris–Lecar neuron model. The middle-left graph shows the true membrane potential $v_{\mathrm{true}}$ (blue solid line), the estimated value $v_{\mathrm{est}}$ (green dashed line), and the target membrane potential $v_{\mathrm{ref}}$ (red dotted line). We found that the estimated value tracked the true value, as shown in the middle-left graph in Figure 8. In particular, the pale purple line represents the simulation results without adopting any control input. Without our proposed method, the Morris–Lecar neuron model converged to a specific value. However, by applying our proposed method, the Morris–Lecar neuron model was controlled toward the desired trajectory, allowing us to switch between neuronal firing and resting states. The bottom-left graph shows the true value $n_{\mathrm{true}}$ (blue solid line) and the estimated value $n_{\mathrm{est}}$ (green dashed line). We can see that the estimated value also closely tracked the true value.

Next, we verified whether our proposed method could estimate the parameters governing the dynamics of the neuron model. The right side of Figure 8 shows the parameters $(g_L,g_{Ca},g_K)$ governing the Morris–Lecar neuronal model. As shown in the three graphs on the right, each estimated parameter value converged to the true value over time, although the estimated values spiked around 500 to 600.

Finally, we verified whether the proposed framework could estimate and control the neuronal dynamics under varying strengths of the control input $u_t$. The top-left graph shows the controllable input current u (blue solid line), the injected current $I_{\mathrm{inj}}$ (green dashed line), the actual net input $I=u+I_{\mathrm{inj}}$, and the limit value (light-blue dash-dotted line). The net input $u=u+I_{\mathrm{inj}}$ always satisfied the norm constraint $|u+I_{\mathrm{inj}}|\le I_{\lim }$. Hence, we can simultaneously estimate and control the nonlinear neuronal dynamics while adhering to the constraint for the control signal.

4. Conclusions

In this paper, we have proposed a probabilistic framework for simultaneously estimating and controlling general dynamics entirely based on the particle filter. By introducing the particle filter not only as a state estimator and a prior estimator for the dynamics but also as a controller, we can directly handle the nonlinearity of the dynamics and the uncertainty of the latent state effectively and efficiently. With adaptive backward resampling, the proposed framework can suppress the degeneracy of the particles with respect to the sufficient statistics and realize accurate parameter estimation. Through experiments on two general models, including nonlinearity in distinct fields, we verified the effectiveness of our proposed framework. By applying our proposed framework to the Lorenz system in a simulation environment, we found that the proposed framework can not only estimate the latent variables from noisy observational data and the parameters governing chaotic behavior but also control the latent state toward the fixed point. Furthermore, by applying our proposed framework to the Morris–Lecar neuron model, we showed that even under noisy partial observation conditions, it can estimate the latent state and parameters and control the latent state toward the desired trajectory. Although our proposed method can simultaneously estimate the latent state and parameters and control nonlinear complex dynamics, our framework cannot handle semi-parametric or non-parametric models directly. In future work, we plan to extend our proposed framework to handle such semi-parametric or non-parametric models.