# Dynamic Expectation Maximization Algorithm for Estimation of Linear Systems with Colored Noise

^{*}

## Abstract

**:**

## 1. Introduction

- Reformulating DEM into an estimation algorithm for LTI systems with colored noise (Section 12).
- Proving that the estimator has theoretical guarantees of convergence for the estimation steps (Section 14).
- Proving through rigorous simulation that DEM outperforms the state-of-the-art system identification methods for parameter estimation under colored noise (Section 16).

## 2. Problem Statement

## 3. Preliminaries

#### 3.1. Generative Model

#### 3.2. Parameters and Hyperparameters

#### 3.3. Colored Noise

#### 3.4. Generalized Motion of the Outputs and Noises

#### 3.5. Notations and Conventions

## 4. Free Energy Objectives

## 5. Laplace Approximation

- It simplifies the internal energy expression $U(\vartheta ,y)$,
- It facilitates an easy computation of the conditional precision ${\mathsf{\Pi}}^{\vartheta}$ (derived in Section 7.2) as the negative curvature of the internal energy at it’s mode ${\mu}^{\vartheta}$.

- the states and inputs, which are time-varying and therefore expressed in generalized coordinates,
- the parameters and hyperparameters, which are time-invariant and not expressed in generalized coordinates.

#### 5.1. Generative Model

#### 5.2. Prior Distributions

#### 5.3. Simplification of the Internal Energy Action $\overline{U}$

## 6. Mean-Field Approximation

#### 6.1. Simplification of the Entropy Action $\overline{H}$

#### 6.2. Mean-Field Terms

## 7. Simplified Free Energy Objectives

#### 7.1. Simplification of Free Action

#### 7.2. Simplification of the Parameter Precisions

#### 7.3. Free Action at Optimal Precision

#### 7.4. Equivalence with the EM Algorithm

- the mean field terms are neglected,
- the generalized motion is not considered, and
- the robot’s priors on $\vartheta $ are not considered.

## 8. Update Rules for Estimation

- a gradient ascent over its free action $\overline{F}$ for the time invariant parameters $\theta $ and $\lambda $,
- a gradient ascent over its free energy F for the time varying parameters X,

#### 8.1. The DEM Algorithm

- D step: (generalized) state and input estimation,
- E step: parameter estimation,
- M step: noise hyperparameter estimation,

#### 8.2. Updated Equations for Estimation

Algorithm 1: Dynamics Expectation Maximization |

#### 8.3. Update Equation for Precision of Estimates

## 9. Gradients of (Log Determinant of) Precision

- It simplifies the precision update rule for hyperparameters given in Equation (51).

## 10. Gradients of Prediction Error

#### 10.1. Gradients of Prediction Error along (Generalized) States

#### 10.2. Gradients of Prediction Error along Parameters

#### 10.3. Gradients of Prediction Error along Hyperparameters

## 11. Gradients of Mean Field Terms

#### 11.1. Gradients of Mean Field Terms along Hyperparameters

#### 11.2. Gradients of Mean Field Terms along Generalized States

#### 11.3. Gradients of Mean Field Terms along Parameters

## 12. The Complete DEM Algorithm

#### 12.1. DEM Estimates

#### 12.2. Precision of Estimates

Algorithm 2: Dynamics Expectation Maximization |

## 13. Translation into Simplified Mathematical Form

#### 13.1. State and Input Estimation as a Linear Observer

#### 13.2. Parameter Estimation—System Identification

#### 13.3. Hyperparameter Update

## 14. Convergence Proof for Parameter and Hyperparameter Estimation

## 15. A Demonstrative Example

#### 15.1. Generative Model

#### 15.2. Priors for Estimation

#### 15.3. Results of Estimation

## 16. Benchmarking

#### 16.1. Evaluation Metric for Parameter Estimation

#### 16.2. Simulation Setup

#### 16.3. Results

## 17. Discussion

## 18. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

LTI system | Linear time invariant system |

DEM | Dynamic Expectation Maximization |

FEP | Free energy principle |

KF | Kalman Filter |

KL divergence | Kullback–Leibler divergence |

## References

- Friston, K. The free-energy principle: A unified brain theory? Nat. Rev. Neurosci.
**2010**, 11, 127–138. [Google Scholar] [CrossRef] [PubMed] - Anil Meera, A.; Wisse, M. Free Energy Principle Based State and Input Observer Design for Linear Systems with Colored Noise. In Proceedings of the 2020 American Control Conference (ACC), Denver, CO, USA, 1–3 July 2020; pp. 5052–5058. [Google Scholar]
- Bos, F.; Anil Meera, A.; Benders, D.; Wisse, M. Free Energy Principle for State and Input Estimation of a Quadcopter Flying in Wind. arXiv
**2021**, arXiv:2109.12052. [Google Scholar] - Pezzato, C.; Ferrari, R.; Corbato, C.H. A novel adaptive controller for robot manipulators based on active inference. IEEE Robot. Autom. Lett.
**2020**, 5, 2973–2980. [Google Scholar] [CrossRef][Green Version] - Pezzato, C.; Hernandez, C.; Wisse, M. Active Inference and Behavior Trees for Reactive Action Planning and Execution in Robotics. arXiv
**2020**, arXiv:2011.09756. [Google Scholar] - Baioumy, M.; Duckworth, P.; Lacerda, B.; Hawes, N. Active inference for integrated state-estimation, control, and learning. arXiv
**2020**, arXiv:2005.05894. [Google Scholar] - Oliver, G.; Lanillos, P.; Cheng, G. Active inference body perception and action for humanoid robots. arXiv
**2019**, arXiv:1906.03022. [Google Scholar] - Çatal, O.; Verbelen, T.; Van de Maele, T.; Dhoedt, B.; Safron, A. Robot navigation as hierarchical active inference. Neural Netw.
**2021**, 142, 192–204. [Google Scholar] [CrossRef] - Lennart, L. System Identification: Theory for the User; PTR Prentice Hall: Upper Saddle River, NJ, USA, 1999; Volume 28. [Google Scholar]
- Zhang, L.Q.; Cichocki, A.; Amari, S. Kalman filter and state-space approach to blind deconvolution. In Neural Networks for Signal Processing X, Proceedings of the 2000 IEEE Signal Processing Society Workshop, Sydney, Australia, 11–13 December 2000; IEEE: Piscataway, NJ, USA, 2000; pp. 425–434. [Google Scholar]
- Han, R.; Bohn, C.; Bauer, G. Blind identification of state-space models in physical coordinates. arXiv
**2021**, arXiv:2108.08498. [Google Scholar] - Abed-Meraim, K.; Qiu, W.; Hua, Y. Blind system identification. Proc. IEEE
**1997**, 85, 1310–1322. [Google Scholar] [CrossRef][Green Version] - Zhang, Y. Unbiased identification of a class of multi-input single-output systems with correlated disturbances using bias compensation methods. Math. Comput. Model.
**2011**, 53, 1810–1819. [Google Scholar] [CrossRef] - Liu, X.; Lu, J. Least squares based iterative identification for a class of multirate systems. Automatica
**2010**, 46, 549–554. [Google Scholar] [CrossRef] - Zheng, W.X. On a least-squares-based algorithm for identification of stochastic linear systems. IEEE Trans. Signal Process.
**1998**, 46, 1631–1638. [Google Scholar] [CrossRef] - Zhang, Y.; Cui, G. Bias compensation methods for stochastic systems with colored noise. Appl. Math. Model.
**2011**, 35, 1709–1716. [Google Scholar] [CrossRef][Green Version] - Cui, T.; Chen, F.; Ding, F.; Sheng, J. Combined estimation of the parameters and states for a multivariable state-space system in presence of colored noise. Int. J. Adapt. Control. Signal Process.
**2020**, 34, 590–613. [Google Scholar] [CrossRef] - Friston, K.J.; Trujillo-Barreto, N.; Daunizeau, J. DEM: A variational treatment of dynamic systems. Neuroimage
**2008**, 41, 849–885. [Google Scholar] [CrossRef] [PubMed] - Van de Laar, T.; Özçelikkale, A.; Wymeersch, H. Application of the free energy principle to estimation and control. arXiv
**2019**, arXiv:1910.09823. [Google Scholar] - Anil Meera, A.; Wisse, M. A Brain Inspired Learning Algorithm for the Perception of a Quadrotor in Wind. arXiv
**2021**, arXiv:2109.11971. [Google Scholar] - Buckley, C.L.; Kim, C.S.; McGregor, S.; Seth, A.K. The free energy principle for action and perception: A mathematical review. J. Math. Psychol.
**2017**, 81, 55–79. [Google Scholar] [CrossRef] - Hohwy, J. The Predictive Mind; Oxford University Press: Oxford, UK, 2013. [Google Scholar]
- Friston, K. The free-energy principle: A rough guide to the brain? Trends Cogn. Sci.
**2009**, 13, 293–301. [Google Scholar] [CrossRef] - Friston, K.J.; Daunizeau, J.; Kilner, J.; Kiebel, S.J. Action and behavior: A free-energy formulation. Biol. Cybern.
**2010**, 102, 227–260. [Google Scholar] [CrossRef] [PubMed][Green Version] - Carhart-Harris, R.L.; Friston, K.J. The default-mode, ego-functions and free-energy: A neurobiological account of Freudian ideas. Brain
**2010**, 133, 1265–1283. [Google Scholar] [CrossRef] - Friston, K.J.; Daunizeau, J.; Kiebel, S.J. Reinforcement learning or active inference? PLoS ONE
**2009**, 4, e6421. [Google Scholar] [CrossRef] [PubMed][Green Version] - Friston, K. Hierarchical models in the brain. PLoS Comput. Biol.
**2008**, 4, e1000211. [Google Scholar] [CrossRef] [PubMed] - Baltieri, M.; Buckley, C.L. PID control as a process of active inference with linear generative models. Entropy
**2019**, 21, 257. [Google Scholar] [CrossRef][Green Version] - Veissière, S.P.; Constant, A.; Ramstead, M.J.; Friston, K.J.; Kirmayer, L.J. Thinking through other minds: A variational approach to cognition and culture. Behav. Brain Sci.
**2020**, 43, e90. [Google Scholar] [CrossRef] [PubMed][Green Version] - Kaufmann, R.; Gupta, P.; Taylor, J. An active inference model of collective intelligence. arXiv
**2021**, arXiv:2104.01066. [Google Scholar] - Miu, E.; Gulley, N.; Laland, K.N.; Rendell, L. Innovation and cumulative culture through tweaks and leaps in online programming contests. Nat. Commun.
**2018**, 9, 1–8. [Google Scholar] [CrossRef] [PubMed][Green Version] - Friston, K.; Kiebel, S. Predictive coding under the free-energy principle. Philos. Trans. R. Soc. Biol. Sci.
**2009**, 364, 1211–1221. [Google Scholar] [CrossRef][Green Version] - Parr, T.; Markovic, D.; Kiebel, S.J.; Friston, K.J. Neuronal message passing using Mean-field, Bethe, and Marginal approximations. Sci. Rep.
**2019**, 9, 1–18. [Google Scholar] [CrossRef] - Van de Laar, T.W.; de Vries, B. Simulating active inference processes by message passing. Front. Robot. AI
**2019**, 6, 20. [Google Scholar] [CrossRef][Green Version] - Friston, K.; FitzGerald, T.; Rigoli, F.; Schwartenbeck, P.; Pezzulo, G. Active inference and learning. Neurosci. Biobehav. Rev.
**2016**, 68, 862–879. [Google Scholar] [CrossRef][Green Version] - Friston, K.J. Variational filtering. NeuroImage
**2008**, 41, 747–766. [Google Scholar] [CrossRef] - Friston, K.; Stephan, K.; Li, B.; Daunizeau, J. Generalised filtering. Math. Probl. Eng.
**2010**, 2010. [Google Scholar] [CrossRef] - Friston, K.; Mattout, J.; Trujillo-Barreto, N.; Ashburner, J.; Penny, W. Variational free energy and the Laplace approximation. Neuroimage
**2007**, 34, 220–234. [Google Scholar] [PubMed] - Blei, D.M.; Kucukelbir, A.; McAuliffe, J.D. Variational inference: A review for statisticians. J. Am. Stat. Assoc.
**2017**, 112, 859–877. [Google Scholar] [CrossRef][Green Version] - Balaji, B.; Friston, K. Bayesian state estimation using generalized coordinates. In Proceedings of the Signal Processing, Sensor Fusion, and Target Recognition XX, Orlando, FL, USA, 5 May 2011; Volume 8050, p. 80501Y. [Google Scholar]
- Anil Meera, A.; Wisse, M. On the convergence of DEM’s linear parameter estimator. In International Workshop on Active Inference; Springer: Berlin/Heidelberg, Germany, 2021; accepted. [Google Scholar]
- Li, B.; Daunizeau, J.; Stephan, K.E.; Penny, W.; Hu, D.; Friston, K. Generalised filtering and stochastic DCM for fMRI. Neuroimage
**2011**, 58, 442–457. [Google Scholar] [CrossRef][Green Version] - Mader, W.; Linke, Y.; Mader, M.; Sommerlade, L.; Timmer, J.; Schelter, B. A numerically efficient implementation of the expectation maximization algorithm for state space models. Appl. Math. Comput.
**2014**, 241, 222–232. [Google Scholar] [CrossRef][Green Version] - Friston, K. A free energy principle for a particular physics. arXiv
**2019**, arXiv:1906.10184. [Google Scholar] - Cara, F.J.; Juan, J.; Alarcón, E. Using the EM algorithm to estimate the state space model for OMAX. Practice
**2014**, 1000, 3. [Google Scholar]

**Figure 1.**A simple block diagram of the robot brain’s inference process using DEM. It uses the measurement data $\mathbf{y}$ generated from the environment (also called generative process). DEM enables the direct fusion of the prior information into the inference process. The concept of generalized coordinates will be detailed in Section 3.1.

**Figure 2.**The DEM algorithm is represented using three coupled steps: D, E, and M steps. The algorithm combines the data from the environment with the robot’s prior beliefs to infer the states, inputs, parameters and hyperparameters of the system. For each parameter update in the E step, the D step updates the (generalized) states and inputs for all times instances, and the M step iterates until hyperparameter convergence, as demonstrated in Algorithm 1. The dynamic process is the generative model in Section 3.1, the priors are the distributions given in Section 5.2 and the generalized coordinates block is defined in Section 3.4. Section 13 will elaborate on the D, E, and M blocks.

**Figure 3.**The DEM algorithm for an LTI system, with the D step simplified as an augmented LTI system given by Equation (90). The D-step block corresponds to the D-step loop in Algorithm 2 and operates at a different frequency from the E and M blocks.

**Figure 4.**The DEM algorithm for an LTI system, with the E step simplified as an augmented LTI system given by Equation (95). The E-step block corresponds to the E-step outer loop in Algorithm 2 and operates at a different frequency when compared to the D and M blocks. The dotted lines illustrate the flow of variables from other blocks and demonstrate the coupled nature of D, E, and M steps. This diagram is illustrative and should not be confused with a control diagram.

**Figure 5.**The results of DEM’s estimation process. (

**a**) The estimated states in blue closely resembles the real states in red. (

**b**) The parameter estimation starts from randomly selected ${\eta}^{\theta}$, marked by red circles and converges with each E step iteration a. (

**c**) Both the hyperparameters start from ${\eta}^{\lambda}=0$, and converge close to the correct value of 8.

**Figure 6.**Maximization of $\overline{F}$ improves the confidence on estimates. (

**a**) Parameter precision ${\mathsf{\Pi}}^{\theta}$. (

**b**) Free action $\overline{F}(a)-\overline{F}(0)$.

**Figure 7.**The sum of all SSE of $\mathsf{\Theta}$ for 100 random systems each, for 5 different noise smoothnesses. DEM outperforms EM, PEM, and SS with minimum SSE for parameter estimation under colored noise.

${\mathit{\theta}}_{1}$ | ${\mathit{\theta}}_{2}$ | ${\mathit{\theta}}_{3}$ | ${\mathit{\theta}}_{4}$ | ${\mathit{\theta}}_{5}$ | ${\mathit{\theta}}_{6}$ | |
---|---|---|---|---|---|---|

Real | 0.048 | 0.753 | −0.761 | −0.218 | 0.360 | 0.077 |

Estimate | 0.034 | 0.714 | −0.769 | −0.219 | 0.333 | 0.098 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Anil Meera, A.; Wisse, M.
Dynamic Expectation Maximization Algorithm for Estimation of Linear Systems with Colored Noise. *Entropy* **2021**, *23*, 1306.
https://doi.org/10.3390/e23101306

**AMA Style**

Anil Meera A, Wisse M.
Dynamic Expectation Maximization Algorithm for Estimation of Linear Systems with Colored Noise. *Entropy*. 2021; 23(10):1306.
https://doi.org/10.3390/e23101306

**Chicago/Turabian Style**

Anil Meera, Ajith, and Martijn Wisse.
2021. "Dynamic Expectation Maximization Algorithm for Estimation of Linear Systems with Colored Noise" *Entropy* 23, no. 10: 1306.
https://doi.org/10.3390/e23101306