#
PID Control as a Process of Active Inference with Linear Generative Models^{ †}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. PID Control

#### 2.1. The Performance-Robustness Trade-Off

- load disturbance response, how a controller reacts to changes in external inputs, e.g., a step input,
- set-point response, how a controller responds to different set-points over time,
- measurement noise response, how noise on the observations impacts the regulation process,

- robustness to model uncertainty, how uncertainty on the plant/environment dynamics affects the controller.

## 3. The Active Inference Framework

## 4. Results

#### 4.1. PID Control as Active Inference

#### 4.2. Responses to External and Internal Changes

#### 4.3. Optimal Tuning of PID Gains

## 5. Discussion

- load disturbance response
- set-point response
- measurement noise response
- robustness to model uncertainty.

## 6. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Appendix A. The Car Model

Description | Value | |
---|---|---|

$s\left(t\right)$ | car position | - |

${r}_{g}$ | gear ratio divided by wheel radius | 12 |

$a\left(t\right)$ | control | - |

${T}_{m}$ | maximum torque | 190 Nm |

$\beta $ | motor constant | $0.4$ |

$\omega $ | engine speed | ${\alpha}_{n}v$ |

${\omega}_{m}$ | speed that gives maximum torque | 420 rad/s |

m | car mass | 100 kg |

g | gravitational acceleration | $9.81$ m/s${}^{2}$ |

$\lambda $ | slope of the road | ${4}^{\circ}$ |

${C}_{r}$ | coefficient of rolling friction | $0.01$ |

$\rho $ | density of the air | $1.3$ kg/m${}^{3}$ |

${C}_{d}$ | aerodynamic drag coefficient | $0.32$ |

A | frontal area of the car | $2.4$ m${}^{2}$ |

## Appendix B. Measurement Noise and Model Uncertainty in Active Inference

**Figure A1.**Performance of PID controllers with a sudden increase in measurement noise. Twenty cars simulated in the case where measurement noise is increased at $t=150$ s during the 300 s simulations. We report aggregate results with the variance from the target value measured over the last $25\%$ ($225<t<300$ s) of a simulation. We show (1) the case for adaptation of the gains of the PI controller (through expected sensory log-precisions, or log-PI gains, ${\mu}_{{\gamma}_{\tilde{z}}}$) interrupted before the measurement noise drastically changes, and (2) the case where the adaptation process persists for the entire duration of the simulations.

## References

- Dayan, P.; Hinton, G.E.; Neal, R.M.; Zemel, R.S. The Helmholtz Machine. Neural Comput.
**1995**, 7, 889–904. [Google Scholar] [CrossRef] [PubMed] - Rao, R.P.; Ballard, D.H. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci.
**1999**, 2, 79–87. [Google Scholar] [CrossRef] [PubMed] - Knill, D.C.; Pouget, A. The Bayesian brain: The role of uncertainty in neural coding and computation. Trends Neurosci.
**2004**, 27, 712–719. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J.; Kilner, J.; Harrison, L. A free energy principle for the brain. J. Physiol.-Paris
**2006**, 100, 70–87. [Google Scholar] [CrossRef] [PubMed] - Clark, A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci.
**2013**, 36, 181–204. [Google Scholar] [PubMed][Green Version] - Hohwy, J. The Predictive Mind; OUP Oxford: Oxford, UK, 2013. [Google Scholar]
- Bogacz, R. A tutorial on the free-energy framework for modelling perception and learning. J. Math. Psychol.
**2017**, 76, 198–211. [Google Scholar] [CrossRef] [PubMed][Green Version] - Buckley, C.L.; Kim, C.S.; McGregor, S.; Seth, A.K. The free energy principle for action and perception: A mathematical review. J. Math. Psychol.
**2017**, 14, 55–79. [Google Scholar] [CrossRef] - Friston, K.J. The free-energy principle: A rough guide to the brain? Trends Cognit. Sci.
**2009**, 13, 293–301. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J. The free-energy principle: A unified brain theory? Nat. Rev. Neurosci.
**2010**, 11, 127–138. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J.; Rigoli, F.; Ognibene, D.; Mathys, C.; Fitzgerald, T.; Pezzulo, G. Active inference and epistemic value. Cognit. Neurosci.
**2015**, 1–28. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J. Hierarchical models in the brain. PLoS Comput. Biol.
**2008**, 4, e1000211. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J.; Daunizeau, J.; Kilner, J.; Kiebel, S.J. Action and behavior: A free-energy formulation. Biol. Cybernet.
**2010**, 102, 227–260. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J.; Mattout, J.; Kilner, J. Action understanding and active inference. Biol. Cybernet.
**2011**, 104, 137–160. [Google Scholar] [CrossRef] [PubMed][Green Version] - Friston, K.J. What is optimal about motor control? Neuron
**2011**, 72, 488–498. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J. Life as we know it. J. R. Soc. Interface
**2013**, 10, 20130475. [Google Scholar] [CrossRef] [PubMed] - Seth, A.K. The Cybernetic Bayesian Brain; Open MIND: Frankfurt am Main, Germany, 2014. [Google Scholar]
- Evans, D.J.; Searles, D.J. The fluctuation theorem. Adv. Phys.
**2002**, 51, 1529–1585. [Google Scholar] [CrossRef] - Friston, K.J. A free energy principle for biological systems. Entropy
**2012**, 14, 2100–2121. [Google Scholar] - Pezzulo, G.; Cisek, P. Navigating the affordance landscape: Feedback control as a process model of behavior and cognition. Trends Cognit. Sci.
**2016**, 20, 414–424. [Google Scholar] [CrossRef] [PubMed] - Körding, K.P.; Wolpert, D.M. Bayesian decision theory in sensorimotor control. Trends Cognit. Sci.
**2006**, 10, 319–326. [Google Scholar] [CrossRef] [PubMed] - Franklin, D.W.; Wolpert, D.M. Computational mechanisms of sensorimotor control. Neuron
**2011**, 72, 425–442. [Google Scholar] [CrossRef] [PubMed] - Friston, K.J.; Daunizeau, J.; Kiebel, S.J. Reinforcement learning or active inference? PLoS ONE
**2009**, 4, e6421. [Google Scholar] [CrossRef] [PubMed] - Clark, A. Surfing Uncertainty: Prediction, Action, and the Embodied Mind; Oxford University Press: Oxford, UK, 2015. [Google Scholar]
- Clark, A. Radical predictive processing. Southern. J. Philos.
**2015**, 53, 3–27. [Google Scholar] [CrossRef] - Baltieri, M.; Buckley, C.L. An active inference implementation of phototaxis. In Proceedings of the 14th European Conference on Artificial Life 2017, Lyon, France, 4–8 September 2017; pp. 36–43. [Google Scholar]
- Baltieri, M.; Buckley, C.L. A Probabilistic Interpretation of PID Controllers Using Active Inference. In From Animals to Animats 15; Manoonpong, P., Larsen, J.C., Xiong, X., Hallam, J., Triesch, J., Eds.; Springer International Publishing: Berlin, Germany, 2018; pp. 15–26. [Google Scholar]
- Clark, A. Being There: Putting Brain, Body, and World Together Again; MIT Press: Cambridge, MA, USA, 1998. [Google Scholar]
- Ashby, W.R. Requisite variety and its implications for the control of complex systems. Cybernetica
**1958**, 1, 83–99. [Google Scholar] - Shannon, C.E. Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec
**1959**, 4, 1. [Google Scholar] - Kalman, R.E. Contributions to the theory of optimal control. Bol. Soc. Mat. Mexicana
**1960**, 5, 102–119. [Google Scholar] - Stengel, R.F. Optimal Control and Estimation; Courier Corporation: North Chelmsford, MA, USA, 1994. [Google Scholar]
- Todorov, E. General duality between optimal control and estimation. In Proceedings of the 47th IEEE Conference on Decision and Control, Cancun, Mexico, 9–11 December 2008; pp. 4286–4292. [Google Scholar]
- Ashby, W.R. An Introduction to Cybernetics; Chapman & Hall Ltd.: London, UK, 1957. [Google Scholar]
- Wiener, N. Cybernetics or Control and Communication in the Animal and the Machine; MIT Press: Cambridge, MA, USA, 1961; Volume 25. [Google Scholar]
- Åström, K.J. PID Controllers: Theory, Design and Tuning; ISA: The Instrumentation, Systems, and Automation Society: Research Triangle Park, NC, USA, 1995. [Google Scholar]
- Ang, K.H.; Chong, G.; Li, Y. PID control system analysis, design, and technology. IEEE Trans. Control Syst. Technol.
**2005**, 13, 559–576. [Google Scholar][Green Version] - Åström, K.J.; Hägglund, T. Advanced PID Control; ISA: The Instrumentation, Systems, and Automation Society: Research Triangle Park, NC, USA, 2006. [Google Scholar]
- Yi, T.M.; Huang, Y.; Simon, M.I.; Doyle, J. Robust perfect adaptation in bacterial chemotaxis through integral feedback control. Proc. Natl. Acad. Sci. USA
**2000**, 97, 4649–4653. [Google Scholar] [CrossRef] [PubMed][Green Version] - Yang, L.; Iglesias, P.A. Positive feedback may cause the biphasic response observed in the chemoattractant-induced response of Dictyostelium cells. Syst. Control Lett.
**2006**, 55, 329–337. [Google Scholar] [CrossRef] [PubMed][Green Version] - Ang, J.; Bagh, S.; Ingalls, B.P.; McMillen, D.R. Considerations for using integral feedback control to construct a perfectly adapting synthetic gene network. J. Theoret. Biol.
**2010**, 266, 723–738. [Google Scholar] [CrossRef] [PubMed] - Ritz, H.; Nassar, M.R.; Frank, M.J.; Shenhav, A. A Control Theoretic Model of Adaptive Learning in Dynamic Environments. J. Cognit. Neurosci.
**2018**, 30, 1405–1421. [Google Scholar] [CrossRef] [PubMed] - Chevalier, M.; Gomez-Schiavon, M.; Ng, A.; El-Samad, H. Design and Analysis of a Proportional-Integral-Derivative Controller with Biological Molecules. bioRxiv
**2018**. [Google Scholar] [CrossRef] - Åström, K.J.; Hägglund, T. The future of PID control. Control Eng. Pract.
**2001**, 9, 1163–1175. [Google Scholar] [CrossRef] - Åström, K.J.; Hägglund, T. Revisiting the Ziegler–Nichols step response method for PID control. J. Process Control
**2004**, 14, 635–650. [Google Scholar] [CrossRef] - Arturo Urquizo. PID Controller—Wikipedia, the Free Encyclopedia. 2011. Available online: https://en.wikipedia.org/wiki/PID_controller (accessed on 30 March 2018).
- Rivera, D.E.; Morari, M.; Skogestad, S. Internal model control: PID controller design. Ind. Eng. Chem. Process Des. Dev.
**1986**, 25, 252–265. [Google Scholar] [CrossRef] - Åström, K.J.; Panagopoulos, H.; Hägglund, T. Design of PI controllers based on non-convex optimization. Automatica
**1998**, 34, 585–601. [Google Scholar] [CrossRef] - Garpinger, O.; Hägglund, T.; Åström, K.J. Performance and robustness trade-offs in PID control. J. Process Control
**2014**, 24, 568–577. [Google Scholar] [CrossRef] - Grimble, M.; Johnson, M. Algorithm for PID controller tuning using LQG cost minimization. In Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251), San Diego, CA, USA, 2–4 June 1999; Volume 6, pp. 4368–4372. [Google Scholar]
- O’Brien, R.T.; Howe, J.M. Optimal PID controller design using standard optimal control techniques. In Proceedings of the 2008 American Control Conference, Seattle, WA, USA, 11–13 June 2008; pp. 4733–4738. [Google Scholar]
- Kolchinsky, A.; Wolpert, D.H. Semantic information, autonomous agency and non-equilibrium statistical physics. Interface Focus
**2018**, 8, 20180041. [Google Scholar] [CrossRef] [PubMed] - Beal, M.J. Variational Algorithms for Approximate Bayesian Inference; University of London: London, UK, 2003. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006. [Google Scholar]
- Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat.
**1951**, 22, 79–86. [Google Scholar] [CrossRef] - Friston, K.J.; Trujillo-Barreto, N.; Daunizeau, J. DEM: A variational treatment of dynamic systems. NeuroImage
**2008**, 41, 849–885. [Google Scholar] [CrossRef] [PubMed][Green Version] - Friston, K.J.; Stephan, K.; Li, B.; Daunizeau, J. Generalised filtering. Math. Probl. Eng.
**2010**, 2010. [Google Scholar] [CrossRef] - Stratonovich, R.L. Topics in the Theory of Random Noise; CRC Press: Boca Raton, FL, USA, 1967; Volume 2. [Google Scholar]
- Jazwinski, A.H. Stochastic Processes and Filtering Theory; Academic Press: Cambridge, MA, USA, 1970; Volume 64. [Google Scholar]
- Fox, R.F. Stochastic calculus in physics. J. Stat. Phys.
**1987**, 46, 1145–1157. [Google Scholar] [CrossRef] - Wong, E.; Zakai, M. On the convergence of ordinary integrals to stochastic integrals. Ann. Math. Stat.
**1965**, 36, 1560–1564. [Google Scholar] [CrossRef] - Moon, W.; Wettlaufer, J. On the interpretation of Stratonovich calculus. New J. Phys.
**2014**, 16, 055017. [Google Scholar] [CrossRef][Green Version] - Van Kampen, N.G. Stochastic Processes in Physics and Chemistry; Elsevier: Amsterdam, The Netherlands, 1992; Volume 1. [Google Scholar]
- Klöden, P.E.; Platen, E. Numerical Solution of Stochastic Differential Equations; Springer: Berlin, Germany, 1992. [Google Scholar]
- Chui, C.K.; Chen, G. Kalman filtering with Real-Time Applications; Springer: Berlin, Germany, 2017. [Google Scholar]
- Friston, K.J.; Mattout, J.; Trujillo-Barreto, N.; Ashburner, J.; Penny, W. Variational free energy and the Laplace approximation. Neuroimage
**2007**, 34, 220–234. [Google Scholar] [CrossRef] [PubMed][Green Version] - MacKay, D.J. Information Theory, Inference and Learning Algorithms; Cambridge University Press: Cambridge, MA, USA, 2003. [Google Scholar]
- Särkkä, S. Bayesian Filtering and Smoothing; Cambridge University Press: Cambridge, MA, USA, 2013; Volume 3. [Google Scholar]
- Kim, C.S. Recognition Dynamics in the Brain under the Free-Energy Principle. Neural Comput.
**2018**, 30, 2616–2659. [Google Scholar] [CrossRef] [PubMed] - Brown, H.; Adams, R.A.; Parees, I.; Edwards, M.; Friston, K.J. Active inference, sensory attenuation and illusions. Cognit. Process.
**2013**, 14, 411–427. [Google Scholar] [CrossRef] [PubMed][Green Version] - Kawato, M. Internal models for motor control and trajectory planning. Curr. Opin. Neurobiol.
**1999**, 9, 718–727. [Google Scholar] [CrossRef] - Wolpert, D.M.; Ghahramani, Z. Computational principles of movement neuroscience. Nat. Neurosci.
**2000**, 3, 1212. [Google Scholar] [CrossRef] [PubMed] - Åström, K.J.; Murray, R.M. Feedback Systems: An Introduction for Scientists and Engineers; Princeton University Press: Princeton, NJ, USA, 2010. [Google Scholar]
- Svrcek, W.Y.; Mahoney, D.P.; Young, B.R.; Mahoney, D.P. A Real-Time Approach to Process Control; Wiley: New York, NY, USA, 2006. [Google Scholar]
- Muzzey, D.; Gómez-Uribe, C.A.; Mettetal, J.T.; van Oudenaarden, A. A systems-level analysis of perfect adaptation in yeast osmoregulation. Cell
**2009**, 138, 160–171. [Google Scholar] [CrossRef] [PubMed] - Sontag, E.D. Adaptation and regulation with signal detection implies internal model. Syst. Control Lett.
**2003**, 50, 119–126. [Google Scholar] [CrossRef][Green Version] - Araki, M.; Taguchi, H. Two-degree-of-freedom PID controllers. Int. J. Control Autom. Syst.
**2003**, 1, 401–411. [Google Scholar] - Johnson, M.A.; Moradi, M.H. PID Control; Springer: Berlin, Germany, 2005. [Google Scholar]
- Hägglund, T. A control-loop performance monitor. Control Eng. Pract.
**1995**, 3, 1543–1551. [Google Scholar] [CrossRef] - Ferguson, T.S. Mathematical Statistics: A Decision Theoretic Approach; Academic Press: Cambridge, MA, USA, 1967. [Google Scholar]
- Brown, L.D. A complete class theorem for statistical problems with finite sample spaces. Ann. Stat.
**1981**, 9, 1289–1300. [Google Scholar] [CrossRef] - Briat, C.; Gupta, A.; Khammash, M. Antithetic integral feedback ensures robust perfect adaptation in noisy biomolecular networks. Cell Syst.
**2016**, 2, 15–26. [Google Scholar] [CrossRef] [PubMed] - Kalman, R.E. A new approach to linear filtering and prediction problems. J. Basic Eng.
**1960**, 82, 35–45. [Google Scholar] [CrossRef]

**Figure 1.**A PID controller [46]. The prediction error $e\left(t\right)$ is given by the difference between a reference signal $r\left(t\right)$, ${y}_{r}$ in our formulation, and the output $y\left(t\right)$ of a process. The different terms, one proportional to the error (P term), one integrating the error over time (I term) and one differentiating it (D term), drive the control signal $u\left(t\right)$.

**Figure 2.**A cruise controller based on PI control under active inference. (

**a**) The response of the car velocity over time with a target state, or prior in our formulation, ${\eta}_{x}=10$ km/h, ${\eta}_{{x}^{\prime}}=0$ km/h${}^{2}$; (

**b**) The acceleration of the car over time with a specified prior ${\eta}_{x}^{\prime}=0$ km/h${}^{2}$; (

**c**) The external force v, introduced at $t=150$ s, models a sudden change in the environmental conditions, for instance wind or change in slope. Action obtained via the minimisation of variational free energy with respect to a and counteracts the effects of v. The motor action is never zero since we assume a constant slope, $\lambda ={4}^{\circ}$ (see Table A1, Appendix A); (

**d**) The model car we implemented, where v could be thought of as a sudden wind or a changing slope.

.43 | .43 |

(a) | (b) |

.43 | .43 |

(c) | (d) |

**Figure 3.**Different responses to load disturbances and set-point changes. The simulations were 300 s long, with an external disturbance/different target velocity introduced at $t=150$ s. Here we report only a 20 s time window around the change in conditions. (

**a**) The same load disturbance ($v=3.0$ km/h${}^{2}$) is applied with varying expected process precisions ${\pi}_{\tilde{w}}$ where ${\pi}_{w}=\{exp(-24),exp(-22),exp(-20)\}$. Expected sensory log-precisions ${\pi}_{\tilde{z}}$ are fixed over the duration of the simulations, with ${\mu}_{{\gamma}_{z}}=1$; (

**b**) A similar example for changes in the target velocity of the car, from ${\eta}_{x}=13$ km/h to ${\eta}_{x}=10$ km/h, tested on varying expected process precisions ${\pi}_{\tilde{w}}$ where ${\pi}_{w}=\{exp(-24),exp(-22),exp(-20)\}$.

.5 | .5 |

(a) | (b) |

**Figure 4.**Optimising PID gains as expected sensory log-precisions ${\mu}_{{\gamma}_{\tilde{z}}}$. This example shows the control of the car velocity before and after the optimisation of ${\mu}_{{\gamma}_{\tilde{z}}}$ (before and after the vertical dash dot black line) is introduced. (

**a**) The velocity of the car; (

**b**) The acceleration of the car; (

**c**) The action of the car, with an external disturbance introduced at $t=150$ s; (

**d**) The optimisation of expected sensory precisions ${\mu}_{{\gamma}_{\tilde{z}}}$ and their convergence to an equilibrium state, after which the optimisation is stopped before introducing an external force. The blue line represents the true log-precision of observation noise in the system, ${\gamma}_{z}={\gamma}_{{z}^{\prime}}=5$.

.5 | .5 |

(a) | (b) |

.5 | .5 |

(c) | (d) |

**Figure 5.**Performance of PID controllers with and without adaptation of the gains based on the minimisation of free energy. The integral absolute error (IAE) is used to measure the effects of the oscillations introduced by a single load disturbance at $t=150$ s (see text for the exact definition of the IAE).

Criterion | Mapped to | Advantages in Active Inference |
---|---|---|

Load disturbance response | ${\mu}_{{\pi}_{\tilde{z}}}$ | Intuitively expressed via the expected inverse variance of the observations (i.e., precision), with low variance implying a fast response and vice versa (see Section 4.2 and Section 4.3) |

Set-point change response | ${\mu}_{{\pi}_{\tilde{w}}}$ | Natural formulation of PID controllers with two degrees of freedom derived from sensory and process precisions and expressed as a Bayesian inference process (see Section 4.2) |

Measurement noise response | ${\mu}_{{\pi}_{\tilde{z}}}$ | Straightforward interpretation of PID gains as (expected) inverse variances of different embedding orders of measurement noise (see Appendix B) |

Robustness to model uncertainty | ${\mu}_{{\pi}_{\tilde{w}}}$ | Direct mapping of model uncertainty to expected variances of the fluctuations, representing unknown dynamics, of the system to control (see Appendix B) |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Baltieri, M.; Buckley, C.L. PID Control as a Process of Active Inference with Linear Generative Models. *Entropy* **2019**, *21*, 257.
https://doi.org/10.3390/e21030257

**AMA Style**

Baltieri M, Buckley CL. PID Control as a Process of Active Inference with Linear Generative Models. *Entropy*. 2019; 21(3):257.
https://doi.org/10.3390/e21030257

**Chicago/Turabian Style**

Baltieri, Manuel, and Christopher L. Buckley. 2019. "PID Control as a Process of Active Inference with Linear Generative Models" *Entropy* 21, no. 3: 257.
https://doi.org/10.3390/e21030257