# Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation

## Abstract

**:**

## 1. Introduction

#### 1.1. Outline

#### 1.2. Feed-Forward Architectures

#### 1.3. The Renormalization Group

## 2. Maxent Distributions and Bayesian Learning

## 3. Deep Multilayer Perceptron

#### Generalized RG Differential Equation of a Neural Network in the Continuum Depth Limit

## 4. Discussion

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Bishop, C.M. Neural Networks for Pattern Recognition; Oxford University Press, Inc.: New York, NY, USA, 1995. [Google Scholar]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- Opper, M.; Haussler, D. Generalization performance of Bayes optimal classification algorithm for learning a perceptron. Phys. Rev. Lett.
**1991**, 66, 2677–2680. [Google Scholar] [CrossRef] [PubMed] - Carleo, G.; Cirac, I.; Cranmer, K.; Daudet, L.; Schuld, M.; Tishby, N.; Vogt-Maranto, L.; Zdeborová, L. Machine learning and the physical sciences. Rev. Mod. Phys.
**2019**, 91, 045002. [Google Scholar] [CrossRef] [Green Version] - Carrasquilla, J.; Melko, R. Machine learning phases of matter. Nat. Phys.
**2017**, 13, 431–434. [Google Scholar] [CrossRef] [Green Version] - Iten, R.; Metger, T.; Wilming, H.; del Rio, L.; Renner, R. Discovering Physical Concepts with Neural Networks. Phys. Rev. Lett.
**2020**, 124, 010508. [Google Scholar] [CrossRef] [Green Version] - Engel, A.; den Broeck, C.V. Statistical Mechanics of Learning; Cambridge University Press: Cambridge, UK, 2001. [Google Scholar]
- Shwartz-Ziv, R.; Tishby, N. Opening the Black Box of Deep Neural Networks via Information. arXiv
**2017**, arXiv:cs.LG/1703.00810. [Google Scholar] - Kinouchi, O.; Caticha, N. Optimal generalization in perceptrons. J. Phys. A
**1992**, 25, 6243. [Google Scholar] [CrossRef] - Biehl, M.; Riegler, P. On-Line Learning with a Preceptron. Europhys. Lett.
**1994**, 28, 525. [Google Scholar] [CrossRef] - Kinouchi, O.; Caticha, N. Lower Bounds for Generalization with Drifting Rules. J. Phys. A
**1993**, 26, 6161. [Google Scholar] [CrossRef] - Copelli, M.; Caticha, N. On-line learning in the Committee Machine. J. Phys. A
**1995**, 28, 1615. [Google Scholar] [CrossRef] - Vicente, R.; Caticha, N. Functional optimization of online algorithms in multilayer neural networks. J. Phys. A Gen. Phys.
**1997**, 30. [Google Scholar] [CrossRef] [Green Version] - Caticha, N.; de Oliveira, E. Gradient descent learning in and out of equilibrium. Phys. Rev. E
**2001**, 63, 061905. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Opper, M. A Bayesian Approach to Online Learning in On-line Learning in Neural Networks; Saad, D., Ed.; Cambridge University Press: Cambridge, UK, 1998. [Google Scholar]
- Solla, S.A.; Winther, O. Optimal online learning: A Bayesian approach. Comput. Phys. Commun.
**1999**, 121–122, 94–97. [Google Scholar] [CrossRef] - Caticha, N.; Vicente, R. Agent-based Social Psychology: From Neurocognitive Processes to Social Data. Adv. Complex Syst.
**2011**, 14, 711–731. [Google Scholar] [CrossRef] [Green Version] - Vicente, R.; Susemihl, A.; Jerico, J.P.; Caticha, N. Moral foundations in an interacting neural networks society: A statistical mechanics analysis. Phys. A Stat. Mech. Its Appl.
**2014**, 400, 124–138. [Google Scholar] [CrossRef] [Green Version] - Caticha, N.; Cesar, J.; Vicente, R. For whom will the Bayesian agents vote? Front. Phys.
**2015**, 3. [Google Scholar] [CrossRef] [Green Version] - Caticha, N.; Alves, F. Trust, Law and Ideology in a NN Agent Model of the US Appellate Courts. Available online: https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2019-72.pdf (accessed on 22 May 2020).
- Wilson, K.G.; Kogut, J. The renormalization group and the ϵ expansion. Phys. Rep.
**1974**, 12, 75–199. [Google Scholar] [CrossRef] - Bény, C. Deep learning and the renormalization group. arXiv
**2013**, arXiv:1301.3124. [Google Scholar] - Mehta, P.; Schwab, D.J. An exact mapping between the Variational Renormalization Group and Deep Learning. arXiv
**2014**, arXiv:1410.3831. [Google Scholar] - Koch-Janusz, M.; Ringel, Z. Mutual information, neural networks and the renormalization group. Nat. Phys.
**2018**, 14, 578–582. [Google Scholar] [CrossRef] [Green Version] - Li, S.H.; Wang, L. Neural Network Renormalization Group. Phys. Rev. Lett.
**2018**, 121, 260601. [Google Scholar] [CrossRef] [Green Version] - Lin, H.W.; Tegmark, M.; Rolnick, D. Why Does Deep and Cheap Learning Work So Well? J. Stat. Phys.
**2017**, 168, 1223–1247. [Google Scholar] [CrossRef] [Green Version] - Rumelhart, D.E.; McClelland, J.L.; PDP Research Group (Eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 2: Psychological and Biological Models; MIT Press: Cambridge, MA, USA, 1986. [Google Scholar]
- Fradkin, E.; Susskind, L. Order and disorder in gauge systems and magnets. Phys. Rev. D
**1979**, 17, 2637. [Google Scholar] [CrossRef] [Green Version] - Kogut, J. An introduction to lattice gauge theory and spin systems. Rev. Mod. Phys.
**1979**, 51, 659. [Google Scholar] [CrossRef] - Fisher, M.E. Renormalization group theory: Its basis and formulation in statistical physics. Rev. Mod. Phys.
**1998**, 70, 653–681. [Google Scholar] [CrossRef] - Pessoa, P.; Caticha, A. Exact Renormalization Groups As a Form of Entropic Dynamics. Entropy
**2018**, 20, 25. [Google Scholar] [CrossRef] [Green Version] - Gardiner, C.W. Handbook of Stochastic Methods; Springer: Berlin/Heidelberg, Germany, 1997. [Google Scholar]
- Annunziato, M.; Borzì, A. Optimal control of probability density functions of stochastic processes. Math. Model. Anal.
**2010**, 15, 393–407. [Google Scholar] [CrossRef] [Green Version] - Annunziato, M.; Borzì, A.; Nobile, F.; Tempone, R. On the Connection between the Hamilton-Jacobi-Bellman and the Fokker-Planck Control Frameworks. Appl. Math.
**2014**, 5, 2476–2484. [Google Scholar] [CrossRef] [Green Version] - Hofstadter, D.R. Godel, Escher, Bach: An Eternal Golden Braid; Basic Books, Inc. Division of Harper Collins: New York, NY, USA, 1979. [Google Scholar]
- Hopfield, J.J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA
**1982**, 79, 2554–2558. [Google Scholar] [CrossRef] [Green Version] - Amit, D.J.; Gutfreund, H.; Sompolinsky, H. Statistical mechanics of neural networks near saturation. Ann. Phys.
**1987**, 173, 30–67. [Google Scholar] [CrossRef] - Kadanoff, L. Scaling laws for Ising models near T(c). Phys. Phys. Fiz.
**1966**, 2, 263–272. [Google Scholar] [CrossRef] [Green Version] - Biehl, M.; Schwarze, H. Learning drifting concepts with neural networks. J. Phys. A Math. Gen.
**1993**, 26, 2651–2665. [Google Scholar] [CrossRef] - de Oliveira, E.A.; Caticha, N. Inference From Aging Information. IEEE Trans. Neural Netw.
**2010**, 21, 1015–1020. [Google Scholar] [CrossRef] [PubMed] - Hasenjager, M.; Ritter, H. Active Learning in Neural Networks. In New Learning Paradigms in Soft Computing; Jain, L.C., Kacprzyk, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2002; pp. 137–169. [Google Scholar]

© 2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Caticha, N.
Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation. *Entropy* **2020**, *22*, 587.
https://doi.org/10.3390/e22050587

**AMA Style**

Caticha N.
Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation. *Entropy*. 2020; 22(5):587.
https://doi.org/10.3390/e22050587

**Chicago/Turabian Style**

Caticha, Nestor.
2020. "Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation" *Entropy* 22, no. 5: 587.
https://doi.org/10.3390/e22050587