Next Article in Journal
A New Surrogating Algorithm by the Complex Graph Fourier Transform (CGFT)
Next Article in Special Issue
Learnability for the Information Bottleneck
Previous Article in Journal
Exergy Analysis of Fluidized Desiccant Cooling System
Article

Gaussian Mean Field Regularizes by Limiting Learned Information

1
Computer Science, University College London, London WC1E 6BT, UK
2
The Swiss AI Lab (IDSIA), University of Lugano (USI) & University of Applied Sciences of Southern Switzerland (SUPSI), 6928 Manno, Switzerland
3
Alan Turing Institute, London NW1 2DB, UK
*
Author to whom correspondence should be addressed.
Entropy 2019, 21(8), 758; https://doi.org/10.3390/e21080758
Received: 14 June 2019 / Revised: 25 July 2019 / Accepted: 1 August 2019 / Published: 3 August 2019
(This article belongs to the Special Issue Information Bottleneck: Theory and Applications in Deep Learning)
Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks. View Full-Text
Keywords: information theory; variational inference; machine learning information theory; variational inference; machine learning
Show Figures

Figure 1

MDPI and ACS Style

Kunze, J.; Kirsch, L.; Ritter, H.; Barber, D. Gaussian Mean Field Regularizes by Limiting Learned Information. Entropy 2019, 21, 758. https://doi.org/10.3390/e21080758

AMA Style

Kunze J, Kirsch L, Ritter H, Barber D. Gaussian Mean Field Regularizes by Limiting Learned Information. Entropy. 2019; 21(8):758. https://doi.org/10.3390/e21080758

Chicago/Turabian Style

Kunze, Julius, Louis Kirsch, Hippolyt Ritter, and David Barber. 2019. "Gaussian Mean Field Regularizes by Limiting Learned Information" Entropy 21, no. 8: 758. https://doi.org/10.3390/e21080758

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop