Gaussian Mean Field Regularizes by Limiting Learned Information
AbstractVariational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks. View Full-Text
Share & Cite This Article
Kunze, J.; Kirsch, L.; Ritter, H.; Barber, D. Gaussian Mean Field Regularizes by Limiting Learned Information. Entropy 2019, 21, 758.
Kunze J, Kirsch L, Ritter H, Barber D. Gaussian Mean Field Regularizes by Limiting Learned Information. Entropy. 2019; 21(8):758.Chicago/Turabian Style
Kunze, Julius; Kirsch, Louis; Ritter, Hippolyt; Barber, David. 2019. "Gaussian Mean Field Regularizes by Limiting Learned Information." Entropy 21, no. 8: 758.
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.