Next Article in Journal
Theory of Non-Equilibrium Heat Transport in Anharmonic Multiprobe Systems at High Temperatures
Next Article in Special Issue
Perfect Density Models Cannot Guarantee Anomaly Detection
Previous Article in Journal
An Error Compensation Method for Improving the Properties of a Digital Henon Map Based on the Generalized Mean Value Theorem of Differentiation
Previous Article in Special Issue
Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Article

Gradient Regularization as Approximate Variational Inference

1
Department of Infomatics, University of Sussex, Brighton BN1 9QJ, UK
2
Department of Computer Science, University of Bristol, Bristol BS8 1UB, UK
*
Author to whom correspondence should be addressed.
Academic Editors: Eric Nalisnick and Dustin Tran
Entropy 2021, 23(12), 1629; https://doi.org/10.3390/e23121629
Received: 20 September 2021 / Revised: 31 October 2021 / Accepted: 1 November 2021 / Published: 3 December 2021
(This article belongs to the Special Issue Probabilistic Methods for Deep Learning)
We developed Variational Laplace for Bayesian neural networks (BNNs), which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. The Variational Laplace objective is simple to evaluate, as it is the log-likelihood plus weight-decay, plus a squared-gradient regularizer. Variational Laplace gave better test performance and expected calibration errors than maximum a posteriori inference and standard sampling-based variational inference, despite using the same variational approximate posterior. Finally, we emphasize the care needed in benchmarking standard VI, as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters. View Full-Text
Keywords: variational inference; Laplace; Bayes; Bayesian neural networks variational inference; Laplace; Bayes; Bayesian neural networks
Show Figures

Figure 1

MDPI and ACS Style

Unlu, A.; Aitchison, L. Gradient Regularization as Approximate Variational Inference. Entropy 2021, 23, 1629. https://doi.org/10.3390/e23121629

AMA Style

Unlu A, Aitchison L. Gradient Regularization as Approximate Variational Inference. Entropy. 2021; 23(12):1629. https://doi.org/10.3390/e23121629

Chicago/Turabian Style

Unlu, Ali, and Laurence Aitchison. 2021. "Gradient Regularization as Approximate Variational Inference" Entropy 23, no. 12: 1629. https://doi.org/10.3390/e23121629

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop