Next Article in Journal
Invalid Microstate Densities for Model Systems Lead to Apparent Violation of Thermodynamic Law
Next Article in Special Issue
Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines
Previous Article in Journal
Laminar-Turbulent Patterning in Transitional Flows
Previous Article in Special Issue
Optimal Nonlinear Estimation in Statistical Manifolds with Application to Sensor Network Localization
Open AccessArticle

Regularizing Neural Networks via Retaining Confident Connections

1
School of Computer Science and Technology, Tianjin University, Tianjin 300072, China
2
Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China
3
Department of Computing and Communications, The Open University, Milton Keynes MK76AA, UK
*
Author to whom correspondence should be addressed.
Entropy 2017, 19(7), 313; https://doi.org/10.3390/e19070313
Received: 30 April 2017 / Revised: 7 June 2017 / Accepted: 23 June 2017 / Published: 30 June 2017
(This article belongs to the Special Issue Information Geometry II)
Regularization of neural networks can alleviate overfitting in the training phase. Current regularization methods, such as Dropout and DropConnect, randomly drop neural nodes or connections based on a uniform prior. Such a data-independent strategy does not take into consideration of the quality of individual unit or connection. In this paper, we aim to develop a data-dependent approach to regularizing neural network in the framework of Information Geometry. A measurement for the quality of connections is proposed, namely confidence. Specifically, the confidence of a connection is derived from its contribution to the Fisher information distance. The network is adjusted by retaining the confident connections and discarding the less confident ones. The adjusted network, named as ConfNet, would carry the majority of variations in the sample data. The relationships among confidence estimation, Maximum Likelihood Estimation and classical model selection criteria (like Akaike information criterion) is investigated and discussed theoretically. Furthermore, a Stochastic ConfNet is designed by adding a self-adaptive probabilistic sampling strategy. The proposed data-dependent regularization methods achieve promising experimental results on three data collections including MNIST, CIFAR-10 and CIFAR-100. View Full-Text
Keywords: information geometry; neural networks; regularization; Fisher information information geometry; neural networks; regularization; Fisher information
Show Figures

Figure 1

MDPI and ACS Style

Zhang, S.; Hou, Y.; Wang, B.; Song, D. Regularizing Neural Networks via Retaining Confident Connections. Entropy 2017, 19, 313.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop