Next Article in Journal
Multivariate Tail Coefficients: Properties and Estimation
Previous Article in Journal
A Measurement Model of Mutual Influence for Information Dissemination
Previous Article in Special Issue
Probabilistic Ensemble of Deep Information Networks

This is an early access version, the complete PDF, HTML, and XML versions will be available soon.

Open AccessArticle

Convergence Behavior of DNNs With Mutual-Information-Based Regularization

IBM Research Zurich, 8803 Rüschlikon, Switzerland
Author to whom correspondence should be addressed.
Entropy 2020, 22(7), 727;
Received: 17 June 2020 / Accepted: 26 June 2020 / Published: 30 June 2020
(This article belongs to the Special Issue Information Bottleneck: Theory and Applications in Deep Learning)
Information theory concepts are leveraged with the goal of better understanding and improving Deep Neural Networks (DNNs). The information plane of neural networks describes the behavior during training of the mutual information at various depths between input/output and hidden-layer variables. Previous analysis revealed that most of the training epochs are spent on compressing the input, in some networks where finiteness of the mutual information can be established. However, the estimation of mutual information is nontrivial for high-dimensional continuous random variables. Therefore, the computation of the mutual information for DNNs and its visualization on the information plane mostly focused on low-complexity fully connected networks. In fact, even the existence of the compression phase in complex DNNs has been questioned and viewed as an open problem. In this paper, we present the convergence of mutual information on the information plane for a high-dimensional VGG-16 Convolutional Neural Network (CNN) by resorting to Mutual Information Neural Estimation (MINE), thus confirming and extending the results obtained with low-dimensional fully connected networks. Furthermore, we demonstrate the benefits of regularizing a network, especially for a large number of training epochs, by adopting mutual information estimates as additional terms in the loss function characteristic of the network. Experimental results show that the regularization stabilizes the test accuracy and significantly reduces its variance.
Keywords: deep neural networks; information bottleneck; regularization methods deep neural networks; information bottleneck; regularization methods
MDPI and ACS Style

Jónsson, H.; Cherubini, G.; Eleftheriou, E. Convergence Behavior of DNNs With Mutual-Information-Based Regularization. Entropy 2020, 22, 727.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Search more from Scilit
Back to TopTop