Next Article in Journal
The Capacity for Correlated Semantic Memories in the Cortex
Next Article in Special Issue
Gaze Information Channel in Cognitive Comprehension of Poster Reading
Previous Article in Journal
Thermo-Economic Optimization of an Idealized Solar Tower Power Plant Combined with MED System
Previous Article in Special Issue
An Information-Theoretic Framework for Evaluating Edge Bundling Visualization
Open AccessArticle

Dissecting Deep Learning Networks—Visualizing Mutual Information

Computer Science Department, Liverpool John Moores University, Liverpool L3 3AF, UK
Institute for Criminal Justice Studies, University of Portsmouth, Portsmouth PO1 2HY, UK
Department of Psychology, Edge Hill University, Ormskirk L39 4QP, UK
Author to whom correspondence should be addressed.
Entropy 2018, 20(11), 823;
Received: 23 August 2018 / Revised: 22 October 2018 / Accepted: 23 October 2018 / Published: 26 October 2018
(This article belongs to the Special Issue Information Theory Application in Visualization)
Deep Learning (DL) networks are recent revolutionary developments in artificial intelligence research. Typical networks are stacked by groups of layers that are further composed of many convolutional kernels or neurons. In network design, many hyper-parameters need to be defined heuristically before training in order to achieve high cross-validation accuracies. However, accuracy evaluation from the output layer alone is not sufficient to specify the roles of the hidden units in associated networks. This results in a significant knowledge gap between DL’s wider applications and its limited theoretical understanding. To narrow the knowledge gap, our study explores visualization techniques to illustrate the mutual information (MI) in DL networks. The MI is a theoretical measurement, reflecting the relationship between two sets of random variables even if their relationship is highly non-linear and hidden in high-dimensional data. Our study aims to understand the roles of DL units in classification performance of the networks. Via a series of experiments using several popular DL networks, it shows that the visualization of MI and its change patterns between the input/output with the hidden layers and basic units can facilitate a better understanding of these DL units’ roles. Our investigation on network convergence suggests a more objective manner to potentially evaluate DL networks. Furthermore, the visualization provides a useful tool to gain insights into the network performance, and thus to potentially facilitate the design of better network architectures by identifying redundancy and less-effective network units. View Full-Text
Keywords: deep learning; convolutional neural networks; information theory; mutual information; visualization deep learning; convolutional neural networks; information theory; mutual information; visualization
Show Figures

Figure 1

MDPI and ACS Style

Fang, H.; Wang, V.; Yamaguchi, M. Dissecting Deep Learning Networks—Visualizing Mutual Information. Entropy 2018, 20, 823.

AMA Style

Fang H, Wang V, Yamaguchi M. Dissecting Deep Learning Networks—Visualizing Mutual Information. Entropy. 2018; 20(11):823.

Chicago/Turabian Style

Fang, Hui; Wang, Victoria; Yamaguchi, Motonori. 2018. "Dissecting Deep Learning Networks—Visualizing Mutual Information" Entropy 20, no. 11: 823.

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Search more from Scilit
Back to TopTop