Next Article in Journal
A Nonparametric Bayesian Approach to the Rare Type Match Problem
Next Article in Special Issue
Prediction and Variable Selection in High-Dimensional Misspecified Binary Classification
Previous Article in Journal
Supercritical Fluid Gaseous and Liquid States: A Review of Experimental Results
Previous Article in Special Issue
Selection Consistency of Lasso-Based Procedures for Misspecified High-Dimensional Binary Model and Random Regressors
Open AccessArticle

Towards a Unified Theory of Learning and Information

Google Research, 8002 Zürich, Switzerland
Entropy 2020, 22(4), 438; https://doi.org/10.3390/e22040438
Received: 10 February 2020 / Revised: 5 April 2020 / Accepted: 6 April 2020 / Published: 13 April 2020
In this paper, we introduce the notion of “learning capacity” for algorithms that learn from data, which is analogous to the Shannon channel capacity for communication systems. We show how “learning capacity” bridges the gap between statistical learning theory and information theory, and we will use it to derive generalization bounds for finite hypothesis spaces, differential privacy, and countable domains, among others. Moreover, we prove that under the Axiom of Choice, the existence of an empirical risk minimization (ERM) rule that has a vanishing learning capacity is equivalent to the assertion that the hypothesis space has a finite Vapnik–Chervonenkis (VC) dimension, thus establishing an equivalence relation between two of the most fundamental concepts in statistical learning theory and information theory. In addition, we show how the learning capacity of an algorithm provides important qualitative results, such as on the relation between generalization and algorithmic stability, information leakage, and data processing. Finally, we conclude by listing some open problems and suggesting future directions of research. View Full-Text
Keywords: statistical learning theory; information theory; entropy; parameter estimation; learning systems; privacy; prediction methods statistical learning theory; information theory; entropy; parameter estimation; learning systems; privacy; prediction methods
Show Figures

Figure 1

MDPI and ACS Style

Alabdulmohsin, I. Towards a Unified Theory of Learning and Information. Entropy 2020, 22, 438.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Search more from Scilit
 
Search
Back to TopTop