Next Article in Journal
Maximum Entropy-Copula Method for Hydrological Risk Analysis under Uncertainty: A Case Study on the Loess Plateau, China
Next Article in Special Issue
Context-Aware Generative Adversarial Privacy
Previous Article in Journal
Random Walk Null Models for Time Series Data
Previous Article in Special Issue
Discovering Potential Correlations via Hypercontractivity
Open AccessFeature PaperArticle

On Lower Bounds for Statistical Learning Theory

Department of Electrical and Computer Engineering, University of Wisconsin-Madison, 1415 Engineering Drive, Madison, WI 53706, USA
Entropy 2017, 19(11), 617; https://doi.org/10.3390/e19110617
Received: 7 September 2017 / Revised: 27 October 2017 / Accepted: 14 November 2017 / Published: 15 November 2017
(This article belongs to the Special Issue Information Theory in Machine Learning and Data Science)
In recent years, tools from information theory have played an increasingly prevalent role in statistical machine learning. In addition to developing efficient, computationally feasible algorithms for analyzing complex datasets, it is of theoretical importance to determine whether such algorithms are “optimal” in the sense that no other algorithm can lead to smaller statistical error. This paper provides a survey of various techniques used to derive information-theoretic lower bounds for estimation and learning. We focus on the settings of parameter and function estimation, community recovery, and online learning for multi-armed bandits. A common theme is that lower bounds are established by relating the statistical learning problem to a channel decoding problem, for which lower bounds may be derived involving information-theoretic quantities such as the mutual information, total variation distance, and Kullback–Leibler divergence. We close by discussing the use of information-theoretic quantities to measure independence in machine learning applications ranging from causality to medical imaging, and mention techniques for estimating these quantities efficiently in a data-driven manner. View Full-Text
Keywords: machine learning; minimax estimation; community recovery; online learning; multi-armed bandits; channel decoding; threshold phenomena machine learning; minimax estimation; community recovery; online learning; multi-armed bandits; channel decoding; threshold phenomena
MDPI and ACS Style

Loh, P.-L. On Lower Bounds for Statistical Learning Theory. Entropy 2017, 19, 617.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop