Next Article in Journal / Special Issue
On Quantifying Semantic Information
Previous Article in Journal
Some Forms of Trust
Previous Article in Special Issue
Information Operators in Categorical Information Spaces
Article

Empirical Information Metrics for Prediction Power and Experiment Planning

by 1,2,3
1
Department of Chemistry & Biochemistry, University of California, Los Angeles, CA 90095, USA
2
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
3
Institute for Genomics & Proteomics, University of California, Los Angeles, CA 90095, USA
Information 2011, 2(1), 17-40; https://doi.org/10.3390/info2010017
Received: 8 October 2010 / Revised: 30 November 2010 / Accepted: 21 December 2010 / Published: 11 January 2011
(This article belongs to the Special Issue What Is Information?)
In principle, information theory could provide useful metrics for statistical inference. In practice this is impeded by divergent assumptions: Information theory assumes the joint distribution of variables of interest is known, whereas in statistical inference it is hidden and is the goal of inference. To integrate these approaches we note a common theme they share, namely the measurement of prediction power. We generalize this concept as an information metric, subject to several requirements: Calculation of the metric must be objective or model-free; unbiased; convergent; probabilistically bounded; and low in computational complexity. Unfortunately, widely used model selection metrics such as Maximum Likelihood, the Akaike Information Criterion and Bayesian Information Criterion do not necessarily meet all these requirements. We define four distinct empirical information metrics measured via sampling, with explicit Law of Large Numbers convergence guarantees, which meet these requirements: Ie, the empirical information, a measure of average prediction power; Ib, the overfitting bias information, which measures selection bias in the modeling procedure; Ip, the potential information, which measures the total remaining information in the observations not yet discovered by the model; and Im, the model information, which measures the model’s extrapolation prediction power. Finally, we show that Ip + Ie, Ip + Im, and Ie — Im are fixed constants for a given observed dataset (i.e. prediction target), independent of the model, and thus represent a fundamental subdivision of the total information contained in the observations. We discuss the application of these metrics to modeling and experiment planning. View Full-Text
Keywords: information metric; inference; sampling; entropy; experiment planning information metric; inference; sampling; entropy; experiment planning
Show Figures

MDPI and ACS Style

Lee, C. Empirical Information Metrics for Prediction Power and Experiment Planning. Information 2011, 2, 17-40. https://doi.org/10.3390/info2010017

AMA Style

Lee C. Empirical Information Metrics for Prediction Power and Experiment Planning. Information. 2011; 2(1):17-40. https://doi.org/10.3390/info2010017

Chicago/Turabian Style

Lee, Christopher. 2011. "Empirical Information Metrics for Prediction Power and Experiment Planning" Information 2, no. 1: 17-40. https://doi.org/10.3390/info2010017

Find Other Styles

Article Access Map by Country/Region

1
Only visits after 24 November 2015 are recorded.
Back to TopTop