Next Article in Journal
Information Pluralism and Some Informative Modes of Ignorance
Next Article in Special Issue
On Quantifying Semantic Information
Previous Article in Journal
Some Forms of Trust
Previous Article in Special Issue
Information Operators in Categorical Information Spaces
Information 2011, 2(1), 17-40; doi:10.3390/info2010017

Empirical Information Metrics for Prediction Power and Experiment Planning

1 Department of Chemistry & Biochemistry, University of California, Los Angeles, CA 90095, USA 2 Department of Computer Science, University of California, Los Angeles, CA 90095, USA 3 Institute for Genomics & Proteomics, University of California, Los Angeles, CA 90095, USA
Received: 8 October 2010 / Revised: 30 November 2010 / Accepted: 21 December 2010 / Published: 11 January 2011
(This article belongs to the Special Issue What Is Information?)
View Full-Text   |   Download PDF [453 KB, uploaded 11 January 2011]   |   Browse Figures


In principle, information theory could provide useful metrics for statistical inference. In practice this is impeded by divergent assumptions: Information theory assumes the joint distribution of variables of interest is known, whereas in statistical inference it is hidden and is the goal of inference. To integrate these approaches we note a common theme they share, namely the measurement of prediction power. We generalize this concept as an information metric, subject to several requirements: Calculation of the metric must be objective or model-free; unbiased; convergent; probabilistically bounded; and low in computational complexity. Unfortunately, widely used model selection metrics such as Maximum Likelihood, the Akaike Information Criterion and Bayesian Information Criterion do not necessarily meet all these requirements. We define four distinct empirical information metrics measured via sampling, with explicit Law of Large Numbers convergence guarantees, which meet these requirements: Ie, the empirical information, a measure of average prediction power; Ib, the overfitting bias information, which measures selection bias in the modeling procedure; Ip, the potential information, which measures the total remaining information in the observations not yet discovered by the model; and Im, the model information, which measures the model’s extrapolation prediction power. Finally, we show that Ip + Ie, Ip + Im, and Ie — Im are fixed constants for a given observed dataset (i.e. prediction target), independent of the model, and thus represent a fundamental subdivision of the total information contained in the observations. We discuss the application of these metrics to modeling and experiment planning.    
Keywords: information metric; inference; sampling; entropy; experiment planning information metric; inference; sampling; entropy; experiment planning
This is an open access article distributed under the Creative Commons Attribution License (CC BY) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Share & Cite This Article

Further Mendeley | CiteULike
Export to BibTeX |
EndNote |
MDPI and ACS Style

Lee, C. Empirical Information Metrics for Prediction Power and Experiment Planning. Information 2011, 2, 17-40.

View more citation formats

Article Metrics

For more information on the journal, click here


[Return to top]
Information EISSN 2078-2489 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert