Next Article in Journal
Information Pluralism and Some Informative Modes of Ignorance
Next Article in Special Issue
On Quantifying Semantic Information
Previous Article in Journal
Some Forms of Trust
Previous Article in Special Issue
Information Operators in Categorical Information Spaces
Article Menu

Export Article

Open AccessArticle
Information 2011, 2(1), 17-40; doi:10.3390/info2010017

Empirical Information Metrics for Prediction Power and Experiment Planning

1
Department of Chemistry & Biochemistry, University of California, Los Angeles, CA 90095, USA
2
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
3
Institute for Genomics & Proteomics, University of California, Los Angeles, CA 90095, USA
Received: 8 October 2010 / Revised: 30 November 2010 / Accepted: 21 December 2010 / Published: 11 January 2011
(This article belongs to the Special Issue What Is Information?)
View Full-Text   |   Download PDF [453 KB, uploaded 11 January 2011]   |  

Abstract

In principle, information theory could provide useful metrics for statistical inference. In practice this is impeded by divergent assumptions: Information theory assumes the joint distribution of variables of interest is known, whereas in statistical inference it is hidden and is the goal of inference. To integrate these approaches we note a common theme they share, namely the measurement of prediction power. We generalize this concept as an information metric, subject to several requirements: Calculation of the metric must be objective or model-free; unbiased; convergent; probabilistically bounded; and low in computational complexity. Unfortunately, widely used model selection metrics such as Maximum Likelihood, the Akaike Information Criterion and Bayesian Information Criterion do not necessarily meet all these requirements. We define four distinct empirical information metrics measured via sampling, with explicit Law of Large Numbers convergence guarantees, which meet these requirements: Ie, the empirical information, a measure of average prediction power; Ib, the overfitting bias information, which measures selection bias in the modeling procedure; Ip, the potential information, which measures the total remaining information in the observations not yet discovered by the model; and Im, the model information, which measures the model’s extrapolation prediction power. Finally, we show that Ip + Ie, Ip + Im, and Ie — Im are fixed constants for a given observed dataset (i.e. prediction target), independent of the model, and thus represent a fundamental subdivision of the total information contained in the observations. We discuss the application of these metrics to modeling and experiment planning.     View Full-Text
Keywords: information metric; inference; sampling; entropy; experiment planning information metric; inference; sampling; entropy; experiment planning
This is an open access article distributed under the Creative Commons Attribution License (CC BY 3.0).

Scifeed alert for new publications

Never miss any articles matching your research from any publisher
  • Get alerts for new papers matching your research
  • Find out the new papers from selected authors
  • Updated daily for 49'000+ journals and 6000+ publishers
  • Define your Scifeed now

SciFeed Share & Cite This Article

MDPI and ACS Style

Lee, C. Empirical Information Metrics for Prediction Power and Experiment Planning. Information 2011, 2, 17-40.

Show more citation formats Show less citations formats

Related Articles

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Information EISSN 2078-2489 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top