Open AccessThis article is
- freely available
Empirical Information Metrics for Prediction Power and Experiment Planning
Department of Chemistry & Biochemistry, University of California, Los Angeles, CA 90095, USA
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
Institute for Genomics & Proteomics, University of California, Los Angeles, CA 90095, USA
Received: 8 October 2010; in revised form: 30 November 2010 / Accepted: 21 December 2010 / Published: 11 January 2011
Abstract: In principle, information theory could provide useful metrics for statistical inference. In practice this is impeded by divergent assumptions: Information theory assumes the joint distribution of variables of interest is known, whereas in statistical inference it is hidden and is the goal of inference. To integrate these approaches we note a common theme they share, namely the measurement of prediction power. We generalize this concept as an information metric, subject to several requirements: Calculation of the metric must be objective or model-free; unbiased; convergent; probabilistically bounded; and low in computational complexity. Unfortunately, widely used model selection metrics such as Maximum Likelihood, the Akaike Information Criterion and Bayesian Information Criterion do not necessarily meet all these requirements. We define four distinct empirical information metrics measured via sampling, with explicit Law of Large Numbers convergence guarantees, which meet these requirements: Ie, the empirical information, a measure of average prediction power; Ib, the overfitting bias information, which measures selection bias in the modeling procedure; Ip, the potential information, which measures the total remaining information in the observations not yet discovered by the model; and Im, the model information, which measures the model’s extrapolation prediction power. Finally, we show that Ip + Ie, Ip + Im, and Ie — Im are fixed constants for a given observed dataset (i.e. prediction target), independent of the model, and thus represent a fundamental subdivision of the total information contained in the observations. We discuss the application of these metrics to modeling and experiment planning.
Keywords: information metric; inference; sampling; entropy; experiment planning
Citations to this Article
Cite This Article
MDPI and ACS Style
Lee, C. Empirical Information Metrics for Prediction Power and Experiment Planning. Information 2011, 2, 17-40.
Lee C. Empirical Information Metrics for Prediction Power and Experiment Planning. Information. 2011; 2(1):17-40.
Lee, Christopher. 2011. "Empirical Information Metrics for Prediction Power and Experiment Planning." Information 2, no. 1: 17-40.