Empirical Information Metrics for Prediction Power and Experiment Planning

Abstract

In principle, information theory could provide useful metrics for statistical inference. In practice this is impeded by divergent assumptions: Information theory assumes the joint distribution of variables of interest is known, whereas in statistical inference it is hidden and is the goal of inference. To integrate these approaches we note a common theme they share, namely the measurement of prediction power. We generalize this concept as an information metric, subject to several requirements: Calculation of the metric must be objective or model-free; unbiased; convergent; probabilistically bounded; and low in computational complexity. Unfortunately, widely used model selection metrics such as Maximum Likelihood, the Akaike Information Criterion and Bayesian Information Criterion do not necessarily meet all these requirements. We define four distinct empirical information metrics measured via sampling, with explicit Law of Large Numbers convergence guarantees, which meet these requirements: I_e, the empirical information, a measure of average prediction power; I_b, the overfitting bias information, which measures selection bias in the modeling procedure; I_p, the potential information, which measures the total remaining information in the observations not yet discovered by the model; and I_m, the model information, which measures the model's extrapolation prediction power. Finally, we show that I_p + I_e, I_p + I_m, and I_e − I_m are fixed constants for a given observed dataset (i.e. prediction target), independent of the model, and thus represent a fundamental subdivision of the total information contained in the observations. We discuss the application of these metrics to modeling and experiment planning.

1. Introduction

1.1. The Need for Information Metrics for Statistical and Scientific Inference

Information theory as formulated by Shannon [1], Kolmogorov and others provides an elegant and general measure of information (or coupling) that connects variables. As such, it might be expected to be universally applied in the “Information Age” (see, for example, the many fields to which it is relevant, described in [2]). Identifying and measuring such information connections between variables lies at the heart of statistical inference (infering accurate models from observed data) and more generally of scientific inference (performing experimental observations to infer increasingly accurate models of the universe).

However, information theory and statistical inference are founded on rather different assumptions, which greatly complicate their union. Statistical inference draws a fundamental distinction between observable variables (operationally defined measurements with no uncertainty) and hidden variables (everything else). It seeks to estimate the likely probability distribution of a hidden variable(s), given a sample of relevant observed variables. Note that from this point of view, probability distributions are themselves hidden, in the sense that they can only be estimated (with some uncertainty) via inference. For example, individual values of an observable are directly observed, but their true distribution can only be inferred from a sample of many such observations.

Traditional information theory, by contrast, assumes as a starting point that the joint probability distribution p(X, Y, Z…) of all variables of interest is completely known, as a prerequisite for beginning any calculations. The basic tools of information theory – entropy, relative entropy, and mutual information – are undefined unless one has the complete joint probability distribution p(X, Y, Z…) in hand. Unfortunately, in statistical inference problems this joint distribution is unknown, and precisely what we are trying to infer.

Thus, while “marrying” information theory and statistical inference is by no means impossible, it requires clear definitions that resolve these basic mismatches in assumptions. In this paper we begin from a common theme that is important to both areas, namely the concept of prediction power, i.e., a model's ability to accurately predict values of the observable variable(s) that it seeks to model. Prediction power metrics have long played a central role in statistical inference. Fisher formulated prediction power as simply the total likelihood of the observations given the model, and developed Maximum Likelihood estimators, based on seeking the specific model that maximizes this quantity. This concept remains central to more recent metrics such as the Akaike Information Criterion (AIC) [3], and Bayesian Information Criterion (BIC) [4], which add “corrections” based on the number of model parameters being fitted.

In this paper we define a set of statistical inference metrics that constitute statistical inference proxies for the fundamental metrics of information theory (such as mutual information, entropy and relative entropy). We show that they are vitally useful for statistical inference (for precisely the same properties that make them useful in information theory), and highlight how they differ from standard statistical inference metrics such as Maximum Likelihood, AIC and BIC. We present a series of metrics that address distinct aspects of statistical inference:

• prediction power, as it is ordinarily defined, as the likelihood of future observations (e.g., “test data”) under a given set of conditions that we have already observed (“training data”).

• bias: A measure of any systematic difference in the model's prediction power on future observations vs. on its original training data.

• completeness: We define a modeling process as “complete” when no further improvements in prediction power are possible (by further varying the model). Thus a completeness metric measures how far we are from obtaining the best possible model.

• extrapolation prediction power:We will introduce a measure of how much the model's prediction power exceeds the prediction power of our existing observation density, when tested on future observations. If this value is zero (or negative) one might reasonably ask to what extent its results can truly be called a “prediction”, but instead are only a summary (or “interpolation”) of our existing observation data.

To clarify the challenges that such metrics must solve, we wish to highlight several characteristics they must possess:

• objective or model-free: One important criterion for such a metric is whether it is model-free; that is, whether or not the calculation of the metric itself involves a process that is equivalent to modeling. If it does, the metric can only be considered to yield a “subjective” evaluation – how well one model fits to the expectations of another model. By contrast, a model-free metric aims to provide an objective measure of how well a model fits the empirical observations. While this criterion may seem very simple to achieve, it poses several challenges, which this paper will seek to clarify.

• unbiased: Like any estimator calculated from a finite sample, these metrics are expected to suffer from sampling errors, but they must be mathematically proven to be free from systematic errors. Such errors are an important source of overfitting problems, and it is important to understand how to exclude them by design.

• convergent: These metrics must provide explicit Law of Large Numbers proofs that they converge to the “true value” in the limit of large sample size. The assumption of convergence is implicit in the use of many methods (such as Maximum Likelihood), but unfortunately the strict requirements of the Law of Large Numbers are sometimes violated, breaking the convergence guarantee and resulting in serious errors. To prevent this, a metric must explicitly show that it meets the requirements of the Law of Large Numbers.

• bounded: These metrics must provide probabilistic bounds that measure the level of uncertainty about their true value, based on the limitations of the available evidence.

• low computational complexity: Ideally, the computational complexity for computing a metric should be O(N log N) or better, where N is the number of sample observations.

In this paper we define a set of metrics obeying these requirements, which we shall refer to as empirical information metrics. As a Supplement, we also provide a tutorial that shows how to calculate these metrics using darwin, an easy-to-use open source software package in Python, available at https://github.com/cjlee112/darwin.

2. Empirical Information

2.1. Standard Prediction Power Metrics

Fisher defined the prediction power of a model Ψ for an observable variable X in terms of the total likelihood of a sample of independent and identically distributed (I.I.D.) draws X₁,X₂, …X_n

$$p\left(X_1,X_2,\dots X_n|\Psi \right)=\prod _{i=1}^n\Psi \left(X_i\right)=exp\left(\sum _{i=1}^{n}log\Psi \left(X_i\right)\right)=exp\left(n\overline{L}\right)$$

where we adopt the convention Ψ(X) ≡ p(X|Ψ) as a shorthand for the probability of an observation given a model, and define the log-likelihood L = log Ψ(X). We follow the standard notation L̅ to indicate its sample mean. Note that we will sometimes write L(Ψ) to emphasize that L is a function of the specific model we are computing.

Fisher's Maximum Likelihood method seeks the model that maximizes the total likelihood or, equivalently, the sample average log-likelihood L̅. Similarly, minimizing the Akaike Information Criterion (AIC) [3]

$$\mathit{\text{AIC}}=2k-2log p\left(x_1,x_2,\dots x_n|\Psi \right)=2k-2n\overline{L}$$

or the Bayesian Information Criterion (BIC) [4]

$$\mathit{\text{BIC}}=Klogn-2n\overline{L}$$

again seeks to maximize the prediction power L̅ while explicitly correcting for model complexity expressed as k, the number of free parameters in the model Ψ.

Vapnik-Chervonenkis theory also supplies a correction factor that penalizes model complexity for classifier problems [6]. For example, consider the simplest case of a binary classifier that predicts the class of each data point with a confidence factor C (by assigning that class a likelihood of $1-\frac{1}{C}$, and the other class a likelihood of $\frac{1}{C}$). In this case the classification error probability on the training data, R_train, converges for large C to $R_{\mathit{\text{train}}}\to -\overline{L}/logC$, and structural risk minimization indicates choosing the model that minimizes the upper bound of the classification error probability:

$$R_{\mathit{\text{VC}}}=\sqrt{\frac{h(1+log\frac{2n}{h})-log\frac{n}{4}}{n}}-\frac{\overline{L}}{logC}$$

where h is the Vapnik-Chervonenkis (VC) dimension of the model (a measure of model complexity), and η is the desired level of confidence for the probabilistic bound.

2.2. Prediction Power and the Law of Large Numbers

These metrics are best understood by highlighting the critical role that the Law of Large Numbers plays in inference metrics. Say we want to find a model Ψ that maximizes the total likelihood of many draws of X, or equivalently the expectation value of the log-likelihood, which depends on the true distribution Ω(X):

$$E\left(L\right)\equiv \sum _X\Omega \left(X\right)log\Psi \left(X\right)$$

where the summation is over all possible values of X (for a continuous variable the summation is replaced by an integral).

Since we do not know the true distribution Ω(X) we cannot use this definition directly. However, we can apply the Law of Large Numbers (LLN) to the log-likelihood of a sample of observations, whose sample average must converge

$$\overline{L}=\frac{1}{n}\sum _{i=1}^n{L}_i=\frac{1}{n}\sum _{i=1}^{n}log\Psi \left(X_i\right)\stackrel{\mathit{\text{LLN}}}{\to }E\left(L\right)$$

as n → ∞, if the sample values L_i are conditionally independent given Ω and identically distributed as L, and the variance Var(L) is finite (the LLN can also be extended to the case of exchangeable observations [5]). Specifically, the Law of Large Numbers guarantees a probabilistic bound on the sample estimator's deviation from the expectation value:

$$p\left(|\overline{L}-E\left(L\right)|\ge \delta \right)\le \frac{\mathit{\text{Var}}\left(L\right)}{n{\delta }^2}$$

So we obtain a lower bound estimate for L at confidence level 1 − ∈ of

$$L_{\in }=\overline{L}-\sqrt{\frac{\mathit{\text{Var}}\left(L\right)}{n\in }}$$

Note that to actually compute this lower bound, we must also use our sample to estimate the variance, which adds another source of error. In practice this is usually not a problem, except for pathological cases (e.g., Var(L) → ∞). For example, to calculate a 95% confidence lower bound:

$$L_{0.05}=\overline{L}-\sqrt{\frac{\overline{\mathit{\text{Var}}\left(L\right)}}{n\left(0.05\right)}}$$

where we have used the shorthand notation $\overline{\mathit{\text{Var}}\left(L\right)}=\overline{{\left(L-\overline{L}\right)}^2}$ to denote the sample estimator of the variance. Note that since the Law of Large Numbers is a general result (i.e., it holds over all possible distributions), it does not necessarily represent the best confidence interval that one can obtain for a specific case. Other methods for computing a confidence interval such as resampling [7], can usually improve on (i.e., increase) this lower bound, but we will not explore such implementation details in this paper.

Since the X_i are indeed conditionally independent given Ω and identically distributed as X, we expect for large sample size n to be able to use L̅ as a proxy for E(L). In that case maximizing L̅ also maximizes E(L), which it is convenient to separate into one term dependent only on Ω and another term dependent on Ψ:

$$E\left(L(\Psi )\right)=\sum _X\Omega \left(X\right)\left(log\frac{\Psi \left(X\right)}{\Omega \left(X\right)}+log\Omega \left(X\right)\right)=\sum _X\Omega \left(X\right)\left(-log\frac{\Omega \left(X\right)}{\Psi \left(X\right)}+log\Omega \left(X\right)\right)=-D\left(\Omega ||\Psi \right)-H\left(\Omega \left(X\right)\right)$$

where D(Ω‖Ψ) is the relative entropy of model Ψ relative to the true distribution Ω, and H(Ω(X)) is the entropy of the true distribution Ω. Since the right hand term is constant with respect to Ψ, this expression is maximized when D(Ω‖Ψ) is minimized, which occurs iff Ψ(X) = Ω(X) for all values of X. This guarantees that choosing the model Ψ* that maximizes E(L) will indeed identify the correct model Ψ*(X) = Ω(X).

2.3. The Problem of Selection Bias

Unfortunately, there is a catch. This guarantee can only be extended to maximization of the sample log-likelihood L̅, if the L_i are identically distributed as L. All of these metrics (L̅, AIC, BIC) were designed for use with model selection; that is, we compute the metric for each of a large set of models, then select the model that maximizes the likelihood (or minimizes the AIC or BIC). And the very nature of model selection introduces bias into the sample likelihoods [8]. Briefly, if the model Ψ was chosen specifically to maximize the values L_i, we cannot assume that the L_i are identically distributed as L. Indeed, we expect that the L_i will be biased to higher values than L in general. Therefore the Law of Large Numbers convergence guarantee collapses, and we cannot prove that model selection using L̅ will yield the true distribution Ω. Vapnik-Chervonenkis theory seeks to protect against this bias by deriving an upper bound on the possible error due to selection bias [6], based on the model's VC dimension.

First, let's examine this problem from an empirical point of view, by simply defining a metric for measuring the bias. We define a test data criterion:

• a set of sample values $X_1^{\prime },X_2^{\prime },\dots X_m^{\prime }$ are valid test data for a model Φ predicting an observable X if the $X_i^{\prime }$ are exchangeable, identically distributed as X, and conditionally independent of Φ given the true distribution Ω, i.e., $P\left(X_i^{\prime },\Phi |\Omega \right)=P\left(X_i^{\prime }|\Omega \right)p\left(\Phi |\Omega \right)$. Equivalently, Φ contains no information about the $X_i^{\prime }$ except via their shared dependence on the hidden distribution Ω. Note that for any model Φ generated by model selection, its training data do not meet this requirement, since Φ is not conditionally independent of the training data given Ω.

We desire an estimator for L̅ − E(L). Since the $X_i^{\prime }$ are identically distribured as X and conditionally independent of Φ given Ω, the $log\Phi \left(X_i^{\prime }\right)$ are identically distributed as log Φ(X) i.e., L. So by the Law of Large Numbers we can define an overfitting bias information metric

$${\overline{I}}_b=\overline{L}-{\overline{L}}_e\stackrel{\mathit{\text{LLN}}}{\to }\overline{L}-E\left(L\right)$$

as m → ∞ , where $\overline{L_e}$ is the sample average of the $log\Phi \left(X_i^{\prime }\right)$ test data log-likelihoods. We will refer to $\overline{L_e}$ as the empirical log-likelihood. Note that whereas Vapnik-Chervonenkis theory provides an upper bound on the bias errors for an entire class of models (i.e., all models with the same VC dimension), I_b measures the actual error due to a specific model's selection bias.

I_b has the corresponding lower bound estimator (under the simplifying assumption that the sample sizes for L and L_e are the same (m = n)):

$$I_{b,\in }={\overline{I}}_b-\sqrt{\frac{\overline{\mathit{\text{var}}\left(L-L_e\right)}}{m\in }}$$

If the model selection procedure has introduced no bias, I_b ≈ 0.

2.4. Example: The BIC Optimal Model for a Small Sample from a Normal Distribution

The BIC adds a correction term k log n to the total log-likelihood, which penalizes against models with larger numbers of parameters. Note that this correction is designed specifically to protect against overfitting. This correction is referred to as the Bayesian Information Criterion because it is based on choosing the model with maximum Bayesian posterior probability, and by this criterion is provably optimal for the exponential family of models [4].

However, several caveats about such corrections should be understood:

• a given correction addresses a particular kind of overfitting, for example, for the AIC and BIC, excessive number of model parameters k.

• a given correction is based on specific assumptions about the model, and may not behave as expected under other conditions;

• Such corrections do not guarantee that the model they select will be optimal, or even unbiased.

As an example, Figure 1: Overfitting analysis of BIC models on a small sample from a normal distribution shows the distribution of L̅ vs.$\overline{L_e}$ for BIC-optimal models generated using a sample of three observations drawn from a unit normal distribution. (Note that in this case BIC-optimality is just equivalent to AIC-optimality and Maximum Likelihood, since the set of all possible normal models share the same value of k = 2). This simple example illustrates several points:

• A large fraction of the models strongly overfit the observations as indicated by a large deviation from the L̅ = $\overline{L_e}$ diagonal.

• L̅ and $\overline{L_e}$ are strongly and non-linearly anti-correlated. That is, the better the apparent fit to the training data, the worse the actual fit to the test data.

2.5. The Empirical Information Metric

Based on these considerations, we use the unbiased estimator L_e to define the empirical information, a signed measure of prediction power relative to the uninformative distribution p(X):

$$\overline{I_e(\Psi )}=\overline{L_e(\Psi )}-\overline{L_e\left(p\right)}=\frac{1}{n}\sum _{i=1}^{n}log\frac{\Psi \left(X_i\right)}{p\left(X_i\right)}$$

The empirical information estimates the improvement in the accuracy of a model Ψ(X) in predicting the test observations. For observable variables X whose uninformative distribution is simply a constant density, $\overline{I_e(\Psi )}$ differs from $\overline{L_e(\Psi )}$ by simply a constant (log R, where R is the size of the range of X). In such cases the lower bound estimator for I_e differs from that of L_e only by this constant:

$$I_{e,\in }={\overline{I}}_e-\sqrt{\frac{\overline{\mathit{\text{var}}\left(L_e\right)}}{n\in }}={\overline{L}}_e-\sqrt{\frac{\overline{\mathit{\text{var}}\left(L_e\right)}}{n\in }}+logR$$

It is important to note a few aspects of the empirical information that arise from the above considerations:

• Note that I_e can be negative, if the model's prediction power is even worse than that of theuninformative distribution.

• Whereas most metrics for model selection such as the AIC and BIC contain correction terms dependent on the model complexity k (or VC dimension h), I_e needs no such corrections because it is unbiased by definition. Excessive model complexity will not increase I_e but instead will reduce it. I_e contains no bias and therefore needs no correction. In this sense, it follows a similar approach as cross-validation [9].

• Note that we do not need to incorporate the sample size directly into the metric definition (as in the case of the BIC [4], Vapnik-Chervonenkis upper-bound error R_vc [6], and “small-sample corrected” versions of the AIC such as the AICc [10]). Instead, the effect of sample size emerges naturally from the Law of Large Numbers lower bound estimator for our empirical information metrics (e.g., L_e,ε, I_b,ε, I_e,ε). Fundamentally, the importance of sample size is simply the uncertainty due to sampling error, and the Law of Large Numbers probabilistic bound captures this in a general way.

2.6. Empirical Information as A Sampleable Form of Mutual Information

Consider the following “mutual information sampling problem”:

• draw a specific inference problem (hidden distribution Ω(X)) from some class of real-world problems (e.g., for weight distributions of different animal species, this step would mean randomly choosing one particular animal species);

• draw training data ${\vec{X}}^t$ and test data X from Ω(X);

• find a way to estimate the mutual information I( ${\vec{X}}^t$; X) on the basis of this single case (single instance of Ω).

The standard definition of mutual information $I({\overrightarrow{X}}^t;X)=E\left(log\frac{p({\overrightarrow{X}}^t,X)}{P({\overrightarrow{X}}^t)P(X)}\right)$ does not enable such a calculation. Even if we draw many pairs ${\vec{X}}^t$, X to estimate this value, we will just get a value of zero, because ${\vec{X}}^t$, X are conditionally independent given Ω. The mutual information I( ${\vec{X}}^t$; X) is defined only over the complete joint distribution p(Ω, ${\vec{X}}^t$, X); it does not appear meaningful to talk about calculating it from a single instance of Ω.

By contrast with mutual information, we do calculate empirical information for a specific value of Ω, i.e., we use it to measure the prediction power of our model Ψ on observations emitted by that specific value of Ω. It is therefore interesting to investigate the relationship of the empirical information vs. the mutual information. We follow the usual information theory approach of taking its expectation value over the complete joint distribution:

$$E\left(I_e(\Psi )\right)=E\left(L_e(\Psi )\right)-E\left(L_e\left(P\right)\right)=E\left(L_e(\Psi )\right)-\sum _{X}p\left(X\right)logp\left(X\right)=E\left(L_e(\Psi )\right)+H\left(X\right)$$

assuming that the uninformative distribution p(X) used in the denominator of I_e matches the true marginal distribution of X. Focusing on the remaining expectation log-likelihood term:

$$E\left(L_e(\Psi )\right)=\sum _{\Omega }\sum _{{\overrightarrow{X}}^t}\sum _{X}p\left(X,{\overrightarrow{X}}^t,\Omega \right)log\Psi \left(X|{\overrightarrow{X}}^t\right)$$

where we take the expectation value over all possible values of the observable X, all possible values of the hidden variable Ω, and all possible training data sets ${\vec{X}}^t$ of size t. Note that we write the model as Ψ(X|${\vec{X}}^t$) to explicitly emphasize its dependence on a set of training data ${\vec{X}}^t$. Since Ω does not appear in the log term we can eliminate it:

$$\begin{array}{c}=\sum _{{\overrightarrow{X}}^t}\sum _{X}p(X,{\overrightarrow{X}}^t)log\Psi (X|{\overrightarrow{X}}^t)\\ =-\sum _{{\overrightarrow{X}}^t}p({\overrightarrow{X}}^t)\sum _{X}p(X|{\overrightarrow{X}}^t)log\frac{p(X|{\overrightarrow{X}}^t)}{\Psi (X|{\overrightarrow{X}}^t)}+\sum _{{\overrightarrow{X}}^t}\sum _{X}p(X,{\overrightarrow{X}}^t)logp(X|{\overrightarrow{X}}^t)\\ =-E_{{\overrightarrow{X}}^t}(D(p(X|{\overrightarrow{X}}^t)\parallel \Psi (X|{\overrightarrow{X}}^t)))-H(X|{\overrightarrow{X}}^t)\end{array}$$

where the first term is a relative entropy of the model vs. the true conditional probability, and the second term is the conditional entropy of the observable vs. the training data. Therefore the expectation value of the empirical information is just:

$$\begin{array}{c}E(I_e(\Psi ))=H(X)-H(X|{\overrightarrow{X}}^t)-E_{{\overrightarrow{X}}^t}(D(p(X|{\overrightarrow{X}}^t)\parallel \Psi (X|{\overrightarrow{X}}^t)))\\ =I(X;{\overrightarrow{X}}^t)-E_{{\overrightarrow{X}}^t}(D(p(X|{\overrightarrow{X}}^t)\parallel \Psi (X|{\overrightarrow{X}}^t)))\end{array}$$

where I(X; ${\vec{X}}^t$) is the mutual information between the training data and the observable. Now consider the following sampling protocol:

• for one specific inference problem (hidden value of Ω), we draw a training dataset ${\vec{X}}^t$, use it to train a model Ψ(X|${\vec{X}}^t$), and measure the empirical information $\overline{I_e(\Psi )}$ on a set of test data ${\vec{X}}^n$ drawn from the same distribution.

• We repeat this procedure for multiple inference problems Ω₍₁₎, Ω₍₂₎, …, Ω_(m), and take the average of their empirical information values $\frac{1}{m}\sum {\overline{I}}_e\stackrel{\mathit{\text{LLN}}}{\to }E(I_e(\Psi ))$.

If the model Ψ(X| ${\vec{X}}^t$) approximates the true conditional distribution p(X| ${\vec{X}}^t$) more and more closely, the relative entropy term D(p(X|${\vec{X}}^t$)‖Ψ(X|${\vec{X}}^t$)) will vanish, and we expect the average of the empirical information values to converge simply to I(X; ${\vec{X}}^t$). Under these conditions, the empirical information becomes a “sampleable form” of the mutual information. Note that the mutual information itself does not have this property; as shown above, the mutual information cannot be computed “piecewise” for individual instances of Ω and then averaged. By contrast, if we compute the empirical information for each inference problem, and then take the average, it will converge to the mutual information.

3. The Problem of Convergence

If we wish to maximize prediction power, our ultimate goal must be convergence, namely that our model will converge to the true, hidden distribution Ω. So we must ask the obvious question, how do we know when we're done? Two basic strategies present themselves:

• self-consistency tests: We can use our model as a reference to test whether the observations exactlymatch its expectations, as must be true if Ψ → Ω.

• convergence distance metric: If we knew the value of the absolute maximum prediction power L(Ω) possible for our target observable X, we could define a distance metric δ = L(Ω) − L(Ψ), which measures how “far” our current model is from convergence, in terms of its relative prediction power.

We will define empirical information metrics for both these approaches.

3.1. The Inference “Halting Problem”

As an example of the need for a convergence metric, we consider the process of Bayesian inference in modeling scientific data. In scientific research, we cannot easily restrict the set of possible models a priori either to closed-form analytic solutions or to finite sets of models that we can fully compute in practical amounts of CPU time. That is, the set of all possible models of the universe is not strictly bounded, and generally can be reduced only by calculating likelihoods for different terms of this set vs. experimental observations.

What is the computational complexity for Bayesian inference to find the correct term Ω or any term within some distance δ of it? We can view this as a form of the Halting Problem, in the sense that it requires a metric that indicates when it has found a term that is less than δ distance from Ω, at which point the algorithm halts. Unfortunately, the standard form of Bayes' Law

$$p(\Psi |\overrightarrow{X})=\frac{p(\overrightarrow{X}|\Psi )p(\Psi )}{{\sum }_{\Psi }p(\overrightarrow{X}|\Psi )p(\Psi )}$$

offers no evident shortcuts: Even if we had calculated all but one last term of the summation, we still would not know whether our best model so far is actually the best model, or even whether it is within distance δ of the best model. In the absence of a halting test, this implies that its computational complexity must simply be that of exhaustive enumeration. This is a serious problem, especially given that the set of all possible models may for scientific inference problems be infinite.

In real-world practice this “halting problem” often grows into an even worse problem of “model misspecification” [11]. That is, Bayesian computational methods typically lack a mechanism for generating all possible models even in theory. Instead they are limited to assuming a specific mathematical form for the model. Unless by good fortune the true distribution exactly fits this mathematical form, the computation will simply exclude it. Therefore, a reliable convergence metric becomes essential as an external indicator for whether the computational model is “misspecified” in this way. It should be noted that this is not addressed by asking whether a given Bayesian modeling process has “converged” in the sense of a Markov Chain Monte Carlo sampling process converging to its stationary distribution [12]. Any such process is still restricted by its assumptions of a specific mathematical form for the model; there is no guarantee that this will contain the correct answer.

3.2. Potential Information

We define I_∞ as the total information content obtainable from a set of observations by considering the infinite set of all possible models. By analogy to the classical physics division of kinetic vs. potential energy components, we divide this into one part representing the model terms we've actually calculated (I_e, the empirical information), and a second part for the remaining uncomputed terms, which we define as I_p, the potential information:

$$I_{\infty }=I_e+I_p$$

I_p therefore represents the maximum amount of information theoretically attainable by computing more terms of the infinite set. Assuming that the true, hidden likelihood is Ω(X) and that our current model (after considering all terms calculated so far) is Ψ(X), then

$$I_p=I_{\infty }-I_e=\sum _{\forall x}\Omega (x)log\frac{\Omega (x)}{p(X)}-\sum _{\forall x}\Omega (x)log\frac{\Psi (x)}{p(X)}$$

where p(X) is the uninformative reference distribution, which cancels, yielding

$$I_p=\sum _{\forall x}\Omega (x)log\Omega (x)-\sum _{\forall x}\Omega (x)log\Psi (x)=-H(\Omega (X))-E(L)$$

We can therefore solve the Inference Halting Problem by deriving an empirical I_p estimator (with a Law of Large Numbers convergence guarantee) that can be calculated without computing any more terms of the infinite model set. This is surprisingly straightforward. The right-hand term can be estimated directly by $-\overline{L_e}$ (the empirical log-likelihood). The left-hand term −H(Ω(X)) is simply the negative entropy of the observable. We evidently need an empirical estimator of the entropy, and specifically of the density Ω(X).

This density estimation problem poses one conceptual problem that requires clarification. Since the ultimate purpose of the potential information calculation is to catch possible errors in modeling, no part of its calculation (such as the empirical entropy calculation) should itself be equivalent to a form of modeling. If we used such a form of modeling to compute the empirical entropy, that would introduce a strongly subjective element, i.e., simply comparing one model (Ψ) versus another (the model used for estimating H_e). To obtain an objective I_p metric, the empirical entropy calculation should be model-free. It should be a purely empirical procedure with a Law of Large Numbers convergence guarantee for large sample size n → ∞.

3.3. The Empirical Entropy

For the case where the observable X is restricted to a set of discrete values, we define an indicator label κ_x(X) which equals 1 if X equals a desired value x, otherwise zero. Then by the Law of Large Numbers

$$\frac{1}{n}\sum _{i=1}^n{K}_x(X_i)\stackrel{\mathit{\text{LLN}}}{\to }E(K_x(X))=\Omega (X=x)(1)+\Omega (X\ne x)(0)=\Omega (x)$$

The empirical entropy estimator follows directly in this case: for n → ∞,

$${\overline{H}}_e=-\frac{1}{n}\sum _{i=1}^{n}log\left(\frac{1}{n}\sum _{j=1}^n\kappa x_i(X_j)\right)\stackrel{\mathit{\text{LLN}}}{\to }H(\Omega (X))$$

For the continuous case, we need an empirical probability density estimator P_e(X). To obtain this we define an indicator function κ_x(X) which equals 1 if X ≤ x, otherwise zero. Then

$$\frac{1}{n}\sum _{i=1}^n{\kappa }_x(X_i)\stackrel{LLN}{\to }E({\kappa }_x(X))={\mathit{\int }}_{-\infty }^{\infty }\Omega (X){\kappa }_x(X)dX={\mathit{\int }}_{-\infty }^x\Omega (X)dX$$

i.e., the cumulative density function c.d.f.(X). Therefore we define

$$\begin{array}{}{\overline{H}}_e=-\frac{1}{n}\sum _{j=1}^{n}log{P}_e(X_j)=-\frac{1}{n}\sum _{j=1}^{n}log\frac{{\sum }_{i=1}^n{\kappa }_{X_j+\delta X/2}(X_i)-{\sum }_{i=1}^n{\kappa }_{X_j+\delta X/2}(X_i)}{n\delta X}& \stackrel{\mathit{\text{LLN}}}{\to }-E\left(log\frac{{\sum }_{i=1}^n{\kappa }_{X_j+\delta X/2}(X_i)-{\sum }_{i=1}^n{\kappa }_{X_j-\delta X/2}(X_i)}{n\delta X}\right)& \to -E\left(log\frac{c.d.f.(X+\delta X/2)-c.d.f.(X-\delta X/2)}{\delta X}\right)\end{array}$$

By construction we choose δx ∝ 1/n → 0 as n → ∞. Then by the Fundamental Theorem of Calculus,

$${\overline{H}}_e\stackrel{\mathit{\text{LLN}}}{\to }-E(log\Omega (X))=-{\int }_{-\infty }^{\infty }\Omega (X)log\Omega (X)\mathit{\text{dX}}=H(\Omega (X))$$

For example, we can construct δx ∝ 1/n as follows: For each sample point X_j, find its m nearest neighbors (sample points), where m is a relatively small constant. Then set

$$\delta x=|X_{j:m}-X_j|+|X_{j:m-1}-X_j|$$

where we use the notation X_j:m to mean the “m -th nearest neighbor of point X_j” . Note that the interval [X_j − δx/2, X_j + δx/2] contains m − 1 sample points (not including X_j itself, to avoid the inherent bias that would introduce; this in turn requires replacing the n in the log-denominator with n − 1). This implementation of the $\overline{H_e}$ calculation is simply:

$${\overline{H}}_e=-\frac{1}{n}\sum _{j=1}^{n}log\frac{m-1}{(n-1)(|X_{j:m}-X_j|+|X_{j:m-1}-X_j|)}$$

There are of course many possible empirical density estimation implementations that could be used; we offer this implementation solely as an illustrative example. This implementation also generalizes to multidimensional data, and thus can be used to estimate mutual information [13,14].

Of course, the empirical entropy has the usual lower bound estimator from the Law of Large Numbers

$$H_{e,\in }={\overline{H}}_e-\sqrt{\frac{\overline{\mathit{\text{Var}}(log{P}_e)}}{n\in }}$$

3.4. Potential Information Estimators

This gives us mean and lower bounds estimators for the potential information

$$\begin{array}{}\overline{I_p}=-\overline{H_e}-\overline{L_e}{I}_{p,\in }=\overline{I_p}-\sqrt{\frac{\overline{\mathit{\text{Var}}(log{P}_e-L_e)}}{n\in }}\end{array}$$

where the variance is computed from P_e and L_e pairs calculated from the same sample of observations.

Note that since the potential information is computed in “observation space” instead of “model space”, the computational complexity of its calculation depends primarily on the observation sample size. This can be very efficient. First of all, the calculation divides into two parts that can be done separately; since the empirical entropy has no dependence on the model Ψ, it need only be calculated once and can then used for computing I_p for many different models. Second, the empirical entropy calculation can have low computational complexity. For the simple implementation outlined above, it is simply O(mn) (where m is a small constant for the nearest-neighbor density calculation; this assumes the observations are already sorted in order. If not, an additional O(n log n) step is required to sort them). For high dimensional data, the computational complexity scales as O(n²), due to the need to calculate pairwise distances. Of course, the details of the computational complexity will vary depending on what empirical entropy implementation is used.

3.5. Convergence to the Kullback-Leibler Distance

In the limit of large sample size, the potential information converges to

$$\overline{I_p}\stackrel{\mathit{\text{LLN}}}{\to }E(log\Omega (X)-log\Psi (X))=D(\Omega \parallel \Psi )$$

which is simply the relative entropy (Kullback-Leibler divergence [15]) of the true distribution vs. the model. (It should be emphasized that computing the Kullback-Leibler divergence directly requires knowing the true distribution, which of course in any inference problem is unknown).

We may thus consider the potential information to represent a distance estimator from the true distribution Ω. Specifically, it estimates the difference in prediction power of our current model vs. that of the true distribution. Thus it solves the Inference Halting metric problem; if we are searching for a model with prediction power within distance δ of the maximum, we simply halt when

$$\overline{I_p}+\sqrt{\frac{\overline{\mathit{\text{Var}}(log{P}_e-L_e)}}{n\in }}\le \delta$$

at whatever level of confidence 1 — ∈ we desire.

The Akaike Information Criterion (AIC) [3] and related information metrics [16] are often referred to as representing the Kullback-Leibler (KL) divergence of the true distribution vs. the model [17]. So it is logical to ask how the potential information differs from these well-known metrics. The AIC and related metrics were designed for model selection problems, in which the observable (characterized by the true distribution Ω) is treated as a fixed constant, and the model is varied in search of the best fit. As shown in part A of Figure 2: Comparing AIC and Potential Information to the Theoretical Kullback-Leibler Divergence, the AIC does indeed correlate directly with the KL divergence D(Ω‖Ψ) under this assumption (holding the true distribution fixed as a constant). Specifically, for a sample of exchangeable observations ${\vec{X}}^n$,

$$\mathit{\text{AIC}}=2k-2\mathit{\text{log}}\Psi ({\overrightarrow{X}}^n)=2k-2n\overline{L(\Psi )}$$

So as n → ∞,

$$\frac{1}{2n}\mathit{\text{AIC}}\stackrel{\mathit{\text{LLN}}}{\to }-E(L(\Psi ))$$

Thus the AIC converges to the negative log-likelihood, whereas the KL divergence D(Ω‖Ψ) = −H(Ω(X)) − E(L(Ψ)) also contains an entropy term −H(Ω(X)). However, if the true distribution Ω(X) is held fixed, then the AIC differs from the KL divergence only by a constant. So for comparing two different models Ψ1, Ψ2, the difference in their AIC values converges to

$$\frac{1}{2n}(\mathit{\text{AIC}}({\Psi }_2)-\mathit{\text{AIC}}({\Psi }_1))\stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel {\Psi }_2)-D(\Omega \parallel {\Psi }_1)$$

This is why the AIC and related likelihood metrics are often treated as a proxy for the KL divergence in model selection.

However, if the true distribution Ω is not treated as a fixed constant, and instead is allowed to vary, this simple relationship breaks. In that case, the AIC no longer correlates with the KL divergence (Figure 2B). By contrast, the potential information metric I_p̅ correlates with the KL divergence under all conditions (Figure 2C). The main difference between the potential information and the AIC is simply the empirical entropy term, which is included in the potential information metric but missing from the AIC:

$$\overline{I_p}-\frac{1}{2n}\mathit{\text{AIC}}=-\overline{H_e}-\frac{\kappa }{n}$$

Thus, the potential information metric (and consequently, the empirical entropy term) is essential for any problem where

• we need an estimate of the absolute value of the Kullback-Leibler divergence, rather than simply comparing its relative value for two models;

• or we need to consider possible variation between different true distributions Ω (or equivalently, different observable variables X). For example, in experiment planning problems, we consider different possible experiments (different observable variables) in order to estimate how much information they are likely to yield [18].

3.6. Unbiased Empirical Posteriors

Standard Bayesian inference can grossly overestimate the posterior probability of a model term, because the sum of calculated terms is biased to underestimate the total p(X) summed over the complete infinite series. The empirical entropy provides a resolution to this problem. By the Asymptotic Equipartition theorem [1], for a sample ${\vec{X}}^N$ = {X₁, X₂, …X_N} of exchangeable observations of size N

$$\frac{1}{N}\sum _{i=1}^{N}logp(X_i)\stackrel{\mathit{\text{LLN}}}{\to }E(logp(X))=\sum _{\forall X}\Omega (X)log\Omega (X)=-H(\Omega (X))$$

and thus we can therefore estimate p( ${\vec{X}}^N$) via

$$p({\overrightarrow{X}}^N)=\prod _{i=1}^{N}p(X_i)\stackrel{\mathit{\text{LLN}}}{\to }exp(-{\mathit{\text{NH}}}_e)$$

This provides an unbiased estimator of the posterior probability of a model term θ

$$p_e(\theta |{\overrightarrow{X}}^N)=\frac{p({\overrightarrow{X}}^N|\theta )p(\theta )}{exp(-{\mathit{\text{NH}}}_e)}$$

We designate this the “empirical posterior” probability of model term θ, with confidence interval:

$$p\left(p_e(\theta |{\overrightarrow{X}}^N)exp(-N\delta )\le p(\theta |{\overrightarrow{X}}^N)\le p_e(\theta |{\overrightarrow{X}}^N)exp(-N\delta )\right)\ge 1-\frac{4\mathit{\text{Var}}(log{P}_e)}{N{\delta }^2}$$

3.7. The Model Self-Consistency Test

We note that a more limited convergence test is possible, by reversing the procedure, and calculating the entropy of the model (which can be done directly, either analytically or by simulation). We define a self-consistency measure

$${\delta }_{\mathit{\text{SC}}}=-H(\Psi (X))-\overline{L_e}$$

where H(Ψ(X)) is the entropy of our model.

For Ψ → Ω, δ_SC → 0. We use this fact to construct a test

$$|-H(\Psi (X))-\overline{L_e}|>\sqrt{\frac{\overline{\mathit{\text{Var}}(L_e)}}{n\in }}$$

for rejecting the null hypothesis that Ψ = Ω at confidence 1 − ∈.

4. Model Information

4.1. What is “Prediction”?

We defined our empirical information metric as a measure of prediction power. However, it seems worthwhile to ask again what exactly we mean by “prediction”. The empirical density estimation procedure outlined above suggests that in the limit of large sample size there is always a trivial way of obtaining perfect prediction power: Copy the empirical density for X as our “likelihood model” for X, and show that it accurately predicts new observations of X. Such a procedure does not seem to qualify as “prediction”; we simply copied the observed density. In this case all the information for the “prediction” came from the observed data, and none at all from the modeling procedure itself. This suggest several conclusions:

• We desire a metric for the intrinsic prediction power of a model, above and beyond just copying the existing observation density. We will refer to this as I_m, the model information.

• Generalizing our original definition of “prediction power”, we wish to maximize our prediction accuracy not only for situations that we have already observed, but also for novel situations that we have never encountered before. In other words, we adopt the conservative position that our data may be incomplete, so we cannot assume that future experience will simply mirror past experience. To maximize future prediction power, we must seek models that predict future observations more accurately than simply interpolating from past observations.

• Of course, we do not know a priori that such models even exist; that is a strictly empirical question. We simply generate models and measure whether they have such intrinsic prediction power, i.e., I_m > 0.

• By definition, such a measurement can only be performed via new observations, e.g., a regionof observation space that we have not observed before. As we will show in a moment, a regionthat has already been observed (thoroughly) cannot yield significant model information, becausethe past observations already provide a good density image for predicting future observations inthis region.

• Thus, we can consider the adoption of a new model to be a cut on the temporal sequence ofobservations, partitioning them into two sets: The “old” observations (those taken before theadoption of the model), and the “new” observations (those taken after the adoption of the model).

4.2. Defining Model Information

The key question of model information is whether the model yields better prediction power than simple interpolation from past observations. As the interpolation reference, we simply use the empirical density calculation defined previously. Specifically, for a model Ψ we define its model information as

$$\overline{I_m(\Psi )}=\overline{L_e(\Psi |\mathit{\text{new}})}+\overline{H_e(\mathit{\text{new}},\mathit{\text{old}})}$$

where $\overline{L_e(\Psi |\mathit{\text{new}})}$ is calculated specifically using the new observations, and we define $\overline{H_e(\mathit{\text{new}},\mathit{\text{old}})}$ = $-\overline{L_e(P_{e,\mathit{\text{old}}}|\mathit{\text{new}})}$ as the empirical cross entropy of the new observations versus the old observations; P_e,old is the empirical density estimator from the old observations. One example implementation (based on the previous empirical density estimator) is

$$\begin{array}{}\overline{H_e(\mathit{\text{new}},\mathit{\text{old}})}=-\frac{1}{n}\sum _{j=1}^{n}log\frac{{\sum }_{i=1}^{n_{\mathit{\text{old}}}}{\kappa }_{X_{j,\mathit{\text{new}}}+\delta x/2}(X_{i,\mathit{\text{old}}})-{\sum }_{i=1}^{n_{\mathit{\text{old}}}}{\kappa }_{X_{j,\mathit{\text{new}}}+\delta x/2}(X_{i,\mathit{\text{old}}})}{n_{\mathit{\text{old}}}\delta x}& \stackrel{\mathit{\text{LLN}}}{\to }-{\mathit{\int }}_{-\infty }^{\infty }\Omega (X)log{P}_{e,\mathit{\text{old}}}(X)\mathit{\text{dX}}\end{array}$$

where X_j,new is the j th observation from the new observation set, X_i,old is the i th observation from the old observations, n is the sample size of the new observations, and n_old is the sample size of the old observations. Many other $\overline{H_e(\mathit{\text{new}},\mathit{\text{old}})}$ estimation implementations are possible. It should be noted that proper normalization of the empirical density is especially important for cross-entropy calculation; however, we will not investigate such implementation details here.

Thus, $\overline{I_m}$ measures whether the model's empirical log-likelihood $\overline{L_e}$ on the new observations exceeds the average log-likelihood of the new observations computed from the old observation density, i.e., $-\overline{H_e(\mathit{\text{new}},\mathit{\text{old}})}$. As for the potential information, we define a lower bound estimator for I_m with confidence level 1 − ∈ based on the Law of Large Numbers:

$$I_{m,\in }=\overline{I_m}-\sqrt{\frac{\overline{Var(L_e-log{P}_{e,old})}}{n\in }}$$

• In the case n_old → 0 we make the density function converge to the uninformative prior based on the detector range for the observable X. That is, if the range of detectable values for X is [0,10] then P_e,_old(X) → 1/10.

• Note that the model information can be negative, indicating that the model has worse prediction power than the old empirical density estimator.

4.3. Example: The Normal Distribution

Figure 3: Model Information of the Normal Distribution. We draw n_old observations from the unit normal distribution N(0,1) and compute the posterior likelihood distribution for this sample. We then draw a new sample of 100 observations from the same distribution and use it to measure I_m for our model. The model information is initially high because the normal model predicts the shape of the distribution much more accurately than simple interpolation from the old observation sample.

4.4. Example: The Binomial Distribution

By contrast, the binomial distribution doesn't yield significant model information, because the observable has only two possible states (success or failure) for the model to predict, and the binomial model's prediction of its probability is just equivalent to the empirical probability in the training data:

$$p(\text{success}|s_{\mathit{\text{old}}},n_{\mathit{\text{old}}})$$

where s_old is the count of successes in the training data, and n_old is the size of the training data set (the +1 and +2 arise from the pseudocount principle, derived by Laplace as his “rule of succession” [19]). Fundamentally, since there is no “shape” for the model to predict (as there would be for a continuous variable, as in the case of the Normal distribution above), there is no way for the model to systematically outperform the empirical distribution.

5. Empirical Information Partition Rules

5.1. The I_p + I_e, I_e − I_m, I_p + I_m Partitions

We now briefly consider the relationships between potential information, empirical information and model information, illustrated in Figure 4: Empirical Information Partition Rules.

• All information originates as potential information. That is, before we have a successful model for a set of observations, our prediction power is no better than random, and this manifests as positive I_p and zero I_e.

• For a given observable X, the sum of I_p + I_e is a constant (i.e., independent of the model Ψ(X)). That is, for any observation sample ${\vec{X}}^n$,

  $$\overline{I_p(\Psi )}+\overline{I_e(\Psi )}=-\overline{H_e}-\overline{L_e(\Psi )}+\overline{L_e(\Psi )}-\overline{L_e(p)}=-\overline{H_e}-\overline{L_e(p)}=\overline{I_p(p)}$$
  where p(X) is the uninformative distribution for X. For large sample size n
  $$\overline{I_p(\Psi )}+\overline{I_e(\Psi )}=\overline{I_p(p)}\stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel p)$$
  which is simply the relative entropy of the true distribution relative to the uninformative distribution p(X).

• Thus potential information is converted to empirical information by modeling. As the model Ψ becomes a more accurate image of the observation density, I_p decreases and I_e increases by the same amount.

• relation to mutual information: It must be emphasized that the mutual information I(X; Ω) is defined only if we know the complete joint distribution p(X, Ω). Since we do not know this joint distribution, we would like a sampling-based estimator for I(X; Ω). We can do this by simply sampling different inference cases Ω₍₁₎, Ω₍₂₎, … Ω_(m) (represented by different observation samples ${\overrightarrow{X}}_{(1)}^n,{\overrightarrow{X}}_{(2)}^n,\cdots {\overrightarrow{X}}_{(m)}^n$ Taking the average of $\overline{I_e(\Psi )}$ + $\overline{I_e(\Psi )}$ over a large number m of inference cases converges:

  $$\begin{array}{}\frac{1}{m}\sum (\overline{I_p}+\overline{I_e})\underrightarrow{\mathit{\text{LLN}}}-E(H(p(X|\Omega )))-E(logp(X))& =\sum _{X,\Omega }p(X,\Omega )logp(X|\Omega )-\sum _{X,\Omega }p(X,\Omega )logp(X)\end{array}$$
  If we explicitly assume that the uninformative distribution used for computing the empirical information matches the true marginal distribution of X, then
  $$=-H(X|\Omega )+H(X)=I(X;\Omega )$$
  Thus, I_p + I_e may be considered to be a “sampleable version of the mutual information”; that is, it can be measured for any individual inference case, and its average over multiple inference problems will converge to the mutual information of the observable vs. hidden variables.

• For a given observable X, the sum of I_e − I_m is a constant. (i.e., independent of the model Ψ(X)).Assuming both I_e, I_m are calculated on the same test data,

  $$\overline{I_e(\Psi )}-\overline{I_m(\Psi )}=\overline{L_e(\Psi )}-\overline{L_e(p)}-\overline{L_e(\Psi )}+\overline{L_e(P_{e,\mathit{\text{old}}})}=-\overline{L_e(P)}+\overline{L_e(P_{e,\mathit{\text{old}}})}$$
  where P_e,_old(X) is the distribution of X computed from past observations (as described above). So for n → ∞
  $$\overline{I_e(\Psi )}-\overline{I_m(\Psi )}\stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel p)-D(\Omega \parallel p_{e,\mathit{\text{old}}})$$
  Thus I_e − I_m measures the amount of information supplied by the past observations (in the form of P_e,_old(X)).

• Moreover, in the asymptotic limit, $\overline{I_e}$ − $\overline{I_m}$ ≥ 0 since for n_old → 0 we guarantee that P_e,_old(X) → p(X) and for n_old → ∞ we have $P_{e,\mathit{\text{old}}}(X)\stackrel{\mathit{\text{LLN}}}{\to }\Omega (X)$.

• Thus, I_m partitions I_e into the part that is simply provided by the training observations themselves, versus the part that actually constitutes “value added” predictive power of the model itself.

• For a given observable X, the sum of I_p + I_m is a constant (i.e., independent of the model Ψ(X)). Specifically, assuming both I_p, I_m are calculated on the same test data,

  $$\begin{array}{}\overline{I_p(\Psi )}+\overline{I_m(\Psi )}=-\overline{H_e}-\overline{L_e(\Psi )}+\overline{L_e(\Psi )}-\overline{L_e(P_{e,\mathit{\text{old}}})}=\overline{I_p(P_{e,\mathit{\text{old}}})}& \stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel P_{e,\mathit{\text{old}}})\end{array}$$
  which simply measures the amount of information available to be learned about the true distribution of X above and beyond that already provided by past observations (in the form of P_e,old(X)).

• Relation of I_m to relative entropy: Note that since $I_p\stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel \Psi )$, this also implies that $I_m\stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel P_{e,\mathit{\text{old}}})-D(\Omega \parallel \Psi )$. This simply restates the principle that the model information representsthe increase in model prediction power relative to the empirical density of the past observations.

5.2. Asymptotic Conversion of Potential and Model Information to Empirical Information

Consider the following asymptotic modeling protocol: For a large sample size n_old → ∞ we simply adopt the empirical density P_e,old as our model Ψ. We then measure I_e,I_p,I_m on a set of new observations.

As n_old → ∞, P_e,old (X) converges to the true density Ω (X), so $H_e(\mathit{\text{new}},\mathit{\text{old}})\stackrel{\mathit{\text{LLN}}}{\to }H(\Omega (X))$ and

$$\overline{I_m}\stackrel{\mathit{\text{LLN}}}{\to }-H(\Omega (X))-D(\Omega \parallel \Psi )+H(\Omega (X))=-D(\Omega \parallel \Psi )\le 0$$

Since the relative entropy is non-negative, the maximum attainable value of the model information drops asymptotically to zero. Moreover, as Ψ(X) = P_e,old(X) also converges to the true density Ω(X), $\overline{I_p}\stackrel{\mathit{\text{LLN}}}{\to }D(\Omega \parallel \Omega )=0$. Since both the model and potential information vanish, by the I_p + I_e and I_e − I_m partition rules, all information is converted exclusively to empirical information.

This scenario illustrates a simple point about the distinct meanings of empirical information vs. model information. The overriding goal of model selection is maximizing empirical information (likelihood). However, this scenario shows that maximizing the empirical information is in a sense trivial if one can collect a large enough observation sample. By contrast, there is no trivial way to produce positive model information; note that the very procedure that automatically maximizes I_e also ensures that I_m ≤ 0.

This suggests several changes in how we think about the value of modeling. In model selection, the value of a model is often thought of in terms of data compression; that is, that the best model encodes the underlying pattern of the data in the most efficient manner possible. Metrics such as the AIC and BIC seek to enforce this principle by adding “correction terms” that penalize the number of model parameters. However, to be truly valuable for prediction, a model should meet this data compression criterion not only retrospectively (i.e., it can yield a more efficient encoding of the past observations) but also prospectively (i.e., it can predict future observations more accurately than simply interpolating from the past observations). Whereas the total empirical information metric fails to draw this distinction, the model information explicitly measures it. That is, it partitions the total I_e into a “trivial” part that represents the prediction power implicit in the observation dataset itself, and a non-trivial part that represents true “predictions” coming from the model.

6. Conclusion

We wish to suggest that these empirical information metrics represent a useful extension of existing statistical inference metrics, because they provide “sampleable” measures of key information theory metrics (such as mutual information and relative entropy), with explicit Law of Large Numbers convergence guarantees. That is, each empirical information metric can be measured via sampling on an individual inference problem (unlike the conventional definition of mutual information); Yet its average value over multiple inference problems will converge to the true, hidden value of its associated metric from information theory (such as the mutual information). On such a foundation, one can begin to recast statistical and scientific inference problems in terms of the very useful and general tools of information theory. For example, the “inference halting problem”, which imposes a variety of problems and limitations in Bayesian inference, can be easily resolved by the potential information metric, which directly measures the distance of the current model from the true distribution in standard information theoretic terms. Similarly, the model information metric measures the “value-added” prediction power of a model relative to its training data.