1. Introduction
Consider a statistician working on a problem
P in which a vector
of real-valued outcomes is to be observed, and—prior to, i.e., without observing
—the statistician’s uncertainty is exchangeable, in the usual sense of being invariant under permutation of the order in which the outcomes are listed in
. This situation has extremely broad real-world applicability, including (but not limited to) the analysis of a completely randomized controlled trial, in which participants—ideally, similar to elements of a population to which it is desired to generalize inferentially—are randomized. Each participant is assigned either to a control group that receives the current best treatment, or to an experimental group that receives a new treatment whose causal effect on
y is of interest. This design, while extremely simple, has proven to be highly useful over the past 90 years, in fields as disparate as agriculture [
1], medicine [
2], and (in contemporary usage)
testing in data science on a massive scale [
3]. We use randomized controlled trials as a motivating example below, but we emphasize that they constitute only one of many settings to which the results of this paper apply.
Focusing just on the experimental group in the randomized controlled trial, the exchangeability inherent in
implies via de Finetti’s Theorem [
4] that the statistician’s state of information may be represented by the hierarchical model
for
, where
F is a cumulative distribution function (CDF) on
and
is a prior on the space of all such CDFs, i.e., the infinite-dimensional probability simplex
. Note that (
1) has uniquely specified the likelihood in a Bayesian nonparametric model for
, and all that remains is specification of
.
Speaking now more generally (not just in the context of a randomized controlled trial), suppose that the nature of the problem
P enables the analyst to identify an alternative statistical problem
in which
where
G is a collection of transformations
g from one problem to another having the property that, without having seen any data,
and
P are the
exact same problem. Then, the prior
under
must be the same as the prior
under
P! Furthermore, since this holds for any
, the result will be, as long as
G is endowed with enough structure, that there is one and only one prior
, for use in
P, that respects the inherent invariance of the problem under study. Bayes’ Rule then implies that there is one and only one posterior distribution under
P. When this occurs—when both the likelihood function and the prior are uniquely specified, as in the example above—we say that the problem
P admits an
optimal Bayesian analysis.
The logic underlying the above argument has been used to motivate and formalize the notion of noninformative priors for decades. Indeed, in the special case where
F is parametric and
G is a group of transformations encoding invariance with respect to monotonically-transformed units of measurement, Jeffreys [
5] derived the resulting prior distribution. As another example, Jaynes [
6] derived the prior distribution for the mean number of arrivals of a Poisson process by using its characterization as a Lévy counting process to specify an appropriate transformation group. Notably, the resulting prior distribution is
not the Jeffreys prior, because the problem’s invariance and corresponding transformation group are different. See Eaton [
7] for additional work on this subject.
Having studied this line of reasoning, it is natural to ponder its generality. In this paper we show that the argument can be made quite general—we prove that the argument’s formal notions
- (a)
can be generalized to include approximately invariant priors in an – sense; and
- (b)
can be extended to infinite-dimensional priors on spaces of CDFs.
We focus on the setting described in (
1) and defer more general situations to future work. In this setting we derive a number of results, ultimately showing that the Dirichlet Process [
8] prior
is an approximately invariant stochastic process for any CDF
on
and sufficiently small
. Together with de Finetti’s Theorem, this demonstrates that the posterior distribution
where
is the empirical CDF, corresponds in a certain sense to an optimal Bayesian analysis—see
Section 3 for more on this point.
Not all approaches to noninformative priors are based on group invariance. Perhaps the earliest approach can be traced back to Laplace [
9], who proposed a Principle of Indifference under which, if all that is known about a quantity
is that
(for some set
of possible values), then the prior should be uniform on
. For example, consider
: the fact that
is not consistent with
for any monotonic nonlinear
f requires that the problem
P under study must uniquely identify the scale on which uniformity should hold for the principle to be valid—this was a major reason for the rise of non-Bayesian theories of inference in the 19th century [
10]. Bernardo [
11] has proposed a notion of noninformative priors that is defined by studying their effect on posterior distributions, and choosing priors that ensure that prior impact is minimized. Jaynes [
12] has proposed the Maximum Entropy Principle, which defines noninformative prior distributions via information-theoretic arguments, for use in settings in which invariance considerations do not lead to a unique prior. All of these notions are different, and applicable to problems where the corresponding notions of noninformativeness arise most naturally.
Most of the work on noninformative priors has focused on the parametric setting, in which the number of unknown quantities is finite. In contrast, Bush et al. [
13] and Lee et al. [
14] have derived results on noninformative priors in Dirichlet Process Mixture models. Their notion of noninformativeness is completely different from our own, as it is a posteriori, i.e., it involves examining the behavior of the posterior distribution under the priors studied. This makes their approach largely complementary to ours: in specifying priors, it is helpful to understand both the prior’s effect on the posterior and the prior’s behavior a priori without considering any data.
Here we study noninformative prior specification from a strictly a priori perspective. We do not consider the prior’s effect on the posterior distribution. There is no data or discussion of computation.
Our motivation is a generalization of the following argument by Jaynes [
12]. Suppose that in the randomized controlled trial described above, the outcome
y of interest is binary. By de Finetti’s Theorem, we know that
is the unique likelihood for (e.g., the treatment group in) this problem. Suppose further that the statistician’s state of information about
external to the data set
is what Jaynes calls “complete initial ignorance” except for the fact that
is such that
Jaynes argues that this state of information is equivalent to the statistician possessing complete initial ignorance about all possible rescaled and renormalized versions of
, namely
for all positive
. Jaynes shows that this leads uniquely to the Haldane prior
where
. Combining this result with the unique Bernoulli likelihood under exchangeability, in our language Jaynes has therefore identified an instance of optimal Bayesian analysis. In what follows we (a) extend Jaynes’s argument to the multinomial setting with
p outcome categories for arbitrary finite
p and (b) show how this generalization leads to a unique noninformative prior on
.
The
posterior and implied
prior—see
Table 1 for the notational conventions used in this work—have not been subject to the same level of formal study as Dirichlet Process Mixture priors and other priors over CDFs, in part due to the simplicity and discrete nature of
. On the other hand, Dirichlet and Dirichlet Process priors with small concentration parameters have been used as low-information priors in a variety of settings (e.g., [
15]), without much formal justification. In this paper we offer a mathematical foundation showing that the use of
is statistically sound.
2. Results
2.1. Preliminaries
To begin our discussion, we first introduce the notion of an invariant distribution, which describes what we mean by the term noninformative.
Definition 1. [Invariant Distribution] A density is invariant with respect to a transformation group
G if for all with , and all measurable sets A,where is the Jacobian of the transformation. Note that in Equation (
8), if we were to instead take
A in the middle and right integrals to be
, we would exactly get the classical integration by substitution formula, which under appropriate conditions is always true. We are interested in the inverse problem: given a set of transformations in
G, does there exist a unique
satisfying (
8)?
In a number of practically-relevant cases,
G is uniquely specified by the context of the problem being studied. If this leads to a unique prior distribution
, and when additionally a unique likelihood also arises, for example via exchangeability, an optimal Bayesian analysis is possible, as defined in
Section 1. It is often the case that the prior distributions that result from this line of reasoning are limits of conjugate families, making them easy to work with—this occurs in our results below, in which the corresponding posterior distributions are Dirichlet.
The above definition is intuitive, but not sufficiently general to be applicable to spaces of functions. There are multiple technical issues:
- (a)
in many cases, cannot be taken to integrate to 1;
- (b)
probability distributions on spaces of functions may not admit Riemann-integrable densities;
- (c)
G may be defined via equivalence classes of transformations, leading to singular Jacobians; and
- (d)
infinite-dimensional measures that are non-normalizable are not well-behaved mathematically.
As a result, the above definition needs to be extended to a measure-theoretic setting. We call a transformation group G acting on a measure space nonsingular if for with , we have , where ≪ denotes absolute continuity of measures.
Definition 2. [Invariant Measure] Let G be a nonsingular transformation group acting on a measure space. We say that a measure π is invariant with respect to
G if for any with and for any measurable subset A we havewhere Ω
is the domain of π, is the indicator function of the set A, and is the Radon–Nikodym derivative of with respect to π. It can be seen by taking
to be absolutely continuous with respect to the Lebesgue measure that Equation (
9) is a direct extension of Equation (
8).
We would ultimately like to extend the above definition to the infinite-dimensional setting. Doing so directly is challenging, because may be non-normalizable, in which case Kolmogorov’s Consistency Theorem and other analytic tools for infinite-dimensional probability measures do not apply. Here we sidestep this problem by instead extending the definition of invariance to allow us to define a sequence of approximately invariant measures, which in our setting can be taken to be probability measures. To do so, two additional definitions are needed.
Definition 3 (ε-invariant Measure).
Let G be a nonsingular transformation group acting on a measure space with invariant measure . We say that a sequence of measures is-invariant with respect to
G if for any with and each measurable subset A, the inequalityimplies thatwhere is a function, implies that , and Ω
is the domain of for all ε. Definition 4 (ε-invariant Process).
Let be a sequence of stochastic processes, and let G be a nonsingular transformation group. Let I be an arbitrary finite subset of the index set of the process, let be the finite-dimensional measure of under I, and let be a finite-dimensional homomorphism of G with invariant measure . We say that the sequence of processes is-invariant
if, for each I, each with and each measurable subset A, the inequalityimplies thatwhere is a function, implies that , is the domain of for all ε, and can be taken to be identical for all I. Definition 4 has been explicitly chosen to formalize the notion of noninformativeness on a space of functions without constructing a non-normalizable infinite-dimensional measure.
To complete our assumptions, we need to specify
G. Our definitions constitute a direct generalization of the transformation group used by Jaynes to derive the Haldane prior for
—see
Section 1.
Definition 5 (Probability Function Transformation Group).
Letbe a nonsingular group of measurable functions under composition acting on the infinite-dimensional simplex . Definition 6 (Probability Vector Transformation Group).
For non-negative integer p and any vector of non-negative constants, letbe a nonsingular group under composition acting on the p-dimensional simplex , where each element represents an equivalence class of the transformations (15). Note that is a p-dimensional homomorphism of —we use this property in our proofs below. It can also readily be seen that for any g, the constants are only determined up to proportionality.
Proposition 1 (Radon–Nikodym Derivative).
For each and , the Radon–Nikodym derivative of with respect to π is Proof. Let
be the Lebesgue measure on the
p-dimensional probability simplex, and define
. Note first that
. Note also that
because the same transformation
g is used in defining
and
. Then, note that
and hence it suffices to consider the transformation
g applied to the Lebesgue measure. Consider an arbitrary hypercube
B. We have
where
are 1-dimensional Lebesgue measures, for which we have that
where
is the one-dimensional projection of the hypercube
B in dimension
i. Consider now the transformation
g. We may decompose
g into
d and
n where
Now consider the effect of
d and
n on
. We have
hence
Therefore
and we can compute the ratio
This holds for all
B, hence the Radon–Nikodym derivative is just
which is the desired result. ☐
Since we are working with non-normalizable measures as improper priors, we cannot rigorously talk about their probability densities. In many cases, such improper priors can be shown to be limits of families of conjugate priors for which the limiting posterior distribution is well-defined, making them usable in practice. To make our discussion of improper priors rigorous, we need the following definition.
Definition 7 (Generalized Density).
Let π be a measure on (for p a positive integer) such that , where λ is Lebesgue measure on . Suppose that the Radon–Nikodym derivative of π with respect to λ is Riemann-integrable, and define a family of functions equal to the Radon–Nikodym derivative up to a proportionality constant. We call any function in this family a generalized density of π.
2.2. Main Results
Remark 1 (Notation).
In the following results, we will assume that is a probability vector of dimension . and will be the transformation groups identified in Definitions 5 and 6, respectively. As noted previously in Table 1, will denote the Dirichlet distribution under the alternative parametrization based on concentration parameter α and mean probability vector . This is equivalent to the usual parameterization in terms of concentration vector α by the identity —we refer to this as the distribution. Similarly, will refer to the Dirichlet Process with concentration parameter α and mean function . We will refer to the improper priors defined via the conjugate limits as of and for arbitrary as and , respectively. We are now ready to introduce our first result. The argument below is a direct generalization of the line of reasoning in Jaynes [
12]: the Haldane prior obtained is a special case of our result for
.
Theorem 1. Among the class of measures that admit generalized densities, the measure π with generalized densitywhich we call , is the unique invariant measure under . Proof. An invariant measure
under
needs to satisfy the equation
where
is the
p-dimensional simplex and
for some
. Since
is assumed to admit a generalized density, we can rewrite (
28) as a Riemann integral. In addition, we substitute in the transformation and Radon–Nikodym derivative, and get
This formula needs to hold for all measurable sets
A, and hence the functions inside the integrals need to be equal pointwise. This yields the functional equation
which will be the main subject of further study. This is a multivariate functional equation that at first may appear fearsome, but is in fact solvable via elementary methods. To solve it, recognizing that (
30) must hold for all probability vectors
and all vectors
of positive constants
, we set
which yields
Then, by swapping
for
, (
32) rearranges into
since the numerator is not a function of any
, and it can easily be checked that all such generalized densities are valid solutions to the original equation. Thus (
33) is the functional equation’s unique solution and therefore the unique invariant measure under
. ☐
The same technique used to solve the functional equation in Theorem 1 can be used to prove a much stronger result: if the functional equation is true approximately, its solutions will approximate those of the exact equation. In the next result we make use of the definition of
stability of a functional equation due to Hyers, Ulam and Rassias—see Jung [
16] for details.
Corollary 1 (Hyers–Ulam–Rassias Stability).
Proof. By repeating the technique from the previous proof, we have
which can be rewritten
where the last inequality is strict because
p is a positive integer. Letting
we get
which is the stability result desired. ☐
This suffices to prove our result for the Dirichlet distribution.
Theorem 2. is an ε-invariant measure under for all .
Proof. By repeating the steps of Theorem 1 and combining them with Corollary 1, we obtain that
is
-invariant under
if and only if it satisfies
Substituting in
, and choosing the constant
of the generalized density
to be the same as for the Dirichlet, we get
where
are the components of the probability vector
, and this expression simplifies to
Since
for all
i, the product is upper bounded by 1 and lower bounded by 0. Thus the inequality holds near zero if
for all
, and since
we get that, as
, we can choose
such that
. Thus,
is
-invariant for all
. ☐
We now extend Theorem 2 to get an analogous result for the Dirichlet Process.
Theorem 3. is an ε-invariant process under for all .
Proof. Consider an arbitrary finite-dimensional index
I with corresponding homomorphism
and finite-dimensional measure
. It follows from Theorem 2 that
is
-invariant with
This inequality depends only on
, so it suffices to show that this constant can be bounded by another constant that is not a function of
p and approaches 0.
is an instance of the inverse multivariate beta function, which is a ratio of gamma functions. It is well known that
where
is the Euler-Mascheroni constant. Therefore, we have
as
. Thus, for each
, we can choose a
to satisfy the required expressions under all finite-dimensional index sets, and
is therefore an
-invariant process. ☐
We conclude our theoretical investigation with a conjecture: the -invariance of all finite-dimensional distributions with a uniform should suffice for invariance with respect to the original group acting on the infinite-dimensional space.
Conjecture 4. A stochastic process is an ε-invariant process if and only if the measure of its sample paths is an ε-invariant measure.
One approach to attempting a proof of this conjecture would involve appropriately extending Kolmogorov’s Consistency Theorem to
-finite infinite-dimensional measures. This can be done, but the notions involved are quite technical—see Yamasaki [
17] for more details.
3. Discussion
To see how our results may be applied, consider again the randomized controlled trial of
Section 1, and suppose now that the outcome
for participant
i in the experimental group is categorical with
p levels. Under exchangeability, a minor extension of de Finetti’s Theorem for dichotomous outcomes then yields that the likelihood can be expressed as
in which MN
is the multinomial distribution with parameters
k and
. Theorem 1 implies that, modulo inherent abuse of notation under improper priors,
is the unique prior that obeys the fundamental invariance possessed by the problem—namely, invariance with respect to all transformations of probability vectors that preserve normalization. Thus we have extended Jaynes’s result for binomial outcomes to the multinomial setting, yielding another instance of optimal Bayesian analysis.
Generalizing to the setting where
is an exchangeable sequence of real-valued outcomes, de Finetti’s most general representation theorem implies that
is the unique likelihood. If little is known about
F, and it is therefore approximately invariant under all measurable functions—i.e., under
, see Definition 5—the prior given by Theorem 3 is
By the usual conjugate updating in the Dirichlet Process setting, the posterior on
F given
with the prior in (
50) is
in which
is the empirical CDF based on
. Since
may be taken as close to zero as one wishes, it is natural to regard
as an instance of approximately optimal Bayesian analysis for all
. Conjecture 4 would strengthen this assertion—provided
can be rigorously constructed as an infinite-dimensional
-finite measure, which is beyond the scope of this work.
Though the simplicity of this analysis may at first make it seem limited, its appeal comes from its extremely general ability to characterize uncertainty. See, e.g., Terenin and Draper [
18] for an example of a
analysis in two randomized controlled trials in e-commerce, one with sample sizes in the tens of millions. Furthermore, sampling from
on a discrete domain has recently been shown in a completely different setting—see Appendix B of Terenin et al. [
19]—to be asymptotically equivalent to the widely-used frequentist bootstrap of Efron [
20]. This also applies to the Bayesian bootstrap of Rubin [
21], since it is asymptotically equivalent to the frequentist version. Our analysis provides a Bayesian nonparametric justification for this class of methods.
Bayesian analysis cannot proceed without the specification of a stochastic model—prior and sampling distribution—relating known quantities to unknown quantities: data to parameters. One of the great challenges of applied statistics is that the model is not necessarily uniquely determined by the context of the problem under study, giving rise to model uncertainty, which if not assessed and correctly propagated can cause badly calibrated and unreliable inference, prediction and decision—see, e.g., Draper [
22]. Perhaps the simplest way to avoid model uncertainty is to recognize settings in which it does not exist—situations where broad and simple mathematical assumptions, rendered true by problem context, lead to unique posterior distributions. Our term for this is
optimal Bayesian analysis. It seems worthwhile (a) to catalog situations in which optimal analysis is possible and (b) to work to extend the list of such situations—Theorems 1 and 3 are two contributions to this effort.