The Objective Bayesian Probability that an Unknown Positive Real Variable Is Greater Than a Known Is 1/2

Abstract

If there are two dependent positive real variables ${\mathfrak{x}}_1$ and ${\mathfrak{x}}_2$, and only ${\mathfrak{x}}_1$ is known, what is the probability that ${\mathfrak{x}}_2$ is larger versus smaller than ${\mathfrak{x}}_1$? There is no uniquely correct answer according to “frequentist” and “subjective Bayesian” definitions of probability. Here we derive the answer given the “objective Bayesian” definition developed by Jeffreys, Cox, and Jaynes. We declare the standard distance metric in one dimension, $d(A,B)\equiv |A-B|$, and the uniform prior distribution, as axioms. If neither variable is known, $P(x_2<x_1)=P(x_2>x_1)$. This appears obvious, since the state spaces $x_2<x_1$ and $x_2>x_1$ have equal size. However, if ${\mathfrak{x}}_1$ is known and ${\mathfrak{x}}_2$ unknown, there are infinitely more numbers in the space $x_2>{\mathfrak{x}}_1$ than $x_2<{\mathfrak{x}}_1$. Despite this asymmetry, we prove $P(x_2<{\mathfrak{x}}_1\mid {\mathfrak{x}}_1)=P(x_2>{\mathfrak{x}}_1\mid {\mathfrak{x}}_1)$, so that ${\mathfrak{x}}_1$ is the median of $p\left(x_2\right|{\mathfrak{x}}_1)$, and ${\mathfrak{x}}_1$ is statistically independent of ratio ${\mathfrak{x}}_2/{\mathfrak{x}}_1$. We present three proofs that apply to all members of a set of distributions. Each member is distinguished by the form of dependence between variables implicit within a statistical model (gamma, Gaussian, etc.), but all exhibit two symmetries in the joint distribution $p(x_1,x_2)$ that are required in the absence of prior information: exchangeability of variables, and non-informative priors over the marginal distributions $p\left(x_1\right)$ and $p\left(x_2\right)$. We relate our conclusion to physical models of prediction and intelligence, where the known ’sample’ could be the present internal energy within a sensor, and the unknown the energy in its external sensory cause or future motor effect.

1. Introduction

We consider a problem of prediction (or inference or estimation). We seek to identify

$$P(x_2<{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1)={\int }_0^{{\mathit{𝖝}}_1}p(x_2\mid {\mathit{𝖝}}_1)d{x}_2,$$

(1)

where ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2$ are positive and finite real numbers that exhibit a statistical dependence $({\mathit{𝖝}}_i\in x_i\subseteq \mathbb{R}+)$, and there is no additional information that is relevant to probability. These variables could represent physical sizes or magnitudes, such as distances, volumes, or energies. Given only one known size (a sample), what is the probability that an unknown size is larger versus smaller? The answer is not obvious, since there are infinitely more possible sizes corresponding to ‘larger’ than ‘smaller’ (Figure 1).

Figure 1. A portion of the state space of two positive real variables ${\mathit{𝖝}}_1\in x_1$ and ${\mathit{𝖝}}_2\in x_2$ (the space has no upper bound). The sub-spaces highlighted correspond to the propositions $x_2<x_1$ (light blue), $x_2>x_1$ (light red), ${\mathit{𝖝}}_1=a$, and ${\mathit{𝖝}}_2=a$. Exchangeability of $x_1$ and $x_2$ (Equation (14)) corresponds to symmetry of probabilities around the identity line $x_1=x_2$; thus $P(x_2<x_1)=P(x_2>x_1)$, and $p({\mathit{𝖝}}_1=a)=p({\mathit{𝖝}}_2=a)$. If ${\mathit{𝖝}}_1=a$ is known, the subspace $x_2>{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1=a$ is infinitely larger than $x_2<{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1=a$. However, we prove that ${\mathit{𝖝}}_2\in x_2$ remains equally likely to be larger versus smaller than ${\mathit{𝖝}}_1=a$.

That such a basic question does not already have a recognized answer is explained not by its mathematical difficulty, but by the remarkable controversy that has surrounded the definition of probability over the last century. We cannot hope to overcome longstanding disputes here, but we try to clarify issues essential to our conclusion (at the cost of introducing substantial text to what would otherwise be concise and routine mathematics).

The question we pose does not involve any frequency; thus it is entirely nonsensical if probability measures frequency. The “frequentist” definition of probability dominates conventional statistics. It was promoted by Fisher and others in an effort to associate probability with objectivity and ontology, but it has severe faults and limitations [1]. We follow instead the Bayesian definition, in which probability concerns prediction (inference) and information (evidence) and epistemology.1

A precise definition of Bayesian probability requires one to specify the exact mathematical relation used to derive probability from information (using criteria such as indifference, transformation invariance, and maximum entropy). This divides Bayesians into two camps. The “objectivists” consider the relation between information and probability to be deterministic [1,2,3,4,5,6,7,8,9,10,11,12], whereas the “subjectivists” consider it to be indeterminate [13,14,15,16,17,18,19,20] (for discussion, see References [1,11,12,17]). We follow the objective Bayesian approach, developed most notably by Jeffreys, Cox, and Jaynes [1,2,3,4,5,6,7].

All Bayesians agree that information is subjective, insofar as it varies across observers (in our view, an observer is information, and information is matter, therefore being local in space and time). The objectivists consider probability to be objective in the sense that properly derived probabilities rationally and correctly quantify information and prediction, just as numbers can objectively quantify distance and energy. Therefore, once the information in our propositions (e.g., $a<{\mathit{𝖝}}_i<b$) is rigorously defined, and sufficient axioms are established, there should exist a unique and objectively correct probability $P(x_2>{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1)$ that is fully determined by the information. In contrast, subjective Bayesians do not accept that there exists such a uniquely correct probability [13,14,16,17,18,20]; therefore they are predisposed to reject our conclusion.2

Objectivity requires logical consistency. All rigorous Bayesians demand a mathematical system in which an observer with fixed information is logically consistent (e.g., never assigns distinct probabilities to the same proposition). However, subjectivists do not demand logical consistency across observers (Section 3.2). They accept that a Bayesian probability could differ between two observers despite each having identical evidence (see Reference [14], pp. 83–87). In that sense, they see probability as a local and personal measure of evidence, with the relation of probability to evidence across observers being indeterminate. In contrast, the objective Bayesian ideal is a universal mathematics of probability in which the same rules apply to all observers,3 so that probability measures evidence, and evidence uniquely determines probability.

The objective ideal can only be achieved by adopting a sufficient set of axioms (Section 3). We see failure to do so as the critical factor explaining why subjectivists believe the relation between information and probability to be indeterminate (Section 3.2). A mathematical system must be defined by axioms that are neither proven nor provable (they cannot be falsified). For example, there are axioms that define real numbers, and Kolmogorov’s axiom that a probability is a positive real number. These axioms are subjective insofar as others could have been chosen, and there is not a uniquely correct set of axioms.4 There may be any number of axiomatic systems that are each internally consistent and able to correctly (objectively) measure evidence across all observers. We believe the system of Jeffreys, Cox, and Jaynes to be one of these. Thus, we deduce $P(x_2>{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1)=1/2$ from our axioms, but we do not claim that our axioms are uniquely correct. We claim only that they are commonly used and useful standards.

We are interested in knowledge of only a single ’sample’ because we believe it to be a simple yet realistic model of prediction by a physical observer, such as a sensor in a brain or robot (Section 10) [21,22]. Jaynes often used the example of an “ill informed robot” to clarify the relation of probabilities to information [1]. Unlike a robot, a scientist given only one sample can choose to collect more data before calculating probabilities and communicating results to other scientists. In contrast, a robot must continually predict and act upon its dynamically varying environment in “real time” given whatever internal information it happens to have here and now within its sensors and actuators. If the only information in a sensor at a single moment in time is a single energy ${\mathit{𝖝}}_1$, then the probability that an external or future energy ${\mathit{𝖝}}_2$ is larger must be conditional on only ${\mathit{𝖝}}_1$, as expressed by $P(x_2>{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1)$. At that moment, ${\mathit{𝖝}}_1$ is fixed, and it is not an option for the sensor to gain more information or to remain agnostic. This example helps to clarify the role of probability theory, for which the fundamental problem is not to determine what the relevant information is, or should be, but rather to derive probabilities once the information has been precisely and explicitly defined.

2. Notation

We use bold and stylized font for variables (samples) ${\mathit{𝖃}}_1,{\mathit{𝖃}}_2,\dots ,{\mathit{𝖃}}_n$, and standard font for their corresponding state spaces $X_1,X_2,\dots ,X_n$ (note that these are not technically “random variables”; see Section 4.2). Therefore ${\mathit{𝖃}}_i$ is an element from the set of real numbers $X_i$ (${\mathit{𝖃}}_i\in X_i\subseteq \mathbb{R}$). Whereas these represent dependent variables, ${𝒴}_i\in Y_i\subseteq \mathbb{R}$ are independent of one another (Section 6.3). The subscripts are arbitrary and exchangeable labels; therefore the implication of a sequence is an unintended consequence of this notation.

We denote arbitrary numbers $A,B,C,D\in \mathbb{R}$. For each uppercase letter representing real-numbered “location”, we have a corresponding lowercase letter representing a positive real “scale”, $x_i=exp\left(X_i\right)$ (Section 4.2). Therefore lowercase letters symbolize positive real numbers (except ${\mathit{s}}_i\in s$, which symbolizes a physical size and is not a number).

As described in Section 4, all probabilities are conditional on prior information I and J, as well as K in cases involving both $x_1$ and $x_2$ (e.g., $p(x_1,x_2)$ and $p\left(x_2\right|{\mathit{𝖝}}_1)$). For simplicity, this conditionality is not shown in notation.

We use uppercase P for the probability of discrete propositions, with defined intervals, $a<{\mathit{𝖝}}_i<b$, as in the case of a cumulative distribution function (CDF). We use the lowercase p for a probability density function (PDF), which is the derivative of its corresponding CDF, $p\left(x_i\right)=dP\left(x_i\right)/d{x}_i$. Because our simplified notation omits the variable of interest and the propositions, for clarity, we define it for CDF and PDF as

$$\begin{array}{ccc}\hfill P\left(x_i\right)& \equiv & P({\mathit{𝖝}}_i<x_i)=F_{\mathit{𝖝}i}\left(x_i\right),\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill p\left(x_i\right)& \equiv & p({\mathit{𝖝}}_i=x_i)=f_{\mathit{𝖝}i}\left(x_i\right),\hfill \end{array}$$

(3)

where $x_i$ represents the set of possible values of the number ${\mathit{𝖝}}_i$; thus ${\mathit{𝖝}}_i=x_i$ represents an infinite set of propositions, one for each real number. For example, if we had discrete units of 0.1 rather than continuous real numbers, then $[{\mathit{𝖝}}_i=x_i]\equiv [{\mathit{𝖝}}_i=0.1,{\mathit{𝖝}}_i=0.2$, ${\mathit{𝖝}}_i=0.3,\dots ]$. Analogous notation applies to locations; thus, $P\left(X_i\right)\equiv P({\mathit{𝖃}}_1<X_i)=F_{\mathit{𝖃}i}$$\left(X_i\right)$.

A joint distribution over two variables is a marginal distribution of the full joint distribution (following the sum rule),

$$p(x_1,x_2)={\int }_0^{\infty }d{x}_3{\int }_0^{\infty }d{x}_4\cdots {\int }_0^{\infty }d{x}_{n}p(x_1,x_2,x_3,\dots ,x_n),$$

(4)

where n approaches infinity. Likewise, $p\left(x_1\right)$ is a marginal distribution of $p(x_1,x_2)$.

3. Axioms

Like other Bayesians, we take as axioms the product and sum rules of Cox (the product rule being Bayes’s Theorem), and Kolmogorov’s first axiom that a probability is a positive real number [1,4,5]. Kolmogorov’s second and third axioms together stipulate that a mutually exclusive set of probabilities must sum to one. Like Cox and Jaynes, we apply this normalization rule where convenient (including the conclusion in our title). However, we do not elevate it to an axiom, since to do so would be unnecessarily restrictive and inconvenient (see Reference [11]).

3.1. Axioms Determine the Non-Informative Prior Distribution

Since we choose to quantify a physical size using a real number ${\mathit{𝖃}}_i$, we need axioms that define real numbers. In particular, we need to measure the size of a subset of real numbers $A<{\mathit{𝖃}}_i<B$ (a partition of the state space). For this, we would like to have a volume or integral that is invariant (a Haar measure) under addition (translation invariance; see Section 5). Since our particular interest concerns metric spaces, this Haar measure is essentially the same as a distance metric. We, therefore, choose our measure in one dimension to be the standard distance metric, which is almost universally assumed when using real numbers.

$$\begin{array}{c}\hfill d(A,B)\equiv |A-B|.\end{array}$$

(5)

Thus the (uncountable) “number” of values ${\mathit{𝖃}}_i$ could take between A and B is $|A-B|$. We must also specify probability as a function of distance. Our probability measure need not necessarily be the same as our distance measure, as a matter of logic. However, the obvious choice is to equate them, so that the probability that ${\mathit{𝖃}}_i$ lies within a distance will be that distance. For $A<B$,

$$\begin{array}{c}\hfill P(A<{\mathit{𝖃}}_i<B)=d(A,B)\equiv B-A.\end{array}$$

(6)

We declare Equations (5) and (6) to be axioms. These may be seen as innocuous and trivial, but by explicitly asserting them to be axioms, we eliminate the appearance of indeterminacy that underlies the subjectivist view.

The axiom in Equation (6) can be expressed as a cumulative distribution $P({\mathit{𝖃}}_i<X_i)=X_i$, or in our more concise notation, $P\left(X_i\right)=X_i$ (Equation (2)). Its derivative is the uniform density, $p\left(X_i\right)=1$ (Equation (3)) (Section 6.1). This is known as a “non-informative prior” (or “reference prior” or “null distribution”), a term for probabilities that are, or are intended to be, conditional on a complete absence of information. Although the uniform prior has long been used, it is the only portion of our proofs that has been a matter of controversy, as we discuss next.

3.2. Controversies Concerning the Uniform Distribution

Whereas we have derived the uniform prior over real numbers, $p\left(X_i\right)=1$, from the standard distance metric, it has long been justified by “the principle of indifference”. We state this principle as the choice to represent equal evidence with equal probabilities, together with the rationale that a complete absence of evidence must correspond to equal evidence for each proposition (we treat each possible number equally). Although this principle has always been used, it has long been alleged to result in logical contradictions [23]. Whereas indifference only applies in the complete absence of evidence, alleged counterexamples invariably and unwittingly depend upon implicit evidence that discriminates amongst propositions and results in non-uniformity once properly utilized [1,9,24]. An example arises here because we consider two alternative numerical representations (parameterizations) of the same size, $x_i=exp\left(X_i\right)$ and $X_i=log\left(x_i\right)$ (Section 4.2). Assigning uniformity over both $X_i$ and $x_i$ would result in logical contradictions. Instead, we begin with axioms that assign uniformity over $X_i$, and then introduce information $x_i=exp\left(X_i\right)$, so that $p\left(x_i\right)$ is not uniform (Section 6.2).

Subjectivists have defended use of the uniform prior as a practical matter but have argued or implied that uniquely correct non-informative priors do not exist [13,14,16,17,18]. Objectivists have argued that the uniform prior is uniquely correct in the absence of information [1,2,3,9,11]. Here, we resolve this issue, at least as a practical matter, by declaring the standard distance metric and the uniform density as axioms. These axioms may be “uniquely correct” in the sense that they are the least complex and most convenient. However, we have no need to make that claim, and simplicity and convenience are not essential to an objective and logically consistent mathematics of probability. (see footnote 4).

Specifically, subjectivists have criticized the uniform prior $p\left(X_i\right)=1$ on the grounds that it is not uniquely correct because, whether one derives uniformity from the principle of indifference or location invariance (Equation (16)), it has the general form $p\left(X_i\right)=c$, where c is any positive constant. Thus, the non-informative prior appears indeterminate, and if different observers choose different values of c, there will appear to be logical inconsistency across observers (see Reference [14], pp. 83–87). This trivial but undesirable sort of subjectivity is analogous to one observer choosing the metric system and another the English system of measurement. It is overcome by adopting an axiom that fixes c as a universal constant to apply to all observers. We have achieved the same end merely by declaring the standard distance metric an axiom, from which we deduce $p\left(X_i\right)=1$ (Section 3.1). Our proofs would be unaltered were we to choose any specific constant $c>0$. However, $c\ne 1$ would be formerly equivalent to defining a non-standard distance metric, $d(A,B)\equiv c|A-B|$.

The uniform density has also caused confusion because it is unnormalizable (“improper”) and hence not a “probability” as defined by Kolmogorov’s axioms (Section 3). The semantic issue of whether to denote the function $p\left(X_i\right)=1$ a “probability” is not critical (we do, but Norton does not [9]). The deeper issue concerns the rationale for, and form of, normalization. In geometry, one can only assign a number to a single distance between points in an arbitrary manner. The assignment of ‘1’ to ‘a meter’ is convenient but is otherwise arbitrary and meaningless. Numbers acquire meaning in applied mathematics only when there is a ratio of two sizes. Thus ‘8 m’ represents, by definition, the ratio 8/1. The numerator and denominator are each arbitrary and meaningless in isolation, but the ratio is not; therefore science measures relative size as ratio (or log ratio). The same applies to probability, as in $P(2<{\mathit{𝖃}}_i<10)/P(1<{\mathit{𝖃}}_i<2)=8/1$, which has a perfectly clear meaning, even though both numerator and denominator are each “improper probabilities” that mean nothing in isolation. Whereas this ratio compares the probability of one ‘part’ to another, the standard ratio that has been chosen by convention to define a “proper probability” is ‘part / whole’, $P\left({\mathit{𝖃}}_i^{+}\right)/(P\left({\mathit{𝖃}}_i^{+}\right)+P\left({\mathit{𝖃}}_i^{-}\right))$, where ‘+’ and ‘−’ indicate true and false propositions. Kolmogorov elevated this form of normalization to an axiom, but, like Cox and other objective Bayesians, we do not [1,4,5]. However, the conditional probabilities that are our ultimate interest, $p\left(X_2\right|X_1)=p(X_1,X_2)/p\left(X_1\right)$, do normalize in the conventional manner, since, although the numerator and denominator on the right are each “improper” (unnormalizable), the conditional probability is their ratio and is a properly normalized density.

4. Defining the Prior Information

We try to define the prior information as rigorously and explicitly as possible, since failure to do so has been a major source of confusion and conflict regarding probabilities. We distinguish information I about physical sizes, information J about our choice of numerical representation of size (the variables), and information K about the statistical dependence between variables. For clarity, we also distinguish the prior information from its absence (ignorance). Although it is the information that we seek to quantify with probabilities, it is its absence that leads directly to the mathematical expressions of symmetry that are the antecedents of our proofs.

4.1. Information I about Size

Information I is prior knowledge of physical ‘size’. Our working definition of a ‘size’ ${\mathit{s}}_i$ is a physical quantity that we postulate to exist, such as a distance, volume, or energy, which is usually understood to be positive and continuous. A ‘size’ need not have extension and is not distinguished from a ‘magnitude’. Critically, since a single size can be quantified by various numerical representations, and size is a relative rather than absolute quantity, a specific size ${\mathit{s}}_i\in s$ is not uniquely associated with a specific number, and the space of possible sizes s is not a unique set of numbers. Information I concerns only prior knowledge of physical size ${\mathit{s}}_i\in s$, not its numerical representation (such as ${\mathit{𝖝}}_i\in x_i\subseteq \mathbb{R}+$).

We define I as the conjunction of three postulates.

• There exist a finite and countable number of distinct sizes ${\mathit{s}}_1,{\mathit{s}}_2,\dots {\mathit{s}}_n$.
• Each existing size ${\mathit{s}}_i$ is an element of a totally ordered space of possible sizes s.
• For each existing size ${\mathit{s}}_i$, there are larger and smaller possible sizes within s.

$$I\equiv \mathrm{Postulates}1-3\mathrm{are}\mathrm{true}.$$

(7)

Information I is so minimal that it could reasonably be called “the complete absence of information”. Being so minimal, I appears vague, but its consequence becomes more apparent once we consider its many corollaries of ignorance. These constrain our choice of numerical representation (Section 4.2) and provide the sufficient antecedents for our proofs (Section 4.4).

4.2. Information J about Numerical Measures of Size

We should not confuse our information I about physical sizes with our information J about our choice of numerical representation. J corresponds to our knowledge of numbers, which includes our chosen axioms. We will therefore use the term “size” for a physical attribute, and “variable” for a number we use to quantify it (we quantify size ${\mathit{s}}_i$ with variables ${\mathit{𝖃}}_i$ and ${\mathit{𝖝}}_i$). Our choice of numerical representation J cannot have any influence on the actual sizes, or our information I about them; thus probabilities $P\left(x_i\right)$ must be invariant to our choice of numerical representation (parameterization) (Section 5.1) (although the density $p\left(x_i\right)$ does depend on J).

Information I implies certain properties that are desirable if not necessary in our choice of J. Postulate 1 indicates that there must be an integer number of sizes (samples) that actually exist, $n\in \mathbb{N}$. Since no number is postulated, n is unknown, and its state space is countably infinite. For example, each size could be the distance between a pair of particles, and the number of pairs n is unknown. For each of these n sizes, we define below two corresponding numerical quantities (variables). Postulate 2 specifies no minimal distance between neighboring sizes in the ‘totally ordered space’ s; thus s is continuous. Postulate 3 specifies that, for any given size ${\mathit{s}}_i$, larger and smaller sizes are possible. Thus, no bound is known for space s, and the range of possibilities is infinite. We therefore choose to represent a physical size ${\mathit{s}}_i\in s$ with real numbered variables (${\mathit{𝖝}}_i\in x_i$ and ${\mathit{𝖃}}_i\in X_i$), with more positive numbers representing larger sizes. We emphasize that, although the true value of each variable is finite, that value is entirely unknown given prior information I and J; therefore its state space is uncountably infinite.

We choose two numerical representations, one measuring relative size as difference, and the other as ratio, related to one another by exponential and logarithmic functions.

$$\begin{array}{ccc}\hfill y/x& =& exp\left(Y\right)/exp\left(X_i\right)=exp(Y-X),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill Y-X& =& log\left(y\right)-log\left(x\right)=log(y/x).\hfill \end{array}$$

(9)

We define information J as the conjunction of these equations, together with the standard distance metric in one dimension.

$$\begin{array}{c}\hfill J\equiv \mathrm{Equations}\left(5\right),\left(8\right),\mathrm{and}\left(9\right).\end{array}$$

(10)

We therefore represent a single physical size ${\mathit{s}}_i$ with two numbers (variables), “scale” ${\mathit{𝖝}}_i\in R^{+}$ and “location” ${\mathit{𝖃}}_i\in R$, where ${\mathit{𝖝}}_i=e^{{\mathit{𝖃}}_i}$. In common practice, a “scale” ${\mathit{𝖝}}_i$ typically corresponds to a ratio (e.g., ${\mathit{𝖝}}_1=8$ m is the ratio $8/1$, by definition); therefore, a “location” corresponds to a log ratio, ${\mathit{𝖃}}_i=log\left({\mathit{𝖝}}_i\right)$. Because we are free to choose any parameterization, and only one is necessary for a proof, it is not essential that we justify the relationship between parameterizations. However, the exponential relationship arises naturally and is at least a convenient choice.5

Since we follow the objective Bayesian definition of probability, our variables are not “random variables” in the standard sense (see Reference [1], especially p. 500). This is in part just semantic, since “random” is ill-defined in standard use and promotes misunderstanding. The technical distinction is that the standard definition of a “random variable” assumes there to be a function (known or unknown) that “randomly samples” ${\mathit{𝖝}}_i$ from its state space $x_i$ following some “true probability distribution” (which could be determined by a physical process). We do not assume there to be any such function or true distribution. Information I and J leaves us entirely agnostic about any potential process that might have generated the physical size we quantify with ${\mathit{𝖝}}_i$. Such a process could just as well be deterministic as random, and even if we knew it to be one or the other, this alone would not alter the probabilities of concern here.

4.3. Information K about Dependence of Variables

Information K postulates that the variables are not independent but does not specify the exact form of dependence. The dependence between variables is typically specified as part of a statistical model. It relates to a measure of the state space in two or more dimensions. For example, the Euclidean distance metric, which exhibits rotational invariance, underlies what we call “Gaussian dependence”.

For independent variables ${𝒴}_1=log\left({𝓎}_1\right)$ and ${𝒴}_2=log\left({𝓎}_2\right)$, $p(Y_2\mid {𝒴}_1)=p\left(Y_2\right)$, and therefore our conclusion that $P({𝒴}_2<{𝒴}_1\mid {𝒴}_1)=P({𝒴}_2>{𝒴}_1\mid {𝒴}_1)$ is obvious (although since $p(Y_2\mid {𝒴}_1)$ is not normalizable, ${𝒴}_1$ is not a unique median). Given our measures of distance (Equation (5)) and probability (Equation (6)) in one dimension, the measure in this two-dimensional state space is simply the product

$$\begin{array}{c}\hfill P(Y_1,Y_2)=P\left(Y_1\right)P\left(Y_2\right)=Y_1{Y}_2,\end{array}$$

(11)

where $P(Y_1,Y_2)\equiv P({𝒴}_1<Y_1,{𝒴}_2<Y_2)$ is the joint CDF in our simplified notation (Equation (2)). Information K postulates a dependence, which we define by the negation of Equation (11):

$$\begin{array}{c}\hfill P(X_1,X_2)\ne P\left(X_1\right)P\left(X_2\right)=X_1{X}_2.\end{array}$$

(12)

We further define K to be any form of dependence (measure) consistent with the two corollaries specified below that result from the conjunction of information I, J, and K (Section 4.4). These corollaries mandate symmetries (Section 5) that defines a set $\mathbb{S}$ of joint distributions $p(X_1,X_2)$ and $p(x_1,x_2)$. Our proofs apply to every member of this set. They demonstrate that every member of the set has the form $P(X_1,X_2)=h(X_2-X_1)=h(X_1-X_2)$, where h is any function consistent with our corollaries. Figure 2 illustrates three exemplary distributions in set $\mathbb{S}$. In Section 7, we present in detail the special case of an “exponential” dependence or model.

Figure 2. Exemplary distributions for exponential (red), Gaussian (blue), and chi (cyan) dependencies. (A) Equiprobability contours of $p(X_1,X_2)$, which we prove are parallel to the identity line for all distributions in set $\mathbb{S}$ (Equation (29)). (B) Contours of $p(x_1,x_2)$. (C) Conditional distributions $p\left(X_2\right|{\mathit{𝖃}}_1)$. This is uniform in the case of independence (black). The distributions given exponential and Gaussian dependence are the standard logistic (Equation (25)) and Cauchy distributions, respectively, with location parameter ${\mathit{𝖝}}_1$ and scale parameter 1. (D) $p\left(x_2\right|{\mathit{𝖝}}_1)$. This is $1/x_2$ in the case of independence (black). The exponential dependence results in the log-logistic distribution with scale parameter ${\mathit{𝖝}}_1$ and shape parameter 1 (Equation (26)).

4.4. Corollaries of Ignorance Implied by I, J, and K

The conjunction of information I, J, and K is all prior knowledge, and its logical corollary is ignorance of all else. The consequence of ignorance is symmetry, which provides constraints that we use to derive probabilities.

Corollary. 

There is no information beyond that in I, J, and K.

1.

There is no information discriminating the location or scale of one variable from another.

2.

There is no information about the location or scale of any variable.

Corollaries 1 and 2 make explicit two sorts of ignorance that are necessary and sufficient for our proofs. Corollary 1 requires that we treat variables equally, meaning that they are “exchangeable” (Equation (14)). Corollary 2 is sufficient to establish location and scale invariance (Equations (15)–(18)) and to derive non-informative priors over single variables (Equations (19)–(22)).

5. Invariance of Prior Probabilities

Given the minimal information defined above, prior probabilities must be invariant to the choice of parameters, exchange of variables, and changes to location and scale [1,2,3,8,9,10,11,13,14,16,26,27,28,29].

5.1. Invariance to Change of Parameters

Probability must not vary with our choice of parameters (Section 4.2). Therefore,

$$P(A<{\mathit{𝖃}}_i<B)=P(a<{\mathit{𝖝}}_i<b),$$

(13)

where the left and right sides represent size as locations and scales, respectively, related by $(e^A,e^B,e^{X_i})=(a,b,x_i)$. These probabilities are equal because these are two distinct formulations of the same proposition (this invariance does not extend to densities $p\left(X_i\right)$ and $p\left(x_i\right)$, which have different domains, and thus different forms).

5.2. Exchangeability of Variables

Corollary 1 requires that we treat all variables equally. Therefore $p(x_1,x_2)$ must exhibit “exchangeability of variables,”

$$p({\mathit{𝖝}}_1=a,{\mathit{𝖝}}_2=b)=p({\mathit{𝖝}}_1=b,{\mathit{𝖝}}_2=a),$$

(14)

for all positive numbers a and b. Exchangeability corresponds to symmetry of $p(x_1,x_2)$ around the identity line, $x_1=x_2$ (Figure 1 and Figure 2B), and likewise for $p(X_1,X_2)$ (Figure 2A). Here, we show only two variables, but exchangeability applies to all n variables in the joint distribution $p(x_1,\dots ,x_n)$. Although n will be finite, it is unknown and unbounded (Section 4.1, postulate 1); we therefore consider the consequences of allowing n to approach infinity. As a result, we have “infinite exchangeability” of variables and can use de Finetti’s theorem, which ensures the existence of a set of parameters $\varphi$ that, if known, cause the variables to be independent and identically distributed (i.i.d.) (see Section 8.3) [13,16,26] (see Chapter 4.3.3 in Reference [16]).

5.3. Location and Scale Invariance

Since corollary 2 stipulates no prior information about size, the prior probability that ${\mathit{𝖃}}_i$ lies in an interval must be invariant to a change in the location of that interval (as in choosing a different origin). Therefore, we have location (translation) invariance

$$P(A<{\mathit{𝖃}}_i<B)=P(A+C<{\mathit{𝖃}}_i<B+C),$$

(15)

for any number $C\in \mathbb{R}$. We can express this instead as integrals over densities, ${\int }_A^{B}p\left(X_i\right)d{X}_i={\int }_{A+C}^{B+C}p\left(X_i\right)d{X}_i$, or

$$p\left(X_i\right)=p(X_i+C).$$

(16)

Because we must have invariance to a change of parameters (Equation (13)), we can rewrite Equation (15) given $x_i=e^{X_i}$, so that location invariance becomes scale invariance

$$P(a<{\mathit{𝖝}}_i<b)=P(ca<{\mathit{𝖝}}_i<cb).$$

(17)

Therefore ${\int }_a^{b}p\left(x_i\right)d{x}_i={\int }_{ca}^{cb}p\left(x_i\right)d{x}_i$, and scale invariance with respect to densities is

$$p\left(x_i\right)=cp\left(c{x}_i\right).$$

(18)

Location and scale invariance (Equations (15)–(18)) are sufficient for deriving these non-informative priors [1,2,3], although these priors and invariances can also be deduced from our axioms (Equations (5) and (6)) and choice of parameters (Equations (8) and (9)).

6. Prior Probabilities over Single Variables

6.1. The Prior over Locations

We express the non-informative prior over location (Equation (6)) more concisely as a CDF (following Equation (2)),

$$P\left(X_i\right)=X_i,$$

(19)

and its derivative, the uniform PDF

$$p\left(X_i\right)=1.$$

(20)

The uniform density is unique in exhibiting location (translation) invariance (Equation (16)), which provides an alternative derivation of it as the non-informative prior over real numbers [1,2,3]. Note that, since we derived the uniform prior from the standard distance metric (Equation (5)), it also applies to a positive variable, $p\left(\right|X_i\left|\right)=1$.

6.2. The Prior over Scales

Because the non-informative prior over scale is less obvious than the uniform prior, we provide three derivations of it. First, because probabilities must be invariant to a change in parameters (Equation (13)), we can find the prior over scales simply by transforming locations, $x_i=e_i^X$ (Equation (8)) (see Reference [14], p. 82). Therefore, $P(A<{\mathit{𝖃}}_i<B)=B-A$ (Equation (6)) becomes $P(a<{\mathit{𝖝}}_i<b)=logb-loga$, and the CDF is

$$P\left(x_i\right)=log\left(x_i\right),$$

(21)

and the corresponding PDF is $p\left(x_i\right)=dP\left(x_i\right)/d{x}_i$

$$p\left(x_i\right)=\frac{1}{x_i}.$$

(22)

A second and more famous derivation is based on the fact that $p\left(x_i\right)=c/x_i$, where $c>0$ is a constant, is the unique distribution that exhibits the scale invariance defined by Equation (18) [1,2,3]. The value of c is irrelevant to our proofs (Section 6.4), but $c=1$ is implied by Equations (19), (20), and Equation (8). Thus we have Equation (22), known as “Jeffreys prior over scale” (which applies whether the unknown is called a “scale parameter” or “variable”). A third derivation below considers a ratio of independent variables.

6.3. Independence and the Prior over Differences and Ratios

The priors over single variables derived above are sufficient for our proofs. However, we can gain additional insight by deriving the non-informative prior over difference and ratio. Complete absence of information requires that we do not have information K (Section 4.3) specifying a dependence, and instead have independent variables $({𝒴}_1,{𝒴}_2)\in (Y_1,Y_2)$.

Given the joint CDF (Equation (11)), the joint PDF is uniform, $p(Y_1,Y_2)=p\left(Y_1\right)p\left(Y_2\right)=1$; therefore uniformity applies to all sums and differences,

$$p\left(Y_1\right)=p\left(Y_2\right)=p(Y_1+Y_2)=p(Y_1-Y_2)=p(Y_2-Y_1)=1,$$

(23)

which are also independent of one another. Transforming locations to scales (Equations (8) and (13)), we have the analogous equation $p\left(y_1\right)=p\left(y_2\right)=p\left(y_1{y}_2\right)=p(y_1/y_2)=p(y_2/y_1)=1$, and the product $y_1{y}_2$ is independent of ratios $y_1/y_2$ and $y_2/y_1$. Thus non-informative priors over products and ratios, and sums and differences, have the same form as shown above for scales and locations, respectively. Indeed, we introduce no information about location or scale if we define $X_i\equiv Y_i-Y_{i+1}$ and $x_i\equiv y_i/y_{i+1}$, and all numbered equations remain valid if we do so.

6.4. The Uniform Prior over Positive Variables

Here, we derive Jeffreys non-informative prior over scale (Equations (21) and (22)) from the uniform prior over positive and independent real numbers. Although less formal than the derivation above (Section 6.2), this derivation may be more appealing conceptually to some since it does not rely on negative numbers and the exponential function.

We begin with uniformity and independence of positive variables, $p\left(\right|Y_1|,|Y_2\left|\right)=p\left(\right|Y_1\left|\right)p\left(\right|Y_2\left|\right)=1$. This gives $p\left(\right|Y_1\left|\right)=p\left(\right|Y_2\left|\right)=1$ (Equation (23)), and exchangeability (Equation (14)) requires $p\left(\right|Y_1|/|Y_2\left|\right)=p\left(\right|Y_2|/|Y_1\left|\right)$. Combining these, we have for the subset of propositions $\left(\right|Y_1|,|Y_2\left|\right)\in (1,2,3)$, $p(1/3)=p(1/2)=p(2/3)=p(1/1)=p(3/2)=p(2/1)=p(3/1)$. This is easily seen to correspond to uniform density over the logarithm, $p\left(log\right(|Y_1|/|Y_2\left|\right)=1$ (‘1’ is required by Equation (5), but any constant is sufficient for present purposes). By defining $z\equiv |Y_1|/|Y_2|$ and $Z\equiv log\left(z\right)$, this becomes $p\left(Z\right)=1$ (Equation (20)), which we have shown above to be sufficient to derive $P\left(Z\right)=Z$ (Equation (19)) and $p\left(z\right)=1/z$ (Equation (22)) (Section 6.2). Therefore Jeffreys’s prior over scale (Equation (22)) is the distribution over a ratio of two independent and uniformly distributed positive real numbers.

This derivation appears to have failed to account for the possible combinations of $|Y_1|$ and $|Y_2|$. For example, it would appear that a ratio ‘1’ could be achieved by any element of the set ‘1/1’, ‘2/2’, ‘3/3’,..., whereas a more extreme ratio, such as ‘1/527’, can be achieved only by a smaller set of combinations. However, the apparent multiplicity of combinations (degeneracy) is illusory here, a counter-intuitive property of independence and a complete absence of information. For example, rather that representing a distance between two geometric points with a single number $|{\mathit{𝖃}}_1|$, we could represent it by assigning two arbitrary numbers ${𝒴}_1$ and ${𝒴}_2$ to the two points, so that $|{\mathit{𝖃}}_1|=|{𝒴}_1-{𝒴}_2|$. The question then arises whether combinations of ‘${𝒴}_1,{𝒴}_2$’, such as ‘$1,1$’ and ‘$2,2$’, are distinguishable (degenerate) states, or indistinguishable states (Bose-Einstein statistics being a consequence of the latter). If one postulates that a single distance is associated with a uniquely correct number, but a single geometric point can only be assigned a number arbitrarily (as typically implied in trigonometry, in which similar triangles of different scales are distinguished, but identical triangles in different locations are not), then ‘$1,1$’ and ‘$2,2$’ are not distinguishable states, and there is only one way to achieve one distance $|{\mathit{𝖃}}_1|=|{𝒴}_1-{𝒴}_2|$. Thus, the state space ‘$Y_1,Y_2$’, like that of $X_1$, is effectively one-dimensional. Choosing to parameterize with ‘$Y_1,Y_2$’ creates what could be called “pseudodegeneracy”, since we cannot expand the size or dimensionality of a state space by choosing to introduce a superfluous and potentially misleading variable. This contrasts with our case of dependent variables ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2$, for which we treat ‘$1,1$’ and ‘$2,2$’ as two distinct combinations.

7. Exemplary Case of an Exponential Dependence

Before general proofs, we present here examples of dependence, and we summarize each of our three proofs for the special case of an “exponential” dependence or model.

7.1. Three Exemplary Forms of Dependence

Figure 2 shows distributions for three forms of dependence in set $\mathbb{S}$ (Section 4.3). The joint distribution over two variables given an exponential dependence is

$$p(x_1,x_2)=\frac{1}{{(x_1+x_2)}^2},$$

(24)

whereas “Gaussian” and “chi” dependence yields $p(x_1,x_2)={(x_1^2+x_2^2)}^{-1}$ and $p(x_1,x_2)=\left(x_1{x}_2\right){(x_1^2+x_2^2)}^{-2}$, respectively. Transformation of Equation (24) according to $x_i=e^{X_i}$ (using Equation (13) after converting from PDF to CDF) gives

$$\begin{array}{c}\hfill p(X_1,X_2)=\frac{e^{X_1-X_2}}{{(1+e^{X_1-X_2})}^2}.\end{array}$$

(25)

The sufficient statistic for the exponential model given samples ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2$ is $({\mathit{𝖝}}_1+{\mathit{𝖝}}_2)/2$, and the scale parameter given an infinite “population” of samples is $\lambda ={lim}_{n\to \infty }$$({\mathit{𝖝}}_1+\dots +{\mathit{𝖝}}_{\mathrm{n}}$$)/n$. If $\lambda$ is known, the sampling distribution is “the exponential distribution”, $p\left(x_i\right|\lambda )=(1/\lambda )exp(-x_i/\lambda )$. Therefore, we refer to this dependence (or statistical model) as ‘exponential’, even though $\lambda$ is unknown in our case (we have a non-informative prior over $\lambda$). We apply the same rationale in denoting our “Gaussian” and “chi” dependencies.

7.2. Proofs 1 and 2 Given Exponential Dependence

If ${\mathit{𝖝}}_1$ is known, Equation (24) becomes $p\left(x_2\right|{\mathit{𝖝}}_1)={\mathit{𝖝}}_1/{({\mathit{𝖝}}_1+x_2)}^2$ (given the product rule $p\left(x_2\right|{\mathit{𝖝}}_1)=p({\mathit{𝖝}}_1,x_2)/p\left({\mathit{𝖝}}_1\right)$, and $p\left({\mathit{𝖝}}_1\right)=1/{\mathit{𝖝}}_1$). Its integral is the CDF $P\left(x_2\right|{\mathit{𝖝}}_1)=x_2/({\mathit{𝖝}}_1+x_2)=1/(1+{\mathit{𝖝}}_1/x_2)$, proving that ${\mathit{𝖝}}_1$ is the median (Equation (1)). This distribution is known as‘log-logistic’ with scale parameter ${\mathit{𝖝}}_1$ and shape parameter 1. However, proofs 1 and 2 take advantage of the greater symmetry present in $P(X_1,X_2)$ compared to $p(x_1,x_2)$ (although proof 2 begins with the latter). Logarithmic transformation converts the log-logistic distribution $p\left(x_2\right|{\mathit{𝖝}}_1)$ to the logistic distribution $p\left(X_2\right|{\mathit{𝖃}}_1)$ (Equation (25)), with location parameter ${\mathit{𝖃}}_1=log\left({\mathit{𝖝}}_1\right)$ and scale parameter 1. It can readily be shown that $p(X_2-{\mathit{𝖃}}_1|{\mathit{𝖃}}_1)=p({\mathit{𝖃}}_1-X_2|{\mathit{𝖃}}_1)$ (Figure 2A,C), indicating that the logistic distribution is an even function, $f(-t)=f\left(t\right)$. Therefore, ${\mathit{𝖃}}_1$ is its median, as well as mean and mode. Since the median is invariant to a change of parameters (Equation (13)), ${\mathit{𝖝}}_1=e^{{\mathit{𝖃}}_1}$ is the median of the log-logistic distribution.

7.3. Proof 3 Given Exponential Dependence

Proof 3 uses the “population” scale parameter $\lambda ={lim}_{n\to \infty }(x_1+\dots +x_n)/n$, for which the sampling distribution is the exponential distribution, $p\left(x_i\right|\lambda )=(1/\lambda )exp(-x_i|\lambda )$. From this and the prior over a scale parameter $p\left(\lambda \right)=1/\lambda$ (Section 6.2), the posterior predictive distribution is

$$\begin{array}{c}\hfill p(x_2\mid {\mathit{𝖝}}_1)={\int }_0^{\infty }\frac{p(x_2\mid \lambda )p({\mathit{𝖝}}_1\mid \lambda )p\left(\lambda \right)}{p\left({\mathit{𝖝}}_1\right)}d\lambda =\frac{{\mathit{𝖝}}_1}{{({\mathit{𝖝}}_1+x_2)}^2},\end{array}$$

(26)

which is again the log-logistic distribution (Section 7.2) (Figure 2B,D). The median is ${\mathit{𝖝}}_1$, since ${\int }_0^{{\mathit{𝖝}}_1}{\mathit{𝖝}}_1/{({\mathit{𝖝}}_1+x_2)}^{2}d{x}_2=1/2$. One can use the same approach to confirm that this is also true given any particular statistical model of positive real variables that is famous enough to have a name (e.g., gamma, chi, half-normal), so long as there is no prior information about scale (as in the case that $\lambda =1)$.

8. General Proof

Proofs 1–3 all proceed from corollary 1, expressed in exchangeability (Equation (14)), and corollary 2, expressed in location and scale invariance (Equations (15)–(18)), and prior probabilities (Equations (6) and (19)–(22)). Proof 1 starts by observing that $p(X_2-X_1)$ must be an even function. Proof 2 applies scale invariance across all dimensions of $p(x_1,x_2,x_3,\dots )$. Proof 3 uses a scale parameter that renders the variables conditionally independent and identically distributed (i.i.d.). Proof 1 is the simplest and proof 3 the most complex. It is interesting to note that we derived them in the opposite order chronologically, which is explained primarily by the fact that proof 3 uses the most common Bayesian methods.

We also note that proofs 2 and 3 begin with strictly positive numbers $x_i>0$, and proof 3 never uses negative numbers. Furthermore, whereas we chose to present our axioms using the standard distance metric for all real numbers (locations), scale invariance (Equation (18)) is an alternative antecedent for our proofs. It can be understood as the consequence of beginning with a multiplicative metric, rather than the standard metric based on difference.

8.1. Proof 1 from the Joint Distribution $p(X_1,X_2)$

To characterize $p(X_1,X_2)$, it is useful to consider sums and differences, $X_1+X_2=A$ and $X_1-X_2=B$ (note that $p(A,B)$ corresponds to rotating $p(X_1,X_2)$ by 90 degrees). Given exchangeability (Equation (14)) from corollary 1, $p(X_1,X_2)$ must be evenly distributed around the identity line for all forms of dependence (statistical models) in set $\mathbb{S}$, so that it is invariant to exchange of $X_1$ and $X_2$. Therefore,

$$\begin{array}{c}\hfill p(X_1-X_2)=p(X_2-X_1),\end{array}$$

(27)

which must be an even function, $f(-t)=f\left(t\right)$.

Given corollary 2, we know nothing about the location of either $X_1$ or $X_2$, and knowing that they exhibit a dependence contributes no information about their joint location (Section 4.3). Therefore, we know nothing about their sum, resulting in uniformity,

$$\begin{array}{c}\hfill p(X_1+X_2)=1,\end{array}$$

(28)

so that all lines through $p(X_1,X_2)$ perpendicular to the identity line $X_1=X_2$ are equally probable, $p(X_1+X_2=C)=p(X_1+X_2=D)$. Uniformity over sums (Equation (28)) is easily seen to be a consequence of location invariance (Equation (16)), since $p\left(X_1\right)=1$ (Equation (20)), and we must have invariance of the distribution whether we add to $X_1$ either an arbitrary number C or location $X_2$.

The combination of Equations (27) and (28) requires that $p(X_1,X_2)$ must be a function only of the difference between variables for every statistical dependence (model) in set $\mathbb{S}$,

$$\begin{array}{c}\hfill p(X_1,X_2)=f(X_1-X_2)=f(X_2-X_1),\end{array}$$

(29)

meaning that equiprobability contours must be parallel to the identity line, as shown in Figure 2A.

Given a known sample ${\mathit{𝖃}}_1\in X_1$, the conditional distribution $p\left(X_2\right|{\mathit{𝖃}}_1)$ is a slice through $p(X_1,X_2)$ perpendicular to the $X_1$ axis. It can be seen in Figure 2A that the distribution over the difference is identical for all values of ${\mathit{𝖃}}_1$.

$$\begin{array}{c}\hfill p(X_2-{\mathit{𝖃}}_1\mid {\mathit{𝖃}}_1)=p(X_2-{\mathit{𝖃}}_1).\end{array}$$

(30)

This proves that ${\mathit{𝖃}}_1$ and ${\mathit{𝖃}}_2-{\mathit{𝖃}}_1$ have no information about one another and are therefore independent. From this and Equation (27), $p\left(X_2\right|{\mathit{𝖃}}_1)$ must be an even function $p(X_2-{\mathit{𝖃}}_1|{\mathit{𝖃}}_1)=p({\mathit{𝖃}}_1-X_2|{\mathit{𝖃}}_1)$ with median ${\mathit{𝖃}}_1$ (Figure 2C), proving our conclusion

$$\begin{array}{c}\hfill P(X_2<{\mathit{𝖃}}_1\mid {\mathit{𝖃}}_1)=P(X_2>{\mathit{𝖃}}_1\mid {\mathit{𝖃}}_1).\end{array}$$

(31)

Converting to scales (Equations (8) and (13)) shows that ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2/{\mathit{𝖝}}_1$ must also be independent, and proves our conclusion that a known positive variable is the median over an unknown.

$$\begin{array}{c}\hfill P(x_2<{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1)=P(x_2>{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1).\end{array}$$

(32)

The conclusion in our title, that these probabilities equal $1/2$, will be true only if $p(x_2/{\mathit{𝖝}}_1\mid {\mathit{𝖝}}_1)$ has a finite integral (is normalizable). All forms of dependence of which we are aware do result in normalizable conditional distributions, but we do not rule out the possibility of a dependence in set $\mathbb{S}$ that is an exception. If there is such a case, equations Equations (31) and (32) remain valid, but there will be no unique median.

8.2. Proof 2 from the Joint Distribution $p(x_1,x_2,\dots ,x_n)$

The prior over scales (Equation (22)) can be derived as the unique distribution exhibiting invariance following scale transformation (Equation (18)), exemplifying the method of “transformation groups” [1,8,9,10,11,27,28,29]. Here we transform the scale of all variables simultaneously

$$p(x_1,\dots ,x_n)d{x}_1,\dots ,d{x}_n=p(x_1^{\prime },\dots ,x_n^{\prime })d{x}_1^{\prime },\dots ,d{x}_n^{\prime },$$

(33)

where $x_i^{\prime }=c{x}_i,0<c<\infty$. This requires that the joint distribution exhibit a symmetry that could be termed “multidimensional scale invariance”, so that $p\left(x_1\right)={\int }_0^{\infty }d{x}_2\dots {\int }_0^{\infty }d{x}_{n}p(x_1,\dots ,x_n)=1/x_1$ (where subscript ‘1’ is exchangeable with any other subscript). It is also sufficient to ensure that the joint distribution is a homogeneous function, so that multiplying each variable by c is equivalent to dividing the joint distribution by $c^n$,

$$p(x_1,\dots ,x_n)=c^{n}p(c{x}_1,\dots ,c{x}_n).$$

(34)

We define the set $\mathbb{S}$ of joint distributions to be those that satisfy both this multidimensional scale invariance (Equation (34)) and exchangeability (Equation (14)). We can find a general form for the set by first applying Euler’s homogeneous function theorem to Equation (34). In the case of two variables, this gives

$$2p(x_1,x_2)+\sum _{i=1}^2{x}_i\frac{\partial p(x_1,x_2)}{\partial x_i}=0,$$

(35)

and the general solution is $p(x_1,x_2)=(1/x_1^2)h_1(x_2/x_1)=(1/x_2^2)h_2(x_1/x_2)$, where $h_1$ and $h_2$ are two arbitrary functions. Whereas this equation was deduced solely from scale invariance (Equation (34)) (required by corollary 2), we next apply exchangeability of variables (Equation (14)) (required by corollary 1). This shows that $(1/x_1^2)h_1(x_2/x_1)=(1/x_2^2)h_1(x_1/x_2)$, and likewise for $h_2$, so that $h_1\left(x\right)=h_2\left(x\right)$. Thus there is only one arbitrary function, $h\left(x\right)\equiv h_1\left(x\right)=h_2\left(x\right)$, and the general solution for two variables becomes

$$p(x_1,x_2)=\frac{1}{x_1^2}h\left(\frac{x_2}{x_1}\right)=\frac{1}{x_2^2}h\left(\frac{x_1}{x_2}\right),$$

(36)

where $h\left(x\right)$ is any arbitrary function that satisfies this equation. We can find the general form of $p\left(x_2\right|{\mathit{𝖝}}_1)$ by dividing Equation (36) by $p\left(x_1\right)=1/x_1$ (Equation (22)), which yields

$$p(x_2\mid {\mathit{𝖝}}_1)d{x}_2=\frac{1}{{\mathit{𝖝}}_1}h\left(\frac{x_2}{{\mathit{𝖝}}_1}\right)d{x}_2=\frac{{\mathit{𝖝}}_1}{x_2^2}h\left(\frac{{\mathit{𝖝}}_1}{x_2}\right)d{x}_2.$$

(37)

Following logarithmic transformation $X_i=log{x}_i$, this becomes

$$p(X_2\mid {\mathit{𝖃}}_1)d{X}_2=\frac{e^{X_2}}{e^{{\mathit{𝖃}}_1}}h\left(\frac{e^{X_2}}{e^{{\mathit{𝖃}}_1}}\right)d{X}_2=\frac{e^{{\mathit{𝖃}}_1}}{e^{X_2}}h\left(\frac{e^{{\mathit{𝖃}}_1}}{e^{X_2}}\right)d{X}_2.$$

(38)

Since $e^{X_1}/e^{X_2}=e^{X_2-X_1}$, we can substitute $T=X_2-X_1$ to give $e^{T}h\left(e^T\right)=e^{-T}h\left(e^{-T}\right)$. This satisfies the definition of an even function, proving that ${\mathit{𝖃}}_1$ is the median of $p\left(X_2\right|{\mathit{𝖃}}_1)$. Since the median is invariant to the transformation $x_i=e^{X_i}$, proof 2 is completed in the same manner as proof 1 (Equations (31) and (32)).

8.3. Proof 3 from a Scale Parameter

Because of corollary 1, we have exchangeability (Equation (14)), and the theorem of de Finetti ensures the existence of a set of parameters $\varphi$ that, if known, renders the variables independent and identically distributed (i.i.d.) (Section 5.2).

$$\begin{array}{c}\hfill p(x_1,x_2)={\int }_{\Phi }p\left(x_1\right|\varphi )p\left(x_2\right|\varphi )p\left(\varphi \right)d\varphi .\end{array}$$

(39)

This set will include a scale parameter6 $\lambda$, and the non-informative prior will be $p\left(\lambda \right)=1/\lambda$ (Section 6.2). We denote other parameters in set $\varphi$ as $\theta$, so that $\varphi =(\theta ,\lambda )$. The posterior predictive distribution can be expressed as

$$\begin{array}{c}\hfill p(x_2\mid {\mathit{𝖝}}_1)=\frac{1}{p\left({\mathit{𝖝}}_1\right)}{\int }_{\Theta }d\theta {\int }_0^{\infty }d\lambda p({\mathit{𝖝}}_1\mid \theta ,\lambda )p(x_2\mid \theta ,\lambda )p(\theta \mid \lambda )p\left(\lambda \right).\end{array}$$

(40)

The general form of $p\left(x_i\right|\theta ,\lambda )$, which can be found using the method of transformation groups [1], is

$$\begin{array}{c}\hfill p(x_i\mid \theta ,\lambda )=\frac{1}{\lambda }{h}_{\theta }\left(\frac{x_i}{\lambda }\right),\end{array}$$

(41)

where $h_{\theta }\left(x\right)$ is an arbitrary normalizable function, the specific form of which depends on the types and values of parameters in $\theta$ (for example, the exponential distribution $h_{\theta }\left(t\right)=e^{-t}$ or gamma distribution $h_{\theta }\left(t\right)=t^{k-1}{e}^{-t}/\Gamma \left(k\right)$).

The remaining terms in Equation (40) are $p\left(\theta \right|\lambda )$ and $p\left({\mathit{𝖝}}_1\right)$. As a scale parameter, $\lambda$ in Equation (41) determines only the width of the distribution, and is not informative with respect to its shape, which is determined by $\theta$. Thus, $\lambda$ and $\theta$ are independent such that $p\left(\theta \right|\lambda )=p(\theta )$. We need not assume anything about the form of $p\left(\theta \right)$, except that it normalizes so that ${\int }_{\Theta }d\theta p\left(\theta \right)=1$.7

All terms in Equation (40) are now determined. Using the fact that ${\int }_0^{{\mathit{𝖝}}_1}d{x}_{2}p(x_2\mid \theta ,\lambda )=H_{\theta }({\mathit{𝖝}}_1/\lambda )$, where $H_{\theta }\left(x\right)$ is the cumulative distribution of $h_{\theta }\left(x\right)$, and by substituting $t={\mathit{𝖝}}_1/\lambda$, the median of the predictive distribution is

$$\begin{array}{cc}\hfill {\int }_0^{{\mathit{𝖝}}_1}d{x}_{2}p(x_2\mid {\mathit{𝖝}}_1)& ={\int }_{\Theta }d\theta p\left(\theta \right){\int }_0^{\infty }d\lambda \frac{{\mathit{𝖝}}_1}{{\lambda }^2}{h}_{\theta }\left(\frac{{\mathit{𝖝}}_1}{\lambda }\right)H_{\theta }\left(\frac{{\mathit{𝖝}}_1}{\lambda }\right),\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill & ={\int }_{\Theta }d\theta p\left(\theta \right){\int }_0^{\infty }dt{h}_{\theta }\left(t\right)H_{\theta }\left(t\right).\hfill \end{array}$$

(43)

Since $h_{\theta }\left(t\right)H_{\theta }\left(t\right)=(d/dt)(H_{\theta }{\left(t\right)}^2/2)$, it can be shown that ${\int }_0^{\infty }dt{h}_{\theta }\left(t\right)H_{\theta }\left(t\right)=1/2$. The same procedure can be used to demonstrate ${\int }_{{\mathit{𝖝}}_1}^{\infty }d{x}_{2}p\left(x_2\right|{\mathit{𝖝}}_1)=(1/2){\int }_{\Theta }d\theta p\left(\theta \right)$. This proves our conclusion

$${\int }_0^{{\mathit{𝖝}}_1}d{x}_{2}p(x_2\mid {\mathit{𝖝}}_1)={\int }_{{\mathit{𝖝}}_1}^{\infty }d{x}_{2}p(x_2\mid {\mathit{𝖝}}_1)=\frac{1}{2},$$

(44)

where ‘1/2’ applies because we have assumed in proof 3 that $p\left(x_2\right|{\mathit{𝖝}}_1)$ is normalized (see Section 8.1).

9. The Effect of Additional Information

A single known number is the median over an unknown only as a result of ignorance (corollaries 1 and 2), and therefore additional information can result in loss of this symmetry. For example, $\omega$ is not the median of $p(x_{m+1}\mid \omega )$ if $\omega ={\mathit{𝖝}}_1+{\mathit{𝖝}}_2+\dots +{\mathit{𝖝}}_m$ and $m > 1$. This is because $\omega$ and $x_{m+1}$ are not exchangeable in the joint distribution $p(\omega ,x_{m+1})$. An interesting special case is that m variables are known and are equal to one another, $p(x_{m+1}\mid {\mathit{𝖝}}_1={\mathit{𝖝}}_2=\dots {\mathit{𝖝}}_m)$. One might expect that the median will be the one known number, but this is only true if $m=1$, at least for our exemplary forms of dependence. Scale information increases with m, causing the distribution to skew towards smaller numbers (Figure 3A). If ${\mathit{𝖝}}_1={\mathit{𝖝}}_2=\dots {\mathit{𝖝}}_m$$=1$, the median converges to a number between 0 and 1 that is determined by the form of dependence (Figure 3B). Given the exponential dependence, the median is $m(2^{-m}-1)$, which is ≈ 0.828 for $m=2$, and converges to $ln2\approx 0.69$ as m goes to infinity.

Figure 3. A known number is not the median if there is additional information about location or scale. We illustrate the special case that there are m knowns, all equal to one another. (A) The density over locations $p(X_{m+1}\mid {\mathit{𝖃}}_1=,{\mathit{𝖃}}_2=\dots ,{\mathit{𝖃}}_m=0)$ for m of $1,2,$ and ∞, given the exponential dependence. Vertical lines indicate medians. We show distributions over location rather than scale so that the asymmetry is easily seen. (B) The median scale ${\mathit{𝖝}}_1=e^{{\mathit{𝖃}}_1}$ decreases as m increases, for exponential (red), Gaussian (blue), and chi (cyan) dependencies.

There are forms of additional information that allow a single known to remain the median. For example, additional information could specify that ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2$ are each the mean or sum of m variables.8 We still must treat ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2$ equally, and we still know nothing about their scale; therefore, ${\mathit{𝖝}}_1$ is still the median of $p\left(x_2\right|{\mathit{𝖝}}_1)$. This remains true whether the number of summed variables m is itself known or unknown.

10. Discussion

Our proof advances the objective Bayesian foundation of probability theory, which seeks to quantify information (knowledge or evidence) with the same rigor that has been attained in other branches of applied mathematics. The ideal is to start from axioms, derive non-informative prior probabilities, and to then deduce probabilities conditional on any additional evidence. A probability should measure evidence just as objectively as numbers measure physical quantities (distance, energy, etc.).

The only aspect of our proofs that has been controversial is the issue of whether absence of information uniquely determines a prior distribution over real numbers. Formal criteria, including indifference, location invariance, and maximum entropy, have all been used to derive the uniform density $p\left(X_i\right)=c$, where $c>0$ (Section 6). Subjectivists view c as indeterminate, thus being a source of subjectivity (see Reference [14], pp. 83–87), whereas objectivists view it merely as a positive constant to be specified by convention (our proofs hold for any choice of c). In contrast, we deduced $p\left(X_i\right)=1$ from the standard distance metric, which we declared an axiom (Equation (5)) (Section 3). In our opinion, the indeterminacy of the subjectivist view is the logical result of insufficient axioms.

Our conclusion appears surprising, since for any known and finite positive number, the space of larger numbers is infinitely larger than the space of smaller positive numbers (Figure 1). However, probability measures evidence, not its absence, and the space of possibilities only exists as the consequence of the absence of evidence (ignorance). A physical ‘size’ is evidence about other physical sizes, and we postulate that physical size and evidence exist (Section 4.1). This postulate implies that the absence of evidence (ignorance), and the corresponding state space, does not exist (or at least has a secondary and contingent status) and therefore carries no weight in reasoning. Reason requires that two physical sizes be treated equally (Section 4.4), and there is no evidence that one is larger than the other, even if one is known. In this way our proof that a known number ${\mathit{𝖝}}_1$ (quantifying size) is the median over an unknown ${\mathit{𝖝}}_2$ can be understood intuitively. Our result can also be understood by considering that there is no meaningful (non-arbitrary) number associated with a single “absolute” size (Section 3.2), but only with ratios of sizes, and we must have $p({\mathit{𝖝}}_2/{\mathit{𝖝}}_1)=p({\mathit{𝖝}}_1/{\mathit{𝖝}}_2)$ (Section 6.4), regardless of whether ${\mathit{𝖝}}_1$ is known. Indeed, we have proven that ${\mathit{𝖝}}_1$ has no information about ${\mathit{𝖝}}_2/{\mathit{𝖝}}_1$ or ${\mathit{𝖝}}_1/{\mathit{𝖝}}_2$, and vice versa (Section 8.1), even though ${\mathit{𝖝}}_1$ and ${\mathit{𝖝}}_2$ are dependent variables and thus have information about one another.

Practical application of our result requires us to consider what information we actually have. The probability that the unknown distance to star B is greater than the known distance to star A is $1/2$, and the probability is also $1/2$ that the unknown energy of particle B is greater than the known energy of particle A. However, this is only true in the absence of additional information. We do realistically have additional information that is implicit in the terms ‘star’ and ‘particle,’ since we know that the distance to a star is large, and the mass of a particle is small, relative to the human scale. Before inquiring about stars or particles, we already know the length and mass of a human, and that additional information changes the problem. Furthermore, we know the human scale with more certainty than we can ever know a much larger or smaller scale.

All scientific measurement and inference of size begins with knowledge of the human scale that is formalized in our standard units (m, kg, s, etc.). These standard units are only slightly arbitrary, since they were chosen from the infinite space of possible sizes to be extremely close to the human scale. Therefore members of the set of distributions that we have characterized here, each conditional on one known size, would be appropriate as prior distributions given only knowledge of standard units. One meter would then be the median over an unknown distance ${\mathit{𝖝}}_2$. If ${\mathit{𝖝}}_2$ is then observed, and we ask “what is another distance ${\mathit{𝖝}}_3$?”, we already know two distances, and neither will be the median (our central conclusion does not apply) (Figure 3) (Section 9).

A beautiful aspect of Bayesian probabilities is that they can objectively describe any knowledge of any observer, at least in principle. Probabilities are typically used to describe the knowledge of scientists, but they can also be used to describe the information in a cognitive or physical model of an observer. This is the basis of Bayesian accounts of biology and psychology that view an organism as a collection of observers that predict and cause the future [21,30,31,32,33,34,35,36,37,38,39,40].

All brains must solve inference problems that are fundamentally the same as those facing scientists. When humans must estimate the distance to a visual object in a minimally informative sensory context, such as a light source in the night sky, they perceive the most likely distance to be approximately one meter, and bias towards one meter also influences perception in more complex and typical sensory environments [41,42]. This is elegantly explained by the fact that the most frequently observed distance to visual objects in a natural human environment is about a meter [31]. The brain is believed to have learned the human scale from experience, and to integrate this prior information with incoming sensory ‘data’ to infer distances, in accord with Bayesian principles [30,31,32]. Therefore, a simple cognitive model, in which the only prior information of a human observer is a single distance of $1$ m, or an energy of $1$ joule (kg m² s⁻²), might account reasonably well for average psychophysical data under typical conditions.

Such a cognitive model is extremely simplified, since the knowledge in a brain is diverse and changes dynamically over time. However, we do not think it is overly simplistic for a model of a physical observer to be just a single quantity at a single moment in time. The brain is a multitude of local physical observers, but our initial concern is just a single observer at a single moment [21,22,33,34,35]. The problem is greatly simplified by the fact that we equate an observer with information, and information with a physical quantity that is local in space and time, and is described by a single number. For example, a single neuron at a single moment has a single energy in the electrical potential across its membrane.9 Given this known and present internal energy, what is the probability distribution over the past external energy that caused it, or the future energy that will be its effect? Here, we have provided a partial answer by characterizing a set of candidate distributions, and demonstrating that a known energy will be the median over an unknown energy.