Relative Belief Inferences from Decision Theory

Abstract

Relative belief inferences are shown to arise as Bayes rules or limiting Bayes rules. These inferences are invariant under reparameterizations and possess a number of optimal properties. In particular, relative belief inferences are based on a direct measure of statistical evidence.

1. Introduction

We consider a sampling model for data x, as given by a collection of densities $\left\{f\right(·\left|\theta \right):\theta \in \Theta \}$ with respect to a support measure $\mu$ on a sample space $\mathcal{X},$ and a proper prior, given by density $\pi$ with respect to support measure $\nu$ on $\Theta .$ When the data $x\in \mathcal{X}$ are observed, these ingredients lead to the posterior distribution on $\Theta$ with density given by $\pi \left(\theta \right|x)=\pi (\theta \left)f\right(x\left|\theta \right)/m\left(x\right)$ with respect to support measure $\nu ,$ where

$$m\left(x\right)={\int }_{\Theta }\pi \left(\theta \right)f\left(x\right|\theta )\nu \left(d\theta \right)$$

is the prior predictive density of the data. In addition, there is a quantity of interest $\psi =\Psi \left(\theta \right),$ where

$$\Psi :\Theta \to \Psi (\Theta )=\{\psi :\psi =\Psi (\theta )\mathrm{for}\mathrm{some}\theta \in \Theta \},$$

for which inferences, such as an estimate $\psi \left(x\right)$ or a hypothesis assessment $H_0:\Psi \left(\theta \right)={\psi }_0,$ are required. Let ${\pi }_{\Psi }$ denote the marginal prior density of $\psi$ and

$$m\left(x\right|\psi )={\int }_{\Theta }f\left(x\right|\theta )\Pi \left(d\theta \right|\psi )$$

be the conditional prior predictive of the data after integrating out the nuisance parameters via the prior conditional distribution of $\theta$ given $\Psi \left(\theta \right)=\psi .$ Bayesian inferences for $\psi$ are then based on the ingredients $(\{m(·|\psi ):\psi \in \Psi (\Theta )\},{\pi }_{\Psi },x)$ alone or by adding a loss function $L.$ Note that the probability measure associated with m will be denoted by M and the probability measure associated with $m(·|\psi )$ will be denoted by $M(·|\psi )$ when these are used in the paper.

A natural question arises, namely, how are we to determine the inferences for $\psi$, namely, an estimate $\psi \left(x\right)$, or assess the hypothesis $H_0:\Psi \left(\theta \right)={\psi }_0$ based upon these ingredients? Several approaches have been put forward to answering this question. Two broad categories can be described, namely, the evidential/inferential approach and the behavioristic/decision-theoretic approach.

The evidential approach can be characterized as having the goal of letting the evidence in the data x determine the inferences and can be subdivided into frequentist, pure likelihood, and Bayesian theories. Central to this is the need to somehow characterize the concept of statistical evidence. The frequentist theory only uses the ingredients $\left(\right\{f(·|\theta ):\theta \in \Theta \},x)$ together with the idea that inferences are to be graded based on their behavior in hypothetical repeated sampling experiments. Despite the impressive accomplishments of Alan Birnbaum in attempting to formulate a definition of statistical evidence, (see [1]), it is fair to say that there is still no such generally acceptable definition within the frequentist context. The pure likelihood theory is also based on the ingredients $\left(\right\{f(·|\theta ):\theta \in \Theta \},x)$ but the idea of using repeated sampling characteristics to determine the inferences is dropped and the likelihood function $L(·|x):\Theta \to [0,\infty ),$ as defined by $L\left(\theta \right|x)=cf(x\left|\theta \right)$ for any positive constant $c,$ is taken to be the proper characterization of statistical evidence. All inferences are then determined by the likelihood; for example, see the discussion in [2]. Again, there are gaps in this treatment, as it is unclear when the likelihood function provides evidence in favor of or against a particular value of $\theta$ being the true value, and it is unclear how the likelihood is to be used for marginal parameters $\psi =\Psi \left(\theta \right).$ The Bayesian approach based on the ingredients $(\{m(·|\psi ):\psi \in \Psi (\Theta )\},{\pi }_{\Psi },x)$ is more successful at characterizing statistical evidence concerning $\psi$ through the principle of evidence, which, loosely speaking, suggests that if the data lead to the posterior probability of an event being greater than (less than) the corresponding prior probability, then there is evidence in favor of (against) the event being true. A precise statement of the principle of evidence is provided in Section 2.3. A full theory of inference based on this idea, and called relative belief, has been developed over a number of years (see [3]), and is sketched in Section 2.3. A much fuller discussion of the issues and developments within the context of the evidential approach to developing statistical theory can be found in [4].

The decision-theoretic approach can also be divided into frequentist and Bayesian theories. The frequentist approach is based on the ingredients $\left(\right\{f(·|\theta ):\theta \in \Theta \},x)$ together with a loss function $L:\Theta \times \Psi (\Theta )\to [0,\infty )$ where $L(\theta ,\Psi (\theta \left)\right)=0$ for all $\theta$ and, generally, $L(\theta ,\psi )$ represents the loss or penalty incurred when $\psi$ is chosen as the true value of $\Psi \left(\theta \right).$ The idea then is to look for a decision procedure $\delta \left(x\right)$ that performs well with respect to the average loss or risk $R(\theta ,\delta )={\int }_{\mathcal{X}}L(\theta ,\delta \left(x\right))f_{\theta }\left(x\right)\mu \left(dx\right),$ namely, choose a $\delta$ that makes $R(\theta ,\delta )$ small uniformly in $\theta .$ The frequentist decision theory, however, is not always successful in determining a suitable $\delta .$ The Bayesian theory of decision considers the prior risk $r\left(\delta \right)={\int }_{\Theta }R(\theta ,\delta )\pi \left(\theta \right)\nu \left(d\theta \right)$ and is generally successful in determining a $\delta$ that minimizes $r\left(\delta \right)$; this is referred to as the Bayes rule. The Bayesian theory of decision has been axiomatized (see [5]); this provides considerable support for this approach.

If the various approaches to determining inferences all lead to more or less the same answers, then there would be little controversy, but unfortunately, this is not the case. The goal of this paper is to show that relative belief inferences can also be developed within the context of decision theory even though their primary motivation is through the characterization of statistical evidence. It is of historical relevance and interest that two of the founders of the statistical discipline, Fisher and Neyman, disagreed profoundly on the purpose of statistical analyses. Fisher saw the purpose of statistics as summarizing what the evidence in the observed data says about questions of scientific interest, while Neyman described the purpose in behavioristic or decision-theoretic terms, where the goal is to minimize average losses in repeated performances. Ref. [6] described this debate, which continues to be part of the statistical profession. The significance of the results of this paper is that it demonstrates that relative belief inference allows for a possible resolution of this conflict and, as we will now discuss, also resolves a general criticism of decision theory.

A natural requirement for statistical analysis is that all the ingredients chosen by the statistician need to be checkable against the observed data to ensure they align with the objective data (or, at least, are correctly collected). The choice of the model, prior, and loss functions are typically subjective decisions made by the analyst, and many consider such subjectivity to be at odds with the demands of science. However, both the model and the prior can be checked against the observed data to determine whether the choices made are contradicted by the data. Model checking has long been an acceptable, even necessary, part of statistical practice. In recent years, methods have been developed to check for a conflict between the prior and data. These methods determine if the prior placed the bulk of its mass in a region of the parameter space unsupported by the data as containing the true value of the parameter. Ref. [7] contained a discussion about checking for prior-data conflicts and also on what to do when a prior fails its checks. While such checking does not establish the objectivity of these elements, it at least allows the objective data to comment on the relevance of the choices made. However, it is unclear how one can check the loss function L using the data, and this ambiguity may be considered a flaw in decision theory, particularly for scientific applications.

There are, however loss functions that are considered intrinsic and that avoid this criticism. For example, Ref. [8] proposed using the intrinsic loss function based on a measure of distance between sampling distributions. Ref. [9] proposed using the intrinsic loss function based on the Kullback–Leibler divergence $KL(f(·|\theta )\Vert f(·|{\theta }^{\prime }))$ between $f(·|\theta )$ and $f(·|{\theta }^{\prime }).$ When $\psi =\theta$, the intrinsic loss function is given by

$$L(\theta ,{\theta }^{\prime })=min(KL(f(·|\theta )\Vert f(·|{\theta }^{\prime })),KL(f(·|{\theta }^{\prime })\Vert f(·|\theta ))).$$

For a marginal parameter $\psi$, the intrinsic loss function is $L(\theta ,\psi )={inf}_{{\theta }^{\prime }\in {\Psi }^{-1}\left\{\psi \right\}}L(\theta ,{\theta }^{\prime }).$ These loss functions are intrinsic because they are based on the sampling model, allowing their suitability to be verified through model checking. The loss functions used to derive relative belief inferences for $\psi =\Psi \left(\theta \right)$ are based upon the prior ${\pi }_{\Psi },$ see Section 3 for the definitions, and so are also intrinsic and checkable against the data while checking for prior-data conflict.

In some contexts, relative belief inferences are Bayes rules, but in a general context, they are seen to arise as the limits of Bayes rules. This approach has some historical antecedents. For example, in [10], it is shown that the MLE is asymptotically a Bayes rule, but this conclusion is drawn under a fixed loss function, with increasing amounts of data and a sequence of priors. In the context discussed here, however, the data amount is fixed, as are the model and prior, but there is a sequence of loss functions, all based on a single fixed prior. The loss functions relevant for deriving relative belief inferences are similar to those used to justify maximum a posteriori (MAP) inferences. It can be demonstrated that under certain conditions, MAP inferences emerge as the limits of Bayes rules through a sequence of loss functions,

$$L_{\lambda }(\theta ,\psi )=I_{B_{\lambda }^c\left(\psi \right)}(\Psi \left(\theta \right))$$

(1)

where $\lambda >0,B_{\lambda }(\Psi \left(\theta \right))$ is the ball of radius $\lambda$ centered at $\psi$, $I_A$ denotes the indicator function for set A and, in the continuous case, the support measure on $\Psi (\Theta )$ is volume measure (see [11]). MAP inferences are not invariant under reparameterizations and such invariance can be considered as a desirable property of any inference method. Relative belief inferences, however, are invariant under reparameterizations.

Section 2 is concerned with describing the general characteristics of three approaches to deriving Bayesian inferences. Section 3 and Section 4 show how relative belief estimation and prediction inferences can be seen to arise from decision theory and Section 5 does this for credible regions and hypothesis assessment. In particular, it is shown here that relative belief estimators, as used in practice, are admissible. The contents of Section 3, Section 4 and Section 5 are original contributions by the authors that were derived some years ago but not published. Some of this discussion has appeared in Ref. [3] and is included here to provide a complete exposition of the relationship between relative belief and decision theory. All proofs of theorems and corollaries are in Appendix A, except for cases where $\Psi (\Theta )$ is finite, as these are quite straightforward and provide motivation for the more complicated contexts.

It should be emphasized that the authors do not consider the fact that relative belief inferences can be derived within the context of decision theory as the primary justification for the approach. Rather, the justification lies within the Bayesian context, which leads—via the principle of evidence—to a clear characterization of statistical evidence. The specific loss functions used, while appealing, are not essential to this characterization. The fact that relative belief inferences are consistent with two of the major themes of statistical research pursued over the years, in our view, provides substantial support for their appropriateness.

2. Bayesian Inference

Some approaches to deriving Bayesian inferences will now be described in detail.

2.1. Bayesian Decision Theory

An ingredient that is commonly added to $\left(\right\{f(·|\theta ):\theta \in \Theta \},\pi ,x)$ is a loss function, namely, $L:\Theta \times \Psi (\Theta )\to [0,\infty )$, satisfying $L(\theta ,\psi )=L({\theta }^{\prime },\psi )$ whenever $\Psi \left(\theta \right)=\Psi \left({\theta }^{\prime }\right)$ and $L(\theta ,\psi )=0$ only when $\psi =\Psi \left(\theta \right).$ The goal is to find a procedure, say $\delta \left(x\right)\in \Psi (\Theta ),$ which in some sense minimizes the loss $L(\theta ,\delta (x\left)\right)$ based on the joint distribution of $(\theta ,x).$ Given the assumptions on L, the loss function can instead be thought of as a map $L:\Psi (\Theta )\times \Psi (\Theta )\to [0,\infty )$ with $L(\psi ,{\psi }^{\prime })=0$ iff $\psi ={\psi }^{\prime }$ and the ingredients can be represented as $(\{m(·|\psi ):\psi \in \Psi (\Theta )\},{\pi }_{\Psi },L,x).$

The goal of a decision analysis is then to find a decision function $\delta :\mathcal{X}\to \Psi (\Theta )$ that minimizes the prior risk,

$$r\left(\delta \right)={\int }_{\Psi (\Theta )}{\int }_{\mathcal{X}}L(\psi ,\delta \left(x\right))M\left(dx\right|\psi ){\Pi }_{\Psi }\left(d\psi \right)={\int }_{\mathcal{X}}r\left(\delta \right|x)M\left(dx\right),$$

where $r\left(\delta \right|x)={\int }_{\Psi (\Theta )}L(\psi ,\delta \left(x\right)){\Pi }_{\Psi }\left(d\psi \right|x)$ is the posterior risk. Such a $\delta$ is called a Bayes rule and clearly a $\delta$ that minimizes $r\left(\delta \right|x)$ for each x is a Bayes rule. Further discussion of the Bayesian decision theory can be found in [12].

As noted in [9], a decision formulation also leads to credible regions for $\psi ,$ namely, a $\gamma$-lowest posterior loss-credible region is defined by

$$D_{\gamma }\left(x\right)=\{\psi :r\left(\psi \right|x)\le d_{\gamma }\left(x\right)\},$$

(2)

where $d_{\gamma }\left(x\right)=inf\{k:{\Pi }_{\Psi }\left(\{\psi :r\left(\psi \right|x)\le k\}\right|x)\ge \gamma .$ Note that $\psi$ in (2) is interpreted as the decision function that takes the value $\psi$ constantly in $x.$ Clearly, as $\gamma \to 0$, set $D_{\gamma }\left(x\right)$ converges to the value of a Bayes rule at $x.$ For example, with quadratic loss, the Bayes rule is given by the posterior mean and a $\gamma$-lowest posterior loss region is the smallest sphere centered at the mean containing (at least) $\gamma$ of the posterior probability.

2.2. MAP Inferences

The highest posterior density (HPD) or MAP-based approach to determining inferences constructs credible regions of the following form

$$H_{\gamma }\left(x\right)=\{\psi :{\pi }_{\Psi }\left(\psi \right|x)\ge h_{\gamma }\left(x\right)\},$$

(3)

where ${\pi }_{\Psi }(·|x)$ is the marginal posterior density with respect to a support measure ${\nu }_{\Psi }$ on $\Psi (\Theta ),$ and $h_{\gamma }\left(x\right)$ is chosen so that $h_{\gamma }\left(x\right)=sup\{k:{\Pi }_{\Psi }\left(\{\psi :{\pi }_{\Psi }\left(\psi \right|x)\ge k\}\right|x)\ge \gamma \}.$ It follows from (3) that, to assess the hypothesis $H_0:\Psi \left(\theta \right)={\psi }_0,$ we can use the tail probability given by $1-inf\{\gamma :{\psi }_0\in H_{\gamma }\left(x\right)\}.$ Furthermore, the class of sets $H_{\gamma }\left(x\right)$ is naturally “centered” at the posterior mode (when it exists uniquely) as $H_{\gamma }\left(x\right)$ converges to this point as $\gamma \to 0.$ The use of the posterior mode as an estimator is commonly referred to as MAP estimation. We can then think of the size of set $H_{\gamma }\left(x\right),$ say for $\gamma =0.95,$ as a measure of how accurate the MAP estimator is in a given context. Furthermore, when $\Psi (\Theta )$ is an open subset of Euclidean space, then $H_{\gamma }\left(x\right)$ minimizes the volume among all $\gamma$-credible regions.

It is well-known that HPD inferences suffer from a defect. In particular, in the continuous case, MAP inferences are not invariant under reparameterization. For example, this means that, if ${\psi }_{MAP}\left(x\right)$ is the MAP estimate of $\psi$, then it is not necessarily true that ${\rm Y}\left({\psi }_{MAP}\left(x\right)\right)$ is the MAP estimate of $\tau ={\rm Y}\left(\psi \right)$ when ${\rm Y}$ is a 1-1 smooth transformation. The non-invariance of a statistical procedure seems very unnatural as it implies that the statistical analysis depends on the parameterization and typically there does not seem to be a good reason for this. Note too that estimates based upon taking posterior expectations will also suffer from this lack of invariance. It is also the case that MAP inferences are not based on a direct characterization of statistical evidence. Both of these issues motivate the development of relative belief inferences.

One justification for MAP inference is decision-theoretic via the loss functions defined in (1). It is common, however, to also consider posterior probabilities of events as expressions of evidence and so think of this approach as evidential in nature. Posterior probabilities, however, express beliefs rather than evidence. For instance, the posterior probability of an event may be very small yet larger than its prior probability, indicating that the data have increased belief in the event’s occurrence. This would suggest that the data provide evidence in favor of the event being true, rather than evidence against it, even though the posterior probability remains small. It appears that evidence is better characterized by how the data change beliefs, rather than by the beliefs themselves.

2.3. Relative Belief Inferences

Relative belief inferences, like MAP inferences, are based on the ingredients $(\{m(·|\psi ):\psi \in \Psi (\Theta )\},{\pi }_{\Psi },x).$ Note that underlying both approaches is the principle (axiom) of conditional probability that says that initial beliefs about $\psi ,$ as expressed by the prior ${\pi }_{\Psi },$ must be replaced by conditional beliefs as expressed by the posterior ${\pi }_{\Psi }(·|x).$ In this approach, however, a measure of statistical evidence is used given by the relative belief ratio,

$$R{B}_{\Psi }\left(\psi \right|x)={\displaystyle \frac{{\pi }_{\Psi }\left(\psi \right|x)}{{\pi }_{\Psi }\left(\psi \right)}}={\displaystyle \frac{m\left(x\right|\psi )}{m\left(x\right)}}.$$

(4)

The relative belief ratio produces the following conclusions: if $R{B}_{\Psi }\left(\psi \right|x)>1,$ then there is evidence in favor of $\psi$ being the true value, if $R{B}_{\Psi }\left(\psi \right|x)<1,$ there is evidence against $\psi$ being the true value, and if $R{B}_{\Psi }\left(\psi \right|x)=1,$ then there is no evidence either way. These implications follow from a very simple principle of inference.

Principle of evidence: for probability model $(\Omega ,\mathcal{F},P),$ if $C\in \mathcal{F}$ is observed to be true, where $P\left(C\right)>0,$ then there is evidence in favor of $A\in \mathcal{F}$ being true if $P\left(A\right|C)>P(A),$ evidence against $A\in \mathcal{F}$ being true if $P\left(A\right|C)<P(A)$, and no evidence either way if $P\left(A\right|C)=P(A).$

This principle seems obvious when ${\Pi }_{\Psi }$ is a discrete probability measure. For the continuous case, where ${\Pi }_{\Psi }\left(\left\{\psi \right\}\right)=0,$ let $N_{ϵ}\left(\psi \right)$ be a sequence of neighborhoods of $\psi$ converging nicely to $\psi$ as $ϵ\to 0$ (see [13]), then under weak conditions, e.g., ${\pi }_{\Psi }$ is continuous and positive at $\psi ,$

$$\underset{ϵ\to 0}{lim}R{B}_{\Psi }\left(N_{ϵ}\left(\psi \right)\right|x)=\underset{ϵ\to 0}{lim}{\displaystyle \frac{{\Pi }_{\Psi }\left(N_{ϵ}\left(\psi \right)\right|x)}{{\Pi }_{\Psi }\left(N_{ϵ}\left(\psi \right)\right)}}={\displaystyle \frac{{\pi }_{\Psi }\left(\psi \right|x)}{{\pi }_{\Psi }\left(\psi \right)}}=R{B}_{\Psi }\left(\psi \right|x)$$

and this justifies the general interpretation of $R{B}_{\Psi }\left(\psi \right|x)$ as a measure of evidence. The relative belief ratio determines the inferences.

A natural estimate of $\psi$ is the relative belief estimate,

$${\psi }_{RB}\left(x\right)=arg\underset{\psi }{sup}R{B}_{\Psi }\left(\psi \right|x)$$

as it has maximum evidence in favor. To assess the accuracy of ${\psi }_{RB}\left(x\right)$, consider the plausible region $P{l}_{\Psi }\left(x\right)=\{\psi :R{B}_{\Psi }\left(\psi \right|x)>1\},$ which is the set of $\psi$ values with evidence supporting them as the true value. The size of $P{l}_{\Psi }\left(x\right)$ along with its posterior content ${\Pi }_{\Psi }\left(P{l}_{\Psi }\left(x\right)\right|x),$ which measures the belief that the true value is in $P{l}_{\Psi }\left(x\right),$ provides an assessment of accuracy. So, if $P{l}_{\Psi }\left(x\right)$ is “small” and ${\Pi }_{\Psi }\left(P{l}_{\Psi }\left(x\right)\right|x)\approx 1,$ then ${\psi }_{RB}\left(x\right)$ is to be considered an accurate estimate of $\psi$ but not otherwise. A relative belief $\gamma$-credible region is given by

$$C_{\Psi ,\gamma }\left(x\right)=\{\psi :R{B}_{\Psi }\left(\psi \right|x)\ge c_{\gamma }\left(x\right)\},$$

where $c_{\gamma }\left(x\right)=sup\{c:{\Pi }_{\Psi }(R{B}_{\Psi }\left(\psi \right|x)\ge c|x)\ge \gamma \},$ for $\psi$ can also be quoted provided $C_{\Psi ,\gamma }\left(x\right)\subset P{l}_{\Psi }\left(x\right).$ The containment is necessary, as otherwise, $C_{\Psi ,\gamma }\left(x\right)$ would contain a value $\psi$ for which there is evidence against $\psi$ being the true value.

To assess the hypothesis $H_0:\Psi \left(\theta \right)={\psi }_0,$ the value $R{B}_{\Psi }\left({\psi }_0\right|x)$ indicates whether there is evidence in favor of or against $H_0.$ The strength of this evidence can be measured by the posterior probability ${\Pi }_{\Psi }\left(\left\{{\psi }_0\right\}\right|x),$ as this measures the belief in what the evidence says. So, if $R{B}_{\Psi }\left({\psi }_0\right|x)>1$ and ${\Pi }_{\Psi }\left(\left\{{\psi }_0\right\}\right|x)\approx 1,$ then there is strong evidence that $H_0$ is true, while when $R{B}_{\Psi }\left({\psi }_0\right|x)<1$ and ${\Pi }_{\Psi }\left(\left\{{\psi }_0\right\}\right|x)\approx 0,$ there is strong evidence that $H_0$ is false. Since ${\Pi }_{\Psi }\left(\left\{{\psi }_0\right\}\right)$ can be small, even 0 in the continuous case, it makes more sense to measure the strength of the evidence in such a case by

$$St{r}_{\Psi }\left({\psi }_0\right|x)={\Pi }_{\Psi }(R{B}_{\Psi }\left(\psi \right|x)\le R{B}_{\Psi }\left({\psi }_0\right|x)|x).$$

If $R{B}_{\Psi }\left({\psi }_0\right|x)>1$ and $St{r}_{\Psi }\left({\psi }_0\right|x)\approx 1,$ then the evidence is strong that ${\psi }_0$ is the true value as there is little belief that the true value of $\psi$ has more evidence in its favor than ${\psi }_0.$ If $R{B}_{\Psi }\left({\psi }_0\right|x)<1$ and $St{r}_{\Psi }\left({\psi }_0\right|x)\approx 0,$ then the evidence is strong that ${\psi }_0$ is not the true value as there is a widespread belief that the true value of $\psi$ has more evidence in its favor than ${\psi }_0.$ There is no reason to quote a single number to measure the strength; both ${\Pi }_{\Psi }\left(\left\{{\psi }_0\right\}\right|x)$ and $St{r}_{\Psi }\left({\psi }_0\right|x)$ can be quoted when relevant.

An important aspect of both $St{r}_{\Psi }\left({\psi }_0\right|x)$ and ${\Pi }_{\Psi }\left(P{l}_{\Psi }\left(x\right)\right|x)$ is what happens as the data increase. To ensure that these behave appropriately, namely, $St{r}_{\Psi }\left({\psi }_0\right|x)\to 0\left(\mathrm{or}1\right)$ when $H_0$ is false (or true) and ${\Pi }_{\Psi }\left(P{l}_{\Psi }\left(x\right)\right|x)\to 1,$ it is necessary to take into account the difference that matters, $\delta .$ By this, we mean that there is a distance measure $d_{\Psi }$ on $\Psi (\Theta )\times \Psi (\Theta )$ such that if $d_{\Psi }(\psi ,{\psi }^{\prime })\le \delta ,$ then in terms of the application, these values are considered equivalent. Such a $\delta$ always exists because measurements are always taken to finite accuracy. For example, if $\psi$ is real-valued, then there is a grid of values $\dots {\psi }_{-2},{\psi }_{-1},{\psi }_0,{\psi }_1,{\psi }_2,\dots$ separated by $\delta$, and inferences are determined using the relative belief ratios of the intervals $[{\psi }_i-\delta /2,{\psi }_i+\delta /2).$ In effect, $H_0$ is now $H_0:\Psi \left(\theta \right)\in [{\psi }_0-\delta /2,{\psi }_0+\delta /2).$ When the computations are carried out in this way, then $St{r}_{\Psi }\left({\psi }_0\right|x)$ and ${\Pi }_{\Psi }\left(P{l}_{\Psi }\left(x\right)\right|x)$ do what is required. As a particular instance of this, see the results in Section 4, where such discretization plays a key role.

It is easy to see that the class of relative-belief credible regions $\{C_{\Psi ,\gamma }\left(x\right):\gamma \in [0,1]\}$ for $\psi$ is independent of the marginal prior ${\pi }_{\Psi }.$ When a value $\gamma \in [0,1]$ is specified, however, set $C_{\Psi ,\gamma }\left(x\right)$ depends on ${\pi }_{\Psi }$ through $c_{\gamma }\left(x\right).$ So, the form of relative belief inferences about $\psi$ is completely robust to the choice of ${\pi }_{\Psi }$ but the quantification of the uncertainty in the inferences is not. For example, when $\psi =\Psi \left(\theta \right)=\theta ,$ then ${\theta }_{RB}\left(x\right)$ is the MLE; however, in general, ${\psi }_{RB}\left(x\right)$ is the maximizer of the integrated likelihood $m\left(x\right|\psi ).$ Similarly, relative belief regions are likelihood regions in cases of the full parameter and integrated likelihood regions. As such, likelihood regions can be seen as essentially Bayesian in character with a clear and precise characterization of evidence through the relative belief ratio, and now have probability assignments through the posterior. A relative belief ratio, $R{B}_{\Psi }\left(\psi \right|x),$ while proportional to an integrated likelihood, cannot be multiplied by an arbitrary positive constant—as with a likelihood—without losing its interpretation in measuring statistical evidence. It has been established in [14] that relative belief inferences for $\psi$ are optimally robust to the prior ${\pi }_{\Psi }.$

As can be seen from (4), relative belief inferences are always invariant under smooth reparameterizations; this is at least one reason why they are preferable to MAP inferences. Any rule for measuring evidence, which satisfies the principle of evidence, also produces valid estimates, as these lie in $P{l}_{\Psi }\left(x\right)$ and so will have the same “accuracy” as ${\psi }_{RB}\left(x\right).$ For example, if instead of the relative belief ratio, the difference ${\pi }_{\Psi }\left(\psi \right|x)-{\pi }_{\Psi }\left(\psi \right)$ is used as the measure of evidence with a cut-off of 0, then this satisfies the principle of evidence, but the estimate is no longer necessarily invariant under reparameterizations. The Bayes factor with a cut-off of 1 is also a valid measure of evidence but there are a number of reasons why the relative belief ratio is to be preferred to the Bayes factor for general inferences (see [15]).

We will now consider a simple example that illustrates the various concepts just discussed.

Example 1. 

Location normal

Suppose that we have a sample $x=(x_1,\dots ,x_n)$ from a $N(\theta ,{\sigma }_0^2)$ where the mean $\theta \in \Theta ={\mathbb{R}}^1$ is unknown and the variance ${\sigma }_0^2$ is assumed known. Suppose interest lies in making inferences about $\psi =\Psi \left(\theta \right)=\theta ,$ and the prior π on θ is given by a $N({\theta }_0,{\tau }_0^2)$ distribution. In this context, $\overline{x}$ serves as a minimal sufficient statistic, allowing the focus to be restricted to the $N(\theta ,{\sigma }_0^2/n)$ model while ignoring the remaining aspects of the data, at least for inference. Certainly the residuals $(x_1-\overline{x},\dots ,x_n-\overline{x})$ are relevant for model checking. The prior predictive density m of $\overline{x}$ is then given by the density of a $N({\theta }_0,{\tau }_0^2+{\sigma }_0^2/n)$ distribution and, as discussed in [7], this is relevant for checking for prior-data conflicts via the tail probability $M(m\left(\overline{X}\right)\le m\left(\overline{x}\right))$ with small values indicating the existence of a conflict.

The posterior $\Pi (·|x)$ of θ is given by

$$\theta |\overline{x}\sim N\left({\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1}(n\overline{x}/{\sigma }_0^2+{\theta }_0/{\tau }_0^2),{\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1}\right).$$

As such, ${\theta }_{MAP}\left(\overline{x}\right)={\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1}(n\overline{x}/{\sigma }_0^2+{\theta }_0/{\tau }_0^2).$ If, as is common, squared error loss is employed, then the Bayes rule for estimating θ is given by ${\theta }_{MAP}\left(\overline{x}\right),$ as this is also the posterior mean. On the other hand, the relative belief ratio is given by

$$\begin{array}{cc}\hfill RB\left(\theta \right|\overline{x})& ={\displaystyle \frac{\pi \left(\theta \right|\overline{x})}{\pi \left(\theta \right)}}={\displaystyle \frac{m\left(\overline{x}\right|\theta )}{m\left(\theta \right)}}\hfill \\ \hfill & ={\left({\displaystyle \frac{{\tau }_0^2+{\sigma }_0^2/n}{{\sigma }_0^2/n}}\right)}^{1/2}{\displaystyle \frac{exp(-n{(\overline{x}-\theta )}^2/2{\sigma }_0^2)}{exp(-{({\tau }_0^2+{\sigma }_0^2/n)}^{-1}{(\overline{x}-{\theta }_0)}^2/2)}}\hfill \end{array}$$

as, since there are no nuisance parameters, $m(·|\theta )$ equals the sampling density of $\overline{x}.$

From this, it is immediate that ${\theta }_{RB}\left(\overline{x}\right)=\overline{x}$, which is the MLE, a result that is generally true for relative belief when estimating the full model parameter. The plausible interval for θ is then, putting $r_0=({\sigma }_0^2/n)/({\tau }_0^2+{\sigma }_0^2/n),$ given by

$$Pl\left(\overline{x}\right)=\{\theta :RB\left(\theta \right|\overline{x})>1\}=\left(\overline{x}\pm {\sigma }_0\sqrt{{\displaystyle \frac{{({\tau }_0^2+{\sigma }_0^2/n)}^{-1}{(\overline{x}-{\theta }_0)}^2-log\left(r_0\right)}{n}}}\right)$$

and note that $-log\left(r_0\right)>0$, so this interval is always defined. The length of $Pl\left(\overline{x}\right)$ and its posterior content, computed using $\Pi (·|x)$, provide a measure of the accuracy of ${\theta }_{RB}\left(\overline{x}\right).$ Notice that $Pl\left(\overline{x}\right)$ converges almost surely to the degenerate interval consisting of the true value of θ and the posterior content of the interval converges to 1 as $n\to \infty .$

To assess a hypothesis, say $H_0:\theta ={\theta }_0,$ the relevant relative belief ratio is as follows:

$$RB\left({\theta }_0\right|\overline{x})=r_0^{1/2}exp\left(-{\displaystyle \frac{r_0-1}{r_0}}{\displaystyle \frac{n}{{\sigma }_0^2}}{\displaystyle \frac{{(\overline{x}-{\theta }_0)}^2}{2}}\right).$$

This gives evidence in favor (or against) $H_0$ when

$${\displaystyle \frac{n{(\overline{x}-{\theta }_0)}^2}{{\sigma }_0^2}}<(>){\displaystyle \frac{r_{0}log{r}_0}{r_0-1}}$$

and note that the right-hand side is always positive. The strength of this evidence is given by

$$\begin{array}{cc}\hfill & Str\left({\theta }_0\right|x)=\Pi (RB\left(\theta \right|x)\le RB\left({\theta }_0\right|x)|x)=\Pi \left(\right|\overline{x}-\theta |\ge |\overline{x}-{\theta }_0\left|\right|x)\hfill \\ \hfill & =F(\overline{x}+|\overline{x}-{\theta }_0|;{\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1}(n\overline{x}/{\sigma }_0^2+{\theta }_0/{\tau }_0^2),{\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1})-\hfill \\ \hfill & F(\overline{x}-|\overline{x}-{\theta }_0|;{\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1}(n\overline{x}/{\sigma }_0^2+{\theta }_0/{\tau }_0^2),{\left(n/{\sigma }_0^2+1/{\tau }_0^2\right)}^{-1}),\hfill \end{array}$$

where $F(·;\mu ,\lambda )$ denotes the $N(\mu ,\lambda )$ cdf. As $n\to \infty ,$ the strength converges to 0 (the strongest possible evidence against) when $H_0$ is false and converges to 1 (the strongest possible evidence in favor) when $H_0$ is true.

3. Estimation: Discrete Parameter Space

The following theorem presents the basic definition of the loss function for the parameter of interest $\psi =\Psi \left(\theta \right)$ when the set of possible values of $\psi$, namely, $\Psi (\Theta )=\{\psi :\psi =\Psi (\theta )\mathrm{for}\mathrm{some}\theta \in \Theta \}$, is finite. This establishes an important optimality result. The indicator function for the set A is denoted as $I_A$.

Theorem 1. 

Suppose that ${\pi }_{\Psi }\left(\psi \right)>0$ for every $\psi \in \Psi (\Theta )$, $\Psi (\Theta )$ is finite with ${\nu }_{\Psi }$ equal to counting measure on $\Psi (\Theta ).$ Then for the loss function

$$L_{RB}(\theta ,\psi )={\displaystyle \frac{I_{{\left\{\psi \right\}}^c}(\Psi \left(\theta \right))}{{\pi }_{\Psi }(\Psi \left(\theta \right))}},$$

(5)

the relative belief estimator ${\psi }_{RB}$ is a Bayes rule.

Proof. 

We have that

$$\begin{array}{cc}\hfill r\left(\delta \right|x)& ={\int }_{\Psi (\Theta )}{\displaystyle \frac{I_{{\left\{\delta \left(x\right)\right\}}^c}\left(\psi \right)}{{\pi }_{\Psi }\left(\psi \right)}}{\Pi }_{\Psi }\left(d\psi \right|x)\hfill \\ \hfill & ={\int }_{\Psi (\Theta )}R{B}_{\Psi }\left(\psi \right|x){\nu }_{\Psi }\left(d\psi \right)-R{B}_{\Psi }\left(\delta \left(x\right)\right|x).\hfill \end{array}$$

(6)

Since $\Psi (\Theta )$ is finite, the first term in (6) is finite and a Bayes rule at x is given by the value $\delta \left(x\right)$ that maximizes the second term. Therefore, ${\psi }_{RB}$ is a Bayes rule. □

The loss function $L_{RB}$ seems very natural. Beliefs about the true value of $\psi$ are expressed by the prior ${\pi }_{\Psi }$. As such, consider values where ${\pi }_{\Psi }\left(\psi \right)$ is very low and $\psi$ is indeed false. It would then be misleading if inferences suggested such a value as being true. So it is appropriate for such values to bear large losses. In a sense, the statistician is acknowledging what such values are by the choice of the prior. Of course, the prior may be wrong in the sense that the bulk of its mass is placed in a region where the true value of $\psi$ does not lie. This is why checking for prior-data conflict, before conducting inference, is always recommended. Procedures for checking priors were discussed in [16,17], and an approach to replacing priors found to be at fault was developed in [7]. The loss function $L_{RB}$ motivates the other losses for relative belief discussed here, making this comment relevant to those losses as well.

The prior risk of $\delta$ satisfies the following,

$$r\left(\delta \right)={\int }_{\Psi (\Theta )}{\int }_{\mathcal{X}}{\displaystyle \frac{I_{{\left\{\delta \left(x\right)\right\}}^c}\left(\psi \right)}{{\pi }_{\Psi }\left(\psi \right)}}M\left(dx\right|\psi ){\Pi }_{\Psi }\left(d\psi \right)=\sum _{\psi \in \Psi (\Theta )}M(\delta \left(x\right)\ne \psi |\psi ),$$

(7)

where $M(·|\psi )$ is the conditional prior predictive probability measure of the data given $\psi$, so (7) is the sum of the conditional prior error probabilities over all $\psi$ values. If instead the loss function is taken to be $L_{MAP}(\theta ,\psi )=I_{{\left\{\psi \right\}}^c}(\Psi \left(\theta \right)),$ as in (1), then the same proof used in Theorem 1 establishes that ${\psi }_{MAP}$ is a Bayes rule with respect to this loss, and the prior risk is given by,

$$\sum _{\psi }M(\delta \left(x\right)\ne \psi |\psi ){\pi }_{\Psi }\left(\psi \right),$$

(8)

which represents the prior probability of making an error. Both $L_{MAP}$ and $L_{RB}$ are two-valued loss functions but, when an incorrect decision is made, the loss is constant in $\Psi \left(\theta \right)$ for $L_{MAP}$, while it equals the reciprocal of the prior probability of $\Psi \left(\theta \right)$ for $L_{RB}$. So, $L_{RB}$ penalizes an incorrect decision much more severely when the true value of $\Psi \left(\theta \right)$ is in the tails of the prior. Note that ${\psi }_{MAP}={\psi }_{RB}$ when ${\Pi }_{\Psi }$ is uniform. It is evident that (7) serves as an upper bound to (8), indicating that controlling losses based on $L_{RB}$ automatically controls the losses based on $L_{MAP}.$

As already noted, $R{B}_{\Psi }\left(\psi \right|x)$ is proportional to the integrated likelihood of $\psi .$ So, under the conditions of Theorem 1, the maximum integrated likelihood estimator is a Bayes rule. Furthermore, the Bayes rule is the same for every choice of ${\pi }_{\Psi }$ and only depends on the full prior through the conditional prior $\Pi (·|\psi )$ placed on the nuisance parameters. When $\psi =\theta$, then ${\theta }_{RB}\left(x\right)$ is the MLE of $\theta$ and so the MLE of $\theta$ is a Bayes rule for every prior $\pi .$

Note that when $\Psi (\Theta )=\{{\psi }_0,{\psi }_1\}$, then $R{B}_{\Psi }\left({\psi }_0\right|x)>(<)1$ iff $R{B}_{\Psi }\left({\psi }_1\right|x)<(>)1$, so ${\psi }_{RB}\left(x\right)={\psi }_0$ when $R{B}_{\Psi }\left({\psi }_0\right|x)>1$, and ${\psi }_{RB}\left(x\right)={\psi }_1$ otherwise. This is the classical context for hypothesis testing, where ${\psi }_{RB}\left(x\right)={\psi }_0$ can be viewed as acceptance of the hypothesis $H_0:\theta \in {\Psi }^{-1}\left\{{\psi }_0\right\}$, and ${\psi }_{RB}\left(x\right)={\psi }_1$ as rejection of $H_0.$ Theorem 1 establishes that relative belief offers a Bayes rule for the hypothesis testing problem.

The loss function (5) does not provide meaningful results when $\Psi (\Theta )$ is infinite as (7) shows that $r\left(\delta \right)$ will be infinite. So, we modify (5) via a parameter $\eta >0$ and define the loss function as follows,

$$L_{RB,\eta }(\theta ,\psi )={\displaystyle \frac{I_{{\left\{\psi \right\}}^c}(\Psi \left(\theta \right))}{max(\eta ,{\pi }_{\Psi }(\Psi \left(\theta \right)))}}.$$

(9)

Note that $L_{RB,\eta }$ is bounded by $1/\eta$. This loss function is like (5) but does not allow for arbitrarily large losses. The following result shows that we can restrict attention to values of $\eta$ that are sufficiently small.

Theorem 2. 

Suppose that ${\pi }_{\Psi }\left(\psi \right)>0$ for every $\psi \in \Psi (\Theta ),$ where $\Psi (\Theta )$ is countable with ${\nu }_{\Psi }$ equal to counting measure, and that ${\psi }_{RB}\left(x\right)$ is the unique maximizer of $R{B}_{\Psi }\left(\psi \right|x)$ for all $x.$ For the loss function (9) and Bayes rule ${\delta }_{\eta },$ then ${\delta }_{\eta }\left(x\right)\to {\psi }_{RB}\left(x\right)$ as $\eta \to 0,$ for every $x\in \mathcal{X}.$

The proof of Theorem 2 also establishes the following result.

Corollary 1. 

For all sufficiently small η, the value of a Bayes rule at x is given by ${\psi }_{RB}\left(x\right).$

The following is an immediate consequence of Theorem 1 and Corollary 1 as ${\psi }_{RB}$ is a Bayes rule.

Corollary 2. 

${\psi }_{RB}$ is an admissible estimator with respect to the loss function $L_{RB}$ when $\Psi (\Theta )$ is finite, and with respect to loss $L_{RB,\eta },$ when η is sufficiently small, and $\Psi (\Theta )$ is countable.

In a general estimation problem, $\delta$ is risk-unbiased with respect to a loss function L if $E_{\theta }\left(L({\theta }^{\prime },\delta \left(x\right))\right)\ge E_{\theta }\left(L(\theta ,\delta \left(x\right))\right)$ for all ${\theta }^{\prime },\theta \in \Theta .$ This says that, on average, $\delta \left(x\right)$ is closer to the true value than any other value when we interpret $L(\theta ,\delta (x\left)\right)$ as a measure of distance between $\delta \left(x\right)$ and $\Psi \left(\theta \right).$ A definition of Bayesian-unbiasedness for $\delta$ with respect to L is given by the inequality,

$${\int }_{\Theta }{\int }_{\Theta }{E}_{\theta }\left(L({\theta }^{\prime },\delta \left(x\right))\right)\Pi \left(d\theta \right)\Pi \left(d{\theta }^{\prime }\right)\ge {\int }_{\Theta }{E}_{\theta }\left(L(\theta ,\delta \left(x\right))\right)\Pi \left(d\theta \right)=r\left(\delta \right),$$

as this retains the idea of being closer, on average, to the true value than a false value. We will now consider a family of loss functions defined as follows,

$$L(\theta ,\psi )=I_{{\left\{\psi \right\}}^c}(\Psi \left(\theta \right))h(\Psi \left(\theta \right))$$

(10)

where h is a nonnegative function satisfying ${\int }_{\Theta }h(\Psi \left(\theta \right))\Pi \left(d\theta \right)<\infty$. This includes $L_{RB}$ and $L_{MAP}$ when $\Psi (\Theta )$ is finite and $L_{RB,\eta }.$

Theorem 3. 

If $\Psi (\Theta )$ is finite or countable, then ${\psi }_{RB}\left(x\right)$ is Bayesian-unbiased under the loss function (10).

Suppose after observing x, there is a need to predict a future (or concealed) value $y\in \mathcal{Y}$, where $y\sim g_{\delta \left(\theta \right)}\left(y\right|x),$ a density with respect to the support measure ${\mu }_{\mathcal{Y}}$ on $\mathcal{Y},$ and it is assumed that the true value of $\theta$ in the model for x, gives the true value of $\delta \left(\theta \right).$ The prior predictive density of y is given by $q\left(y\right)={\int }_{\Theta }{\int }_{\mathcal{X}}\pi \left(\theta \right)f_{\theta }\left(x\right)g_{\delta \left(\theta \right)}\left(y\right|x)\mu \left(dx\right)\nu \left(d\theta \right)$ while the posterior predictive density is $q\left(y\right|x)={\int }_{\Theta }\pi \left(\theta \right|x)g_{\delta \left(\theta \right)}\left(y\right|x)\nu \left(d\theta \right).$ The relative belief ratio for a future value of y is, thus, $R{B}_Y\left(y\right|x)=q\left(y\right|x)/q\left(y\right)$, and the relative belief prediction, namely, the value that maximizes $R{B}_Y(·|x),$ is denoted as $y_{RB}\left(x\right).$ When $\mathcal{Y}$ is finite, using the same argument as in Theorem 1, $y_{RB}$ is a Bayes rule under the loss function $L_{RB}(y,y^{\prime })=I_{{\left\{y\right\}}^c}\left(y^{\prime }\right)/q\left(y\right).$ Also, it can be demonstrated that $y_{RB}$ is a limit of the Bayes rule when $\mathcal{Y}$ is countable.

We will now consider a common application where $\Psi (\Theta )$ is finite.

Example 2. 

Classification

For a classification problem, there are k categories $\{{\psi }_1,\dots ,{\psi }_k\},$ prescribed by a function $\Psi ,$ where ${\pi }_{\Psi }\left({\psi }_i\right)>0$ for each $i.$ Estimating ψ is then equivalent to classifying the data as having come from one of the distributions in the classes specified by ${\Psi }^{-1}\left\{{\psi }_i\right\}.$ The standard Bayesian solution to this problem is to use ${\psi }_{MAP}\left(x\right)$ as the classifier. From (8), we have that ${\psi }_{MAP}\left(x\right)$ minimizes the prior probability of misclassification, while from (7), ${\psi }_{RB}\left(x\right)$ minimizes the sum of the probabilities of misclassification. The essence of the difference is that ${\psi }_{RB}\left(x\right)$ treats the misclassification errors equally while ${\psi }_{MAP}\left(x\right)$ weights the errors by their prior probabilities.

The following shows that minimizing the sum of error probabilities is often more appropriate than minimizing the weighted sum. Suppose that $k=2$ and $x\sim$ Bernoulli$\left({\psi }_0\right)$ or $x\sim$ Bernoulli$\left({\psi }_1\right)$ with $\pi \left({\psi }_0\right)=1-ϵ$ and $\pi \left({\psi }_1\right)=ϵ$ representing the known proportions of individuals coming from population 0 or 1. For example, consider ${\psi }_0$ as the probability of a positive diagnostic test for a disease in the non-diseased population while ${\psi }_1$ is this probability for the diseased population. Suppose that ${\psi }_0/{\psi }_1$ is very small, indicating that the test is successful at identifying the disease while not yielding many false positives and that ϵ is very small, so the disease is rare. The challenge then becomes assigning a randomly chosen individual to a population based on their test results.

The posterior is given by $\pi \left({\psi }_0\right|1)={\psi }_0(1-ϵ)/({\psi }_0(1-ϵ)+{\psi }_1ϵ)$ and $\pi \left({\psi }_0\right|0)=(1-{\psi }_0)(1-ϵ)/((1-{\psi }_0)(1-ϵ)+(1-{\psi }_1)ϵ).$ Therefore,

$$\begin{array}{cc}\hfill {\psi }_{MAP}\left(1\right)& =\left\{\begin{array}{cc}\hfill {\psi }_0& if{\psi }_0/{\psi }_1>ϵ/(1-ϵ)\\ \hfill {\psi }_1& otherwise\hfill \end{array}\right.\hfill \\ \hfill {\psi }_{MAP}\left(0\right)& =\left\{\begin{array}{cc}\hfill {\psi }_0& if(1-{\psi }_0)/(1-{\psi }_1)>ϵ/(1-ϵ)\\ \hfill {\psi }_1& otherwise\hfill \end{array}\right.\hfill \end{array}$$

This implies that ${\psi }_{MAP}$ will always classify a person to the non-diseased population when ϵ is small enough, e.g., when ${\psi }_0=0.05,{\psi }_1=0.80,$ and $ϵ<0.0625.$ In contrast, in this situation, ${\psi }_{RB}$ always classifies an individual with a positive test to the diseased population and the non-diseased population for a negative test. Since $M(·|{\psi }_i)$ is the Bernoulli$\left({\psi }_i\right)$ distribution, when ${\psi }_0<{\psi }_1$ and ϵ are small enough, we have the following:

$$\begin{array}{cc}\hfill M({\psi }_{MAP}& \ne {\psi }_0\left|{\psi }_0\right)+M({\psi }_{MAP}\ne {\psi }_1|{\psi }_1)=0+1=1,\hfill \\ \hfill M({\psi }_{RB}& \ne {\psi }_0\left|{\psi }_0\right)+M({\psi }_{RB}\ne {\psi }_1|{\psi }_1)={\psi }_0+(1-{\psi }_1)=0.25.\hfill \end{array}$$

This clearly illustrates the difference between these two procedures as ${\psi }_{RB}$ does better than ${\psi }_{MAP}$ on the diseased population when ${\psi }_0$ is small and ${\psi }_1$ is large, as would be the case for a good diagnostic. Of course, ${\psi }_{MAP}$ minimizes the overall error rate, but at the price of ignoring the most important class in this problem, namely, those who have the disease. Note that this example can be extended to the situation where we need to estimate the ${\psi }_i$ based on samples from the respective populations, but this will not materially affect the overall conclusions.

We will now consider a situation where $(x,c)$ is such that $x|c\sim f_c,c|ϵ\sim$ Bernoulli$\left(ϵ\right)$, where $f_0$ and $f_1$ are known but ϵ is unknown with a prior $\pi .$ This is a generalization of the previous discussion, where ϵ is assumed to be known. Then, based on a sample $(x_1,c_1),\dots ,(x_n,c_n)$ from the joint distribution, the goal is to predict the value $c_{n+1}$ for a newly observed $x_{n+1}.$

The prior of c is $q\left(c\right)={\int }_0^1{(1-ϵ)}^{1-c}{ϵ}^c\pi \left(ϵ\right)dϵ$, and if $ϵ\sim$ beta$(\alpha ,\beta ),$ the prior predictive of $c_{n+1}$ is Bernoulli$(\alpha /(\alpha +\beta \left)\right).$ The posterior predictive density of $c_{n+1}$ equals, where $\overline{c}=n^{-1}{\sum }_{i=1}^n{c}_i,$

$$\begin{array}{cc}\hfill & q\left(c\right|(x_1,c_1),\dots ,(x_n,c_n),x_{n+1})\hfill \\ \hfill & \propto {\left(f_0\left(x_{n+1}\right)\right)}^{1-c}{\left(f_1\left(x_{n+1}\right)\right)}^c{\int }_0^1{ϵ}^{n\overline{c}+c}{(1-ϵ)}^{n(1-\overline{c})+(1-c)}\pi \left(ϵ\right)dϵ\hfill \\ \hfill & =f_c\left(x_{n+1}\right)\Gamma \left(\alpha +n\overline{c}+c\right)\Gamma (\beta +n(1-\overline{c})+1-c).\hfill \end{array}$$

It follows that, suppressing the dependence on the data, we have the following,

$$\begin{array}{cc}\hfill c_{MAP}& =\left\{\begin{array}{cc}1& if{\displaystyle \frac{f_1\left(x_{n+1}\right)}{f_0\left(x_{n+1}\right)}}{\displaystyle \frac{\left(\alpha +n\overline{c}\right)}{\left(\beta +n(1-\overline{c})\right)}}>1\hfill \\ 0& otherwise,\hfill \end{array}\right.\hfill \\ \hfill c_{RB}& =\left\{\begin{array}{cc}1& if{\displaystyle \frac{f_1\left(x_{n+1}\right)}{f_0\left(x_{n+1}\right)}}{\displaystyle \frac{\beta \left(\alpha +n\overline{c}\right)}{\alpha \left(\beta +n(1-\overline{c})\right)}}<1\hfill \\ 0& otherwise\hfill \end{array}\right.\hfill \end{array}$$

(11)

Note that $c_{MAP}$ and $c_{RB}$ are identical whenever $\alpha =\beta .$

From these formulas, it is apparent that a substantial difference will arise between $c_{MAP}$ and $c_{RB}$ when either α or β is much bigger than the other. As in Example 2, these correspond to situations where we believe that ϵ or $1-ϵ$ is very small. Suppose we take $\alpha =1$ and let β be relatively large, as this corresponds to knowing a priori that ϵ is very small. Then (11) implies that $c_{MAP}\le c_{RB}$ and so $c_{RB}=1$ whenever $c_{MAP}=1.$ A similar conclusion arises when we take $\beta =1$ and $\alpha <1.$

To see what kind of improvement is possible, we consider a simulation study. Let $f_0$ be a $N(0,1)$ density, $f_1$ be a $N(\mu ,1)$ density, and $n=10$ and the prior on ϵ be beta$(1,\beta ).$ Table 1 presents the Bayes risks for $c_{MAP}$ and $c_{RB}$ for various choices of β when $\mu =1.$ When $\beta =1$, they are equivalent, but we see that as β rises, the performance of $c_{MAP}$ deteriorates while $c_{RB}$ improves. Large values of β correspond to having information where ϵ is small. When $\beta =14$, about $0.50$ of the prior probability is to the left of $0.05$; with $\beta =32$, about $0.80$ of the prior probability is to the left of $0.05$; and with $\beta =100$, about $0.99$ of the prior probability is to the left of $0.05.$ We see that the misclassification rates for the small group $(c=1)$ stay about the same for $c_{RB}$ as β increases while they deteriorate markedly for $c_{MAP}$ as the MAP procedure basically ignores the small group.

Table 1. Conditional prior probabilities of misclassification for $c_{MAP}$ and $c_{RB}$ for various values of $\beta$ in Example 2 when $\alpha =1$, $\mu =1$, and n = 10.

$\mathit{\beta }$	${\mathit{M}}_0({\mathit{c}}_{\mathbf{MAP}}\ne 0)+{\mathit{M}}_1({\mathit{c}}_{\mathbf{MAP}}\ne 1)$	${\mathit{M}}_0({\mathit{c}}_{\mathbf{RB}}\ne 0)+{\mathit{M}}_1({\mathit{c}}_{\mathbf{RB}}\ne 1)$
1	$0.386+0.390=0.776$	$0.386+0.390=0.776$
14	$0.002+0.975=0.977$	$0.285+0.380=0.665$
32	$0.000+0.997=0.997$	$0.292+0.349=0.641$
100	$0.000+1.000=1.000$	$0.300+0.324=0.624$

We also investigated other choices for n and $\mu .$ There is very little change as n increases. When μ moves toward 0 and μ moves away from 0, the error rates go up and go down, as one would expect; $c_{RB}$ always dominates $c_{MAP}.$

4. Estimation: Continuous Parameter Space

When $\psi$ has a continuous prior distribution, the argument in Theorem 2 does not work, as ${\Pi }_{\Psi }\left(\left\{\delta \left(x\right)\right\}\right|x)=0.$ There are several possible ways to proceed but one approach is to use a discretization of the problem that uses Theorem 2. For this, we will assume that the spaces involved are locally Euclidean, the mappings are sufficiently smooth, and take the support measures to be the analogs of Euclidean volume on the respective spaces. While the argument presented is broadly applicable, it has been simplified in this context by assuming that all spaces are open subsets of Euclidean spaces, with the support measures being the Euclidean volume on these sets.

For each $\lambda >0$, suppose there is a discretization $\{B_{\lambda }\left(\psi \right):\psi \in \Psi (\Theta )\}$ of $\Psi (\Theta )$ into a countable number of subsets with the following properties: $\psi \in B_{\lambda }\left(\psi \right),{\Pi }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)>0$ and ${sup}_{\psi \in \Psi }$diam$\left(B_{\lambda }\left(\psi \right)\right)\to 0$ as $\lambda \to 0.$ So, if ${\psi }^{\prime }\in B_{\lambda }\left(\psi \right),$ then $B_{\lambda }\left({\psi }^{\prime }\right)=B_{\lambda }\left(\psi \right).$ For example, $B_{\lambda }\left(\psi \right)$ could be equal volume rectangles in $R^k.$ Further, we assume that ${\Pi }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)/{\nu }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)\to {\pi }_{\Psi }\left(\psi \right)$ as $\lambda \to 0$ for every $\psi .$ This will hold whenever ${\pi }_{\Psi }$ is continuous everywhere and $B_{\lambda }\left(\psi \right)$ converges nicely to $\left\{\psi \right\}$ as $\lambda \to 0.$ Let ${\psi }_{\lambda }\left(\psi \right)$ denote a point in $B_{\lambda }\left(\psi \right)$ such that ${\psi }_{\lambda }\left(\psi \right)={\psi }_{\lambda }\left({\psi }^{\prime }\right)$ whenever $\psi ,{\psi }^{\prime }\in \in B_{\lambda }\left(\psi \right)$ and put ${\Psi }_{\lambda }=\{{\psi }_{\lambda }\left(\psi \right):\psi \in \Psi (\Theta )\}.$ So, ${\Psi }_{\lambda }$ is a discretized version of $\Psi (\Theta ).$ We will call this a regular discretization of $\Psi (\Theta ).$ The discretized prior on ${\Psi }_{\lambda }$ is ${\pi }_{\Psi ,\lambda }\left({\psi }_{\lambda }\left(\psi \right)\right)={\Pi }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)$ and the discretized posterior is ${\pi }_{\Psi ,\lambda }\left({\psi }_{\lambda }\left(\psi \right)\right|x)={\Pi }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right|x).$

The loss function for the discretized problem is defined in Theorem 2 as follows,

$$L_{RB,\lambda ,\eta }(\theta ,{\psi }_{\lambda }\left(\psi \right))={\displaystyle \frac{I_{\left\{{\psi }_{\lambda }\left(\psi \right)\right\}}\left({\psi }_{\lambda }(\Psi \left(\theta \right))\right)}{max(\eta ,{\pi }_{\Psi ,\lambda }\left({\psi }_{\lambda }(\Psi \left(\theta \right))\right))}}$$

(12)

and let ${\delta }_{\lambda ,\eta }\left(x\right)$ denote a Bayes rule for this problem.

Theorem 4. 

Suppose that ${\pi }_{\Psi }$ is positive and continuous and we have a regular discretization of $\Psi .$ Furthermore, suppose that ${\psi }_{RB}\left(x\right)$ is the unique maximizer of $R{B}_{\Psi }\left(\psi \right|x)$ and for any $ϵ>0$,

$$\underset{\{\psi :||\psi -{\psi }_{RB}\left(x\right)||\ge ϵ\}}{sup}R{B}_{\Psi }\left(\psi \right|x)<R{B}_{\Psi }\left({\psi }_{RB}\left(x\right)\right|x).$$

Then, there exists $\eta \left(\lambda \right)\downarrow 0$ as $\lambda \to 0$ such that a Bayes rule ${\delta }_{\lambda ,\eta \left(\lambda \right)}\left(x\right),$ under the loss $L_{RB,\lambda ,\eta \left(\lambda \right)},$ converges to ${\psi }_{RB}\left(x\right)$ as $\lambda \to 0$ for all $x.$

Theorem 4 states that ${\psi }_{RB}\left(x\right)$ is a limit of Bayes rules. So, when $\Psi \left(\theta \right)=\theta$, we have the result that the MLE is a limit of the Bayes rule, and more generally, the MLE from an integrated likelihood is a limit of Bayes rules. The regularity conditions stated in Theorem 4 hold in many common statistical problems.

Now let ${\widehat{\psi }}_{\lambda }\left(x\right)$ be the relative belief estimate from the discretized problem, i.e., ${\widehat{\psi }}_{\lambda }\left(x\right)$ maximizes $R{B}_{\Psi }\left(B_{\lambda }\left(\psi \right)\right|x)$ as a function of $\psi \in {\Psi }_{\lambda }.$ The following is immediate from the proof of Theorem 4, Theorem 3, and Corollary 2:

Corollary 3. 

${\widehat{\psi }}_{\lambda }$ is admissible and Bayesian-unbiased for the discretized problem, and ${\widehat{\psi }}_{\lambda }\left(x\right)\to {\psi }_{RB}\left(x\right)$ as $\lambda \to 0$ for every $x.$

By similar arguments, an analog of Theorem 4 for ${\psi }_{MAP}$ can be established. In this case, a simpler development can be followed in certain situations by using the loss function $I_{B_{\lambda }^c\left(\psi \right)}(\Psi \left(\theta \right))$. For this, the posterior risk of $\delta$ in the discretized problem, is given by $1-{\Pi }_{\Psi }\left(B_{\lambda }\left(\delta \left(x\right)\right)\right|x)=1-{\pi }_{\Psi }\left({\delta }^{\prime }\left(x\right)\right|x){\nu }_{\Psi }\left(B_{\lambda }\left(\delta \left(x\right)\right)\right)$ for some ${\delta }^{\prime }\left(x\right)\in B_{\lambda }\left(\delta \left(x\right)\right).$ Now suppose $B_{\lambda }\left(\psi \right)$ is a cube centered at $\psi$ of edge length $\delta .$ Suppose that for each $ϵ>0$, there exists $\lambda \left(ϵ\right)>0$, such that, when $\left|\right|\psi -{\psi }_{MAP}\left(x\right)\left|\right|>\lambda \left(ϵ\right)$, then

$$p{i}_{\Psi }\left(\psi \right|x)<\underset{{\psi }^{\prime }\in B_{\lambda \left(ϵ\right)}\left({\psi }_{MAP}\left(x\right)\right)}{inf}{\pi }_{\Psi }\left({\psi }^{\prime }\right|x).$$

Since ${\nu }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)$ is constant, a Bayes rule ${\delta }_{\lambda \left(ϵ\right)}$ must then satisfy $\left|\right|{\delta }_{\lambda \left(ϵ\right)}\left(x\right)-{\psi }_{MAP}\left(x\right)\left|\right|<ϵ$. This proves that ${\psi }_{MAP}$ is a limit of the Bayes rules. By contrast, for the loss

$$I_{B_{\lambda }^c\left(\psi \right)}(\Psi \left(\theta \right))/{\Pi }_{\Psi }\left(B_{\lambda }(\Psi \left(\theta \right))\right),$$

the posterior risk of $\delta$ is given by,

$${\int }_{\Psi }{\left\{{\Pi }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)\right\}}^{-1}{\Pi }_{\Psi }\left(d\psi \right|x)-{\int }_{B_{\lambda }\left(\delta \left(x\right)\right)}{\left\{{\Pi }_{\Psi }\left(B_{\lambda }\left(\psi \right)\right)\right\}}^{-1}{\Pi }_{\Psi }\left(d\psi \right|x)$$

and the first term is generally unbounded unless $\Psi (\Theta )$ is compact.

We will now consider an important example.

Example 3. 

Regression

Suppose that $y=X\beta +e$, where $y\in R^n,X\in R^{n\times k}$ is fixed of rank $k,\beta \in R^{n\times k},$ and $e\sim N_n(0,{\sigma }^{2}I).$ To simplify the discussion, we will assume that ${\sigma }^2$ is known but this is not necessary. Let π be a prior density for $\beta .$ For every $\pi ,$ having observed $(X,y),$ then ${\beta }_{RB}\left(y\right)=b={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }y$, the MLE of $\beta .$

It is interesting to contrast this result with more standard Bayesian estimates such as MAP or the posterior mean. For example, suppose that $\beta \sim N_k(0,{\tau }^{2}I).$ Then the posterior distribution of $\beta$ is $N_k({\beta }_{post}\left(y\right),{\Sigma }_{post})$, where

$${\beta }_{post}\left(y\right)={\Sigma }_{post}\left({\sigma }^{-2}{X}^{\prime }Xb\right),{\Sigma }_{post}={({\tau }^{-2}I+{\sigma }^{-2}{X}^{\prime }X)}^{-1}$$

and note that ${\beta }_{MAP}\left(y\right)={\beta }_{post}\left(y\right).$ Writing the spectral decomposition of $X^{\prime }X$ as $X^{\prime }X=Q\Lambda Q^{\prime }$, we have that

$$\left|\right|{\beta }_{MAP}\left(y\right)\left|\right|=\left|\right|{(I+({\sigma }^2/{\tau }^2){\Lambda }^{-1})}^{-1}{Q}^{\prime }b\left|\right|.$$

Since $\left|\right|b\left|\right|=\left|\right|Q^{\prime }b\left|\right|$ and $1/(1+{\tau }^2{\lambda }_i/{\sigma }^2)<1$ for each $i,$ this implies that ${\beta }_{MAP}\left(y\right)$ shrinks the MLE toward the prior mean of $0.$ When the columns of X are orthonormal, then ${\beta }_{MAP}\left(y\right)=r{(1+r)}^{-1}b$, where $r={\tau }^2/{\sigma }^2$, and so the shrinkage is substantial unless ${\tau }^2$ is much larger than ${\sigma }^2.$ This shrinkage is often cited as a positive attribute of these estimates. Consider now the situation where the true value of β is some distance from the mean. In this case, it seems wrong to move β toward the prior mean; thus, it is not clear whether shrinking the MLE is necessarily a good thing, particularly as this requires giving up invariance.

Suppose that estimating the mean response $\psi =\Psi \left(\beta \right)=w^{\prime }\beta$ at w for the predictors is required. The prior distribution of ψ is $N(0,{\sigma }_{\psi }^2)=N(0,{\tau }^2{w}^{\prime }w)$ and the posterior distribution is $N({\psi }_{MAP}\left(y\right),{\sigma }_{\psi ,post}^2)=N(w^{\prime }{\beta }_{MAP}\left(y\right),w^{\prime }{\Sigma }_{post}\left(\beta \right)w).$ Note the following relationships,

$${\sigma }_{\psi }^2-{\sigma }_{\psi ,post}^2=w^{\prime }({\tau }^{2}I-{\Sigma }_{post})w={\tau }^2{w}^{\prime }{Q}^{\prime }(I-{(I+({\tau }^2/{\sigma }^2)\Lambda )}^{-1})Qw>0$$

since $1/(1+{\tau }^2{\lambda }_i/{\sigma }^2)<1$ for each $i.$ Therefore, maximizing the ratio of the posterior to prior densities leads to

$${\psi }_{RB}\left(y\right)={(1-{\sigma }_{\psi ,post}^2/{\sigma }_{\psi }^2)}^{-1}{\psi }_{MAP}\left(y\right).$$

(13)

Then ${\sigma }_{\psi }^2>{\sigma }_{\psi ,post}^2$ implies $|{\psi }_{RB}\left(y\right)|>|{\psi }_{MAP}\left(y\right)|.$ Note that when ${\sigma }_{\psi ,post}^2$ is much smaller than ${\sigma }_{\psi }^2,$ in other words, the posterior is much more concentrated than the prior, then ${\psi }_{RB}\left(y\right)$ and ${\psi }_{MAP}\left(y\right)$ are very similar. In general, ${\psi }_{RB}\left(y\right)$ is not equal to $w^{\prime }b,$ the plug-in MLE of $\psi ,$ although it is the MLE from the integrated likelihood. Moreover, ${\psi }_{RB}\left(y\right)\to w^{\prime }b$ as ${\tau }^2\to \infty$, and when X has orthonormal columns, ${\psi }_{RB}\left(y\right)=w^{\prime }b.$

Suppose predicting a response z at the predictor value $w\in R^k$ is required. When $\beta \sim N_k(0,{\tau }^{2}I)$, the prior distribution of z is $z\sim N(0,{\sigma }^2+{\tau }^2{w}^{\prime }w)=N(0,{\sigma }_z^2)$ and the posterior distribution is $N({\mu }_{post}\left(z\right),{\sigma }_{post}^2\left(z\right))$, where we have that

$${\mu }_{post}\left(z\right)=w^{\prime }{\beta }_{post}\left(y\right),{\sigma }_{post}^2\left(z\right)={\sigma }^2+w^{\prime }{\Sigma }_{post}w.$$

To obtain $z_{RB}\left(y\right)$, it is necessary to maximize the ratio of the posterior to the prior densities of z; this leads to

$$z_{RB}\left(y\right)={(1-{\sigma }_{post}^2\left(z\right)/{\sigma }_{prior}^2\left(z\right))}^{-1}{\mu }_{post}\left(z\right).$$

(14)

Note that ${\sigma }_z^2-{\sigma }_{post}^2\left(z\right)={\sigma }_z^2\left(w^{\prime }\beta \right)-{\sigma }_{post}^2\left(w^{\prime }\beta \right)>0$; thus, $|z_{RB}\left(y\right)|>|{\mu }_{post}\left(z\right)|$, and $z_{RB}$ is further from the prior mean than $z_{MAP}\left(y\right)={\mu }_{post}\left(z\right).$ Also, when ${\sigma }_{post}^2\left(z\right)$ is small, then $z_{RB}\left(y\right)$ and $z_{MAP}\left(y\right)$ are very similar. Finally, comparing (13) and (14), we have that

$$z_{RB}\left(y\right)=({\sigma }_{prior}^2\left(z\right)/{\sigma }_{post}^2\left(\psi \right))w^{\prime }{\psi }_{RB}\left(y\right)=(1+{\sigma }^2/{\tau }^2){\psi }_{RB}\left(y\right)$$

and so ${\psi }_{RB}\left(y\right)$ at w is more dispersed than the $z_{RB}\left(y\right)$ estimate of the mean at w; this makes good sense as we have to take into account the additional variation due to prediction. By contrast, $w_{MAP}\left(y\right)={\psi }_{MAP}\left(y\right).$

5. Credible Regions and Hypothesis Assessment

Recall that a $\gamma$-relative-belief credible region for $\psi =\Psi \left(\theta \right)$ is given by $C_{\Psi ,\gamma }\left(x\right)=\{\psi :R{B}_{\Psi }\left(\psi \right|x)\ge c_{\gamma }\left(x\right)\}$, where $c_{\gamma }\left(x\right)=sup\{c:{\Pi }_{\Psi }(R{B}_{\Psi }\left(\psi \right|x)\ge c|x)\ge \gamma \}.$ There is some arbitrariness in the choice of the greater than or equal sign to define the credible region as it could have been defined as $C_{\Psi ,\gamma }\left(x\right)=\{\psi :R{B}_{\Psi }\left(\psi \right|x)>c_{\gamma }\left(x\right)\}$, where $c_{\gamma }\left(x\right)=inf\{c:{\Pi }_{\Psi }(R{B}_{\Psi }\left(\psi \right|x)\le c|x)\le 1-\gamma \}.$ In the latter case, $c_{\gamma }\left(x\right)$ is the $(1-\gamma )$-th quantile of the posterior distribution of the relative belief ratio. This definition has some advantages as using this implies that the plausible region satisfies $P{l}_{\Psi }\left(x\right)=C_{\Psi ,\gamma }\left(x\right)$, where $\gamma ={\Pi }_{\Psi }\left(P{l}_{\Psi }\left(x\right)\right|x).$ Also, the strength of the evidence concerning the hypothesis $H_0:\Psi \left(\theta \right)={\psi }_0$ satisfies $St{r}_{\Psi }\left({\psi }_0\right|x)=1-{\Pi }_{\Psi }\left(C_{\Psi ,\gamma }\left(x\right)\right|x)$ where $\gamma =1-St{r}_{\Psi }\left({\psi }_0\right|x).$ The key point here is the close relationship between relative-belief credible regions, the plausible region, and the strength calculation. Thus, any decision-theoretic interpretation applicable to relative-belief credible regions also pertains to the plausible region and the strength of the evidence. Throughout this section, we will retain the definition for $C_{\Psi ,\gamma }\left(x\right)$ provided in Section 2.3.

Now consider the lowest posterior loss $\gamma$-credible regions that arise from the prior-based loss functions considered here.

Theorem 5. 

Suppose that ${\pi }_{\Psi }\left(\psi \right)>0$ for every $\psi \in \Psi (\Theta )$, where $\Psi (\Theta )$ is finite with ${\nu }_{\Psi }$ equal to the counting measure. Then $C_{\Psi ,\gamma }\left(x\right)$ is a γ-lowest posterior loss-credible region for the loss function $L_{RB}.$

Proof. 

From (2) and (6), the $\gamma$-lowest posterior loss-credible region is given by

$$D_{\gamma }\left(x\right)=\left\{\psi :R{B}_{\Psi }\left(\psi \right|x)\ge {\int }_{\Psi }R{B}_{\Psi }\left(\zeta \right|x){\nu }_{\Psi }\left(d\zeta \right)-d_{\gamma }\left(x\right)\right\}$$

and $d_{\gamma }\left(x\right)=sup\{d:{\Pi }_{\Psi }(\{\psi :r\left(\psi \right|x)\le d|x)\ge \gamma \}.$ As ${\int }_{\Psi }R{B}_{\Psi }\left(z\right|x){\nu }_{\Psi }\left(dz\right)$ is independent of $\psi$ it is clearly equivalent to define this region via $C_{\Psi ,\gamma }\left(x\right)=\left\{\psi :R{B}_{\Psi }\left(\psi \right|x)\ge c_{\gamma }\left(x\right)\right\},$ namely, $D_{\gamma }\left(x\right)=C_{\Psi ,\gamma }\left(x\right).$ □

Now consider the case where $\Psi$ is countable and we use the loss function $L_{RB,\eta }.$ Following the proof of Theorem 5, we see that a $\gamma$-lowest posterior loss region takes the form,

$$D_{\eta ,\gamma }\left(x\right)=\left\{\psi :{\pi }_{\Psi }\left(\psi \right|x)/max(\eta ,{\pi }_{\Psi }\left(\psi \right))\ge d_{\eta ,\gamma }\left(x\right)\right\},$$

where $d_{\eta ,\gamma }\left(x\right)=sup\{d:{\Pi }_{\Psi }\left(\{\psi :{\pi }_{\Psi }\left(\psi \right|x)/max(\eta ,{\pi }_{\Psi }\left(\psi \right))\ge d|x)\right\}\ge \gamma \}.$

Theorem 6. 

Suppose that ${\pi }_{\Psi }\left(\psi \right)>0$ for every $\psi \in \Psi ,$ the Ψ is countable with ${\nu }_{\Psi }$ equal to counting measure. For the loss function $L_{RB,\eta }$, $C_{\Psi ,\gamma }\left(x\right)\subset lim{inf}_{\eta \to 0}{D}_{\eta ,\gamma }\left(x\right)$ whenever γ is such that ${\Pi }_{\Psi }\left(C_{\Psi ,\gamma }\left(x\right)\right|x)=\gamma$, and $lim{sup}_{\eta \to 0}{D}_{\eta ,\gamma }\left(x\right)\subset C_{\Psi ,{\gamma }^{\prime }}\left(x\right)$ whenever ${\gamma }^{\prime }>\gamma$ and ${\Pi }_{\Psi }\left(C_{\Psi ,{\gamma }^{\prime }}\left(x\right)\right|x)={\gamma }^{\prime }.$

While Theorem 6 does not establish the exact convergence ${lim}_{\eta \to 0}{D}_{\eta ,\gamma }\left(x\right)=C_{\gamma }\left(x\right)$, it is likely that this does hold under quite general circumstances due to the discreteness. Theorem 6 shows that limit points of the class of sets $D_{\eta ,\gamma }\left(x\right)$ always contain $C_{\Psi ,\gamma }\left(x\right)$ and their posterior probability content differs from $\gamma$ by at most ${\gamma }^{\prime }-\gamma$, where ${\gamma }^{\prime }>\gamma$ is the next largest value for which we have exact content.

Now, consider the continuous case with a regular discretization. For $S^{*}\subset {\Psi }_{\lambda }=\{{\psi }_{\lambda }\left(\psi \right):{\psi }_{\lambda }\left(\psi \right)\in B_{\lambda }\left(\psi \right)\},$ namely, $S^{*}$ is a subset of a discretized version of $\Psi (\Theta ),$ we define the un-discretized version of $S^{*}$ to be $S={\cup }_{\psi \in S^{*}}{B}_{\lambda }\left(\psi \right).$ Now, let $C_{\Psi ,\lambda ,\gamma }^{*}\left(x\right)$ be the $\gamma$-relative belief region for the discretized problem and let $C_{\Psi ,\lambda ,\gamma }\left(x\right)$ be its un-discretized version. Note that in a continuous context, we will consider two sets as equal if they differ only by a set of measure 0 with respect to ${\Pi }_{\Psi }.$ The following result says that a $\gamma$-relative belief-credible region for the discretized problem, after un-discretizing, converges to the $\gamma$-relative belief region for the original problem.

Theorem 7. 

Suppose that ${\pi }_{\Psi }$ is positive and continuous, there is regular discretization of $\Psi (\Theta )$ and $R{B}_{\Psi }\left(\psi \right|x)$ has a continuous posterior distribution. Then, ${lim}_{\lambda \to 0}{C}_{\Psi ,\lambda ,\gamma }\left(x\right)=C_{\Psi ,\gamma }\left(x\right).$

While Theorem 7 is interesting in its own right, it can also be used to prove that relative belief regions are limits of the lowest posterior loss regions.

Let $D_{\eta ,\lambda ,\gamma }^{*}\left(x\right)$ be the $\gamma$-lowest posterior loss region obtained for the discretized problem using loss function (12), and let $D_{\eta ,\lambda ,\gamma }\left(x\right)$ be the un-discretized version.

Theorem 8. 

Suppose that ${\pi }_{\Psi }$ is positive and continuous, we have a regular discretization of Ψ, and $R{B}_{\Psi }\left(\psi \right|x)$ has a continuous posterior distribution. Then, we have that

$$C_{\Psi ,\gamma }\left(x\right)=\underset{\lambda \to 0}{lim}\underset{\eta \to 0}{liminf}{D}_{\Psi ,\gamma }\left(x\right)=\underset{\lambda \to 0}{lim}\underset{\eta \to 0}{limsup}{D}_{\Psi ,\gamma }\left(x\right).$$

In [18,19], additional properties of relative belief regions are developed. For example, it has been shown that a $\gamma$-relative belief region $C_{\Psi ,\gamma }\left(x\right)$ for $\psi$, satisfying ${\Pi }_{\Psi }\left(C_{\Psi ,\gamma }\left(x\right)\right|x)=\gamma$, minimizes ${\Pi }_{\Psi }\left(B\right)$ among all (measurable) subsets of $\Psi$ satisfying ${\Pi }_{\Psi }\left(B\right|x)\ge \gamma .$ So, a $\gamma$-relative belief region is the smallest among all $\gamma$-credible regions for $\psi$, where the size is measured using the prior measure. This property has several consequences. For example, the prior probability that a region $B\left(x\right)\subset \Psi (\Theta )$ contains a false value from the prior is given by ${\int }_{\Theta }{\int }_{\Psi }{F}_{\theta }(\psi \in B\left(x\right)){\Pi }_{\Psi }\left(d\psi \right)\Pi \left(d\theta \right)$, where a false value is a value of $\psi \sim {\Pi }_{\Psi }$ generated independently of $(\theta ,x)\sim {\Pi }_{\Psi }\times F_{\theta }.$ It can be demonstrated that a $\gamma$-relative belief region minimizes this probability among all $\gamma$-credible regions for $\psi$ and is always unbiased in the sense that the probability of covering a false value is bounded above by $\gamma .$ Furthermore, a $\gamma$-relative belief region maximizes the relative belief ratio ${\Pi }_{\Psi }\left(B\right|x)/{\Pi }_{\Psi }\left(B\right)$ and the Bayes factor ${\Pi }_{\Psi }\left(B\right|x){\Pi }_{\Psi }\left(B^c\right)/{\Pi }_{\Psi }\left(B^c\right|x){\Pi }_{\Psi }\left(B\right)$ among all regions $B\subset \Psi$ with ${\Pi }_{\Psi }\left(B\right)={\Pi }_{\Psi }\left(C_{\Psi ,\gamma }\left(x\right)\right|x).$

While the results in this section focus on obtaining credible regions for parameters, similar results can be proven for the construction of prediction regions.

6. Conclusions

Relative belief inferences are based on a clear characterization of statistical evidence and are closely related to likelihood inferences. This, along with their invariance and optimality properties, positions these as prime candidates for appropriate inferences in Bayesian contexts. This paper shows that relative belief inferences also arise naturally in a decision-theoretic formulation using loss functions based on the prior. So, relative belief inferences represent a degree of unification between the evidential and decision-theoretic approaches to deriving statistical inferences.