On Consistency of the Bayes Estimator of the Density

Abstract

Under mild conditions, strong consistency of the Bayes estimator of the density is proved. Moreover, the Bayes risk (for some common loss functions) of the Bayes estimator of the density (i.e., the posterior predictive density) goes to zero as the sample size goes to ∞. In passing, a similar result is obtained for the estimation of the sampling distribution.

1. Introduction

In a statistical context, since the expression the probability of an event A (usually denoted $P_{\theta }\left(A\right)$) depends on the unknown parameter, it is really a misuse of language. Before performing the experiment, this expression can be assigned a natural meaning from a Bayesian perspective as the prior predictive probability of A since it is the prior mean of the probabilities $P_{\theta }\left(A\right)$. However, in accordance with Bayesian philosophy, once the experiment has been carried out and the value $\omega$ has been observed, a more appropriate estimate of $P_{\theta }\left(A\right)$ is the posterior predictive probability given $\omega$ of A. The author has recently proved ([1]) that not only is this the Bayes estimator of $P_{\theta }\left(A\right)$ but that the posterior predictive distribution (resp. the posterior predictive density) is the Bayes estimator of the sampling distribution $P_{\theta }$ (resp. the density $p_{\theta }$) for the squared variation total (resp. the squared $L^1$) loss function in the Bayesian experiment corresponding to an n-sized sample of the unknown distribution. It should be noted that the loss functions considered derive in a natural way from the commonly used squared error loss function when estimating a real function of the parameter.

The posterior predictive distribution is the cornerstone of Predictive Inference, which seeks to make inferences about a new unknown observation from a preceding random sample (see [2,3]). With that idea in mind, it has also been used in other areas such as model selection, testing for discordancy, goodness of fit, perturbation analysis, and classification (see additional fields of application in [1,2,3,4,5]). Furthermore, in [1], it has been presented as a solution for the Bayesian density estimation problem, giving several examples to illustrate the results and, in particular, to calculate a posterior predictive density. [3] provide many other examples of determining the posterior predictive distribution. But in practice, explicit evaluation of the posterior predictive distribution may be cumbersome, and its simulation may become preferable. The aforementioned work of [3] also constitutes a good reference for such simulation methods, and hence for the computation of the Bayes estimators of the density and the sampling distribution.

We would refer to the references cited in [1] for other statistical uses of the posterior predictive distribution and some useful ways to calculate it.

In this communication, we shall explore the asymptotic behaviour of the posterior predictive density as the Bayes estimator of the density, showing its strong consistency and that the Bayes risk goes to 0 as n goes to ∞.

2. The Framework

Let

$$(\mathsf{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$$

be a Bayesian experiment (where Q denotes de prior distribution on the parameter space $(\mathrm{\Theta },\mathcal{T})$), and consider the infinite product Bayesian experiment

$$({\mathsf{\Omega }}^{\mathbb{N}},{\mathcal{A}}^{\mathbb{N}},\{P_{\theta }^{\mathbb{N}}:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$$

corresponding to an infinite sample of the unknown distribution $P_{\theta }$. Let us write

$$I(\omega ,\theta ):=\omega ,\hspace{1em}J(\omega ,\theta ):=\theta ,\hspace{1em}{I}_n(\omega ,\theta ):={\omega }_n\hspace{1em}\mathrm{and}\hspace{1em}{I}_{\left(n\right)}\left(\omega \right):={\omega }_{\left(n\right)}:=({\omega }_1,\dots ,{\omega }_n)$$

for integer n.

We suppose that $P^{\mathbb{N}}(\theta ,A):=P_{\theta }^{\mathbb{N}}\left(A\right)$ is a Markov kernel. Let

$${\mathrm{\Pi }}_{\mathbb{N}}:=P^{\mathbb{N}}\otimes Q$$

be the joint distribution of the parameter and the observations, i.e.,

$${\mathrm{\Pi }}_{\mathbb{N}}(A\times T)={\int }_T{P}_{\theta }^{\mathbb{N}}\left(A\right)dQ\left(\theta \right),\hspace{1em}A\in {\mathcal{A}}^{\mathbb{N}}, T\in \mathcal{T}.$$

As $Q:={\mathrm{\Pi }}_{\mathbb{N}}^J$ (i.e., the probability distribution of J with respect to ${\mathrm{\Pi }}_{\mathbb{N}}$), $P_{\theta }^{\mathbb{N}}$ is a version of the conditional distribution (regular conditional probability) ${\mathrm{\Pi }}_{\mathbb{N}}^{I|J=\theta }$. Analogously, $P_{\theta }^n$ is a version of the conditional distribution ${\mathrm{\Pi }}_{\mathbb{N}}^{I_{\left(n\right)}|J=\theta }$.

Let ${\beta }_{Q,\mathbb{N}}^{\ast }:={\mathrm{\Pi }}_{\mathbb{N}}^I$, the prior predictive distribution in ${\mathsf{\Omega }}^{\mathbb{N}}$ (so that ${\beta }_{Q,\mathbb{N}}^{\ast }\left(A\right)$ is the prior mean of the probabilities $P_{\theta }^{\mathbb{N}}\left(A\right)$). Similarly, write ${\beta }_{Q,n}^{\ast }:={\mathrm{\Pi }}_{\mathbb{N}}^{I_{\left(n\right)}}$ for the prior predictive distribution in ${\mathsf{\Omega }}^n$. So, the posterior distribution $P_{\omega ,\mathbb{N}}^{\ast }:={\mathrm{\Pi }}_{\mathbb{N}}^{J|I=\omega }$ given $\omega \in {\mathsf{\Omega }}^{\mathbb{N}}$ satisfies

$${\mathrm{\Pi }}_{\mathbb{N}}(A\times T)={\int }_T{P}_{\theta }^{\mathbb{N}}\left(A\right)dQ\left(\theta \right)={\int }_A{P}_{\omega ,\mathbb{N}}^{\ast }\left(T\right)d{\beta }_{Q,\mathbb{N}}^{\ast }\left(\omega \right),\hspace{1em}A\in {\mathcal{A}}^{\mathbb{N}},\hspace{1em}T\in \mathcal{T}.$$

Denote by $P_{{\omega }_{\left(n\right)},n}^{\ast }:={\mathrm{\Pi }}_{\mathbb{N}}^{J|I_{\left(n\right)}={\omega }_{\left(n\right)}}$ for ${\omega }_{\left(n\right)}\in {\mathsf{\Omega }}^n$ the posterior distribution given ${\omega }_{\left(n\right)}\in {\mathsf{\Omega }}^n$.

Write $P_{{\omega }_{\left(n\right)},n}^{\ast P}$ for the posterior predictive distribution given ${\omega }_{\left(n\right)}\in {\mathsf{\Omega }}^n$ defined for $A\in \mathcal{A}$ as

$$P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)={\int }_{\mathrm{\Theta }}{P}_{\theta }\left(A\right)d{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(\theta \right).$$

So $P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)$ is nothing but the posterior mean given ${\omega }_{\left(n\right)}\in {\mathsf{\Omega }}^n$ of the probabilities $P_{\theta }\left(A\right)$.

In the dominated case, we can assume without loss of generality that the dominating measure $\mu$ is a probability measure (because of (1) below). We write $p_{\theta }=d{P}_{\theta }/d\mu$. The likelihood function $\mathcal{L}(\omega ,\theta ):=p_{\theta }\left(\omega \right)$ is assumed to be $\mathcal{A}\times \mathcal{T}$-measurable.

We have that, for all n and every event $A\in \mathcal{A}$,

$$\begin{array}{c}\begin{array}{cc}\hfill P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)& ={\int }_{\mathrm{\Theta }}{P}_{\theta }\left(A\right)d{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(\theta \right)={\int }_{\mathrm{\Theta }}{\int }_A{p}_{\theta }\left({\omega }^{\prime }\right)d\mu \left({\omega }^{\prime }\right)d{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(\theta \right)\hfill \\ & ={\int }_A{\int }_{\mathrm{\Theta }}{p}_{\theta }\left({\omega }^{\prime }\right)d{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(\theta \right)d\mu \left({\omega }^{\prime }\right),\hfill \end{array}\end{array}$$

which proves that

$$p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right):={\int }_{\mathrm{\Theta }}{p}_{\theta }\left({\omega }^{\prime }\right)d{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(\theta \right)$$

is a $\mu$-density of $P_{{\omega }_{\left(n\right)},n}^{\ast P}$ that we recognize as the posterior predictive density on $\mathsf{\Omega }$ given ${\omega }_{\left(n\right)}$.

In the same way,

$$p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right):={\int }_{\mathrm{\Theta }}{p}_{\theta }\left({\omega }^{\prime }\right)d{P}_{\omega ,\mathbb{N}}^{\ast }\left(\theta \right)$$

is a $\mu$-density of $P_{\omega ,\mathbb{N}}^{\ast P}$, the posterior predictive density on $\mathsf{\Omega }$ given $\omega \in {\mathsf{\Omega }}^{\mathbb{N}}$.

In the following, we will assume the following additional regularity conditions:

(i)

$(\mathsf{\Omega },\mathcal{A})$ is a standard Borel space;

(ii)

$\mathrm{\Theta }$ is a Borel subset of a Polish space and $\mathcal{T}$ is its Borel $\sigma$-field;

(iii)

$\{P_{\theta }:\theta \in \mathrm{\Theta }\}$ is identifiable.

According to [1], the posterior predictive distribution $P_{{\omega }_{\left(n\right)},n}^{\ast P}$ (resp. the posterior predictive density $p_{{\omega }_{\left(n\right)},n}^{\ast P}$) is the Bayes estimator of the sampling distribution $P_{\theta }$ (resp. the density $p_{\theta }$) for the squared variation total (resp. the squared $L^1$) loss function in the product experiment $({\mathsf{\Omega }}^n,{\mathcal{A}}^n,\{P_{\theta }^n:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$. Analogously, the posterior predictive distribution $P_{\omega ,\mathbb{N}}^{\ast P}$ (resp. the posterior predictive density $p_{\omega ,\mathbb{N}}^{\ast P}$) is the Bayes estimator of the sampling distribution $P_{\theta }$ (resp. the density $p_{\theta }$) for the squared variation total (resp. the squared $L^1$) loss function in the product experiment $({\mathsf{\Omega }}^{\mathbb{N}},{\mathcal{A}}^{\mathbb{N}},\{P_{\theta }^{\mathbb{N}}:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$.

As a particular case of a well known result about the total variation distance between two probability measures and the $L^1$-distance between their densities, we have that

$$\underset{A\in \mathcal{A}}{\sup }\left|P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)-P_{\theta }\left(A\right)\right|=\frac{1}{2}{\int }_{\mathsf{\Omega }}\left|p_{{\omega }_{\left(n\right)},n}^{\ast P}-p_{\theta }\right|d\mu .$$

(1)

3. The Main Result

We ask whether the Bayes risk of the Bayes estimator $P_{{\omega }_{\left(n\right)},n}^{\ast P}$ of the sampling distribution $P_{\theta }$ goes to zero when $n\to \infty$, i.e., whether

$$\underset{n}{\lim }{\int }_{{\mathsf{\Omega }}^{\mathbb{N}}\times \mathrm{\Theta }}\underset{A\in \mathcal{A}}{\sup }{\left|P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)-P_{\theta }\left(A\right)\right|}^{2}d{\mathrm{\Pi }}_{\mathbb{N}}(\omega ,\theta )=0.$$

In terms of densities, the question is whether the Bayes risk of the Bayes estimator $p_{{\omega }_{\left(n\right)},n}^{\ast P}$ of the density $p_{\theta }$ goes to zero when $n\to \infty$, i.e., whether

$$\underset{n}{\lim }{\int }_{{\mathsf{\Omega }}^{\mathbb{N}}\times \mathrm{\Theta }}{\left({\int }_{\mathsf{\Omega }}\left|p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)-p_{\theta }\left({\omega }^{\prime }\right)\right|d\mu \left({\omega }^{\prime }\right)\right)}^{2}d{\mathrm{\Pi }}_{\mathbb{N}}(\omega ,\theta )=0.$$

Let us consider the auxiliary Bayesian experiment

$$(\mathsf{\Omega }\times {\mathsf{\Omega }}^{\mathbb{N}},\mathcal{A}\times {\mathcal{A}}^{\mathbb{N}},\{\mu \times P_{\theta }^{\mathbb{N}}:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\}).$$

For ${\omega }^{\prime }\in \mathsf{\Omega }$, $\omega \in {\mathsf{\Omega }}^n$ and $\theta \in \mathrm{\Theta }$, we will continue to write $I({\omega }^{\prime },\omega ,\theta )=\omega$ and $J({\omega }^{\prime },\omega ,\theta )=\theta$, and now we write $I^{\prime }({\omega }^{\prime },\omega ,\theta )={\omega }^{\prime }$.

The new prior predictive distribution is $\mu \times {\beta }_{Q,n}^{\ast }$ since

$${(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})}^{(I^{\prime },I_{\left(n\right)})}(A^{\prime }\times A_{\left(n\right)})=\mu \left(A^{\prime }\right)·{\beta }_{Q,n}^{\ast }\left(A_{\left(n\right)}\right)=(\mu \times {\beta }_{Q,n}^{\ast })(A^{\prime }\times A_{\left(n\right)}).$$

To compute the new posterior distributions, notice that

$$\begin{array}{c}(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})(A^{\prime }\times I_{\left(n\right)}^{-1}\left(A_{\left(n\right)}\right)\times T)=\\ {\int }_{A^{\prime }\times I_{\left(n\right)}^{-1}\left(A_{\left(n\right)}\right)}{(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})}^{J|(I^{\prime },I_{\left(n\right)})=({\omega }^{\prime },{\omega }_{\left(n\right)})}\left(T\right)d{(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})}^{(I^{\prime },I_{\left(n\right)})}({\omega }^{\prime },{\omega }_{\left(n\right)}).\end{array}$$

On the other hand,

$$\begin{array}{c}(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})(A^{\prime }\times I_{\left(n\right)}^{-1}\left(A_{\left(n\right)}\right)\times T)=\mu \left(A^{\prime }\right)·{\mathrm{\Pi }}_{\mathbb{N}}(I_{\left(n\right)}^{-1}\left(A_{\left(n\right)}\right)\times T)=\\ \mu \left(A^{\prime }\right)·{\int }_{A_{\left(n\right)}}{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(T\right)d{\beta }_{Q,n}^{\ast }\left({\omega }_{\left(n\right)}\right)={\int }_{A^{\prime }\times A_{\left(n\right)}}{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left(T\right)d(\mu \times {\beta }_{Q,n}^{\ast })({\omega }^{\prime },{\omega }_{\left(n\right)}).\end{array}$$

So,

$$P_{{\omega }_{\left(n\right)},n}^{\ast }={(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})}^{J|(I^{\prime },I_{\left(n\right)})=({\omega }^{\prime },{\omega }_{\left(n\right)})}.$$

It follows that if $f\in L^1\left(Q\right)$ then

$$E_{P_{{\omega }_{\left(n\right)},n}^{\ast }}\left(f\right)=E_{\mu \times {\mathrm{\Pi }}_{\mathbb{N}}}[f\mid (I^{\prime },I_{\left(n\right)})=({\omega }^{\prime },{\omega }_{\left(n\right)})].$$

when ${\mathcal{A}}_{\left(n\right)}^{\prime }:={(I^{\prime },I_{\left(n\right)})}^{-1}(\mathcal{A}\times {\mathcal{A}}^n)$, we have that ${\left({\mathcal{A}}_{\left(n\right)}^{\prime }\right)}_n$ is an increasing sequence of sub-$\sigma$-fields of $\mathcal{A}\times {\mathcal{A}}^{\mathbb{N}}$ such that $\mathcal{A}\times {\mathcal{A}}^{\mathbb{N}}=\sigma \left({\cup }_n{\mathcal{A}}_{\left(n\right)}^{\prime }\right)$. According to the martingale convergence theorem of Lévy, if Y is $(\mathcal{A}\times {\mathcal{A}}^{\mathbb{N}}\times \mathcal{T})$-measurable and $\mu \times {\mathrm{\Pi }}_{\mathbb{N}}$-integrable then

$$E_{\mu \times {\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|{\mathcal{A}}_{\left(n\right)}^{\prime })$$

converges $(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})$-a.e. and in $L^1(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})$ to $Y=E_{\mu \times {\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|{\mathcal{A}}^{\prime }\times {\mathcal{A}}^{\mathbb{N}})$.

Let us consider the $\mu \times {\mathrm{\Pi }}_{\mathbb{N}}$-integrable function

$$Y({\omega }^{\prime },\omega ,\theta ):=p_{\theta }\left({\omega }^{\prime }\right).$$

We shall see that

$$p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)=E_{\mu \times {\mathrm{\Pi }}_{\mathbb{N}}}(Y\mid (I^{\prime },I)=({\omega }^{\prime },\omega )).$$

(2)

Indeed, given $A^{\prime }\in \mathcal{A}$ and $A\in {\mathcal{A}}^{\mathbb{N}}$, we have that

$$\begin{array}{c}\begin{array}{cc}\hfill {\int }_{{(I^{\prime },I)}^{-1}(A^{\prime }\times A)}& p_{\theta }\left({\omega }^{\prime }\right)d(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})({\omega }^{\prime },\omega ,\theta )={\int }_A{\int }_{\mathrm{\Theta }}{\int }_{A^{\prime }}{p}_{\theta }\left({\omega }^{\prime }\right)d\mu \left({\omega }^{\prime }\right)d{P}_{\omega ,\mathbb{N}}^{\ast }\left(\theta \right)d{\beta }_{Q,\mathbb{N}}^{\ast }\left(\omega \right)\hfill \\ & ={\int }_A{\int }_{\mathrm{\Theta }}{P}_{\theta }\left(A^{\prime }\right)d{P}_{\omega ,\mathbb{N}}^{\ast }\left(\theta \right)d{\beta }_{Q,\mathbb{N}}\left(\omega \right)={\int }_A{P}_{\omega ,\mathbb{N}}^{\ast P}\left(A^{\prime }\right)d{\beta }_{Q,\mathbb{N}}^{\ast }\left(\omega \right)\hfill \\ & ={\int }_{A^{\prime }}{\int }_A{p}_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)d\mu \left({\omega }^{\prime }\right)d{\beta }_{Q,\mathbb{N}}^{\ast }\left(\omega \right)={\int }_{A^{\prime }\times A}{p}_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)d{(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})}^{(I^{\prime },I)}({\omega }^{\prime },\omega ),\hfill \end{array}\end{array}$$

which proves (2).

Analogously, it can be shown that

$$p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=E_{\mu \times {\mathrm{\Pi }}_{\mathbb{N}}}(Y\mid (I^{\prime },I_{\left(n\right)})=({\omega }^{\prime },{\omega }_{\left(n\right)})).$$

(3)

Hence, it follows from the aforementioned theorem of Lévy that

$$\underset{n}{\lim }{p}_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right),\hspace{1em}(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})-\mathrm{a}.\mathrm{e}.$$

(4)

and

$$\underset{n}{\lim }{\int }_{\mathsf{\Omega }\times {\mathsf{\Omega }}^{\mathbb{N}}\times \mathrm{\Theta }}\left|p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)-p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)\right|d(\mu \times {\mathrm{\Pi }}_{\mathbb{N}})({\omega }^{\prime },\omega ,\theta )=0,$$

i.e.,

$$\underset{n}{\lim }{\int }_{{\mathsf{\Omega }}^{\mathbb{N}}\times \mathrm{\Theta }}{\int }_{\mathsf{\Omega }}\left|p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)-p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)\right|d\mu \left({\omega }^{\prime }\right)d{\mathrm{\Pi }}_{\mathbb{N}}(\omega ,\theta )=0.$$

(5)

On the other hand, as a consequence of a known theorem of Doob (see Theorem 6.9 and Proposition 6.10 of [4], pp. 129, 130, we have that, for every ${\omega }^{\prime }\in \mathsf{\Omega }$,

$$\underset{n}{\lim }{\int }_{\mathrm{\Theta }}{p}_{{\theta }^{\prime }}\left({\omega }^{\prime }\right)d{P}_{{\omega }_{\left(n\right)},n}^{\ast }\left({\theta }^{\prime }\right)=p_{\theta }\left({\omega }^{\prime }\right),\hspace{1em}{P}_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

for Q-almost every $\theta$. Hence

$$\underset{n}{\lim }{p}_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=p_{\theta }\left({\omega }^{\prime }\right),\hspace{1em}{P}_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

for Q-almost every $\theta$, i.e., given ${\omega }^{\prime }\in \mathsf{\Omega }$ there exists $T_{{\omega }^{\prime }}\in \mathcal{T}$ such that $Q\left(T_{{\omega }^{\prime }}\right)=0$ and, $\forall \theta \notin T_{{\omega }^{\prime }}$,

$$\underset{n}{\lim }{p}_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=p_{\theta }\left({\omega }^{\prime }\right),\hspace{1em}{P}_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

So, for $\theta \notin T_{{\omega }^{\prime }}$, there exists $N_{\theta ,{\omega }^{\prime }}\in {\mathcal{A}}^{\mathbb{N}}$ such that $P_{\theta }^{\mathbb{N}}\left(N_{\theta ,{\omega }^{\prime }}\right)=0$ and

$$\underset{n}{\lim }{p}_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=p_{\theta }\left({\omega }^{\prime }\right),\hspace{1em}\forall \omega \notin N_{\theta ,{\omega }^{\prime }},\forall \theta \notin T_{{\omega }^{\prime }},\forall {\omega }^{\prime }\in \mathsf{\Omega }.$$

In particular,

$$\underset{n}{\lim }{p}_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=p_{\theta }\left({\omega }^{\prime }\right),\hspace{1em}\mu \times P_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

(6)

From (4) and (6), it follows that $p_{\theta }\left({\omega }^{\prime }\right)=p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)$, $\mu \times P_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$

From this and (5), it follows that

$$\underset{n}{\lim }{\int }_{{\mathsf{\Omega }}^{\mathbb{N}}\times \mathrm{\Theta }}{\int }_{\mathsf{\Omega }}\left|p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)-p_{\theta }\left({\omega }^{\prime }\right)\right|d\mu \left({\omega }^{\prime }\right)d{\mathrm{\Pi }}_{\mathbb{N}}(\omega ,\theta )=0,$$

i.e., the risk of the Bayes estimator of the density for the $L^1$ loss function goes to 0 when $n\to \infty$.

It follows from this and (1) that

$$\underset{n}{\lim }{\int }_{{\mathsf{\Omega }}^{\mathbb{N}}\times \mathrm{\Theta }}\underset{A\in \mathcal{A}}{\sup }\left|P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)-P_{\theta }\left(A\right)\right|d{\mathrm{\Pi }}_{\mathbb{N}}(\omega ,\theta )=0,$$

i.e., the risk of the Bayes estimator of the sampling distribution $P_{\theta }$ for the variation total loss function goes to 0 when $n\to \infty$.

We ask whether these results remain true for the squared versions of the loss functions. The answer is affirmative because of the following general result: Let $\left(X_n\right)$ be a sequence of r.r.v. on a probability space $(\mathsf{\Omega },\mathcal{A},P)$ such that ${\lim }_n\int \left|X_n\right|dP=0$. If there exists $a>0$ such that $|X_n|\le a$, for all n, then ${\lim }_n\int {\left|X_n\right|}^{2}dP=0$ because

$$0\le \int |X_n{|}^{2}dP\le a\int \left|X_n\right|dP{\to }_{n}0.$$

In our case $a=2$, $P:={\mathrm{\Pi }}_{\mathbb{N}}$ and

$$X_n:={\int }_{\mathsf{\Omega }}\left|p_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)-p_{\omega ,\mathbb{N}}^{\ast P}\left({\omega }^{\prime }\right)\right|d\mu \left({\omega }^{\prime }\right),\hspace{1em}\mathrm{or}\hspace{1em}{X}_n:=\underset{A\in \mathcal{A}}{\sup }\left|P_{{\omega }_{\left(n\right)},n}^{\ast P}\left(A\right)-P_{\theta }\left(A\right)\right|.$$

So, we have proved the following result.

Theorem 1.

Let $(\mathsf{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$ be a Bayesian experiment dominated by a σ-finite measure μ. Let us assume that $(\mathsf{\Omega },\mathcal{A})$ is a standard Borel space, and that Θ is a Borel subset of a Polish space and $\mathcal{T}$ is its Borel σ-field. Assume also that the likelihood function $\mathcal{L}(\omega ,\theta ):=p_{\theta }\left(\omega \right)=\frac{d{P}_{\theta }}{d\mu }\left(\omega \right)$ is $\mathcal{A}\times \mathcal{T}$-measurable and the family $\{P_{\theta }:\theta \in \mathrm{\Theta }\}$ is identifiable. Then:

(a) 

The posterior predictive density $p_{{\omega }_{\left(n\right)},n}^{\ast P}$ is the Bayes estimator of the density $p_{\theta }$ in the product experiment $({\mathsf{\Omega }}^n,{\mathcal{A}}^n,\{P_{\theta }^n:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$ for the squared $L^1$ loss function. Moreover the risk function converges to 0 for both the $L^1$ loss function and the squared $L^1$ loss function.

(b) 

The posterior predictive distribution $P_{{\omega }_{\left(n\right)},n}^{\ast P}$ is the Bayes estimator of the sampling distribution $P_{\theta }$ in the product experiment $({\mathsf{\Omega }}^n,{\mathcal{A}}^n,\{P_{\theta }^n:\theta \in (\mathrm{\Theta },\mathcal{T},Q)\})$ for the squared variation total loss function. Moreover the risk function converges to 0 for both the variation total loss function and the squared variation total loss function.

(c) 

The posterior predictive density is a strongly consistent estimator of the density $p_{\theta }$, i.e.,

$$\underset{n}{\lim }{p}_{{\omega }_{\left(n\right)},n}^{\ast P}\left({\omega }^{\prime }\right)=p_{\theta }\left({\omega }^{\prime }\right),\hspace{1em}\mu \times P_{\theta }^{\mathbb{N}}-a.e.$$

for Q-almost every $\theta \in \mathrm{\Theta }$.