Optimal Bayesian Estimation of a Regression Curve, a Conditional Density, and a Conditional Distribution

Abstract

In this paper, several related estimation problems are addressed from a Bayesian point of view, and optimal estimators are obtained for each of them when some natural loss functions are considered. The problems considered are the estimation of a regression curve, a conditional distribution function, a conditional density, and even the conditional distribution itself. These problems are posed in a sufficiently general framework to cover continuous and discrete, univariate and multivariate, and parametric and nonparametric cases, without the need to use a specific prior distribution. The loss functions considered come naturally from the quadratic error loss function commonly used in estimating a real function of the unknown parameter. The cornerstone of these Bayes estimators is the posterior predictive distribution. Some examples are provided to illustrate the results.

1. Introduction

In Statistics, the expression the probability of an event A (written $P_{\theta }\left(A\right)$) is, in general, ambiguous, as it depends on the unknown parameter $\theta$. Before conducting the experiment, a Bayesian statistician, provided with the prior distribution, possesses a natural candidate—the prior predictive probability of A—since it is the prior mean of the probabilities of A. However, in accordance with Bayesian philosophy, after the experiment has been performed and the data $\omega$ observed, a reasonable estimation is the posterior predictive probability of A given $\omega$ because it is the posterior mean of the probabilities of A given $\omega$. It can be shown that not only is this the Bayes estimator of the probability $P_{\theta }\left(A\right)$ of A for the squared error loss function but also that the posterior predictive distribution is the Bayes estimator of the sampling probability distribution $P_{\theta }$ for the squared variation total loss function and that the posterior predictive density is the Bayes estimator of its density for the $L^1$-squared loss function. Note that these loss functions should be considered natural in the sense that they are derived directly from the quadratic error loss function commonly used in the estimation of a real function of the parameter. Ref. [1] contains precise statements and proofs of these results, which are nothing but a functional generalization of Theorem 1.1 (more specifically of its Corollary 1.2.(a)) of [2], p. 228, which yields the Bayes estimator of a real function of the parameter for the squared error loss function.

This communication addresses the estimation of a regression curve and some related problems, such as the estimation of a conditional density or a conditional distribution function or even the conditional distribution itself from a Bayesian perspective. It should, therefore, be considered as the conditional counterpart of [1], and the results to be presented below as the functional extension of [2], Theorem 1.1, for the conditional case. Thus, it is unsurprising that the posterior predictive distribution is the cornerstone for the estimation problems to be discussed below. Some examples illustrating the results will be presented in Section 7. See [1] and the references therein for other examples of the determination of the posterior predictive distribution. In practice, however, the explicit evaluation of the posterior predictive distribution could well be cumbersome, and its simulation may become preferable. Ref. [3] is a good reference for such simulation methods, and hence, for the computation of the Bayes estimators of the conditional density and the regression curve.

The posterior predictive distribution has been presented as the base of Predictive Inference, which seeks to make inferences about a new unknown observation from the previous random sample in contrast with the greater emphasis that statistical inference, since its mathematical foundations in the early twentieth century, puts on parameter estimation and contrast (see [4] or [3]). With that idea in mind, it has also been used in other areas, such as model selection, testing for discordancy, goodness of fit, perturbation analysis, or classification (see additional fields of application in [4,5]), but never as a possible solution for the Bayesian problems of estimating an unconditional or conditional density. The reader is referred to the references within [1] for other uses of the posterior predictive distribution in Bayesian statistics.

To summarize the contribution of this work, I want to emphasize that the problems of estimating a density (conditional or not) or a regression curve are of central importance in Nonparametric Inference and Functional Data Analysis (for example, see [6] or [7], and the references they contain). Although nobody expects an optimal result for these problems in a frequentist environment, this article together with [1] produces optimal solutions for them in a Bayesian framework. The reader should note that these are not just theorems of existence and uniqueness of solutions; rather, on the contrary, the results obtained explicit formulas for the solutions based on the posterior predictive distribution. Note also that there is enough literature on how to calculate it, exactly or approximately.

Section 2 sets out the proper statistical framework for tackling the problems, i.e., the proper Bayesian experiment (conceived also as a probability space along the lines suggested by [8], for example.)

Section 3 deals with the problem of Bayesian estimation of a conditional distribution when the squared total variation loss function is used and Theorem 1 gives the Bayes estimator in terms of the posterior predictive distribution.

Section 4 takes advantage of Theorem 1 to solve the problem of the Bayesian estimation of a conditional density using the $L^1$-squared loss function, obtaining the Bayes estimator of the conditional density (see Theorem 2).

Section 5 and Section 6 deal with the problems of Bayesian estimation of a conditional distribution function and a regression curve in the real case. Theorems 3 and 4 yieds the solutions.

Section 7 provide some examples to illustrate the application of all these theorems.

For ease of reading, the proofs are postponed until Section 8. This is followed by an appendix (Appendix A) explaining the notation and concepts used in the text.

We shall place ourselves from this point onwards in a general framework for Bayesian inference, as described in [9].

2. The Framework

Let $(\Omega ,\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})$ be a Bayesian statistical experiment, and $X_i:(\Omega ,\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})\to ({\Omega }_i,{\mathcal{A}}_i)$, $i=1,2$, two statistics. Consider the Bayesian experiment image of $(X_1,X_2)$:

$$({\Omega }_1\times {\Omega }_2,{\mathcal{A}}_1\times {\mathcal{A}}_2,\{P_{\theta }^{(X_1,X_2)}:\theta \in (\Theta ,\mathcal{T},Q)\}).$$

In what follows, we shall assume that $P^{(X_1,X_2)}(\theta ,A_{12}):=P_{\theta }^{(X_1,X_2)}\left(A_{12}\right)$, $\theta \in \Theta ,A_{12}\in {\mathcal{A}}_1\times {\mathcal{A}}_2$ is a Markov kernel and write $R_{\theta }=P_{\theta }^{(X_1,X_2)}$.

The Bayesian experiment corresponding to a sample of size n of the joint distribution of $(X_1,X_2)$ is:

$$\left({({\Omega }_1\times {\Omega }_2)}^n,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n,\left\{R_{\theta }^n:\theta \in (\Theta ,\mathcal{T},Q)\right\}\right).$$

(1)

We write $R^n(\theta ,A_{12,n}^{\prime })=R_{\theta }^n\left(A_{12,n}^{\prime }\right)$ for $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and

$${\Pi }_{12,n}:=Q\otimes R^n$$

for the joint distribution of the parameter and the sample:

$${\Pi }_{12,n}(A_{12,n}^{\prime }\times T)={\int }_T{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right),A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n,T\in \mathcal{T}.$$

(2)

The corresponding prior predictive distribution ${\beta }_{12,n}^{*}$ is:

$${\beta }_{12,n}^{*}\left(A_{12,n}^{\prime }\right)={\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right),A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n.$$

(3)

The posterior distribution is a Markov kernel:

$$R_n^{*}:({({\Omega }_1\times {\Omega }_2)}^n,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n)\succ ⟶(\Theta ,\mathcal{T})$$

such that, for all $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and $T\in \mathcal{T}$,

$${\Pi }_{12,n}(A_{12,n}^{\prime }\times T)={\int }_T{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right)={\int }_{A_{12,n}^{\prime }}{R}_n^{*}(x^{\prime },T)d{\beta }_{12,n}^{*}\left(x^{\prime }\right).$$

Let us write $R_{n,x^{\prime }}^{*}\left(T\right):=R_n^{*}(x^{\prime },T)$.

The posterior predictive distribution on ${\mathcal{A}}_1\times {\mathcal{A}}_2$ is the Markov kernel:

$${R_n^{*}}^R:({({\Omega }_1\times {\Omega }_2)}^n,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n)\succ ⟶({\Omega }_1\times {\Omega }_2,{\mathcal{A}}_1\times {\mathcal{A}}_2)$$

defined, for $x^{\prime }\in {({\Omega }_1\times {\Omega }_2)}^n$, by:

$${R_n^{*}}^R(x^{\prime },A_{12}):={\int }_{\Theta }{R}_{\theta }\left(A_{12}\right)d{R}_{n,x^{\prime }}^{*}\left(\theta \right).$$

It follows that, with obvious notation:

$${\int }_{{\Omega }_1\times {\Omega }_2}f\left(x\right)d{R_{n,x^{\prime }}^{*}}^R\left(x\right)={\int }_{\Theta }{\int }_{{\Omega }_1\times {\Omega }_2}f\left(x\right)d{R}_{\theta }\left(x\right)d{R}_{n,x^{\prime }}^{*}\left(\theta \right)$$

for any non-negative or integrable real random variable (r.r.v. for short) f.

We can also consider the posterior predictive distribution on ${({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ defined as the Markov kernel:

$${R_n^{*}}^{R^n}:({({\Omega }_1\times {\Omega }_2)}^n,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n)\succ ⟶({({\Omega }_1\times {\Omega }_2)}^n,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n)$$

such that:

$${R_n^{*}}^{R^n}(x^{\prime },A_{12,n}^{\prime }):={\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)d{R}_{n,x^{\prime }}^{*}\left(\theta \right).$$

According to Theorem 1 of [1], this is the Bayes estimator of the distribution $R_{\theta }^n$ for the squared total variation function:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^n\times \Theta }\underset{A_{12,n}\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n}{sup}{\left|{R_{n,x^{\prime }}^{*}}^{R^n}\left(A_{12,n}\right)-R_{\theta }^n\left(A_{12,n}\right)\right|}^{2}d{\Pi }_{12,n}(x^{\prime },\theta )\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^n\times \Theta }\underset{A_{12,n}\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n}{sup}{\left|M(x^{\prime },A_{12,n})-R_{\theta }^n\left(A_{12,n}\right)\right|}^{2}d{\Pi }_{12,n}(x^{\prime },\theta ),\end{array}$$

for every Markov kernel $M:{({\Omega }_1\times {\Omega }_2,{\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\succ ⟶{({\Omega }_1\times {\Omega }_2,{\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$.

It can be readily checked that:

$${\left[{R_{n,x^{\prime }}^{*}}^{R^n}\right]}^{{\pi }_1^{\prime }}={R_{n,x^{\prime }}^{*}}^R,$$

where ${\pi }_1^{\prime }\left(x^{\prime }\right):=x_1^{\prime }:=(x_{11}^{\prime },x_{21}^{\prime })$ for $x^{\prime }\in {({\Omega }_1\times {\Omega }_2)}^n$. Then, Theorem 2 of [1] shows that:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^n\times \Theta }\underset{A_{12}\in {\mathcal{A}}_1\times {\mathcal{A}}_2}{sup}{\left|{R_{n,x^{\prime }}^{*}}^R\left(A_{12}\right)-R_{\theta }\left(A_{12}\right)\right|}^{2}d{\Pi }_{12,n}(x^{\prime },\theta )\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^n\times \Theta }\underset{A_{12}\in {\mathcal{A}}_1\times {\mathcal{A}}_2}{sup}{\left|M(x^{\prime },A_{12})-R_{\theta }\left(A_{12}\right)\right|}^{2}d{\Pi }_{12,n}(x^{\prime },\theta ),\end{array}$$

for every Markov kernel $M:{({\Omega }_1\times {\Omega }_2,{\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\succ ⟶({\Omega }_1\times {\Omega }_2,{\mathcal{A}}_1\times {\mathcal{A}}_2)$.

We introduce some notation for $(x^{\prime },x,\theta )\in {({\Omega }_1\times {\Omega }_2)}^n\times ({\Omega }_1\times {\Omega }_2)\times \Theta$:

$$\begin{array}{c}{\pi }^{\prime }(x^{\prime },x,\theta ):=x^{\prime },{\pi }_i^{\prime }(x^{\prime },x,\theta ):=x_i^{\prime }:=(x_{i1}^{\prime },x_{i2}^{\prime }),1\le i\le n,\\ \pi (x^{\prime },x,\theta ):=x,{\pi }_i(x^{\prime },x,\theta ):=x_i,i=1,2,\\ q(x^{\prime },x,\theta ):=\theta .\end{array}$$

Let us consider the probability space:

$$\begin{array}{c}({({\Omega }_1\times {\Omega }_2)}^n\times ({\Omega }_1\times {\Omega }_2)\times \Theta ,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times ({\mathcal{A}}_1\times {\mathcal{A}}_2)\times \mathcal{T},{\Pi }_n),\end{array}$$

(4)

where:

$$\begin{array}{c}{\Pi }_n(A_{12,n}^{\prime }\times A_{12}\times T)={\int }_T{R}_{\theta }\left(A_{12}\right)R_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right)\end{array}$$

(5)

when $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$, $A_{12}\in {\mathcal{A}}_1\times {\mathcal{A}}_2$ and $T\in \mathcal{T}$.

Thus, for a r.r.v. f on: $({({\Omega }_1\times {\Omega }_2)}^n\times ({\Omega }_1\times {\Omega }_2)\times \Theta ,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times ({\mathcal{A}}_1\times {\mathcal{A}}_2)\times \mathcal{T})$,

$$\begin{array}{c}\int fd{\Pi }_n={\int }_{\Theta }{\int }_{{({\Omega }_1\times {\Omega }_2)}^n}{\int }_{({\Omega }_1\times {\Omega }_2)}f(x^{\prime },x,\theta )d{R}_{\theta }\left(x\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)\end{array}$$

(6)

provided that the integral exists. Moreover, for a r.r.v. h on $(({\Omega }_1\times {\Omega }_2)\times \Theta ,({\mathcal{A}}_1\times {\mathcal{A}}_2)\times \mathcal{T})$:

$$\int hd{\Pi }_n={\int }_{\Theta }{\int }_{{\Omega }_1\times {\Omega }_2}h(x,\theta )d{R}_{\theta }\left(x\right)dQ\left(\theta \right)={\int }_{{\Omega }_1\times {\Omega }_2}{\int }_{\Theta }h(x,\theta )d{R}_{1,x}^{*}\left(\theta \right)d{\beta }_{12,1}^{*}\left(x\right).$$

The following proposition is straightforward.

Proposition 1.

Given $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$, $A_{12}\in {\mathcal{A}}_1\times {\mathcal{A}}_2$ and $T\in \mathcal{T}$, we have that:

$$\begin{array}{c}{\Pi }_n^{({\pi }^{\prime },q)}(A_{12.n}^{\prime }\times T)={\Pi }_{12,n}(A_{12,n}^{\prime }\times T)={\int }_T{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right)={\int }_{A_{12,n}^{\prime }}{R}_{n,x^{\prime }}^{*}\left(T\right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right),\\ {\Pi }_n^{(\pi ,q)}(A_{12}\times T)={\Pi }_{12,1}(A_{12}\times T)={\int }_T{R}_{\theta }\left(A_{12}\right)dQ\left(\theta \right)={\int }_{A_{12}}{R}_{1,x}^{*}\left(T\right)d{\beta }_{12,1}^{*}\left(x\right).\end{array}$$

Moreover:

$$\begin{array}{c}{\Pi }_n^q=Q,{\Pi }_n^{({\pi }^{\prime },q)}={\Pi }_{12,n},{\Pi }_n^{{\pi }^{\prime }}={\beta }_{12,n}^{*},{\Pi }_n^{(\pi ,q)}={\Pi }_{12,1},{\Pi }_n^{\pi }={\beta }_{12,1}^{*},\\ {\Pi }_n^{{\pi }^{\prime }|q=\theta }=R_{\theta }^n,{\Pi }_n^{\pi |q=\theta }=R_{\theta },{\Pi }_n^{q|{\pi }^{\prime }=x^{\prime }}=R_{n,x^{\prime }}^{*},{\Pi }_n^{q|\pi =x}=R_{1,x}^{*},\\ P_{\theta }^{X_1}=R_{\theta }^{{\pi }_1},P_{\theta }^{X_2|X_1=x_1}=R_{\theta }^{{\pi }_2|{\pi }_1=x_1},E_{P_{\theta }}\left(X_2\right|X_1=x_1)=E_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1),\end{array}$$

where the last equality refers to the case where $X_2$ is a real statistic with a finite mean.

In particular, the probability space (4) contains all the basic ingredients of the Bayesian experiment (1), i.e., the prior distribution, the sampling probabilities, the posterior distributions, and the prior predictive distribution. In addition, it becomes the natural framework in which to address the estimation problems of this communication, as we shall see in what follows.

3. Bayes Estimator of the Conditional Distribution

An estimator of the conditional distribution $P_{\theta }^{X_2|X_1}$ from an n-sized sample of the joint distribution of $(X_1,X_2)$ is a Markov kernel:

$$M:({({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1)\succ ⟶({\Omega }_2,{\mathcal{A}}_2)$$

such that, for observed $x^{\prime }=((x_{11}^{\prime },x_{21}^{\prime }),\cdots ,(x_{1n}^{\prime },x_{2n}^{\prime }))\in {({\Omega }_1\times {\Omega }_2)}^n$, $M(x^{\prime },x_1,·)$ is a probability measure on ${\mathcal{A}}_2$ that can be considered to be an estimation of the conditional distribution $P_{\theta }^{X_2|X_1=x_1}$ for a given $x_1\in {\Omega }_1$.

From a Bayesian point of view, the Bayes estimator of the conditional distribution $P^{X_2|X_1}=R^{{\pi }_2|{\pi }_1}$ is a Markov kernel:

$$M:({({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1)\succ ⟶({\Omega }_2,{\mathcal{A}}_2)$$

minimizing the Bayes risk:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^n\times \Theta }{\int }_{{\Omega }_1}\underset{A_2\in {\mathcal{A}}_2}{sup}|M(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{\Pi }_{12,n}(x^{\prime },\theta )=\\ {\int }_{\Theta }{\int }_{{({\Omega }_1\times {\Omega }_2)}^n}{\int }_{{\Omega }_1}\underset{A_2\in {\mathcal{A}}_2}{sup}|M(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^n\times ({\Omega }_1\times {\Omega }_2)\times \Theta }\underset{A_2\in {\mathcal{A}}_2}{sup}|M(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta ).\end{array}$$

The following result yields the Bayes estimator of the conditional distribution $P_{\theta }^{X_2|X_1}$ from the posterior predictive distribution.

Theorem 1.

Assume that the σ-field ${\mathcal{A}}_2$ is separable. Then, the conditional distribution of ${\pi }_2$ given ${\pi }_1=x_1$ with respect to the posterior predictive distribution ${R_{n,x^{\prime }}^{*}}^R$:

$$M_n^{*}(x^{\prime },x_1,A_2):={\left({R_{n,x^{\prime }}^{*}}^R\right)}^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right),$$

is the Bayes estimator of the conditional distribution $R^{{\pi }_2|{\pi }_1}$ for the squared total variation loss function:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }\underset{A_2\in {\mathcal{A}}_2}{sup}|M_n^{*}(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }\underset{A_2\in {\mathcal{A}}_2}{sup}|M(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\end{array}$$

for any estimator M of the conditional distribution $R^{{\pi }_2|{\pi }_1}$.

Fix an event $A_2\in {\mathcal{A}}_2$ and define:

$$H_{A_2}(x^{\prime },x_1,\theta ):=P_{\theta }^{X_2|X_1=x_1}\left(A_2\right)=R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right).$$

Jensen’s inequality could help reach a proof of the theorem if the following result can be proved.

Lemma 1.

Given $A_2\in {\mathcal{A}}_2$:

$$E_{{\Pi }_n}\left(H_{A_2}\right|{\pi }^{\prime }=x^{\prime },{\pi }_1=x_1)=M_n^{*}(x^{\prime },x_1,A_2),$$

i.e., for all $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and all $A_1\in {\mathcal{A}}_1$:

$$\begin{array}{c}{\int }_{A_{12,n}^{\prime }\times A_1\times {\Omega }_2\times \Theta }{R}_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{\Pi }_n(x^{\prime },x,\theta )={\int }_{A_{12,n}^{\prime }\times A_1}{\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{{\Pi }_n}^{({\pi }^{\prime },{\pi }_1)}(x^{\prime },x_1).\end{array}$$

4. Bayes Estimator of the Conditional Density

When the joint distribution $R_{\theta }=P_{\theta }^{(X_1,X_2)}$ has a density $f_{\theta }$ with respect to the product of two $\sigma$-finite measures ${\mu }_1$ and ${\mu }_2$ on ${\mathcal{A}}_1$ and ${\mathcal{A}}_2$, resp., the conditional density is:

$$f_{\theta }^{X_2|X_1=x_1}\left(x_2\right):=\frac{f_{\theta }(x_1,x_2)}{f_{\theta ,X_1}\left(x_1\right)}$$

for almost every $x_1$, where $f_{\theta ,X_1}\left(x_1\right)$ stands for the marginal density of $X_1$.

An estimator of the conditional density $f_{\theta }^{X_2|X_1}$ from an n-sized sample of the joint distribution of $(X_1,X_2)$ is a map:

$$m:({({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1\times {\Omega }_2,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1\times {\mathcal{A}}_2)\to (\mathbb{R},\mathcal{R})$$

such that, being observed $x^{\prime }=((x_{11}^{\prime },x_{21}^{\prime }),\cdots ,(x_{1n}^{\prime },x_{2n}^{\prime }))\in {({\Omega }_1\times {\Omega }_2)}^n$, $m(x^{\prime },x_1,·)$ is considered to be an estimation of the conditional density $f_{\theta }^{X_2|X_1=x_1}$ of $X_2$ given $X_1=x_1$.

It is well known (see, for instance [7], p. 126)) that, given two probability measures $P_1$ and $P_2$ on a measurable space $(\Omega ,\mathcal{A})$ having densities $p_1$ and $p_2$ with respect to a $\sigma$-finite measure $\mu$:

$$\underset{A\in \mathcal{A}}{sup}|P_1\left(A\right)-P_2\left(A\right)|=\frac{1}{2}{\int }_{\Omega }|p_1-p_2|d\mu .$$

Thus, the Bayesian estimation of the conditional distribution $P_{\theta }^{X_2|X_1=x_1}=R_{\theta }^{{\pi }_2|{\pi }_1=x_1}$ for the squared total variation loss function corresponds to the Bayesian estimation of its density $f_{\theta }^{X_2|X_1=x_1}$ for the $L^1$-squared loss function. Hence, according to Theorem 1, the Bayes estimator of the conditional density $f_{\theta }^{X_2|X_1=x_1}$ for the $L^1$-squared loss function is the ${\mu }_2$-density ${f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}$ of the conditional distribution:

$${\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_2|{\pi }_1=x_1}.$$

Note that:

$$\begin{array}{c}{R_{n,x^{\prime }}^{*}}^R(A_1\times A_2)={\int }_{\Theta }{R}_{\theta }(A_1\times A_2)d{R}_{n,x^{\prime }}^{*}\left(\theta \right)=\\ {\int }_{\Theta }{\int }_{A_1\times A_2}{f}_{\theta }(x_1,x_2)d({\mu }_1\times {\mu }_2)(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)=\\ {\int }_{A_1\times A_2}{\int }_{\Theta }{f}_{\theta }(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)d({\mu }_1\times {\mu }_2)(x_1,x_2)\end{array}$$

where $r_{n,x^{\prime }}^{*}\left(\theta \right)$ denotes the Q-density of the posterior distribution $R_{n,x^{\prime }}^{*}$. Thus, $r_{n,x^{\prime }}^{*}\left(\theta \right)$ is of the form $K\left(x^{\prime }\right)f_{n,\theta }\left(x^{\prime }\right)$, where:

$$f_{n,\theta }\left(x^{\prime }\right):=\prod _{i=1}^n{f}_{\theta }\left(x_i^{\prime }\right)$$

is the ${({\mu }_1\times {\mu }_2)}^n$-density of $R_{\theta }^n$. Hence, the ${\mu }_1\times {\mu }_2$-density of the posterior predictive distribution ${R_{x^{\prime }}^{*}}^R$ is:

$$f_{n,x^{\prime }}^{*}(x_1,x_2):={\int }_{\Theta }{f}_{\theta }(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right),$$

and its first marginal is:

$$f_{n,x^{\prime },1}^{*}\left(x_1\right):={\int }_{{\Omega }_2}{\int }_{\Theta }{f}_{\theta }(x_1,t)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)d{\mu }_2\left(t\right).$$

Thus, we have proved the following result.

Theorem 2.

Assume that ${\mathcal{A}}_2$ is separable. The Bayes estimator of the conditional density $f_{\theta }^{X_2|X_1}$ for the $L^1$-squared loss function is the ${\mu }_2$-density:

$${f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(x_2\right):=\frac{f_{n,x^{\prime }}^{*}(x_1,x_2)}{f_{n,x^{\prime },1}^{*}\left(x_1\right)}=\frac{{\int }_{\Theta }{f}_{\theta }(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)}{{\int }_{{\Omega }_2}{\int }_{\Theta }{f}_{\theta }(x_1,t)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)d{\mu }_2\left(t\right)}$$

of the conditional distribution ${\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_2|{\pi }_1}$ of ${\pi }_2$ given ${\pi }_1$ with respect to the posterior predictive distribution ${R_{n,x^{\prime }}^{*}}^R$:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }{\left({\int }_{{\Omega }_2}\left|{f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(t\right)-f_{\theta }^{X_2|X_1=x_1}\left(t\right)\right|d{\mu }_2\left(t\right)\right)}^{2}d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }{\left({\int }_{{\Omega }_2}\left|m(x^{\prime },x_1,t)-f_{\theta }^{X_2|X_1=x_1}\left(t\right)\right|d{\mu }_2\left(t\right)\right)}^{2}d{\Pi }_n(x^{\prime },x,\theta ),\end{array}$$

for any estimator m of the conditional density.

5. Bayes Estimator of the Conditional Distribution Function

When $X_2$ is a r.r.v., we may be interested in the estimation of the conditional distribution function of $X_2$ given $X_1=x_1$:

$$F_{\theta }(x_1,t):=P_{\theta }(X_2\le t|X_1=x_1)=R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(\right]-\infty ,t\left]\right).$$

An estimator of such a conditional distribution function from an n-sized sample of $R_{\theta }$ is a map of the form:

$$F:(x^{\prime },x_1,t)\in {({\Omega }_1\times \mathbb{R})}^n\times {\Omega }_1\times \mathbb{R}\mapsto F(x^{\prime },x_1,t):=M(x^{\prime },x_1,]-\infty ,t\left]\right)\in [0,1]$$

for a Markov kernel:

$$M:({({\Omega }_1\times \mathbb{R})}^n\times {\Omega }_1,{({\mathcal{A}}_1\times \mathcal{R})}^n\times {\mathcal{A}}_1)\succ ⟶(\mathbb{R},\mathcal{R}).$$

An optimal estimator of the conditional distribution function $F_{\theta }$ for the $L^{\infty }$-squared loss function from a Bayesian point of view (i.e., a Bayes estimator) is an estimator $F_n^{*}$ minimizing the Bayes risk, i.e., such that:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }\underset{t\in \mathbb{R}}{sup}{|F_n^{*}(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }\underset{t\in \mathbb{R}}{sup}{|F(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\end{array}$$

for any estimator F of the conditional distribution function $F_{\theta }$.

A natural candidate is the conditional distribution function for the posterior predictive distribution, as is stated in the following theorem.

Theorem 3.

The posterior predictive conditional distribution function:

$$F_n^{*}(x^{\prime },x_1,t):={\left({R_{n,x^{\prime }}^{*}}^R\right)}^{{\pi }_2|{\pi }_1=x_1}\left(\right]-\infty ,t\left]\right)$$

is the Bayes estimator of the conditional distribution function $F_{\theta }$ for the $L^{\infty }$-squared loss function.

6. Bayes Estimator of a Regression Curve

Now assume that $X_2$ is a squared-integrable r.r.v. Thus, $({\Omega }_2,{\mathcal{A}}_2)=(\mathbb{R},\mathcal{R})$. The regression curve of $X_2$ given $X_1$ is the map $x_1\in {\Omega }_1\mapsto r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)$. An estimator of the regression curve $r_{\theta }$ from a sample of size n of the joint distribution of $(X_1,X_2)$ is a statistic:

$$m:(x^{\prime },x_1)\in {({\Omega }_1\times \mathbb{R})}^n\times {\Omega }_1⟼m(x^{\prime },x_1)\in \mathbb{R},$$

so that, being observed $x^{\prime }\in {({\Omega }_1\times \mathbb{R})}^n$, $m(x^{\prime },·)$ is the estimation of $r_{\theta }$.

From a frequentist point of view, the simplest way to evaluate the error in estimating an unknown regression curve is to use the expectation of the quadratic deviation (see [6], p. 120):

$$\begin{array}{c}{E}_{\theta }\left[{\int }_{{\Omega }_1}{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^{2}d{P}_{\theta }^{X_1}\left(x_1\right)\right]=\\ {\int }_{{({\Omega }_1\times \mathbb{R})}^n}{\int }_{{\Omega }_1}{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^{2}d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right).\end{array}$$

From a Bayesian point of view, the Bayes estimator of the regression curve $r_{\theta }$ should minimize the Bayes risk (i.e., the prior mean of the expectation of the quadratic deviation):

$$\begin{array}{c}{\int }_{\Theta }{\int }_{{({\Omega }_1\times \mathbb{R})}^n}{\int }_{{\Omega }_1}{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^{2}d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ E_{{\Pi }_n}\left[{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right].\end{array}$$

The following result solves the problem of estimating the regression curve from a Bayesian point of view.

Theorem 4.

The regression curve of ${\pi }_2$ on ${\pi }_1$ with respect to the posterior predictive distribution ${R_{n,x^{\prime }}^{*}}^R$:

$$m_n^{*}(x^{\prime },x_1):=E_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)$$

is the Bayes estimator of the regression curve $r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)$ for the squared error loss function:

$$E_{{\Pi }_n}\left[{(m_n^{*}(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]\le E_{{\Pi }_n}\left[{(m_n(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]$$

for any other estimator $m_n$ of the regression curve $r_{\theta }$.

Remark 1.

(Estimation of the regression curve when densities are available.) According to the previous results, when $R_{\theta }$ has density $f_{\theta }$ with respect to the product ${\mu }_1\times {\mu }_2$ of two σ-finite measures, the ${\mu }_2$-density ${f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}$ of the conditional distribution ${\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_2|{\pi }_1=x_1}$ is:

$${f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(x_2\right):=\frac{f_{n,x^{\prime }}^{*}(x_1,x_2)}{f_{n,x^{\prime },1}^{*}\left(x_1\right)}=\frac{{\int }_{\Theta }{f}_{\theta }(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)}{{\int }_{\mathbb{R}}{\int }_{\Theta }{f}_{\theta }(x_1,t)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)d{\mu }_2\left(t\right)},$$

which is the Bayes estimator of the conditional density $f_{\theta }^{X_2|X_1}$. Hence, the Bayes estimator of the regression curve can be computed as:

$$\begin{array}{c}{m}_n^{*}(x^{\prime },x_1):=E_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)=\\ {\int }_{\mathbb{R}}{x}_2·{f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(x_2\right)d{\mu }_2\left(x_2\right)=\\ \frac{{\int }_{\mathbb{R}}{x}_2{\int }_{\Theta }{f}_{\theta }(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)d{\mu }_2\left(x_2\right)}{{\int }_{\mathbb{R}}{\int }_{\Theta }{f}_{\theta }(x_1,x_2)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)d{\mu }_2\left(x_2\right)}.\end{array}$$

7. Examples

Example 1.

Let us assume that, for $\theta ,\lambda ,x_1>0$, $P_{\theta }^{X_1}=G(1,{\theta }^{-1})$, $P_{\theta }^{X_2|X_1=x_1}=G(1,{\left(\theta x_1\right)}^{-1})$, and $Q=G(1,{\lambda }^{-1})$, where $G(\alpha ,\beta )$ denotes the gamma distribution of parameters $\alpha ,\beta >0$ (where β stands for the scale parameter). Hence, the joint density of $X_1$ and $X_2$ is:

$$f_{\theta }(x_1,x_2)={\theta }^2{x}_{1}exp\{-\theta x_1(1+x_2)\}{I}_{{]0,\infty [}^2}(x_1,x_2).$$

Then the density of $R_{\theta }^n$ is:

$$f_{n,\theta }\left(x^{\prime }\right)={\theta }^{2n}·{\prod }_{i=1}^n{x}_{i1}^{\prime }·exp\left\{-\theta {\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })\right\}·I_{{]0,\infty [}^{2n}}\left(x^{\prime }\right),$$

and the posterior Q-density given $x^{\prime }$ is:

$$\frac{d{R}_{n,x^{\prime }}^{*}\left(\theta \right)}{dQ}=:r_{n,x^{\prime }}^{*}\left(\theta \right)=K\left(x^{\prime }\right)f_{n,\theta }\left(x^{\prime }\right)$$

where $K\left(x^{\prime }\right)={\left[{\int }_0^{\infty }{f}_{n,\theta }\left(x^{\prime }\right)dQ\left(\theta \right)\right]}^{-1}$.

Hence, the posterior predictive density given $x^{\prime }\in {]0,\infty [}^{2n}$ is:

$$\begin{array}{c}{f}_{n,x^{\prime }}^{*}\left(x\right)={\int }_0^{\infty }{f}_{\theta }\left(x\right)r_{n,x^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)=\\ \lambda K\left(x^{\prime }\right)x_1\prod _{i=1}^n{x}_{i1}^{\prime }{\int }_0^{\infty }{\theta }^{2n+2}exp\left\{-\theta [\lambda +x_1(1+x_2)+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]\right\}d\theta ·I_{{]0,\infty [}^2}\left(x\right).\end{array}$$

Since:

$${\int }_0^{\infty }{\theta }^{n}exp\{-a\theta \}d\theta =\frac{n!}{a^{n+1}},$$

we have that:

$$f_{n,x^{\prime }}^{*}\left(x\right)=\frac{(2n+2)!\lambda K\left(x^{\prime }\right)x_1{\prod }_{i=1}^n{x}_{i1}^{\prime }}{{[\lambda +x_1(1+x_2)+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]}^{(2n+3)}}$$

and its first marginal is:

$$f_{n,x^{\prime },1}^{*}\left(x_1\right)={\int }_{\mathbb{R}}{f}_{n,x^{\prime }}^{*}(x_1,x_2)d{x}_2={\int }_0^{\infty }\frac{A}{{(Bt+C)}^m}dt=\frac{A}{(m-1)B{C}^{m-1}}$$

where:

$$m=2n+3,A=(2n+2)!\lambda K\left(x^{\prime }\right)x_1\prod _{i=1}^n{x}_{i1}^{\prime },B=x_1,andC=\lambda +x_1+\sum _{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime }).$$

Thus:

$$f_{n,x^{\prime },1}^{*}\left(x_1\right)=\frac{(2n+2)!\lambda K\left(x^{\prime }\right)x_1{\prod }_{i=1}^n{x}_{i1}^{\prime }}{(2n+2)x_1{[\lambda +x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]}^{2n+2}}.$$

The Bayes estimator of the conditional density $f_{\theta }^{X_2|X_1=x_1}\left(x_2\right)=\theta x_{1}exp\{-\theta x_1{x}_2\}{I}_{]0,\infty [}\left(x_2\right)$ is, for $x_1,x_2>0$:

$$\begin{array}{c}{f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(x_2\right):=\frac{f_{n,x^{\prime }}^{*}(x_1,x_2)}{f_{n,x^{\prime },1}^{*}\left(x_1\right)}=\\ (2n+2)x_1\frac{{[\lambda +x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]}^{2n+2}}{{[\lambda +x_1(1+x_2)+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]}^{2n+3}}=\\ \frac{(2n+2)x_1{a}_n{(x^{\prime },x_1)}^{2n+2}}{{(x_1{x}_2+a_n(x^{\prime },x_1))}^{2n+3}}\end{array}$$

where:

$$a_n(x^{\prime },x_1)=\lambda +x_1+\sum _{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime }).$$

The Bayes estimator of the conditional distribution function:

$$F_{\theta }(x_1,t):=P_{\theta }(X_2\le t|X_1=x_1)$$

is, for $t>0$:

$$\begin{array}{c}\begin{array}{cc}\hfill F_n^{*}(x^{\prime },x_1,t)& ={\int }_0^t(2n+2)x_1\frac{{[\lambda +x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]}^{2n+2}}{{[\lambda +x_1(1+x_2)+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })]}^{2n+3}}d{x}_2\hfill \\ & ={\int }_0^t\frac{(2n+2)x_1{a}_n{(x^{\prime },x_1)}^{2n+2}}{{(x_1{x}_2+a_n(x^{\prime },x_1))}^{2n+3}}d{x}_2\hfill \\ & =a_n{(x^{\prime },x_1)}^{2n+2}\left(\frac{1}{a_n{(x^{\prime },x_1)}^{2n+2}}-\frac{1}{{(x_{1}t+a_n(x^{\prime },x_1))}^{2n+2}}\right)\hfill \\ & =1-{\left(1+\frac{x_{1}t}{a_n(x^{\prime },x_1)}\right)}^{-2n-2}.\hfill \end{array}\end{array}$$

The Bayes estimator of the regression curve $r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)=\frac{1}{\theta x_1}$ is, for $x_1>0$:

$$m_n^{*}(x^{\prime },x_1)={\int }_0^{\infty }{x}_2·{f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(x_2\right)d{x}_2=\frac{\lambda +x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })}{(2n+1)x_1}=\frac{a_n(x^{\prime },x_1)}{(2n+1)x_1}.$$

Example 2.

Let us assume that $X_1$ has a Bernoulli distribution $Bi(1,\theta )$ of unknown parameter $\theta \in ]0,1[$ (i.e., $P_{\theta }^{X_1}=Bi(1,\theta )$) and, given $X_1=k_1\in \{0,1\}$, $X_2$ has distribution $Bi(1,1-\theta )$ when $k_1=0$ and $Bi(1,\theta )$ when $k_1=1$, i.e., $P_{\theta }^{X_2|X_1=k_1}=Bi(1,k_1+(1-2k_1)(1-\theta ))$. We can think of tossing a coin with probability θ of getting heads ($=1$) and making a second toss of this coin if it comes up heads on the first toss, or tossing a second coin with probability $1-\theta$ of making heads if the first toss is tails (=0). Consider the uniform distribution on $]0,1[$ as the prior distribution Q.

Then, the joint probability function of $X_1$ and $X_2$ is:

$$\begin{array}{c}\begin{array}{cc}\hfill f_{\theta }(k_1,k_2)& ={\theta }^{k_1}{(1-\theta )}^{1-k_1}{[k_1+(1-2k_1)(1-\theta )]}^{k_2}{[1-k_1-(1-2k_1)(1-\theta )]}^{1-k_2}\hfill \\ & =\left\{\begin{array}{c}\theta (1-\theta )if{k}_2=0,\hfill \\ {(1-\theta )}^{2}if{k}_1=0,k_2=1,\hfill \\ {\theta }^{2}if{k}_1=1,k_2=1.\hfill \end{array}\right.\hfill \end{array}\end{array}$$

The probability function of $R_{\theta }^n$ is:

$$f_{n,\theta }\left(k^{\prime }\right)=\prod _{i=1}^n{f}_{\theta }\left(k_i^{\prime }\right)={\theta }^{a_n\left(k^{\prime }\right)}{(1-\theta )}^{b_n\left(k^{\prime }\right)}$$

for $k^{\prime }=(k_1^{\prime },\cdots ,k_n^{\prime })=(k_{11}^{\prime },k_{12}^{\prime },\cdots ,k_{n1}^{\prime },k_{n2}^{\prime })\in {\{0,1\}}^{2n}$, where:

$$a_n\left(k^{\prime }\right)=n_{+0}\left(k^{\prime }\right)+2n_{11}\left(k^{\prime }\right),b_n\left(k^{\prime }\right)=n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right),$$

being $n_{j_1{j}_2}\left(k^{\prime }\right)$ the number of indices $i\in \{1,\cdots ,n\}$ such that $(k_{i1}^{\prime },k_{i2}^{\prime })=(j_1,j_2)$ and $n_{+j}=n_{0j}+n_{1j}$ for $j=0,1$. Note that $a_n\left(k^{\prime }\right)+b_n\left(k^{\prime }\right)=2n$.

Hence, the posterior Q-density given $k^{\prime }$ is:

$$r_{n,k^{\prime }}^{*}\left(\theta \right)=\frac{d{R}_{n,k^{\prime }}^{*}}{dQ}\left(\theta \right)=K\left(k^{\prime }\right)f_{n,\theta }\left(k^{\prime }\right)=K\left(k^{\prime }\right){\theta }^{a_n\left(k^{\prime }\right)}{(1-\theta )}^{b_n\left(k^{\prime }\right)}$$

where:

$$K\left(k^{\prime }\right)={\left[{\int }_0^1{f}_{n,\theta }\left(k^{\prime }\right)dQ\left(\theta \right)\right]}^{-1}=\frac{1}{B(a_n\left(k^{\prime }\right)+1,b_n\left(k^{\prime }\right)+1)},$$

where $B(\alpha ,\beta )$ stands for the beta function.

Thus, the posterior predictive density given $k^{\prime }\in {\{0,1\}}^{2n}$ is:

$$\begin{array}{c}\begin{array}{cc}\hfill f_{n,k^{\prime }}^{*}(k_1,k_2)& ={\int }_0^1{f}_{\theta }(k_1,k_2)r_{n,k^{\prime }}^{*}\left(\theta \right)dQ\left(\theta \right)\hfill \\ & =K\left(k^{\prime }\right){\int }_0^1{\theta }^{a_n\left(k^{\prime }\right)+k_1}{(1-\theta )}^{b_n\left(k^{\prime }\right)+k_2}d\theta \hfill \\ & =K\left(k^{\prime }\right)B(a_n\left(k^{\prime }\right)+k_1+1,b_n\left(k^{\prime }\right)+k_2+1),\hfill \end{array}\end{array}$$

and its first marginal is:

$$\begin{array}{c}{f}_{n,k^{\prime },1}^{*}\left(k_1\right)=K\left(k^{\prime }\right)[B(a_n\left(k^{\prime }\right)+k_1+1,b_n\left(k^{\prime }\right)+1)+B(a_n\left(k^{\prime }\right)+k_1+1,b_n\left(k^{\prime }\right)+2)].\end{array}$$

The Bayes estimator of the conditional probability function:

$$f_{\theta }^{X_2|X_1=k_1}\left(k_2\right)={[k_1+(1-2k_1)(1-\theta )]}^{k_2}{[1-k_1-(1-2k_1)(1-\theta )]}^{1-k_2}$$

is

$$\begin{array}{c}{f_{n,x^{\prime }}^{*}}^{X_2|X_1=k_1}\left(k_2\right):=\frac{f_{n,k^{\prime }}^{*}(k_1,k_2)}{f_{n,k^{\prime },1}^{*}\left(k_1\right)}=\left\{\begin{array}{cc}\frac{2n+2}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+3}\hfill & if{k}_1=k_2=0,\hfill \\ \frac{n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+1}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+3}\hfill & if{k}_1=0,k_2=1,\hfill \\ \frac{2n+3}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+4}\hfill & if{k}_1=1,k_2=0,\hfill \\ \frac{n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+1}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+4}\hfill & if{k}_1=k_2=1.\hfill \end{array}\right.\end{array}$$

The Bayes estimator of the conditional mean $r_{\theta }\left(k_1\right):=E_{\theta }\left(X_2\right|X_1=k_1)={\theta }^{k_1}{(1-\theta )}^{1-k_1}$ is, for $k_1=0,1$:

$$m_n^{*}(k^{\prime },k_1)={f_{n,k^{\prime }}^{*}}^{X_2|X_1=k_1}\left(1\right)=\left\{\begin{array}{cc}\frac{n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+1}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+3}\hfill & if{k}_1=0,\hfill \\ \frac{n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+1}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+4}\hfill & if{k}_1=1.\hfill \end{array}\right.$$

Example 3.

Let $(X_1,X_2)$ have bivariate normal distribution with density:

$$\begin{array}{c}{f}_{\theta }\left(x\right):=\frac{1}{2\pi {\sigma }^2\sqrt{1-{\rho }^2}}exp\left\{-\frac{1}{2{\sigma }^2(1-{\rho }^2)}[{(x_1-\theta )}^2-2\rho (x_1-\theta )(x_2-\theta )+{(x_2-\theta )}^2]\right\}\\ =\frac{1}{2\pi {\sigma }^2\sqrt{1-{\rho }^2}}exp\left\{-\frac{1}{2{\sigma }^2(1-{\rho }^2)}[x_1^2+x_2^2-2\rho x_1{x}_2-2(1-\rho )(x_1+x_2)\theta +2(1-\rho ){\theta }^2]\right\},\end{array}$$

where $\sigma >0$ and $\rho \in ]-1,1[$ are assumed to be known. Thus:

$$\begin{array}{c}{R}_{\theta }=N_2\left(\left(\begin{array}{c}\theta \\ \theta \end{array}\right),{\sigma }^2\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)\right),X_1,X_2{\sim }_{\theta }N(\theta ,{\sigma }^2)\\ P_{\theta }^{X_2|X_1=x_1}=N((1-\rho )\theta +\rho x_1,{\sigma }^2\sqrt{1-{\rho }^2}),E_{\theta }\left(X_2\right|X_1=x_1)=(1-\rho )\theta +\rho x_1.\end{array}$$

Hence, for $x^{\prime }=(x_1^{\prime },\cdots ,x_n^{\prime })=(x_{11}^{\prime },x_{12}^{\prime },\cdots ,x_{n1}^{\prime },x_{n2}^{\prime })\in {\left({\mathbb{R}}^2\right)}^n$:

$$\begin{array}{c}{f}_{n,\theta }\left(x^{\prime }\right)=\prod _{i=1}^n{f}_{\theta }\left(x_i^{\prime }\right)=\\ \frac{1}{{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^n}exp\left\{-\frac{1}{2{\sigma }^2(1-{\rho }^2)}{\sum }_{i=1}^n\left[{(x_{i1}^{\prime }-\theta )}^2-2\rho (x_{i1}^{\prime }-\theta )(x_{i2}^{\prime }-\theta )+{(x_{i2}^{\prime }-\theta )}^2\right]\right\}.\end{array}$$

Let us consider the prior distribution $Q=N(\mu ,{\tau }^2)$ whose density is:

$$g\left(\theta \right)=\frac{1}{\tau \sqrt{2\pi }}exp\left\{-\frac{1}{2{\tau }^2}{(\theta -\mu )}^2\right\}.$$

The posterior Q-density is:

$$\begin{array}{c}{r}_{n,x^{\prime }}^{*}\left(\theta \right):=\frac{d{R}_{n,x^{\prime }}^{*}}{dQ}\left(\theta \right)=K_1\left(x^{\prime }\right)f_{n,\theta }\left(x^{\prime }\right)=\\ \frac{K_1\left(x^{\prime }\right)}{{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^n}exp\left\{-\frac{1}{2{\sigma }^2(1-{\rho }^2)}[2n(1-\rho ){\theta }^2-2(1-\rho )s_1\left(x^{\prime }\right)\theta +s_2\left(x^{\prime }\right)-2\rho p\left(x^{\prime }\right)]\right\}\end{array}$$

where:

$$\begin{array}{c}{s}_1\left(x^{\prime }\right):=\sum _i(x_{i1}^{\prime }+x_{i2}^{\prime }),s_2\left(x^{\prime }\right):=\sum _i({x_{i1}^{\prime }}^2+{x_{i2}^{\prime }}^2),p\left(x^{\prime }\right)=\sum _i{x}_{i1}^{\prime }{x}_{i2}^{\prime },\\ K_1\left(x^{\prime }\right)={\left[{\int }_{\mathbb{R}}{f}_{n,x^{\prime }}\left(\theta \right)dQ\left(\theta \right)\right]}^{-1}.\end{array}$$

Note that, writing $c_n(\rho ,\sigma ,\tau )={\left[\tau \sqrt{2\pi }{\left(2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right)}^n\right]}^{-1}$:

$$\begin{array}{c}{\int }_{\mathbb{R}}{f}_{n,x^{\prime }}\left(\theta \right)dQ\left(\theta \right)=c_n(\rho ,\sigma ,\tau )·\\ {\int }_{\mathbb{R}}\left\{-\frac{1}{2{\sigma }^2(1-{\rho }^2)}[2n(1-\rho ){\theta }^2-2(1-\rho )s_1\left(x^{\prime }\right)\theta +s_2\left(x^{\prime }\right)-2\rho p\left(x^{\prime }\right)]-\frac{1}{2{\tau }^2}{(\theta -\mu )}^2]\right\}d\theta \\ =c_n(\rho ,\sigma ,\tau ){\int }_{\mathbb{R}}exp\{-(A_1{\theta }^2-B_1\left(x^{\prime }\right)\theta +C_1\left(x^{\prime }\right))\}d\theta \\ =\frac{exp\left\{-\left(C_1\left(x^{\prime }\right)-\frac{B_1^2\left(x^{\prime }\right)}{4A_1}\right)\right\}}{\tau \sqrt{2\pi }{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^n}{\int }_{\mathbb{R}}exp\left\{-A_1{\left(\theta -\frac{B_1\left(x^{\prime }\right)}{2A_1}\right)}^2\right\}d\theta \\ =\frac{exp\left\{-\left(C_1\left(x^{\prime }\right)-\frac{B_1^2\left(x^{\prime }\right)}{4A_1}\right)\right\}}{\tau \sqrt{2A_1}{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^n}\end{array}$$

where:

$$\begin{array}{c}{A}_1=\frac{n}{{\sigma }^2(1+\rho )}+\frac{1}{2{\tau }^2},B_1\left(x^{\prime }\right)=\frac{s_1\left(x^{\prime }\right)}{{\sigma }^2(1+\rho )}+\frac{\mu }{{\tau }^2},C_1\left(x^{\prime }\right)=\frac{s_2\left(x^{\prime }\right)-2\rho p\left(x^{\prime }\right)}{2{\sigma }^2(1-{\rho }^2)}+\frac{{\mu }^2}{2{\tau }^2}.\end{array}$$

Hence:

$$K_1\left(x^{\prime }\right)=\tau \sqrt{2A_1}{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^{n}exp\left\{C_1\left(x^{\prime }\right)-\frac{B_1^2\left(x^{\prime }\right)}{4A_1}\right\}.$$

The posterior predictive density given $x^{\prime }\in {\left({\mathbb{R}}^2\right)}^n$ is:

$$\begin{array}{c}{f}_{n,x^{\prime }}^{*}\left(x\right):={\int }_{\mathbb{R}}{f}_{\theta }\left(x\right)r_{n,x^{\prime }}^{*}\left(\theta \right)g\left(\theta \right)d\theta =\frac{K_1\left(x^{\prime }\right)}{\tau \sqrt{2\pi }{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^{n+1}}·\\ {\int }_{\mathbb{R}}exp\left\{-\frac{2(n+1)(1-\rho ){\theta }^2-2(1-\rho )(x_1+x_2+s_1\left(x^{\prime }\right))\theta +(x_1^2+x_2^2+s_2\left(x^{\prime }\right))-2\rho (x_1{x}_2+p\left(x^{\prime }\right))}{2{\sigma }^2(1-{\rho }^2)}\right.\\ \left.-\frac{({\theta }^2-2\mu \theta +{\mu }^2)}{2{\tau }^2}\right\}d\theta =\\ K_2\left(x^{\prime }\right){\int }_{\mathbb{R}}exp\{-(A_2{\theta }^2-B_2(x,x^{\prime })\theta +C_2(x,x^{\prime }))\}d\theta =\\ K_2\left(x^{\prime }\right)exp\left\{-\left(C_2(x,x^{\prime })-\frac{B_2^2(x,x^{\prime })}{4A_2}\right)\right\}{\int }_{\mathbb{R}}exp\left\{-A_2{\left(\theta -\frac{B_2(x,x^{\prime })}{2A_2}\right)}^2\right\}d\theta =\\ K_2\left(x^{\prime }\right)\sqrt{\frac{\pi }{A_2}}exp\left\{-\left(C_2(x,x^{\prime })-\frac{B_2^2(x,x^{\prime })}{4A_2}\right)\right\}\end{array}$$

where:

$$\begin{array}{c}{K}_2\left(x^{\prime }\right)=\frac{K_1\left(x^{\prime }\right)}{\tau \sqrt{2\pi }{\left[2\pi {\sigma }^2\sqrt{1-{\rho }^2}\right]}^{n+1}}=\frac{\sqrt{A_1}exp\left\{C_1\left(x^{\prime }\right)-\frac{B_1^2\left(x^{\prime }\right)}{4A_1}\right\}}{2{\pi }^{3/2}{\sigma }^2\sqrt{1-{\rho }^2}},A_2=\frac{(n+1)(1+\rho )}{{\sigma }^2}+\frac{1}{2{\tau }^2}\\ B_2(x.x^{\prime })=\frac{x_1+x_2+s_1\left(x^{\prime }\right)}{{\sigma }^2(1+\rho )}+\frac{\mu }{{\tau }^2},C_2(x,x^{\prime })=\frac{x_1^2+x_2^2+s_2\left(x^{\prime }\right)-2\rho (x_1{x}_2+p\left(x^{\prime }\right))}{2{\sigma }^2(1-{\rho }^2)}+\frac{{\mu }^2}{{\tau }^2}\end{array}$$

We can write:

$$\begin{array}{c}{C}_2(x,x^{\prime })-\frac{B_2^2(x,x^{\prime })}{4A_2}=A_3{x}_1^2+A_3{x}_2^2+B_3{x}_1{x}_2+C_3\left(x^{\prime }\right)(x_1+x_2)+D_3\left(x^{\prime }\right)\end{array}$$

where:

$$\begin{array}{c}{A}_3=\frac{1}{2{\sigma }^2(1-{\rho }^2)}-\frac{{\tau }^2}{{\sigma }^2{(1+\rho )}^2[4(n+1)(1+\rho ){\tau }^2+2{\sigma }^2]},\\ B_3=-\frac{\rho }{{\sigma }^2(1-{\rho }^2)}-\frac{{\tau }^2}{{\sigma }^2{(1+\rho )}^2[2(n+1)(1+\rho ){\tau }^2+{\sigma }^2]},\\ C_3\left(x^{\prime }\right)=-\frac{{\tau }^2{s}_1\left(x^{\prime }\right)+\mu {\sigma }^2(1+\rho )}{{\sigma }^2{(1+\rho )}^2[2(n+1)(1+\rho ){\tau }^2+{\sigma }^2]},\\ D_3\left(x^{\prime }\right)=\frac{{\tau }^2{s}_2\left(x^{\prime }\right)-2\rho {\tau }^{2}p\left(x^{\prime }\right)+2{\mu }^2{\sigma }^2(1-{\rho }^2)}{2{\tau }^2{\sigma }^2(1-{\rho }^2)}-\frac{{\tau }^4{s}_1{\left(x^{\prime }\right)}^2+{\mu }^2{\sigma }^4{(1+\rho )}^2+2\mu {\sigma }^2(1+\rho )s_1\left(x^{\prime }\right)}{{\tau }^2{\sigma }^2{(1+\rho )}^2[4(n+1)(1+\rho ){\tau }^2+2{\sigma }^2]}.\end{array}$$

It is readily shown that $A_3>0$. It follows that the posterior predictive density $f_{n,x^{\prime }}^{*}$ is the density of a normal bivariate distribution $N_2(m\left(x^{\prime }\right),\Sigma )$ where:

$$m\left(x^{\prime }\right)=\left(\genfrac{}{}{0pt}{}{m_1\left(x^{\prime }\right)}{m_2\left(x^{\prime }\right)}\right),\Sigma ={\sigma }_1^2\left(\begin{array}{cc}1& {\rho }_1\\ {\rho }_1& 1\end{array}\right)$$

being:

$${\rho }_1=\frac{B_3}{2A_3},{\sigma }_1^2=\frac{2A_3}{4A_3^2-B_3^2},m_1\left(x^{\prime }\right)=m_2\left(x^{\prime }\right)=-\frac{C_3\left(x^{\prime }\right)}{2(1-{\rho }_1)}.$$

It is easy to see that, as was to be expected, $|{\rho }_1|<1$ and ${\sigma }_1^2>0$. If we denote:

$$a_n(\rho ,\sigma ,\tau ):=2(n+1)(1+\rho )+\frac{{\sigma }^2}{{\tau }^2}$$

we can write:

$$\begin{array}{c}{\rho }_1=-\frac{a_n(\rho ,\sigma ,\tau )+\frac{1-\rho }{1+\rho }}{a_n(\rho ,\sigma ,\tau )-\frac{1-\rho }{1+\rho }}·\rho ,{\sigma }_1^2=\frac{a_n(\rho ,\sigma ,\tau )}{a_n(\rho ,\sigma ,\tau )-\frac{1-\rho }{1+\rho }}·{\sigma }^2,\\ m_1\left(x^{\prime }\right)=m_2\left(x^{\prime }\right)=\frac{s_1\left(x^{\prime }\right)+(1+\rho )\frac{{\sigma }^2}{{\tau }^2}\mu }{2(1-{\rho }_1){(1+\rho )}^2{\sigma }^2{a}_n(\rho ,\sigma ,\tau )}.\end{array}$$

It follows that the conditional distribution:

$${\left({R_{n,x^{\prime }}^{*}}^R\right)}^{{\pi }_2|{\pi }_1=x_1}:=N\left((1-{\rho }_1)m_1\left(x^{\prime }\right)+{\rho }_1{x}_1,{\sigma }_1^2(1-{\rho }_1^2)\right)$$

is the Bayes estimator of the conditional distribution:

$$P_{\theta }^{X_2|X_1=x_1}=N\left((1-\rho )\theta +\rho x_1,{\sigma }^2(1-{\rho }^2)\right)$$

for the squared total variation function, and its density ${f_{n,x^{\prime }}^{*}}^{{\pi }_2|{\pi }_1=x_1}$ is the Bayes estimator of the conditional density:

$$f_{\theta }^{X_2|X_1=x_1}\left(x_2\right)=\frac{1}{\sigma \sqrt{2\pi (1-{\rho }^2)}}exp\left\{-\frac{1}{2{\sigma }^2(1-{\rho }^2)}{[x_2-(1-\rho )\theta -\rho x_1]}^2\right\}$$

for the $L^1$-squared loss function.Moreover, its mean:

$$E_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)=(1-{\rho }_1)m_1\left(x^{\prime }\right)+{\rho }_1{x}_1$$

is the Bayes estimator of the regression curve:

$$E_{\theta }\left(X_2\right|X_1=x_1)=(1-\rho )\theta +\rho x_1$$

for the squared error loss function.

8. Proofs

Proof of Proposition 1.

It follows from (5) and (2) that ${\Pi }_n^{({\pi }^{\prime },q)}={\Pi }_{12,n}$ because, for every $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and $T\in \mathcal{T}$:

$$\begin{array}{c}{\Pi }_n^{({\pi }^{\prime },q)}(A_{12,n}^{\prime }\times T)={\Pi }_n\left({({\pi }^{\prime },q)}^{-1}(A_{12,n}^{\prime }\times T)\right)={\Pi }_n(A_{12,n}^{\prime }\times ({\Omega }_1\times {\Omega }_2)\times T)=\\ {\int }_T{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right)={\Pi }_{12,n}(A_{12,n}^{\prime }\times T).\end{array}$$

Analogously, we can show that ${\Pi }_n^{(\pi ,q)}={\Pi }_{12,1}$. Furthermore, (5) also proves that:

$$\begin{array}{c}{\Pi }_n^q\left(T\right)={\int }_T{R}_{\theta }({\Omega }_1\times {\Omega }_2)R_{\theta }^n\left({({\Omega }_1\times {\Omega }_2)}^n\right)dQ\left(\theta \right)=Q\left(T\right),\end{array}$$

and:

$$\begin{array}{c}{\Pi }_n^{{\pi }^{\prime }}\left(A_{12,n}^{\prime }\right)={\int }_{\Theta }{R}_{\theta }({\Omega }_1\times {\Omega }_2)R_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right)={\beta }_{12,n}^{*}\left(A_{12,n}^{\prime }\right),\end{array}$$

because of (2${}^{\prime \prime }$). Analogously, it can be proved that ${\Pi }_n^{\pi }={\beta }_{12,1}^{*}$. Moreover, the identity:

$${\Pi }_n^{{\pi }^{\prime }|q=\theta }=R_{\theta }^n$$

follows from the definition of the conditional distribution and the facts that ${\Pi }_n^q=Q$ and

$$\begin{array}{c}{\Pi }_n({\pi }^{\prime }\in A_{12,n}^{\prime },q\in T)={\int }_T{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)dQ\left(\theta \right).\end{array}$$

A similar reasoning proves that ${\Pi }_n^{\pi |q=\theta }=R_{\theta }$. From the definition of the posterior distribution we also have that:

$$\begin{array}{c}{\Pi }_n^{({\pi }^{\prime },q)}(A_{12,n}^{\prime }\times T)={\Pi }_n({\pi }^{\prime }\in A_{12,n}^{\prime },q\in T)={\int }_{A_{12,n}^{\prime }}{R}_{n,x^{\prime }}^{*}\left(T\right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right)\end{array}$$

which proves that ${\Pi }_n^{q|{\pi }^{\prime }=x^{\prime }}=R_{n,x^{\prime }}^{*}$ as ${\Pi }_n^{{\pi }^{\prime }}={\beta }_{12,n}^{*}$. In the same manner, we get ${\Pi }_n^{q|\pi =x}=R_{1,x}^{*}$. Now it is clear that, given $A_i\in {\mathcal{A}}_i$, $i=1,2$:

$$P^{X_1}\left(A_1\right)=P_{\theta }^{(X_1,X_2)}(A_1\times {\Omega }_2)=R_{\theta }(A_1\times {\Omega }_2)=R_{\theta }^{{\pi }_1}\left(A_1\right)$$

and that:

$$\begin{array}{c}{\int }_{A_1}{P}_{\theta }^{X_2|X_1=x_1}\left(A_2\right)d{P}_{\theta }^{X_1}\left(x_1\right)=P_{\theta }^{(X_1,X_2)}(A_1\times A_2)=\\ R_{\theta }(A_1\times A_2)={\int }_{A_1}{R}_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{R}_{\theta }^{{\pi }_1}\left(x_1\right),\end{array}$$

so $P_{\theta }^{X_2|X_1=x_1}=R_{\theta }^{{\pi }_2|{\pi }_1=x_1}$. Finally, by definition of the conditional expectation, given $A_1\in {\mathcal{A}}_1$, we have that, when ${\Omega }_2=\mathbb{R}$:

$$\begin{array}{c}{\int }_{A_1\times \mathbb{R}}{x}_{2}d{R}_{\theta }(x_1,x_2)={\int }_{A_1}{E}_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }^{{\pi }_1}\left(x_1\right).\end{array}$$

However, $R_{\theta }^{{\pi }_1}=P_{\theta }^{X_1}$ and:

$$\begin{array}{c}{\int }_{A_1\times \mathbb{R}}{x}_{2}d{R}_{\theta }(x_1,x_2)={\int }_{X_1^{-1}\left(A_1\right)}{X}_{2}d{P}_{\theta }.\end{array}$$

This proves that $E_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)=E_{P_{\theta }}\left(X_2\right|X_1=x_1)$. □

Proof of Lemma 1.

We must show that, for all $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and all $A_1\in {\mathcal{A}}_1$:

$$\begin{array}{c}{\int }_{A_{12,n}^{\prime }\times A_1\times {\Omega }_2\times \Theta }{R}_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{\Pi }_n(x^{\prime },x,\theta )=\\ {\int }_{A_{12,n}^{\prime }\times A_1}{\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{{\Pi }_n}^{({\pi }^{\prime },{\pi }_1)}(x^{\prime },x_1).\end{array}$$

(7)

According to (6), we have that, for all $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and all $A_1\in {\mathcal{A}}_1$:

$$\begin{array}{c}{\int }_{A_{12,n}^{\prime }\times A_1\times {\Omega }_2\times \Theta }{R}_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{\Pi }_n(x^{\prime },x,\theta )=\\ {\int }_{\Theta }{\int }_{A_{12,n}^{\prime }}{\int }_{A_1}{R}_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ {\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)R_{\theta }(A_1\times A_2)dQ\left(\theta \right).\end{array}$$

(8)

Note that, by definition of conditional distribution:

$$\begin{array}{c}{R_{n,x^{\prime }}^{*}}^R(A_1\times A_2)={\int }_{A_1}{\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right)d{\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{{\pi }_1}\left(x_1\right),\end{array}$$

(9)

and, by definition of posterior predictive distribution:

$${R_{n,x^{\prime }}^{*}}^R(A_1\times A_2)={\int }_{\Theta }{R}_{\theta }(A_1\times A_2)d{R}_{n,x^{\prime }}^{*}\left(\theta \right).$$

Note that, for any $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$, being $R_{n,x^{\prime }}^{*}={\Pi }_n^{q|{\pi }^{\prime }=x^{\prime }}$:

$$\begin{array}{c}{\int }_{A_{12,n}^{\prime }}{\int }_{\Theta }{R}_{\theta }(A_1\times A_2)d{R}_{n,x^{\prime }}^{*}\left(\theta \right)d{\Pi }_n^{{\pi }^{\prime }}\left(x^{\prime }\right)={\int }_{A_{12,n}^{\prime }}{\int }_{\Theta }{R}_{\theta }(A_1\times A_2)d{\Pi }_n^{q|{\pi }^{\prime }=x^{\prime }}\left(\theta \right)d{\Pi }_n^{{\pi }^{\prime }}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }\times \Theta }{R}_{\theta }(A_1\times A_2)d{\Pi }_n^{({\pi }^{\prime },q)}(x^{\prime },\theta )={\int }_{{{\pi }^{\prime }}^{-1}\left(A_{12,n}^{\prime }\right)}{R}_{\theta }(A_1\times A_2)d{\Pi }_n(x^{\prime },x,\theta ),\end{array}$$

proving that:

$${\int }_{\Theta }{R}_{\theta }(A_1\times A_2)d{R}_{n,x^{\prime }}^{*}\left(\theta \right)=E_{{\Pi }_n}\left(r_{A_1\times A_2}\right|{\pi }^{\prime }=x^{\prime })$$

where $r_{A_1\times A_2}\left(\theta \right):=R_{\theta }(A_1\times A_2)$, and hence:

$${R_{n,x^{\prime }}^{*}}^R(A_1\times A_2)=E_{{\Pi }_n}\left(r_{A_1\times A_2}\right|{\pi }^{\prime }=x^{\prime }).$$

Thus, by definition of conditional expectation:

$$\begin{array}{c}{\int }_{A_{12,n}^{\prime }}{R_{n,x^{\prime }}^{*}}^R(A_1\times A_2)d{\Pi }_n^{{\pi }^{\prime }}\left(x^{\prime }\right)={\int }_{A_{12,n}^{\prime }\times {\Omega }_1\times {\Omega }_2\times \Theta }{R}_{\theta }(A_1\times A_2)d{\Pi }_n(x^{\prime },x,\theta ).\end{array}$$

(10)

However, the second term of this equation is:

$$\begin{array}{c}{\int }_{A_{12,n}^{\prime }\times {\Omega }_1\times {\Omega }_2\times \Theta }{R}_{\theta }(A_1\times A_2)d{\Pi }_n(x^{\prime },x,\theta )=\\ {\int }_{\Theta }{\int }_{A_{12,n}^{\prime }}{R}_{\theta }(A_1\times A_2)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)={\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right)R_{\theta }(A_1\times A_2)dQ\left(\theta \right)\end{array}$$

which coincides with (8). This proves the lemma. □

Proof of Theorem 1.

Denoting by ${\parallel ·\parallel }_2$ the usual norm on the Hilbert space $L^2({({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta ,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^{n+1}\times \mathcal{T},{\Pi }_n)$, as a consequence of Jensen’s inequality we have that, given $A_2\in {\mathcal{A}}_2$:

$$\parallel H_{A_2}(x^{\prime },x_1,\theta )-M_n^{*}(x^{\prime },x_1,A_2){\parallel }_2\le {\parallel H_{A_2}(x^{\prime },x_1,\theta )-G(x^{\prime },x_1)\parallel }_2$$

for any real measurable function G on $(({\Omega }_1\times {\Omega }_2)\times {\Omega }_1,{\mathcal{A}}_1\times {\mathcal{A}}_2)\times {\mathcal{A}}_1)$. In particular, for any Markov kernel:

$$M:({({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1,({\mathcal{A}}_1\times {\mathcal{A}}_2)\times {\mathcal{A}}_1)\succ ⟶({\Omega }_2,{\mathcal{A}}_2),$$

we have:

$$\begin{array}{c}\parallel H_{A_2}(x^{\prime },x_1,\theta )-M_n^{*}(x^{\prime },x_1,A_2){\parallel }_2\le {\parallel H_{A_2}(x^{\prime },x_1,\theta )-M(x^{\prime },x_1,A_2)\parallel }_2,\end{array}$$

(11)

i.e.,

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }|M_n^{*}(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }|M(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\end{array}$$

When the $\sigma$-field ${\mathcal{A}}_2$ is separable, there exists a countable field ${\mathcal{A}}_{20}$ such that:

$$\begin{array}{c}C(x^{\prime },x_1,\theta ):=\underset{A_2\in {\mathcal{A}}_2}{sup}|M_n^{*}(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^2=\\ \underset{A_2\in {\mathcal{A}}_{20}}{sup}|M_n^{*}(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^2.\end{array}$$

Hence, $C(x^{\prime },x_1,\theta )$ is ${({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1\times \mathcal{T}$-measurable. Moreover, given $k\in \mathbb{N}$, there exists $A_{2k}(x^{\prime },x_1,\theta )\in {\mathcal{A}}_{20}$ such that:

$$C(x^{\prime },x_1,\theta )-\frac{1}{k}\le |M_n^{*}(x^{\prime },x_1,A_{2k}(x^{\prime },x_1,\theta ))-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_{2k}(x^{\prime },x_1,\theta )\right){|}^2.$$

Now we appeal to the Ryll-Nardzewski and Kuratowski measurable selection theorem as it appears in [10], p. 36. With the notation of that book, we take $({({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1\times \Theta ,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1\times \mathcal{T})$ to be the measurable space $(\Omega ,\mathcal{B})$, and ${\mathcal{A}}_{20}$ (the countable field generating ${\mathcal{A}}_2$) to be the complete separable metric space X. Given $k\in \mathbb{N}$, let us consider the map $S_k:{({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1\times \Theta \to \mathcal{P}\left(X\right)$ defined by:

$$S_k(x^{\prime },x_1,\theta ):=\left\{A_2\in {\mathcal{A}}_{20}:C(x^{\prime },x_1,\theta )-\frac{1}{k}\le |M_n^{*}(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^2\right\}$$

as the map $\Psi$ of that book. We have that $\varnothing \ne S_k(x^{\prime },x_1,\theta )\subset {\mathcal{A}}_{20}$ and $S_k(x^{\prime },x_1,\theta )$ is closed for the discrete topology on ${\mathcal{A}}_{20}$. Moreover, given an open set $U\subset {\mathcal{A}}_{20}$,

$$\{(x^{\prime },x_1,\theta ):S_k(x^{\prime },x_1,\theta )\cap U\ne \varnothing \}\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1\times \mathcal{T},$$

because, given $A_2\in {\mathcal{A}}_{20}$:

$$\begin{array}{c}\{(x^{\prime },x_1,\theta ):S_k(x^{\prime },x_1,\theta )\ni A_2\}=\\ \left\{(x^{\prime },x_1,\theta ):C(x^{\prime },x_1,\theta )-|M_n^{*}(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^2\le \frac{1}{k}\right\}.\end{array}$$

Thus, according to the aforementioned measurable selection theorem, there exists a measurable map:

$$s_k:({({\Omega }_1\times {\Omega }_2)}^n\times {\Omega }_1\times \Theta ,{({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n\times {\mathcal{A}}_1\times \mathcal{T})\to ({\mathcal{A}}_{20},\mathcal{P}\left({\mathcal{A}}_{20}\right))$$

such that:

$$s_k(x^{\prime },x_1,\theta )\in S_k(x^{\prime },x_1,\theta )$$

for every $(x^{\prime },x_1,\theta )$, or, which is the same:

$$C(x^{\prime },x_1,\theta )-\frac{1}{k}\le |M_n^{*}(x^{\prime },x_1,s_k(x^{\prime },x_1,\theta ))-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(s_k(x^{\prime },x_1,\theta )\right){|}^2.$$

Hence:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }C(x^{\prime },x_1,\theta )d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }|M_n^{*}(x^{\prime },x_1,s_k(x^{\prime },x_1,\theta ))-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(s_k(x^{\prime },x_1,\theta )\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta )+\frac{1}{k}\le \\ {\int }_{{({\Omega }_1\times {\Omega }_2)}^{n+1}\times \Theta }\underset{A_2\in {\mathcal{A}}_2}{sup}|M(x^{\prime },x_1,A_2)-R_{\theta }^{{\pi }_2|{\pi }_1=x_1}\left(A_2\right){|}^{2}d{\Pi }_n(x^{\prime },x,\theta )+\frac{1}{k}.\end{array}$$

With k being arbitrary, this shows that $M_n^{*}$ is the Bayes estimator of the conditional distribution $P^{X_2|X_1}(=R^{{\pi }_2|{\pi }_1})$ for the squared total variation loss function. □

Proof of Theorem 3.

According to (11), taking $A_2=]-\infty ,t]$, we have that, given $t\in \mathbb{R}$:

$$\parallel F_{\theta }(x^{\prime },x_1,t)-F_n^{*}(x^{\prime },x_1,t){\parallel }_2\le {\parallel F_{\theta }(x^{\prime },x_1,t)-F(x^{\prime },x_1,t)\parallel }_2$$

for every estimator F of the conditional distribution function $F_{\theta }$, where:

$$F_n^{*}(x^{\prime },x_1,t):=M_n^{*}(x^{\prime },x_1,]-\infty ,t\left]\right)={\left({R_{n,x^{\prime }}^{*}}^R\right)}^{{\pi }_2|{\pi }_1=x_1}\left(\right]-\infty ,t\left]\right),$$

i.e., for any t and every estimator F of $F_{\theta }$:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }{|F_n^{*}(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }{|F(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|}^{2}d{\Pi }_n(x^{\prime },x,\theta )\end{array}$$

Let $C(x^{\prime },x_1,\theta ):={sup}_{t\in \mathbb{R}}{|F_n^{*}(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|}^2$. Since:

$$\underset{t\in \mathbb{R}}{sup}|F_n^{*}(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|=\underset{r\in \mathbb{Q}}{sup}|F_n^{*}(x^{\prime },x_1,r)-F_{\theta }(x_1,r)|,$$

we have that, given $(x^{\prime },x_1,\theta )$ and $k\in \mathbb{N}$, there exists $r_k\in \mathbb{Q}$ such that:

$$C(x^{\prime },x_1,\theta )\le {|F_n^{*}(x^{\prime },x_1,r_k)-F_{\theta }(x_1,r_k)|}^2+\frac{1}{k}$$

and hence:

$$\begin{array}{c}{\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }C(x^{\prime },x_1,\theta )d{\Pi }_n(x^{\prime },x,\theta )\le \\ {\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }{|F_n^{*}(x^{\prime },x_1,r_k)-F_{\theta }(x_1,r_k)|}^{2}d{\Pi }_n(x^{\prime },x,\theta )+\frac{1}{k}\le \\ {\int }_{{({\Omega }_1\times \mathbb{R})}^{n+1}\times \Theta }\underset{t\in \mathbb{R}}{sup}{|F(x^{\prime },x_1,t)-F_{\theta }(x_1,t)|}^{2}d{\Pi }_n(x^{\prime },x,\theta )+\frac{1}{k}.\end{array}$$

Note that $r_k=r_k(x^{\prime },x_1,\theta )$, and a judicious use of the aforementioned measurable selection theorem could solve the measure-theoretical technicalities in these inequalities, as in the proof of Theorem 1. With k being arbitrary, the result is proved. □

Proof of Theorem 4.

To prove the result, Jensen’s inequality could be helpful if we were to show that:

$$m_n^{*}(x^{\prime },x_1)=E_{{\Pi }_n}\left(F\right|{\pi }^{\prime }=x^{\prime },{\pi }_1=x_1),$$

where $F(x^{\prime },x,\theta ):=r_{\theta }\left(x_1\right)(=E_{P_{\theta }}\left(X_2\right|X_1=x_1)=E_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1))$. By definition of conditional expectation, this is equivalent to proving that, for all $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times \mathcal{R})}^n$ and $A_1\in {\mathcal{A}}_1$:

$${\int }_{A_{12,n}^{\prime }\times A_1\times \mathbb{R}\times \Theta }F(x^{\prime },x,\theta )d{\Pi }_n(x^{\prime },x,\theta )={\int }_{A_{12,n}^{\prime }\times A_1}{m}_1^{*}(x^{\prime },x_1)d{{\Pi }_n}^{({\pi }^{\prime },{\pi }_1)}(x^{\prime },x_1),$$

or, which is the same:

$${\int }_{A_{12,n}^{\prime }\times A_1\times \mathbb{R}\times \Theta }{E}_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)d{\Pi }_n(x^{\prime },x,\theta )={\int }_{A_{12,n}^{\prime }\times A_1}{E}_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{{\Pi }_n}^{({\pi }^{\prime },{\pi }_1)}(x^{\prime },x_1).$$

Let us write $\left(\alpha \right)$ for the first term in this equality and $\left(\beta \right)$ for the second.

Note that:

$$\begin{array}{c}\left(\alpha \right)={\int }_{A_{12,n}^{\prime }\times A_1\times \mathbb{R}\times \Theta }{E}_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)d{\Pi }_n(x^{\prime },x,\theta )=\\ {\int }_{\Theta }{\int }_{A_{12,n}^{\prime }}{\int }_{A_1}{E}_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ {\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right){\int }_{A_1}{E}_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }^{{\pi }_1}\left(x_1\right)dQ\left(\theta \right).\end{array}$$

Moreover:

$$\begin{array}{c}\left(\beta \right)={\int }_{A_{12,n}^{\prime }\times A_1}{E}_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{{\Pi }_n}^{({\pi }^{\prime },{\pi }_1)}(x^{\prime },x_1)=\\ {\int }_{A_{12,n}^{\prime }\times A_1\times \mathbb{R}\times \Theta }{E}_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{\Pi }_n(x^{\prime },x,\theta )=\\ {\int }_{\Theta }{\int }_{A_{12,n}^{\prime }}{\int }_{A_1}{E}_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }^{{\pi }_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ {\int }_{\Theta }{\int }_{A_{12,n}^{\prime }}{h}_{A_1}(x^{\prime },\theta )d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)={\int }_{\Theta \times A_{12,n}^{\prime }}{h}_{A_1}(x^{\prime },\theta )d(Q\otimes R^n)(\theta ,x^{\prime }),\end{array}$$

where:

$$h_{A_1}(x^{\prime },\theta ):={\int }_{A_1}{E}_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }^{{\pi }_1}\left(x_1\right).$$

Since $Q\otimes R^n=R_n^{*}\otimes {\beta }_{12,n}^{*}$, we have:

$$\begin{array}{c}\left(\beta \right)={\int }_{A_{12,n}^{\prime }\times \Theta }{h}_{A_1}(x^{\prime },\theta )d(R_n^{*}\otimes {\beta }_{12,n}^{*})(x^{\prime },\theta )={\int }_{A_{12,n}^{\prime }}{\int }_{\Theta }{h}_{A_1}(x^{\prime },\theta )d{R}_{n,x^{\prime }}^{*}\left(\theta \right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }}{\int }_{\Theta }{\int }_{{\Omega }_1}{I}_{A_1}\left(x\right)E_{{R_{x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }\left(x\right)d{R}_{n,x^{\prime }}^{*}\left(\theta \right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }}{\int }_{{\Omega }_1\times \mathbb{R}}{I}_{A_1}\left(x\right)E_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{R_{n,x^{\prime }}^{*}}^R\left(x\right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }}{\int }_{A_1}{E}_{{R_{n,x^{\prime }}^{*}}^R}\left({\pi }_2\right|{\pi }_1=x_1)d{\left({R_{n,x^{\prime }}^{*}}^R\right)}^{{\pi }_1}\left(x_1\right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }}{\int }_{A_1\times \mathbb{R}}{\pi }_2\left(x\right)d{R_{n,x^{\prime }}^{*}}^R\left(x\right)d{\beta }_{12,n,Q}^{*}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }}{\int }_{\Theta }{\int }_{{\Omega }_1\times \mathbb{R}}{I}_{A_1}\left(x_1\right)x_{2}d{R}_{\theta }\left(x\right)d{R}_{n,x^{\prime }}^{*}\left(\theta \right)d{\beta }_{12,n}^{*}\left(x^{\prime }\right)=\\ {\int }_{A_{12,n}^{\prime }\times \Theta }{g}_{A_1}\left(\theta \right)d(R_n^{*}\otimes {\beta }_{12,n}^{*})(x^{\prime },\theta )\end{array}$$

where:

$$g_{A_1}\left(\theta \right)={\int }_{{\Omega }_1\times \mathbb{R}}{I}_{A_1}\left(x_1\right)x_{2}d{R}_{\theta }\left(x\right).$$

Using again that $Q\otimes R^n=R_n^{*}\otimes {\beta }_{12,n}^{*}$, we have:

$$\begin{array}{c}\left(\beta \right)={\int }_{\Theta \times A_{12,n}^{\prime }}{g}_{A_1}\left(\theta \right)d(Q\otimes R^n)(\theta ,x^{\prime })={\int }_{\Theta }{\int }_{A_{12,n}^{\prime }}{g}_{A_1}\left(\theta \right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ {\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right){\int }_{A_1\times \mathbb{R}}{x}_{2}d{R}_{\theta }\left(x\right)dQ\left(\theta \right)=\\ {\int }_{\Theta }{R}_{\theta }^n\left(A_{12,n}^{\prime }\right){\int }_{A_1}{E}_{R_{\theta }}\left({\pi }_2\right|{\pi }_1=x_1)d{R}_{\theta }^{{\pi }_1}\left(x_1\right)dQ\left(\theta \right),\end{array}$$

which proves that $\left(\alpha \right)=\left(\beta \right)$, and hence, that:

$$m_n^{*}(x^{\prime },x_1)=E_{{\Pi }_n}\left(F\right|{\pi }^{\prime }=x^{\prime },{\pi }_1=x_1).$$

Thus,

$$E_{{\Pi }_n}\left[{(m_n^{*}(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]\le E_{{\Pi }_n}\left[{(m_n(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]$$

for any other estimator $m_n$ of the regression curve $r_{\theta }$, i.e., $m_n^{*}$ is the Bayes estimator of the regression curve $r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)$ for the squared error loss function. □