Asymptotic Behavior of the Bayes Estimator of a Regression Curve

Abstract

In this work, we prove the convergence to 0 in both $L^1$ and $L^2$ of the Bayes estimator of a regression curve (i.e., the conditional expectation of the response variable given the regressor). The strong consistency of the estimator is also derived. The Bayes estimator of a regression curve is the regression curve with respect to the posterior predictive distribution. The result is general enough to cover discrete and continuous cases, parametric or nonparametric, and no specific supposition is made about the prior distribution. Some examples, two of them of a nonparametric nature, are given to illustrate the main result; one of the nonparametric examples exhibits a situation where the estimation of the regression curve has an optimal solution, although the problem of estimating the density is meaningless. An important role in the demonstration of these results is the establishment of a probability space as an adequate framework to address the problem of estimating regression curves from the Bayesian point of view, putting at our disposal powerful probabilistic tools in that endeavor.

1. Introduction

Given a random variable $X_1$—the independent variable, regressor, or predictor—and a real random variable $X_2$—the dependent variable or response—the so called regression curve of $X_2$ given $X_1$ is the map $r\left(x_1\right):=E\left(X_2\right|X_1=x_1)$, the function of $X_1$ that best approximates $X_2$ in the least squares sense; therefore, it becomes an essential tool in the study of the relationship between these two variables. Many statistical problems in practice, especially those related to prediction, require the estimation of the regression function from data, i.e., from a sample $(x_{1i},x_{2i})$, $i=1,\dots ,n$, of the joint distribution of $X_1$ and $X_2$. This estimation problem has been addressed in a good number of papers in both parametric and nonparametric contexts, from both the frequentist and the Bayesian points of view. In fact, regression techniques are among the most widely used methods in applied statistics.

In a nonparametric frequentist framework, the problem of estimation of the regression curve was first considered in [1,2]. We refer to [3] for this problem in a Bayesian context; it includes some historical notes about Bayesian nonparametric regression and some results about the consistency of the estimates for some specific priors.

Talking about the probability of an event A (written $P_{\theta }\left(A\right)$) in a statistical context is ambiguous, as it depends of the unknown parameter. In a Bayesian context, once the data $\omega$ has been observed, a natural estimate of $P_{\theta }\left(A\right)$ is the posterior predictive probability of A given $\omega$ since it is the posterior mean of the probabilities of A given $\omega$, which, as is well known, is the Bayes estimator of $P_{\theta }\left(A\right)$ for the squared error loss function. This simple fact already justifies the use of the posterior predictive distribution as an estimator of the sampling distribution, but, in reality, much more is true because, as shown in [4], the posterior predictive distribution is the Bayes estimator of the sampling probability distribution $P_{\theta }$ for the squared total variation loss function. It is similar to what happens with the strong law of large numbers and the Glivenko–Cantelli theorem: the first guarantees an almost certain punctual convergence of the empirical distribution function to the unknown population distribution function, but the second yields an almost sure uniform convergence, becoming the fundamental theorem of Mathematical Statistics. The problem of estimation of the density in a Bayesian nonparametric framework is considered as a number of references, such as [5], [6], [7], or [8]. In [4], the problem of estimation of the density from a Bayesian point of view is also addressed, and, under mild conditions, it is shown that the posterior predictive density is the Bayes estimator for the $L^1$-squared loss function, and the convergence to 0 of the Bayes risk (and the strong consistency) of this estimator is shown in [9].

As regards the estimation of the regression curve, or even the conditional density, reference [10] or reference [11] contain sufficient arguments on the usefulness of these problems in practice from a frequentist point of view, problems that go back to [12], although they have not produced much literature since then either. The paper [13] deals, among others, with the problem of the Bayesian estimation of a regression curve and proves that the regression curve, with respect to the posterior predictive distribution, is the Bayes estimator (for the squared error loss function). Here, we wonder about the convergence to 0 of its Bayes risk (and its strong consistency). This is the main goal of the paper, and Theorem 1 below answers the question in the affirmative.

So, the posterior predictive distribution is the key to the estimation problems raised above. It has been presented in the literature as the base of Predictive Inference, which seeks to make inferences about a new unknown observation from the previous random sample instead of estimating an unknown parameter. It should be noted that, in practice, the explicit evaluation of the posterior predictive distribution could be cumbersome, and its simulation may become preferable. The interested reader can find in the papers mentioned above, and the references therein, more information on the problems of estimating the density or regression curve, from both the frequentist and Bayesian perspectives, or about the usefulness of the posterior predictive distribution in Bayesian Inference and its calculation. We place special emphasis on the monographs [3,14,15].

In Section 2, an important and useful achievement of the paper is obtained, as it establishes a probability space as the theoretical framework (i.e., Bayesian experiment) appropriate to address the problem (in the same way that [16] considers the Bayesian experiment as a probability space). In fact, starting from the Bayesian experiment (1) corresponding to a sample of size m (possibly infinite) from the joint distribution of the two variables $X_1$ (predictor) and $X_2$ (response) of interest, the probability space (3) is presented as the appropriate model for the estimation of the regression curve of $X_2$ given $X_1=x_1$ from an m-sized sample of the joint distribution of the two variables. This has allowed us to obtain an explicit expression of the Bayes risk of an estimator of the regression curve and take advantage of powerful probabilistic tools when solving the problem of its asymptotic behavior.

Section 3 includes the aforementioned Theorem 1, whose proof lies in Jensen’s inequality, Lévy’s martingale convergence theorem, and a result by Doob on the consistency of the posterior distribution. The result is general enough to cover discrete and continuous cases, parametric or nonparametric, as the examples provided show, and, unlike what we have been able to find in the literature, no specific supposition is made about the prior distribution.

Section 4 contains the proof of the main result and some auxiliary results. In particular, Lemma 1, the key of the proof of the Theorem, yields a representation of the Bayes estimator of the regression curve as its conditional mean in the Bayesian experiment (3).

Section 5 includes some examples to illustrate the main result of the paper, two of them of a nonparametric nature. The last of these two nonparametric examples shows a situation where estimating the regression curve has a Bayes estimator, although the problem of estimating the density is meaningless; in fact, the estimation of the regression function is performed through the conditional distribution itself.

For ease of reading, we encourage the reader who is not familiar with the terminology or the notation used in the paper to start by reading the Appendix of [17].

2. The Framework

We recall from [13] the appropriate framework to address the problem and update it to incorporate the required asymptotic flavor.

Let $(\mathrm{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})$ be a Bayesian statistical experiment and

$$X_i:(\mathrm{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})\to ({\mathrm{\Omega }}_i,{\mathcal{A}}_i),i=1,2,$$

two statistics. Consider the Bayesian experiment image of $(X_1,X_2)$

$$({\mathrm{\Omega }}_1\times {\mathrm{\Omega }}_2,{\mathcal{A}}_1\times {\mathcal{A}}_2,\{P_{\theta }^{(X_1,X_2)}:\theta \in (\Theta ,\mathcal{T},Q)\}).$$

In the following, we will assume that $P^{(X_1,X_2)}(\theta ,A_{12}):=P_{\theta }^{(X_1,X_2)}\left(A_{12}\right)$, $\theta \in \Theta ,A_{12}\in {\mathcal{A}}_1\times {\mathcal{A}}_2$, and the joint distribution of $X_1$ and $X_2$, is a Markov kernel. Let us write $R_{\theta }=P_{\theta }^{(X_1,X_2)}$ and $p_j\left(x\right):=x_j$ for $j=1,2$, $x:=(x_1,x_2)\in {\mathrm{\Omega }}_1\times {\mathrm{\Omega }}_2$. Hence

$$\begin{array}{c}{P}_{\theta }^{X_1}=R_{\theta }^{p_1}\mathrm{and}{P}_{\theta }^{X_2|X_1=x_1}=R_{\theta }^{p_2|p_1=x_1},\end{array}$$

and, when $X_2$ is a real random variable, $E_{P_{\theta }}\left(X_2\right|X_1=x_1)=E_{R_{\theta }}\left(p_2\right|p_1=x_1)$. In order to alleviate and shorten the notation, we write ${\mathrm{\Omega }}_{12}={\mathrm{\Omega }}_1\times {\mathrm{\Omega }}_2$ and ${\mathcal{A}}_{12}={\mathcal{A}}_1\times {\mathcal{A}}_2$.

Given an integer n, for $m=n$ (respectively, $m=\mathbb{N}$), the Bayesian experiment corresponding to a n-sized sample (respectively, an infinite sample) of the joint distribution of $(X_1,X_2)$ is

$$\begin{array}{c}\left({\mathrm{\Omega }}_{12}^m,{\mathcal{A}}_{12}^m,\left\{R_{\theta }^m:\theta \in (\Theta ,\mathcal{T},Q)\right\}\right).\end{array}$$

(1)

We define the Markov kernel $R^m$ by $R^m(\theta ,A_{12,m}^{\prime }):=R_{\theta }^m\left(A_{12,m}^{\prime }\right)$, for $A_{12,m}^{\prime }\in {\mathcal{A}}_{12}^m$, and denote

$${\mathrm{\Pi }}_{12,m}:=Q\otimes R^m$$

for the joint distribution of the parameter and the sample, i.e.,

$${\mathrm{\Pi }}_{12,m}(A_{12,m}^{\prime }\times T)={\mathrm{\int }}_T{R}_{\theta }^m\left(A_{12,m}^{\prime }\right)dQ\left(\theta \right),A_{12,m}^{\prime }\in {\mathcal{A}}_{12}^m,T\in \mathcal{T}.$$

The corresponding prior predictive distribution ${\beta }_m^{*}$ on ${\mathrm{\Omega }}_{12}^m$ is

$${\beta }_m^{*}\left(A_{12,m}^{\prime }\right)={\int }_{\Theta }{R}_{\theta }^m\left(A_{12,m}^{\prime }\right)dQ\left(\theta \right),A_{12,m}^{\prime }\in {\mathcal{A}}_{12}^m.$$

The posterior distribution is a Markov kernel

$$R_m^{*}:({\mathrm{\Omega }}_{12}^m,{\mathcal{A}}_{12}^m)\succ ⟶(\Theta ,\mathcal{T})$$

such that, for all $A_{12,m}^{\prime }\in {\mathcal{A}}_{12}^m$ and $T\in \mathcal{T}$,

$${\mathrm{\Pi }}_{12,m}(A_{12,m}^{\prime }\times T)={\int }_T{R}_{\theta }^m\left(A_{12,m}^{\prime }\right)dQ\left(\theta \right)={\int }_{A_{12,m}^{\prime }}{R}_m^{*}(x^{\prime },T)d{\beta }_m^{*}\left(x^{\prime }\right).$$

Let us write $R_{m,x^{\prime }}^{*}\left(T\right):=R_m^{*}(x^{\prime },T)$.

The posterior predictive distribution on ${\mathcal{A}}_{12}$ is the Markov kernel

$${R_m^{*}}^R:({\mathrm{\Omega }}_{12}^m,{\mathcal{A}}_{12}^m)\succ ⟶({\mathrm{\Omega }}_{12},{\mathcal{A}}_{12})$$

defined, for $x^{\prime }\in {\mathrm{\Omega }}_{12}^m$, by

$${R_m^{*}}^R(x^{\prime },A_{12}):={\int }_{\Theta }{R}_{\theta }\left(A_{12}\right)d{R}_{m,x^{\prime }}^{*}\left(\theta \right)$$

This way, given $x^{\prime }$, the posterior predictive probability of an event $A_{12}$ is nothing but the posterior mean of $R_{\theta }\left(A_{12}\right)$. It follows that, with obvious notations,

$${\int }_{{\mathrm{\Omega }}_{12}}f\left(x\right)d{R_{m,x^{\prime }}^{*}}^R\left(x\right)={\int }_{\Theta }{\int }_{{\mathrm{\Omega }}_{12}}f\left(x\right)d{R}_{\theta }\left(x\right)d{R}_{m,x^{\prime }}^{*}\left(\theta \right)$$

for any non-negative or integrable real random variable f.

We can also consider the posterior predictive distribution on ${\mathcal{A}}_{12}^m$, defined as the Markov kernel

$${R_m^{*}}^{R^m}:({\mathrm{\Omega }}_{12}^m,{\mathcal{A}}_{12}^m)\succ ⟶({\mathrm{\Omega }}_{12}^m,{\mathcal{A}}_{12}^m)$$

such that

$${R_m^{*}}^{R^m}(x^{\prime },A_{12,m}^{\prime }):={\int }_{\Theta }{R}_{\theta }^m\left(A_{12,m}^{\prime }\right)d{R}_{m,x^{\prime }}^{*}\left(\theta \right).$$

Looking for the appropriate framework to address the problem of estimating the regression curve of the real random variable $X_2$ given $X_1=x_1$, we part from the Bayesian experiment (1) corresponding to a sample $x^{\prime }\in {\mathrm{\Omega }}_{12}^m$ from the joint distribution $R_{\theta }$ of the predictor $X_1$ and the response $X_2$, and we choose $x=(x_1,x_2)\in {\mathrm{\Omega }}_{12}$ (in fact, we only need the first coordinate $x_1$ of x as the argument of the regression curve) from $R_{\theta }$ independently of $x^{\prime }$, which brings us to the product Bayesian experiment

$$\begin{array}{c}\left({\mathrm{\Omega }}_{12}^m\times {\mathrm{\Omega }}_{12},{\mathcal{A}}_{12}^m\times {\mathcal{A}}_{12},\left\{R_{\theta }^m\times R_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\right\}\right).\end{array}$$

(2)

The Bayesian experiment (2) can be identified in a standard way with the probability space

$$\begin{array}{c}({\mathrm{\Omega }}_{12}^m\times {\mathrm{\Omega }}_{12}\times \Theta ,{\mathcal{A}}_{12}^m\times {\mathcal{A}}_{12}\times \mathcal{T},{\mathrm{\Pi }}_m),\end{array}$$

(3)

where ${\mathrm{\Pi }}_m:=(R^m\times R)\otimes Q$, i.e.,

$${\mathrm{\Pi }}_m(A_{12,m}^{\prime }\times A_{12}\times T)={\int }_T{R}_{\theta }\left(A_{12}\right)R_{\theta }^m\left(A_{12,m}^{\prime }\right)dQ\left(\theta \right),$$

when $A_{12,m}^{\prime }\in {\mathcal{A}}_{12}^m$, $A_{12}\in {\mathcal{A}}_{12}$ and $T\in \mathcal{T}$.

So, for a real random variable f on $({\mathrm{\Omega }}_{12}^m\times {\mathrm{\Omega }}_{12}\times \Theta ,{\mathcal{A}}_{12}^m\times {\mathcal{A}}_{12}\times \mathcal{T})$,

$$\begin{array}{c}\int fd{\mathrm{\Pi }}_m={\int }_{\Theta }{\int }_{{\mathrm{\Omega }}_{12}^m}{\int }_{{\mathrm{\Omega }}_{12}}f(x^{\prime },x,\theta )d{R}_{\theta }\left(x\right)d{R}_{\theta }^m\left(x^{\prime }\right)dQ\left(\theta \right),\end{array}$$

provided that the integral exists. Moreover, for a real random variable h on $({\mathrm{\Omega }}_{12}\times \Theta ,{\mathcal{A}}_{12}\times \mathcal{T})$, by definition of the posterior distributions,

$$\int hd{\mathrm{\Pi }}_m={\int }_{\Theta }{\int }_{{\mathrm{\Omega }}_{12}}h(x,\theta )d{R}_{\theta }\left(x\right)dQ\left(\theta \right)={\int }_{{\mathrm{\Omega }}_{12}}{\int }_{\Theta }h(x,\theta )d{R}_{1,x}^{*}\left(\theta \right)d{\beta }_1^{*}\left(x\right).$$

3. The Bayes Estimator of the Regression Curve: Asymptotic Behavior

Now suppose that $({\mathrm{\Omega }}_2,{\mathcal{A}}_2)=(\mathbb{R},\mathcal{R})$. Let $X_2$ be a squared-integrable real random variable, such that $E_{\theta }\left(X_2^2\right)$ has a finite prior mean; in particular, $E_{\theta }\left(X_2\right)$ also has a finite prior mean.

In the regression curve of $X_2$, given that $X_1$ is the map, $x_1\in {\mathrm{\Omega }}_1\mapsto r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)$ An estimator of the regression curve $r_{\theta }$ from a sample of size n of the joint distribution of $(X_1,X_2)$ is a statistic

$$m:(x^{\prime },x_1)\in {({\mathrm{\Omega }}_1\times \mathbb{R})}^n\times {\mathrm{\Omega }}_1⟼m(x^{\prime },x_1)\in \mathbb{R},$$

so that, being observed, the sample $x^{\prime }\in {({\mathrm{\Omega }}_1\times \mathbb{R})}^n$, $m(x^{\prime },·)$ is the estimation of $r_{\theta }$.

From a classical point of view, the simplest way to evaluate the error in estimating an unknown regression curve is to use the expectation of the quadratic deviation (see [18], p. 120):

$$\begin{array}{c}{E}_{\theta }\left[{\int }_{{\mathrm{\Omega }}_1}{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^{2}d{P}_{\theta }^{X_1}\left(x_1\right)\right]=\\ {\int }_{{({\mathrm{\Omega }}_1\times \mathbb{R})}^n}{\int }_{{\mathrm{\Omega }}_1}{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^{2}d{R}_{\theta }^{p_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right).\end{array}$$

From a Bayesian point of view, the Bayes estimator—the optimal estimator—of the regression curve $r_{\theta }$ should minimize the Bayes risk (i.e., the prior mean of the expectation of the quadratic deviation)

$$\begin{array}{c}{\int }_{\Theta }{\int }_{{({\mathrm{\Omega }}_1\times \mathbb{R})}^n}{\int }_{{\mathrm{\Omega }}_1}{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^{2}d{R}_{\theta }^{p_1}\left(x_1\right)d{R}_{\theta }^n\left(x^{\prime }\right)dQ\left(\theta \right)=\\ E_{{\mathrm{\Pi }}_n}\left[{(m(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right].\end{array}$$

So, the Bayesian experiment (3) is the appropriate framewok to address these questions (see also Remark 1).

Recall from [13] that the regression curve of $p_2$ on $p_1$ with respect to the posterior predictive distribution ${R_{n,x^{\prime }}^{*}}^R$, given the data $x^{\prime }$, is the Bayes estimator of the regression curve $r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)$ for the squared error loss function; for the sake of completeness, this proposition is also included as part (i) of Theorem 1 below.

We wonder about the convergence to 0 of the Bayes risk:

$$E_{{\mathrm{\Pi }}_n}\left[{(m_n^{*}(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right].$$

Another question of interest is the consistency of this Bayes estimator.

Lemma 1 below is key to solving the problem, since it shows that the Bayes estimator of the regression curve becomes its conditional mean in the Bayesian experiment (3). What the following theorem really provides is the asymptotic behavior of this estimator: the convergence to zero of its Bayes risks and the strong consistency of the Bayes estimator of the regression curve.

Theorem 1.

Let $(\mathrm{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})$ be a Bayesian statistical experiment and $X_1:(\mathrm{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})\to ({\mathrm{\Omega }}_1,{\mathcal{A}}_1)$ and $X_2:(\mathrm{\Omega },\mathcal{A},\{P_{\theta }:\theta \in (\Theta ,\mathcal{T},Q)\})\to (\mathbb{R},\mathcal{R})$ be two statistics, such that $E_{\theta }\left(X_2^2\right)$ has a finite prior mean. Let us suppose that: (a) $({\mathrm{\Omega }}_1,{\mathcal{A}}_1)$ is a standard Borel space; (b) Θ is a Borel subset of a Polish space, and $\mathcal{T}$ is its Borel σ-field; and (c) $\{R_{\theta }:\theta \in \Theta \}$ is identifiable.

Then,

(i) The regression curve of $p_2$ on $p_1$ with respect to the posterior predictive distribution ${R_{n,x^{\prime }}^{*}}^R$,

$$m_n^{*}(x^{\prime },x_1):=E_{{R_{n,x^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1),$$

is the Bayes estimator of the regression curve $r_{\theta }\left(x_1\right):=E_{\theta }\left(X_2\right|X_1=x_1)$ for the squared error loss function, i.e.,

$$E_{{\mathrm{\Pi }}_n}\left[{(m_n^{*}(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]\le E_{{\mathrm{\Pi }}_n}\left[{(m_n(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]$$

for any other estimator $m_n$ of the regression curve $r_{\theta }$.

(ii) Moreover, $m_n^{*}$ is a strongly consistent estimator of the regression curve, in the sense that

$$\underset{n}{lim}{E}_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{\theta }\left(X_2\right|X_1=x_1),{\mathrm{\Pi }}_{\mathbb{N}}-\mathrm{a}.\mathrm{e}.,$$

where $x_{\left(n\right)}^{\prime }:=(x_1^{\prime },\dots ,x_n^{\prime })$ if $x^{\prime }\in {\mathrm{\Omega }}_{12}^{\mathbb{N}}$.

(iii) Finally, the Bayes risk of $m_n^{*}$ converges to 0 for both the $L^1$ and the $L^2$ loss functions, i.e.,

$$\underset{n}{lim}{E}_{{\mathrm{\Pi }}_{\mathbb{N}}}\left[\right|m_n^{*}(x^{\prime },x_1)-r_{\theta }\left(x_1\right){|}^k]=0,k=1,2.$$

4. Proofs and Auxiliary Results

Let us introduce some notations for different projections in the Bayesian model (3). Given $(x^{\prime },x,\theta )\in {\mathrm{\Omega }}_{12}^m\times {\mathrm{\Omega }}_{12}\times \Theta$, we write

$$\begin{array}{c}{\pi }_m^{\prime }(x^{\prime },x,\theta ):=x^{\prime },{\pi }_m(x^{\prime },x,\theta ):=x,{\pi }_{j,m}(x^{\prime },x,\theta ):=x_j,j=1,2,q_m(x^{\prime },x,\theta ):=\theta \\ {\pi }_{i,m}^{\prime }(x^{\prime },x,\theta ):=x_i^{\prime }:=(x_{i1}^{\prime },x_{i2}^{\prime }),{\pi }_{\left(i\right),m}^{\prime }(x^{\prime },x,\theta ):=(x_1^{\prime },\dots ,x_i^{\prime }),\end{array}$$

for $1\le i\le m$ (read $i\in \mathbb{N}$ if $m=\mathbb{N}$).

The following result is taken from [13].

Proposition 1.

For $n\in \mathbb{N}$,

$$\begin{array}{c}{\mathrm{\Pi }}_{\mathbb{N}}^{({\pi }_{\left(n\right),\mathbb{N}}^{\prime },{\pi }_{1,\mathbb{N}},q_{\mathbb{N}})}={\mathrm{\Pi }}_n,{\mathrm{\Pi }}_{\mathbb{N}}^{({\pi }_{\left(n\right),\mathbb{N}}^{\prime },{\pi }_{1,\mathbb{N}})}={\mathrm{\Pi }}_n^{({\pi }_{\left(n\right),n}^{\prime },{\pi }_{1,n})},\\ {\mathrm{\Pi }}_m^{q_m}=Q,{\mathrm{\Pi }}_m^{({\pi }_m^{\prime },q_m)}={\mathrm{\Pi }}_{12,m},{\mathrm{\Pi }}_m^{{\pi }_m^{\prime }}={\beta }_m^{*},{\mathrm{\Pi }}_m^{({\pi }_m,q_m)}={\mathrm{\Pi }}_{12,1},{\mathrm{\Pi }}_m^{{\pi }_m}={\beta }_1^{*},\\ {\mathrm{\Pi }}_m^{{\pi }_m^{\prime }|q_m=\theta }=R_{\theta }^m,{\mathrm{\Pi }}_m^{{\pi }_m|q_m=\theta }=R_{\theta },{\mathrm{\Pi }}_m^{q_m|{\pi }_m^{\prime }=x^{\prime }}=R_{m,x^{\prime }}^{*},{\mathrm{\Pi }}_m^{q_m|{\pi }_m=x}=R_{1,x^{\prime }}^{*}.\end{array}$$

Remark 1.

It follows from this proposition that the probability space (3) contains all the basic ingredients of the Bayesian experiment (1), i.e., the prior distribution, the sampling probabilities, the posterior distributions, and the prior predictive distribution. When $m=\mathbb{N}$, (3) becomes the natural framework to address the asymptotic problem considered in this paper, since it integrates the sample $x^{\prime }$, the parameter θ, and the argument $x_1$ of the regression function to be estimated into a simple joint probability distribution.

Lemma 1.

Let $Y(x^{\prime },x,\theta ):=E_{\theta }\left(p_2\right|p_1=x_1)$.

(i) For $n\in \mathbb{N}$ and an infinite sample $x^{\prime }$ of the joint distribution $R_{\theta }$ of $X_1$ and $X_2$, we show that

$$\begin{array}{c}{E}_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|({\pi }_{\left(n\right),\mathbb{N}}^{\prime },{\pi }_{1,\mathbb{N}})=(x_{\left(n\right)}^{\prime },x_1)),\end{array}$$

where $x_{\left(n\right)}^{\prime }:=(x_1^{\prime },\dots ,x_n^{\prime })$.

(ii) For an infinite sample $x^{\prime }$ of the distribution $R_{\theta }$, we show that

$$E_{{R_{\mathbb{N},x^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|({\pi }_{\mathbb{N}}^{\prime },{\pi }_{1,\mathbb{N}})=(x^{\prime },x_1)).$$

Proof for Lemma 1. 

(i) According to Lemma 1 of [13], we show that, for all $A_{12,n}^{\prime }\in {({\mathcal{A}}_1\times {\mathcal{A}}_2)}^n$ and all $A_i\in {\mathcal{A}}_i$, $i=1,2$,

$$\begin{array}{c}\begin{array}{c}\hfill {\int }_{A_{12,n}^{\prime }\times A_1\times {\mathrm{\Omega }}_2\times \Theta }{R}_{\theta }^{p_2|p_1=x_1}\left(A_2\right)d{\mathrm{\Pi }}_n(x^{\prime },x,\theta )=\\ \hfill {\int }_{A_{12,n}^{\prime }\times A_1}{\left[{R_{n,x^{\prime }}^{*}}^R\right]}^{p_2|p_1=x_1}\left(A_2\right)d{{\mathrm{\Pi }}_n}^{({\pi }^{\prime },p_1)}(x^{\prime },x_1).\end{array}\end{array}$$

(4)

The proof of (i) follows in a standard way from this and Proposition 1 as

$$E_{\theta }\left(p_2\right|p_1=x_1)={\int }_{\mathbb{R}}{x}_{2}d{R}_{\theta }^{p_2|p_1=x_1}\left(x_2\right)\mathrm{and}{E}_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)={\int }_{\mathbb{R}}{x}_{2}d{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R\left(x_2\right).$$

(ii) The proof of (ii) is analogous. □

Proof for Theorem 1. 

When ${\mathcal{A}}_{\left(n\right)}^{\prime }:={({\pi }_{\left(n\right),\mathbb{N}}^{\prime },{\pi }_{1,\mathbb{N}})}^{-1}({\mathcal{A}}_{12}^n\times {\mathcal{A}}_1)$, we have that ${\left({\mathcal{A}}_{\left(n\right)}^{\prime }\right)}_n$ is an increasing sequence of sub-$\sigma$-fields of ${\mathcal{A}}_{12}^{\mathbb{N}}\times {\mathcal{A}}_1$, such that ${\mathcal{A}}_{12}^{\mathbb{N}}\times {\mathcal{A}}_1=\sigma \left({\cup }_n{\mathcal{A}}_{\left(n\right)}^{\prime }\right)$. According to the martingale convergence theorem of Lévy, if Y is ${\mathcal{A}}_{12}^{\mathbb{N}}\times {\mathcal{A}}_1\times \mathcal{T}$-measurable and ${\mathrm{\Pi }}_{\mathbb{N}}$-integrable, then

$$E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|{\mathcal{A}}_{\left(n\right)}^{\prime })$$

converges ${\mathrm{\Pi }}_{\mathbb{N}}$-a.e. and in $L^1\left({\mathrm{\Pi }}_{\mathbb{N}}\right)$ to $E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|{\mathcal{A}}_{12}^{\mathbb{N}}\times {\mathcal{A}}_1)$.

Let us consider the measurable function

$$Y(x^{\prime },x,\theta ):=E_{\theta }\left(X_2\right|X_1=x_1).$$

Notice that $E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right)={\int }_{\Theta }{E}_{\theta }\left(X_2\right)dQ\left(\theta \right)$, so Y is ${\mathrm{\Pi }}_{\mathbb{N}}$-integrable. Hence, it follows from the aforementioned theorem of Lévy that

$$\begin{array}{c}\underset{n}{lim}{E}_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{{R_{\mathbb{N},x^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1),{\mathrm{\Pi }}_{\mathbb{N}}-\mathrm{a}.\mathrm{e}.\end{array}$$

(5)

and

$$\begin{array}{c}\underset{n}{lim}{\int }_{{\mathrm{\Omega }}_{12}^{\mathbb{N}}\times {\mathrm{\Omega }}_{12}\times \Theta }\left|E_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)-E_{{R_{\mathbb{N},x^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)\right|d{\mathrm{\Pi }}_{\mathbb{N}}(x^{\prime },x,\theta )=0.\end{array}$$

(6)

As a consequence of the known theorem of Doob (see Theorem 6.9 and Proposition 6.10 from [3], pp. 129, 130), for every $x_1\in {\mathrm{\Omega }}_1$,

$$\underset{n}{lim}{\int }_{\Theta }{E}_{{\theta }^{\prime }}\left(X_2\right|X_1=x_1)d{\mathrm{\Pi }}_{\mathbb{N}}^{q_{\mathbb{N}}|({\pi }_{\left(n\right),\mathbb{N}}^{\prime },{\pi }_{1,\mathbb{N}})=(x_{\left(n\right)}^{\prime },x_1)}\left({\theta }^{\prime }\right)=E_{\theta }\left(X_2\right|X_1=x_1),R_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

and for Q, almost every $\theta$. Hence, according to Lemma 1 (i),

$$\underset{n}{lim}{E}_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{\theta }\left(X_2\right|X_1=x_1),R_{\theta }^{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

and for Q, almost every $\theta$.

In particular,

$$\underset{n}{lim}{E}_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{\theta }\left(X_2\right|X_1=x_1),{\mathrm{\Pi }}_{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

In this sense, we can say that the predictive posterior regression curve $E_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)$ of $X_2$, given $X_1=x_1$, is a strongly consistent estimator of the sampling regression curve $E_{\theta }\left(X_2\right|X_1=x_1)$ of $X_2$, given $X_1=x_1$.

From this and (5), we obtain the following:

$$E_{{R_{\mathbb{N},x^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=E_{\theta }\left(X_2\right|X_1=x_1),{\mathrm{\Pi }}_{\mathbb{N}}-\mathrm{a}.\mathrm{e}.$$

According to (6), we obtain the following:

$$\begin{array}{c}\underset{n}{lim}{\int }_{{\mathrm{\Omega }}_{12}^{\mathbb{N}}\times {\mathrm{\Omega }}_{12}\times \Theta }\left|E_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)-E_{\theta }\left(X_2\right|X_1=x_1)\right|d{\mathrm{\Pi }}_{\mathbb{N}}(x^{\prime },x,\theta )=0,\end{array}$$

which proves that the Bayes risk of the optimal estimator $E_{{R_{n,x_{\left(n\right)}^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)$ of the regression curve $E_{\theta }\left(X_2\right|X_1=x_1)$ converges to 0 for the $L^1$-loss function.

We wonder if that also happens for the $L^2$-squared loss function, i.e., if the Bayes risk

$$E_{{\mathrm{\Pi }}_n}\left[{(m_n^{*}(x^{\prime },x_1)-r_{\theta }\left(x_1\right))}^2\right]$$

converges vers 0 when n goes to ∞. Theorem 6.6.9 of [19] shows that the answer is affirmative because

$$m_n^{*}(x^{\prime },x_1)=E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y\right|{\mathcal{A}}_{\left(n\right)}^{\prime })$$

and, by Jensen’s inequality,

$$\begin{array}{c}{E}_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(E_{{\mathrm{\Pi }}_{\mathbb{N}}}{\left(Y\right|{\mathcal{A}}_{\left(n\right)}^{\prime })}^2\right)\le E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y^2\right|{\mathcal{A}}_{\left(n\right)}^{\prime })\right)=E_{{\mathrm{\Pi }}_{\mathbb{N}}}\left(Y^2\right)\le {\int }_{\Theta }{E}_{\theta }\left(X_2^2\right)dQ\left(\theta \right)<\infty .\end{array}$$

This completes the proof. □

5. Examples

Example 1.

Let us suppose that, for $\theta ,\lambda ,x_1>0$, $P_{\theta }^{X_1}=G(1,{\theta }^{-1})$, $P_{\theta }^{X_2|X_1=x_1}=G(1,{\left(\theta x_1\right)}^{-1})$, and $Q=G(1,{\lambda }^{-1})$, where $G(\alpha ,\beta )$ denotes the gamma distribution of parameters $\alpha ,\beta >0$. Hence, the joint density of $X_1$ and $X_2$ is

$$f_{\theta }(x_1,x_2)={\theta }^2{x}_{1}exp\{-\theta x_1(1+x_2)\}{I}_{{]0,\infty [}^2}(x_1,x_2).$$

It is shown in [13], Example 1, that the Bayes estimator of the conditional density function $f_{\theta }^{X_2|X_1=x_1}\left(t\right)=\theta x_{1}texp\{-\theta x_{1}t\}{I}_{]0,\infty [}\left(t\right)$ (for $x_1>0$) is the conditional density of $X_2$, given $X_1=x_1$, with respect to the posterior predictive distribution, i.e.,

$$m_n^{*}(x^{\prime },x_1)={f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(t\right)={\displaystyle \frac{(2n+2)x_1{a}_n{(x^{\prime },x_1)}^{2n+2}}{{(x_{1}t+a_n(x^{\prime },x_1))}^{2n+3}}},$$

where $a_n(x^{\prime },x_1)=\lambda +x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime }).$

Let $\phi :{\mathbb{R}}_{+}\to \mathbb{R}$ be a real bounded measurable function. Hence, $\phi \circ X_2$ satisfies the conditions of Theorem 1. The Bayes estimator of the regression curve

$$r_{\theta }\left(x_1\right):=E_{\theta }(\phi \circ X_2|X_1=x_1)={\int }_0^{\infty }\phi \left(t\right)·f_{\theta }^{X_2|X_1=x_1}\left(t\right)dt$$

is

$$m_n^{*}(x^{\prime },x_1)={\int }_0^{\infty }\phi \left(t\right)·{f_{n,x^{\prime }}^{*}}^{X_2|X_1=x_1}\left(t\right)dt={\int }_0^{\infty }\phi \left(t\right)·{\displaystyle \frac{(2n+2)x_1{a}_n{(x^{\prime },x_1)}^{2n+2}}{{(x_{1}t+a_n(x^{\prime },x_1))}^{2n+3}}}dt.$$

For instance, it is readily shown that, if $\phi =I_{]0,1[}$, then the Bayes estimator is

$$m_n^{*}(x^{\prime },x_1)=1-{\left({\displaystyle \frac{\lambda +x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })}{\lambda +2x_1+{\sum }_{i=1}^n{x}_{i1}^{\prime }(1+x_{i2}^{\prime })}}\right)}^{2n+2}.$$

Theorem 1 shows that this a strongly consistent estimator of the regression curve $r_{\theta }\left(x_1\right)$, and its Bayes risk converges to 0 for both the $L^1$ and $L^2$ loss functions.

Example 2.

Let us suppose that $X_1$ has a Bernoulli distribution of unknown parameter $\theta \in ]0,1[$ (i.e., $P_{\theta }^{X_1}=Bi(1,\theta )$), and, given $X_1=k_1\in \{0,1\}$, $X_2$ has distribution $Bi(1,1-\theta )$ when $k_1=0$ and $Bi(1,\theta )$ when $k_1=1$, i.e., $P_{\theta }^{X_2|X_1=k_1}=Bi(1,k_1+(1-2k_1)(1-\theta ))$. We can think of tossing a coin with probability θ of getting heads ($=1$) and making a second toss of this coin if it comes up heads on the first toss or tossing a second coin with probability $1-\theta$ of making heads if the first toss is tails ($=0$). Consider the uniform distribution on $]0,1[$ as the prior distribution Q.

So, the joint probability function of $X_1$ and $X_2$ is

$$\begin{array}{c}\begin{array}{cc}\hfill f_{\theta }(k_1,k_2)& ={\theta }^{k_1}{(1-\theta )}^{1-k_1}{[k_1+(1-2k_1)(1-\theta )]}^{k_2}{[1-k_1-(1-2k_1)(1-\theta )]}^{1-k_2}\hfill \\ & =\left\{\begin{array}{c}\theta (1-\theta )\mathrm{if}{k}_2=0,\hfill \\ {(1-\theta )}^2\mathrm{if}{k}_1=0,k_2=1,\hfill \\ {\theta }^2\mathrm{if}{k}_1=1,k_2=1.\hfill \end{array}\right.\hfill \end{array}\end{array}$$

It is shown in [13], Example 2, that the Bayes estimator of the conditional mean $r_{\theta }\left(k_1\right):=E_{\theta }\left(X_2\right|X_1=k_1)={\theta }^{k_1}{(1-\theta )}^{1-k_1}$ is, for $k_1=0,1$,

$$m_n^{*}(k^{\prime },k_1)={f_{n,k^{\prime }}^{*}}^{X_2|X_1=k_1}\left(1\right)=\left\{\begin{array}{cc}{\displaystyle \frac{n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+1}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+3}}\hfill & \mathrm{if}{k}_1=0,\hfill \\ {\displaystyle \frac{n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+1}{2n+n_{+0}\left(k^{\prime }\right)+2n_{01}\left(k^{\prime }\right)+4}}\hfill & \mathrm{if}{k}_1=1,\hfill \end{array}\right.$$

$n_{j_1{j}_2}\left(k^{\prime }\right)$ being the number of indices $i\in \{1,\dots ,n\}$, such that $(k_{i1}^{\prime },k_{i2}^{\prime })=(j_1,j_2)$ and $n_{+j}=n_{0j}+n_{1j}$ for $j=0,1$.

Theorem 1 proves that it is a strongly consistent estimator of the conditional mean $r_{\theta }\left(k_1\right)$, and its Bayes risk converges to 0 for both the $L^1$ and $L^2$ loss functions.

Example 3.

Let $(X_1,X_2)$ have a bivariate normal distribution

$$N_2\left(\left(\begin{array}{c}\theta \\ \theta \end{array}\right),{\sigma }^2\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)\right),$$

and consider the prior distribution $Q=N(\mu ,{\tau }^2)$, where $\mu ,\sigma ,\tau ,and\rho$ are supposed to be known, and θ refers to the unknown parameter. It is shown in [13], Example 3, that the conditional mean

$$E_{{R_{n,x^{\prime }}^{*}}^R}\left(p_2\right|p_1=x_1)=(1-{\rho }_1)m_1\left(x^{\prime }\right)+{\rho }_1{x}_1$$

is the Bayes estimator of the regression curve

$$E_{\theta }\left(X_2\right|X_1=x_1)=(1-\rho )\theta +\rho x_1$$

for the squared error loss function, where

$$\begin{array}{c}{\rho }_1=-{\displaystyle \frac{a_n(\rho ,\sigma ,\tau )+\frac{1-\rho }{1+\rho }}{a_n(\rho ,\sigma ,\tau )-\frac{1-\rho }{1+\rho }}}·\rho ,m_1\left(x^{\prime }\right)={\displaystyle \frac{s_1\left(x^{\prime }\right)+(1+\rho )\frac{{\sigma }^2}{{\tau }^2}\mu }{2(1-{\rho }_1){(1+\rho )}^2{\sigma }^2{a}_n(\rho ,\sigma ,\tau )}},\end{array}$$

being

$$\begin{array}{c}{s}_1\left(x^{\prime }\right):=\sum _i(x_{i1}^{\prime }+x_{i2}^{\prime }),a_n(\rho ,\sigma ,\tau ):=2(n+1)(1+\rho )+{\displaystyle \frac{{\sigma }^2}{{\tau }^2}}.\end{array}$$

Theorem 1 proves that it is a strongly consistent estimator of the regression curve, and its Bayes risk converges to 0 for both the $L^1$ and $L^2$ loss functions.

Example 4.

In this (discrete and nonparametric) example, we assume that $(X_1,X_2)$ is a ${\mathbb{N}}_0^2$-valued random variable with unknown arbitrary probability distribution P in $M\left({\mathbb{N}}_0^2\right)$, the set of all probability distributions on $({\mathbb{N}}_0^2,\mathfrak{P}\left({\mathbb{N}}_0^2\right))$; $\mathfrak{P}$ stands for the discrete σ-field. The Bayesian experiment (1) considered in this example for an n-sized sample of the joint distribution P of $(X_1,X_2)$ is

$$({\left({\mathbb{N}}_0^2\right)}^n,\mathfrak{P}{\left({\mathbb{N}}_0^2\right)}^n,\{P^n:P\in (M\left({\mathbb{N}}_0^2\right),{\mathcal{B}}_{M\left({\mathbb{N}}_0^2\right)},D_{\alpha })\})$$

where ${\mathcal{B}}_{M\left({\mathbb{N}}_0^2\right)}$ is the Borel σ-field on $M\left({\mathbb{N}}_0^2\right)$ for the weak topology (for which it becomes a Polish space), α a finite measure on ${\mathbb{N}}_0^2$, and $D_{\alpha }$ the Dirichlet process with base measure α, which plays the role of prior distribution. We refer to [3] or [20] for everything related to Dirichlet processes.

Let $\phi :{\mathbb{N}}_0\to \mathbb{R}$ be a real bounded function, so that $\phi \circ X_2$ satisfies the conditions of Theorem 1. To estimate the regression function

$$E_P(\phi \circ p_2|p_1=k)={\displaystyle \frac{{\sum }_{j\in {\mathbb{N}}_0}\phi \left(j\right)P(k,j)}{{\sum }_{j\in {\mathbb{N}}_0}P(k,j)}}$$

from a sample $x^{\prime }:=(x_1^{\prime },\dots ,x_n^{\prime })\in {\left({\mathbb{N}}_0^2\right)}^n$, we need the posterior predictive distribution given $x^{\prime }$ which, as is known (see [20], for instance), is

$${R_{n,x^{\prime }}^{*}}^R={\displaystyle \frac{\alpha +{\sum }_{i=1}^n{\delta }_{x_i^{\prime }}}{\alpha \left({\mathbb{N}}_0^2\right)+n}},$$

where ${\delta }_{x_i^{\prime }}(k,l)=1$ when $x_i^{\prime }=(k,l)$, and $=0$ otherwise. Then, the Bayes estimator of the regression function $E_P(\phi \circ p_2|p_1=k)={\sum }_{j\in {\mathbb{N}}_0}\phi \left(j\right)P(p_2=j|p_1=k)$ is the regression function of $\phi \circ p_2$, given $p_1=k$, with respect to the posterior predictive distribution, given $x^{\prime }$, i.e.,

$$E_{\left({R_{n,x^{\prime }}^{*}}^R\right)}(\phi \circ p_2|p_1=k)={\displaystyle \frac{{\sum }_{j\in {\mathbb{N}}_0}\phi \left(j\right)[\alpha (k,j)+{\sum }_{i=1}^n{\delta }_{x_i^{\prime }}(k,j)]}{{\sum }_{j\in {\mathbb{N}}_0}[\alpha (k,j)+{\sum }_{i=1}^n{\delta }_{x_i^{\prime }}(k,j)]}}.$$

Theorem 1 proves that it is a strongly consistent estimator of this regression curve, and its Bayes risk converges to 0 for both the $L^1$ and $L^2$ loss functions.

The next example is a continuation of Example 5.5 of [17].

Example 5.

In this (nonparametric with continuous probability measure base for the prior Dirichlet process) example, we assume that $(X_1,X_2)$ is a ${\mathbb{R}}^2$-valued random variable with an unknown arbitrary probability distribution P in $M\left({\mathbb{R}}^2\right)$, and the set of all probability distributions on $({\mathbb{R}}^2,{\mathcal{R}}^2))$; ${\mathcal{R}}^2$ stands for the Borel σ-field on ${\mathbb{R}}^2$. The Bayesian experiment (1) considered in this example, for an n-sized sample of the joint distribution P of $(X_1,X_2)$, is

$$({\left({\mathbb{R}}^2\right)}^n,{\left({\mathcal{R}}^2\right)}^n,\{P^n:P\in (M\left({\mathbb{R}}^2\right),{\mathcal{B}}_{M\left({\mathbb{R}}^2\right)},D_{\alpha })\})$$

where ${\mathcal{B}}_{M\left({\mathbb{R}}^2\right)}$ is the Borel σ-field on $M\left({\mathbb{R}}^2\right)$ for the weak topology (for which it becomes a Polish space), α a probability measure on ${\mathcal{R}}^2$, and $D_{\alpha }$ the Dirichlet process with base measure α, which plays the role of prior distribution. A reference to [3] or [20] for everything related to Dirichlet processes is still appropriate. The posterior predictive distribution, given the sample $x^{\prime }:=(x_1^{\prime },\dots ,x_n^{\prime })\in {\left({\mathbb{R}}^2\right)}^n$, is known to be

$${R_{n,x^{\prime }}^{*}}^R={\displaystyle \frac{\alpha +{\sum }_{i=1}^n{\delta }_{x_i^{\prime }}}{n+1}}={\displaystyle \frac{1}{n+1}}\alpha +{\displaystyle \frac{n}{n+1}}{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\delta }_{x_i^{\prime }},$$

where ${\delta }_{x_i^{\prime }}(x_1,x_2)=1$ when $x_i^{\prime }=(x_1,x_2)$, and $=0$ otherwise. Note that this distribution is a convex combination of the probability measure α, the base measure of the prior distribution, and the empirical measure, a mixture in which the weight of the data increases with the sample size (in fact, it reaches 1). The Bayes estimator of the conditional distribution $P^{X_2|X_1=x_1}$ is the conditional distribution of $p_2$ given $p_1=x_1$ with respect to the posterior predictive distribution, given $x^{\prime }$:

$${\left({R_{n,x^{\prime }}^{*}}^R\right)}^{p_2|p_1=x_1}.$$

It is shown in [17] that this conditional distribution can be calculated, for a Borel set $B\in \mathcal{R}$, as follows:

$$\begin{array}{c}{\left({R_{n,x^{\prime }}^{*}}^R\right)}^{p_2|p_1=x_1}\left(B\right)=\\ I_{{\{x_{11}^{\prime },\dots ,x_{n1}^{\prime }\}}^c}\left(x_1\right)·{\alpha }^{p_2|p_1=x_1}\left(B\right)+I_{\{x_{11}^{\prime },\dots ,x_{n1}^{\prime }\}}\left(x_1\right)·{\displaystyle \frac{{\sum }_{i=1}^n{\delta }_{x_i^{\prime }}(\left\{x_1\right\}\times B)}{{\sum }_{i=1}^n{\delta }_{x_{i1}^{\prime }}\left(x_1\right)}}\end{array}$$

Let us now consider a real bounded measurable function φ on $\mathbb{R}$ (then $\phi \circ p_2$ satisfies the conditions of Theorem 1. The Bayes estimator of the regression function $E_P(\phi \circ p_2|p_1=x_1)$ is the regression function of $\phi \circ p_2$, given $p_1=x_1$, with respect to the posterior predictive distribution, given $x^{\prime }$, i.e.,

$$m_n^{*}(x^{\prime },x_1):=E_{{R_{n,x^{\prime }}^{*}}^R}(\phi \circ p_2|p_1=x_1).$$

So

$$\begin{array}{c}\begin{array}{cc}\hfill m_n^{*}(x^{\prime },x_1)& =I_{{\{x_{11}^{\prime },\dots ,x_{n1}^{\prime }\}}^c}\left(x_1\right)·{\int }_{\mathbb{R}}\phi \left(x_2\right)d{\alpha }^{X_2|X_1=x_1}\left(x_2\right)\hfill \\ & +I_{\{x_{11}^{\prime },\dots ,x_{n1}^{\prime }\}}\left(x_1\right)·{\displaystyle \frac{1}{{\sum }_{i=1}^n{\delta }_{x_{i1}^{\prime }}\left(x_1\right)}}\sum _{i=1}^n{\int }_{\mathbb{R}}\phi \left(x_2\right)d{\delta }_{x_i^{\prime }}(x_1,d{x}_2).\hfill \end{array}\end{array}$$

This way, if $x_1\in {\{x_{11}^{\prime },\dots ,x_{n1}^{\prime }\}}^c$, $m_n^{*}(x^{\prime },x_1)$ is the mean conditional of $\phi \circ p_2$, given $p_1=x_1$ for the base probability measure α (for instance, if α is the product ${\alpha }_1\times {\alpha }_2$ of two probability distributions in $\mathbb{R}$, then $m_n^{*}(x^{\prime },x_1)$ is the mean $E_{{\alpha }_2}\left(\phi \right)$ in this case).

If $x_1\in \{x_{11}^{\prime },\dots ,x_{n1}^{\prime }\}$, denote $S_{x_1}$ as the set of index $1\le i\le n$, such that $x_{i1}^{\prime }=x_1$ and $s_{x_1}$ the number of such indices. In this case,

$$m_n^{*}(x^{\prime },x_1)={\displaystyle \frac{1}{s_{x_1}}}\sum _{i\in S_{x_1}}\phi \left(x_{i2}^{\prime }\right).$$

Therefore, if $x_1$ is (respectively, is not) in the sample support, only the empirical measure (respectively, the base measure α of the prior distribution) is taken into account when estimating the regression curve at the point $x_1$.

Theorem 1 shows that the Bayes risk of this estimator of the regression curve $E_P(\phi \circ p_2|p_1=x_1)$ converges to 0 for both the $L^1$ and $L^2$ loss functions and that it is a strongly consistent estimator of this regression curve.

Remark 2.

In Example 5, the problem of estimating the density has no sense because $M\left({\mathbb{R}}^2\right)$ contains all the probability measures on the plane. In fact, it is known (see [3]) that, even if α is absolutely continuous, as is the case, $D_{\alpha }$ is concentrated in the set of discrete probability measures on ${\mathbb{R}}^2$, which discourages its use as the prior distribution in the set of all density functions. Fortunately, Theorem 3.1 of [17] allows us to address the problem in terms of the conditional distribution and, finally, obtain the Bayes estimator of the regression function without needing to appeal to density.

6. Conclusions

The optimality, in a decisional sense, of the regression curve with respect to the posterior predictive distribution as an estimator of the regression curve for the squared error loss function, together with its asymptotic behavior—consistency and convergence to 0 of the Bayes risk—is an important point in favor of this method of estimating the regression curve against to other estimation methods in a Bayesian context.

A remarkable fact about the results of this paper (and [13]) is that the predictor $X_1$ is an arbitrary random variable (not necessarily a real or n-dimensional random variable). Furthermore, no special assumptions are made about the prior distribution.

An important issue of the paper is the establishment of a certain probability space as the appropriate theoretical framework for the study of the asymptotic behavior of the estimator of the regression curve, allowing for obtaining an explicit expression of the Bayes risk of this estimator and take advantage of powerful probabilistic tools when solving the problem of its asymptotic behavior.