Asymptotically Constant-Risk Predictive Densities When the Distributions of Data and Target Variables Are Different

Abstract

We investigate the asymptotic construction of constant-risk Bayesian predictive densities under the Kullback–Leibler risk when the distributions of data and target variables are different and have a common unknown parameter. It is known that the Kullback–Leibler risk is asymptotically equal to a trace of the product of two matrices: the inverse of the Fisher information matrix for the data and the Fisher information matrix for the target variables. We assume that the trace has a unique maximum point with respect to the parameter. We construct asymptotically constant-risk Bayesian predictive densities using a prior depending on the sample size. Further, we apply the theory to the subminimax estimator problem and the prediction based on the binary regression model.

1. Introduction

Let $x^{\left(N\right)}=\left(x_1,\cdots ,x_N\right)$ be independent N data distributed according to a probability density, p(x|θ), that belongs to a d-dimensional parametric model, {p(x|θ) : θ ∈ Θ}, where $\theta =\left({\theta }^1,\cdots ,{\theta }^d\right)$ is an unknown d-dimensional parameter and Θ is the parameter space. Let y be a target variable distributed according to a probability density, q(y|θ), that belongs to a d-dimensional parametric model, {q(y|θ) : θ ∈ Θ } with the same parameter, θ. Here, we assume that the distributions of the data and the target variables, p(x|θ) and q(y|θ), are different. For simplicity, we assume that the data and the target variables are independent, given by θ.

We construct predictive densities for target variables based on the data. We measure the performance of the predictive density, $\widehat{q}\left(y;x^{\left(N\right)}\right)$, by the Kullback–Leibler divergence, $D\left(q\left(\cdot |\theta \right),\widehat{q}\left(\cdot ;x^{\left(N\right)}\right)\right)$, from the true density, q(y|θ), to the predictive density, $\widehat{q}\left(y;x^{\left(N\right)}\right)$:

$$D\left(q\left(\cdot |\theta \right),\widehat{q}\left(\cdot ;x^{\left(N\right)}\right)\right)={\displaystyle \int q\left(y|\theta \right)\log \frac{q\left(y|\theta \right)}{\widehat{q}\left(\cdot ;x^{\left(N\right)}\right)}dy.}$$

Then, the risk function, $R\left(\theta ,\widehat{q}\left(y;x^{\left(N\right)}\right)\right)$, of the predictive density, $\widehat{q}\left(y;x^{\left(N\right)}\right)$, is given by:

For the construction of predictive densities, we consider the Bayesian predictive density defined by:

$${\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)=\frac{{\displaystyle \int q\left(y|\theta \right)p\left(x^{\left(N\right)}|\theta \right)\pi \left(\theta ;N\right)d\theta }}{{\displaystyle \int p\left(x^{\left(N\right)}|\theta \right)\pi \left(\theta ;N\right)d\theta }},$$

where π(θ; N) is a prior density for θ, possibly depending on the sample size, N. Aitchison [1] showed that, for a given prior density, π(θ; N), the Bayesian predictive density, ${\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)$, is a Bayes solution under the Kullback–Leibler risk. Based on the asymptotics as the sample size goes to infinity, Komaki [2] and Hartigan [3] showed its superiority over any plug-in predictive density, $q\left(y|\widehat{\theta }\right)$, with any estimator, $\widehat{\theta }$. However, there remains a problem of prior selection for constructing better Bayesian predictive densities. Thus, a prior, π(θ; N), must be chosen based on an optimality criterion for actual applications.

Among various criteria, we focus on a criterion of constructing minimax predictive densities under the Kullback–Leibler risk. For simplicity, we refer to the priors generating minimax predictive densities as minimax priors. Minimax priors have been previously studied in various predictive settings; see [4–8]. When the simultaneous distributions of the target variables and the data belong to the submodel of the multinomial distributions, Komaki [7] shows that minimax priors are given as latent information priors maximizing the conditional mutual information between target variables and the parameter given the data. However, the explicit forms of latent information priors are difficult to obtain, and we need asymptotic methods, because they require the maximization on the space of the probability measures on Θ.

Except for [7], these studies on minimax priors are based on the assumption that the distributions, p(x|θ) and q(y|θ), are identical. Let us consider the prediction based on the logistic regression model where the covariates of the data and the target variables are not identical. In this predictive setting, the assumption that the distributions, p(x|θ) and q(y|θ), are identical is no longer valid.

We focus on the minimax priors in predictions where the distributions, p(x|θ) and q(y|θ), are different and have a common unknown parameter. Such a predictive setting has traditionally been considered in statistical prediction and experiment design. It has recently been studied in statistical learning theory; for example, see [9]. Predictive densities where the distributions, p(x|θ) and q(y|θ), are different and have a common unknown parameter are studied by [10–13].

Let $g_{ij}^X\left(\theta \right)$ be the (i, j)-component of the Fisher information matrix of the distribution, p(x|θ), and let $g_{ij}^Y\left(\theta \right)$ be the (i, j)-component of the Fisher information matrix of the distribution, q(y|θ). Let g^X,ij(θ) and g^Y,ij(θ) denote the (i, j)-components of their inverse matrices. We adopt Einstein’s summation convention: if the same indices appear twice in any one term, it implies summation over that index from one to d. For the asymptotics below, we assume that the prior densities, π(θ; N), are smooth.

On the asymptotics as the sample size N goes to infinity, we construct the asymptotically constant-risk prior, π(θ; N), in the sense that the asymptotic risk:

$$R\left(\theta ,{\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)\right)=\frac{1}{N}{R}_1\left(\theta ,{\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)\right)+\frac{1}{N\sqrt{N}}{R}_2\left(\theta ,{\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)\right)+O\left(N^{-2}\right)$$

is constant up to O(N⁻²). Since the proper prior with the constant risk is a minimax prior for any finite sample size, the asymptotically constant-risk prior relates to the minimax prior; in Section 4, we verify that the asymptotically constant-risk prior agrees with the exact minimax prior in binomial examples.

When we use the prior, π(θ), independent of the sample size, N, it is known that the N⁻¹-order term, $R_1\left(\theta ,{\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)\right)$, of the Kullback–Leibler risk is equal to the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$. If the trace does not depend on the parameter, θ, the construction of the asymptotically constant-risk prior is parallel to [6]; see also [13].

However, we consider the settings where there exists a unique maximum point of the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$; for example, these settings appear in predictions based on the binary regression model, where the covariates of the data and the target variables are not identical. In the settings, there do not exist asymptotically constant-risk priors among the priors independent of the sample size, N. The reason is as follows: we consider the prior, π(θ), independent of the sample size, N. Then, the Kullback–Leibler risk of the Bayesian predictive density is expanded as:

$$R\left(\theta ,{\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)\right)=\frac{1}{2N}{g}_{ij}^Y\left(\theta \right)g^{X,ij}\left(\theta \right)+\mathrm{O}\left(N^{-\text{2}}\right)\text{.}$$

Since, in our settings, the first-order term, $g_{ij}^Y\left(\theta \right)g^{X,ij}\left(\theta \right)$, is not constant, the prior independent of the sample size, N, is not an asymptotically constant-risk prior.

When there exists a unique maximum point of the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$, we construct the asymptotically constant-risk prior, π(θ; N), up to O(N⁻²), by making the prior dependent on the sample size, N, as:

$$\frac{\pi \left(\theta ;N\right)}{{\left|g^X\left(\theta \right)\right|}^{1/2}}\propto {\left\{f(\theta )\right\}}^{\sqrt{N}}h\left(\theta \right),$$

where f(θ) and h(θ) are the scalar functions of θ independent of N and |g^X(θ)| denotes the determinant of the Fisher information matrix, g^X(θ).

The key idea is that, if the specified parameter point has more undue risk than the other parameter points, then the more prior weights should be concentrated on that point.

Further, we clarify the subminimax estimator problem based on the mean squared error from the viewpoint of the prediction where the distributions of data and target variables are different and have a common unknown parameter. We obtain the improvement achieved by the minimax estimator over the subminimax estimators up to O(N⁻²). The subminimax estimator problem [14,15] is the problem that, at first glance, there seems to exist asymptotically dominating estimators of the minimax estimator. However, any relationship between such subminimax estimator problems and predictions have not been investigated, and further, in general, the improvement by the minimax estimator over the subminimax estimators has not been investigated.

2. Information Geometrical Notations

In this section, we prepare the information geometrical notations; see [16] for details. We abbreviate ∂/∂θⁱ to ∂_i, where the indices, i, j, k,…, run from one to d. Similarly, we abbreviate ∂²/∂θⁱ∂θ^j, ∂³/∂θⁱ∂θ^j∂θ^k and ∂⁴/∂θⁱ∂θ^j∂θ^k∂θ^l to ∂_ij, ∂_ijk and ∂_ijkl, respectively. We denote the expectations of the random variables, X, Y and X^(N), by E_X[·], E_Y [·] and E_X(N) [·], respectively. We denote their probability densities by p(x|θ), q(y|θ) and p(x^(N)|θ), respectively.

We define the predictive metric proposed by Komaki [13] as:

$${\stackrel{°}{g}}_{ij}\left(\theta \right)=g_{ij}^X\left(\theta \right)g^{Y,kl}{g}_{lj}^X\left(\theta \right).$$

When the parameter is one-dimensional, g_θθ(θ) denotes Fisher information and g^θθ(θ) denotes its inverse. Let $\stackrel{\mathrm{e}}{\Gamma }{}_{ij,k}^X\left(\theta \right)$ and $\stackrel{\mathrm{m}}{\Gamma }{}_{ij,k}^X\left(\theta \right)$ be the quantities given by:

$$\stackrel{\mathrm{e}}{\Gamma }{}_{ij,k}^X\left(\theta \right):={\mathrm{E}}_X\left[{\partial }_{ij}\log p\left(x|\theta \right){\partial }_k\log p\left(x|\theta \right)\right]$$

and:

$$\stackrel{\mathrm{m}}{\Gamma }{}_{ij,k}^X\left(\theta \right):={\displaystyle \int \frac{1}{p\left(x|\theta \right)}\left[{\partial }_{ij}p\left(x|\theta \right){\partial }_{k}p\left(x|\theta \right)\right]dx}.$$

Using these quantities, the e-connection and m-connection coefficients with respect to the parameter, θ, for the model, {p(x|θ) : θ ∈ Θ}, are given by:

$$\stackrel{\mathrm{e}}{\Gamma }{}_{ij}^{X,k}\left(\theta \right):=g^{X,lk}\left(\theta \right)\stackrel{\mathrm{e}}{\Gamma }{}_{ij,l}^X\left(\theta \right)$$

and:

$$\stackrel{\mathrm{m}}{\Gamma }{}_{ij}^{X,k}\left(\theta \right):=g^{X,lk}\left(\theta \right)\stackrel{\mathrm{m}}{\Gamma }{}_{ij,l}^X\left(\theta \right),$$

respectively.

The (0, 3)-tensor, $T_{ijk}^X\left(\theta \right)$, is defined by:

$$T_{ijk}^X\left(\theta \right):={\mathrm{E}}_X\left[{\partial }_i\log p\left(x|\theta \right){\partial }_j\log p\left(x|\theta \right){\partial }_k\log p\left(x|\theta \right)\right].$$

The tensor, $T_{ijk}^X\left(\theta \right)$, also produces a (0, 1)-tensor:

$$T_i^X\left(\theta \right):=T_{ijk}^X\left(\theta \right)g^{X,jk}\left(\theta \right).$$

In the same manner, the information geometrical quantities, $\stackrel{\mathrm{e}}{\Gamma }{}_{ij,k}^Y\left(\theta \right),\stackrel{\mathrm{m}}{\Gamma }{}_{ij,k}^Y\left(\theta \right)$ and $T_{ijk}^Y\left(\theta \right)$, are defined for the model, {q(y|θ) : θ ∈ Θ}.

Let $M_{ij}^k\left(\theta \right)$ be a (1, 2)-tensor defined by:

$$M_{ij}^k\left(\theta \right):=\stackrel{\mathrm{m}}{\Gamma }{}_{ij}^{Y,k}\left(\theta \right)-\stackrel{\mathrm{m}}{\Gamma }{}_{ij}^{X,k}\left(\theta \right).$$

For a derivative, $\left({\partial }_1\upsilon \left(\theta \right),\cdots ,{\partial }_d\upsilon \left(\theta \right)\right)$, of the scalar function, υ(θ), the e-covariant derivative is given by:

$${\stackrel{\mathrm{e}}{\nabla }}_i{\partial }_j\upsilon \left(\theta \right):={\partial }_{ij}\upsilon \left(\theta \right)-\stackrel{\mathrm{e}}{\Gamma }{}_{ij}^{X,k}\left(\theta \right){\partial }_k\upsilon \left(\theta \right).$$

3. Asymptotically Constant-Risk Priors When the Distributions of Data and Target Variables Are Different

In this section, we consider the settings where the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$, has a unique maximum point. We construct the asymptotically constant-risk prior under the Kullback–Leibler risk in the sense that the asymptotic risk up to O(N⁻²) is constant. We find asymptotically constant-risk priors up to O(N⁻²) in two steps: first, expand the Kullback–Leibler risks of Bayesian predictive densities; second, find the prior having an asymptotically constant risk using this expansion.

From now on, we assume the following two conditions for the prior, π(θ; N):

(C1) The prior, π(θ; N), has the form:

$$\frac{\pi \left(\theta ;N\right)}{{\left|g^X\left(\theta \right)\right|}^{1/2}}\propto \exp \left\{\sqrt{N}\log f\left(\theta \right)+\log h\left(\theta \right)\right\},$$

where f(θ) and h(θ) are smooth scalar functions of θ independent of N.

(C2) The unique maximum point of the scalar function, f(θ), is equal to the unique maximum point of the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$.

Based on Conditions (C1) and (C2), we expand the Kullback–Leibler risk of a Bayesian predictive density up to O(N⁻²).

Theorem 1. The Kullback–Leibler risk of a Bayesian predictive density based on the prior, π(θ; N), satisfying Condition (C1), is expanded as:

(1)

The proof is given in the Appendix. The first term in (1) represents that the precision of the estimation is determined by the geometric quantity of the data, g^X;ij(θ), and the metric of the parameter is determined by the geometric quantity of the target variables, $g_{ij}^Y\left(\theta \right)$. Note that each term in (1) is invariant under the reparametrization.

Remark 1. For the subsequent theorem, it is important that at the point, θ_f, maximizing the scalar function, $\log f\left(\theta \right),R\left({\theta }_f,{\widehat{q}}_{\pi }\left(y|x^N\right)\right)$ is given by:

$$\begin{array}{l}R\left({\theta }_f,{\widehat{q}}_{\pi }\left(y|x^N\right)\right)\\ =\frac{1}{2N}\underset{\theta \in \Theta }{\sup }\left\{g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)\right\}+\frac{1}{N\sqrt{N}}\stackrel{°}{g^{ij}}\left({\theta }_f\right){\partial }_{ij}\log f\left({\theta }_f\right)+\mathrm{O}\left(N^{-2}\right).\end{array}$$

(2)

The N^−3/2-order term of this risk is common whenever we use the same scalar function, log f(θ). This term is negative because of the definition of the point, θ_f. Under Condition (C2), θ_f is equal to the unique maximum point, θ_max, of the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$.

Based on (1) and (2), we construct asymptotically constant-risk priors using the solutions of the partial differential equations.

Theorem 2. Suppose that the scalar functions, $\log \tilde{f}\left(\theta \right)$ and, $\log \tilde{h}\left(\theta \right)$ satisfy the following conditions:

(A1)$\log \tilde{f}\left(\theta \right)$ is the solution of the Eikonal equation given by:

$$\stackrel{°}{g^{ij}}\left(\theta \right){\partial }_i\log \tilde{f}\left(\theta \right){\partial }_j\log \tilde{f}\left(\theta \right)=g^{X,ij}\left({\theta }_{max}\right)g_{ij}^Y\left({\theta }_{max}\right)-g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right),$$

(3)

where θ_max is the unique maximum point of the scalar function, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$.

(A2)$\log \tilde{h}\left(\theta \right)$ is the solution of the first-order linear partial equation given by:

(4)

Let π(θ; N) be the prior that is constructed as:

$$\frac{\pi \left(\theta ;N\right)}{{\left|g^X\left(\theta \right)\right|}^{1/2}}\propto \exp \left\{\sqrt{N}\log \tilde{f}\left(\theta \right)+\log \tilde{h}\left(\theta \right)\right\}.$$

Further, suppose that$\log \tilde{f}\left(\theta \right)$ satisfies Condition (C2).

Then, the Bayesian predictive density based on the prior, π(θ; N), has the asymptotically smallest constant risk up to O(N⁻²) among all priors with the form (C1).

Proof. First, we consider the prior, φ(θ; N), constructed as:

$$\frac{\varphi \left(\theta ;N\right)}{{\left|g^X\left(\theta \right)\right|}^{1/2}}\propto \exp \left\{\sqrt{N}\log \tilde{f}\left(\theta \right)\right\}.$$

From Theorem 1, the Kullback–Leibler risk, $R\left(\theta ,{\widehat{q}}_{\varphi }\left(y|x^{\left(N\right)}\right)\right)$, based on the prior, φ(θ; N), is given by:

$$R\left(\theta ,{\widehat{q}}_{\varphi }\left(y|x^{\left(N\right)}\right)\right)=\frac{1}{2N}{g}^{X,ij}\left({\theta }_{max}\right)g_{ij}^Y\left({\theta }_{max}\right)+\mathrm{o}\left(N^{-1}\right).$$

(5)

This is constant up to o(N⁻¹).

Suppose that there exists another prior, φ(θ; N), constructed as:

$$\frac{\phi \left(\theta ;N\right)}{{\left|g^X\left(\theta \right)\right|}^{1/2}}\propto \exp \left\{\sqrt{N}\log f\left(\theta \right)\right\},$$

and the Bayesian predictive density based on the prior, φ(θ; N), has the asymptotically constant risk:

$$R\left(\theta ,{\widehat{q}}_{\phi }\left(y|x^{\left(N\right)}\right)\right)=\frac{k}{2N}+\mathrm{o}\left(N^{-1}\right).$$

From Theorem 1, the prior φ(θ; N) must satisfy the equation:

$$\stackrel{°}{g}{}^{ij}\left(\theta \right){\partial }_i\log f\left(\theta \right){\partial }_j\log f\left(\theta \right)=k-g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right).$$

The left-hand side of the above equation is non-negative, because the matrix, $\stackrel{°}{g}{}^{ij}\left(\theta \right)$, is positive-definite. Hence, the infimum of the constant, k, is equal to $g^{X,ij}\left({\theta }_{max}\right)g_{ij}^Y\left({\theta }_{max}\right)$. From (5), the N⁻¹-order term of the risk based on the prior, ϕ(θ; N), achieves the infimum, $g^{X,ij}\left({\theta }_{max}\right)g_{ij}^Y\left({\theta }_{max}\right)$. Thus, the Bayesian predictive density based on the prior, ϕ(θ; N), has the asymptotically smallest constant risk up to o(N⁻¹).

Second, we consider the prior, π(θ; N), constructed as:

$$\frac{\pi \left(\theta ;N\right)}{{\left|g^X\left(\theta \right)\right|}^{1/2}}\propto \exp \left\{\sqrt{N}\log \tilde{f}\left(\theta \right)+\log \tilde{h}\left(\theta \right)\right\}.$$

The above argument ensures that the prior, π(θ; N), has the asymptotically smallest constant risk up to o(N⁻¹). Thus, we only have to check if the N^−3/2-order term of the risk is the smallest constant. From (2), the N^−3/2-order term of the risk at the point, θ_max, is unchanged by the choice of the scalar function, log h(θ). In other words, the constant N^−3/2-order term must agree with the quantity, $\stackrel{°}{g}{}^{ij}\left({\theta }_{max}\right){\partial }_{ij}\log \tilde{f}\left({\theta }_{max}\right)$. From Theorem 1, if we choose the prior, π(θ; N), the N^−3/2-order term of the risk is the smallest constant, and it agrees with the quantity, $\stackrel{°}{g}{}^{ij}\left({\theta }_{max}\right){\partial }_{ij}\log \tilde{f}\left({\theta }_{max}\right)$. Thus, the prior, π(θ; N), has the asymptotically smallest constant risk up to O(N⁻²). □

Remark 2. In Theorem 2, we choose$\log \tilde{f}\left(\theta \right)$, satisfying Condition (C2) among the solutions of (A1). We consider the model with a one-dimensional parameter, θ. There are four possibilities to the solutions of (A1):

where the double-sign corresponds. From the concavity around θ_max as suggested by (C2), we choose$\log \tilde{f}\left(\theta \right)$ as the solution of the following equation:

(6)

Integrating both sides of Equation (6), the unique function, $\log \tilde{f}\left(\theta \right)$, is obtained.

Remark 3. Compare the Kullback–Leibler risk based on the asymptotically constant-risk prior, π(θ; N), with that based on the prior, λ(θ), independent of the sample size, N. From Theorem 1 and Theorem 2, the Kullback–Leibler risk based on the asymptotically constant-risk prior, π(θ; N), is given as:

(7)

In contrast, the Kullback–Leibler risk based on the prior, λ(θ), is given as:

$$R\left(\theta ,\widehat{q}\mathrm{\lambda }\left(y|x^{\left(N\right)}\right)\right)=\frac{1}{2N}{g}^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)+\mathrm{O}\left(N^{-2}\right).$$

(8)

The N⁻¹-order term in (8) is under the N⁻¹-order term in (7); although the N^−3/2-order term in (8) does not exist, the N^−3/2-order term in (7) is negative. Thus, the maximum of the risk based on the asymptotically constant-risk prior, π(θ; N), is smaller than that of the risk based on the prior, λ(θ). This result is consistent with the minimaxity of selecting the prior that constructs the predictive density with the smallest maximum of the risk.

4. Subminimax Estimator Problem Based on the Mean Squared Error

In this section, we refer to the subminimax estimator problem based on the mean squared error, from the viewpoint of the prediction where the distributions of data and target variables are different and have a common unknown parameter. First, we give a brief review of subminimax estimator problem through the binomial example.

Example. Let us consider the binomial estimation based on the mean squared error, $R_{\text{MSE}}\left(\theta ,\widehat{\theta }\right)$. For any finite sample size, N, the Bayes estimator, ${\widehat{\theta }}_{\pi }$, based on the Beta prior, $\pi \left(\theta ;N\right)\propto {\theta }^{\sqrt{N}/2-1}{\left(1-\theta \right)}^{\sqrt{N}/2-1}$, is minimax under the mean squared error. The mean squared error of the minimax Bayes estimator, ${\widehat{\theta }}_{\pi }$, is given by:

$$R_{\text{MSE}}(\theta ,{\widehat{\theta }}_{\mathrm{\pi }})=\frac{N}{4{(\sqrt{N}+N)}^2}=\frac{1}{4N}-\frac{1}{2N\sqrt{N}}+\mathrm{O}(N^{-2}).$$

(9)

In contrast, the mean squared error of the maximum likelihood estimator, ${\widehat{\theta }}_{\text{MLE}}$, is given by:

$$R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{\text{MLE}}\right)=\frac{\theta \left(1-\theta \right)}{N}.$$

We compare the two estimators, ${\widehat{\theta }}_{\pi }$ and${\widehat{\theta }}_{\text{MLE}}$. In the comparison of the N⁻¹-order terms of the mean squared errors, it seems that the maximum likelihood estimator, ${\widehat{\theta }}_{\text{MLE}}$, dominates the minimax Bayes estimator, ${\widehat{\theta }}_{\pi }$. In other words, the N⁻¹-order term of$R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{\text{MLE}}\right)$ is not greater than that of$R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{\pi }\right)$ for every θ ∈ Θ, and the equality holds when θ = 1/2. This seeming paradox is known as the subminimax estimator problem; see [14,17,18] for details. See also [15] for the conditions that such problems do not occur in estimation.

However, this paradox does not mean the inferiority of the minimax Bayes estimator. This is because, although the mean squared error of the minimax Bayes estimator, ${\widehat{\theta }}_{\pi }$, has the negative N^−3/2-order term, the mean squared error of the maximum likelihood estimator, ${\widehat{\theta }}_{\text{MLE}}$, does not have the N^−3/2-order term. Hence, in comparison to the mean squared errors up to O(N⁻²), the maximum of the mean squared error, $R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{\pi }\right)$, is below the maximum of the mean squared error, $R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{\text{MLE}}\right)$.

Next, we construct the asymptotically constant-risk prior in the estimation based on the mean squared error when the subminimax estimator problem occurs, from the viewpoint of the prediction. We consider the priors, π(θ; N), satisfying (C1). From Lemma 5 in the Appendix, the mean squared error of the Bayes estimator, ${\widehat{\theta }}_{\pi }$, is equal to the Kullback–Leibler risk of the ${\widehat{\theta }}_{\pi }$-plugin predictive density, $q\left(y|{\widehat{\theta }}_{\pi }\right)$, by assuming that the target variable, y, is a d-dimensional Gaussian random variable with the mean vector, θ, and unit variance. Note that $g_{ij}^Y\left(\theta \right)=1,\stackrel{m}{\Gamma }{}_{ij,k}^Y=0$, and $\stackrel{e}{\Gamma }{}_{ij,k}^Y=0$ for $i,j,k=1,\cdots ,d$. Thus, if $g_{ij}^Y\left(\theta \right)g^{X,ij}\left(\theta \right)={\displaystyle \sum {}_{i=1}^d}{g}^{X,ii}\left(\theta \right)$ has a unique maximum point, we obtain the asymptotically constant-risk prior, π(θ; N), up to O(N⁻²) from Lemma 4 in the Appendix and Theorem 2.

Finally, we compare the mean squared error of the asymptotically constant-risk Bayes estimator, ${\widehat{\theta }}_{\pi }$, with that of the maximum likelihood estimator, ${\widehat{\theta }}_{\text{MLE}}$. The mean squared error of the asymptotically constant-risk Bayes estimator, ${\widehat{\theta }}_{\pi }$, is given as:

$$R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{\pi }\right)=\frac{1}{N}{\displaystyle \sum _{i=1}^d{g}^{X,ii}}\left({\theta }_{max}\right)+\frac{2}{N\sqrt{N}}{\displaystyle \sum {}_{k=1}^d}{g}^{X,ik}\left({\theta }_{max}\right)g^{X,jk}\left({\theta }_{max}\right){\partial }_{ij}\log \tilde{f}\left({\theta }_{max}\right)+\mathrm{O}\left(N^{-2}\right).$$

In contrast, the mean squared error of the maximum likelihood estimator, ${\widehat{\theta }}_{\text{MLE}}$, is given as:

$$R_{\text{MSE}}\left(\theta ,{\widehat{\theta }}_{{\rm M}\mathrm{L}{\rm E}}\right)=\frac{1}{N}{\displaystyle \sum {}_{i=1}^d}{g}^{X,ii}\left(\theta \right)+\mathrm{O}\left(N^{-2}\right).$$

See [16,19].

Thus, the maximum of the mean squared error of the asymptotically constant-risk Bayes estimator is smaller than that of estimators by the improvement of order N^−3/2 in proportion to the Hessian of the scalar function, $\log \tilde{f}\left(\theta \right),$ at θ_max. In the prediction where the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$, has a unique maximum point, the same improvement holds (Remark 3).

Example revisited. Using the above results, we consider the binomial estimation based on the mean squared error from the viewpoint of the prediction. The geometrical quantities to be used are given by:

respectively. $\stackrel{\mathrm{m}}{\Gamma }{}_{\theta \theta }^{X,\theta },\stackrel{\mathrm{m}}{\Gamma }{}_{\theta \theta }^{Y,\theta }$ and$T_{\theta \theta \theta }^Y$ vanish, the asymptotically constant-risk prior in the estimation is identical to the asymptotically constant-risk prior in the prediction; compare Theorem 1 with the expansion of$g^{Y,ij}\left(\theta \right){\mathrm{E}}_{X^{\left(N\right)}}\left[\left({\widehat{\theta }}_{\pi }^i-{\theta }^i\right)\left({\widehat{\theta }}_{\pi }^j-{\theta }^j\right)\right]$ in Lemma 4 in the Appendix.

In this example, Equation (3) is given by:

$${\theta }^2{\left(1-\theta \right)}^2{\left\{{\partial }_{\theta }\log \tilde{f}\left(\theta \right)\right\}}^2=\sqrt{\frac{1}{4}-\theta \left(1-\theta \right)},$$

and the solution, $\log \tilde{f}\left(\theta \right)$, is (1/2) log{θ(1 − θ)}. Here, the second-order derivative of the function, $\log \tilde{f}\left(\theta \right)$, is given by:

$${\partial }_{\theta \theta }\log \tilde{f}\left(\theta \right)=\frac{1-2\theta +2{\theta }^2}{2{\theta }^2{\left(1-\theta \right)}^2}.$$

From this, Equation (4) is given by:

$$\frac{1}{2}\theta \left(1-\theta \right)\left(1-2\theta \right){\partial }_{\theta }\log \tilde{h}\left(\theta \right)+{\theta }^2-\theta =\frac{1}{4},$$

and the solution, $\log \tilde{h}\left(\theta \right)$, is (1/2) log{θ(1 − θ)}. Hence, the asymptotically constant-risk prior, π (θ; N), is a Beta prior with the parameters, $\alpha =\sqrt{N}/2$ and$\beta =\sqrt{N}/2$. Note that the asymptotically constant-risk prior coincides with the exact minimax prior. Since g^X,θθ(θ_max) = 1/2 and$g^{X,\theta \theta }\left({\theta }_{max}\right){\partial }_{\theta \theta }\log \tilde{f}\left({\theta }_{max}\right)=-1$, the mean squared error of the asymptotically constant-risk Bayes estimator, ${\widehat{\theta }}_{\pi }$, agrees with (9) up to O(N⁻²).

5. Application to the Prediction of the Binary Regression Model under the Covariate Shift

In this section, we construct asymptotically constant-risk priors in the prediction based on the binary regression model under the covariate shift; see [10].

We consider that we predict a binary response variable, y, based on the binary response variables, x^(N). We assume that the target variable, y, and the data, x^(N), follow the logistic regression models with the same parameter, β, given by:

$$\log \frac{{\Pi }_x}{1-{\Pi }_x}=\alpha +z\beta$$

and:

$$\log \frac{{\Pi }_y}{1-{\Pi }_y}=\tilde{\alpha }+\tilde{z}\beta$$

where Π_x is the success probability of the data and Π_y is the success probability of the target variable. Let α and $\tilde{\alpha }$ denote known constant terms, and let β denote the common unknown parameter. Further, we assume that the covariates, z and $\tilde{z}$, are different.

Using the parameter θ = Π_x, we convert this predictive setting to binomial prediction where the data, x, and the target variable, y, are distributed according to:

$$p\left(x|\theta \right):=\{\begin{array}{ll}\theta & \text{if}x=1,\\ 1-\theta & \text{if}x=0\end{array}$$

and:

respectively. We obtain two Fisher information for x and y as:

$$g_{\theta \theta }^X\left(\theta \right)=\frac{1}{\theta \left(1-\theta \right)}$$

and:

$$g_{\theta \theta }^X\left(\theta \right)={\left(\frac{\tilde{z}}{z}\right)}^2{\mathrm{e}}^{-\tilde{\alpha }+\tilde{z}{z}^{-1}\alpha }\frac{{\left(1-\theta \right)}^{\tilde{z}{z}^{-1}-2}{\theta }^{\tilde{z}{z}^{-1}-2}}{{\left\{{\theta }^{\tilde{z}{z}^{-1}}+{\mathrm{e}}^{-\tilde{\alpha }+\tilde{z}{z}^{-1}\alpha }{\left(1-\theta \right)}^{\tilde{z}{z}^{-1}}\right\}}^2},$$

respectively.

For simplicity, we consider the setting where z = 1, $\tilde{z}=2$ and $\alpha =\tilde{\alpha }=0$. The geometrical quantities for the model, {p(x|θ) : θ ∈ Θ}, are given by:

respectively. In the same manner, the geometrical quantities for the model, {q(y|θ) : θ ∈ Θ}, are given by:

respectively.

Using these quantities, Equation (3) is given by:

$$4\frac{{\theta }^2{\left(1-\theta \right)}^2}{{\left\{{\theta }^2+{\left(1-\theta \right)}^2\right\}}^2}{\left({\partial }_{\theta }\log \tilde{f}\left(\theta \right)\right)}^2=4-4\frac{\theta \left(1-\theta \right)}{{\left\{{\theta }^2+{\left(1-\theta \right)}^2\right\}}^2}.$$

By noting that the maximum point of $g^{X,\theta \theta }\left(\theta \right)g_{\theta \theta }^Y\left(\theta \right)$ is 1/2, the solution, $\log \tilde{f}\left(\theta \right)$, of this equation is given by:

$$\begin{array}{l}\log \tilde{f}\left(\theta \right)=2\sqrt{1-\theta +{\theta }^2}+\log \left\{\theta \left(1-\theta \right)\right\}\\ -\log \left(2-\theta +2\sqrt{1-\theta +{\theta }^2}\right)-\log \left(1-\theta +2\sqrt{1-\theta +{\theta }^2}\right).\end{array}$$

Using this solution, we obtain the solution of Equation (4) given by:

The asymptotically constant-risk priors for the different sample sizes are shown in Figure 1. The prior weight is found to be more concentrated to 1/2 as the sample size, N, grows.

In this example, we obtain the Kullback–Leibler risk of the Bayesian predictive density based on the asymptotically constant-risk prior, π(θ; N), as:

$$R\left(\theta ,{\widehat{q}}_{\pi }\left(y|x^{\left(N\right)}\right)\right)=\frac{2}{N}-\frac{4\sqrt{3}}{N\sqrt{N}}+\mathrm{O}\left(N^{-2}\right).$$

We compare this value with the Bayes risk calculated using the Monte Carlo simulation; see Figure 2. As the sample size, N, grows, the difference appears negligible. Further, we compare this value with the risk itself calculated by the Monte Carlo simulation; see Figure 3. As the sample size, N, grows, the risk becomes more constant.

6. Discussion and Conclusions

We have considered the setting where the quantity, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$—the trace of the product of the inverse Fisher information matrix, g^X,ij(θ), and the Fisher information matrix, $g_{ij}^Y\left(\theta \right)$—has a unique maximum point, and we have investigated the asymptotically constant-risk prior in the sense that the asymptotic risk is constant up to O(N⁻²).

In Section 3, we have considered the prior depending on the sample size, N, and constructed the asymptotically constant-risk prior using Equations (3) and (4). In Section 4, we have clarified the relationship between the subminimax estimator problem based on the mean squared error and the prediction where the distributions of data and target variables are different. In Section 5, we have constructed the asymptotically constant-risk prior in the prediction based on the logistic regression model under the covariate shift.

We have assumed that the trace, $g^{X,ij}\left(\theta \right)g_{ij}^Y\left(\theta \right)$, is finite. However, the trace may diverge in the non-compact parameter space; for example, it diverges under the predictive setting, where the distribution, q(y|θ), of the target variable is the Poisson distribution and the data distribution, p(x|θ), is the exponential distribution, with Θ equivalent to ℝ. Therefore, for our future work, in such a setting, we should adopt criteria other than minimaxity.