Pertinent Prediction Intervals in Linear Regression

Abstract

In linear regression, a point predictor ${\widehat{Y}}_{\mathrm{f}}$ of a future response $Y_{\mathrm{f}}$ associated with a regressor value of interest ${\underline{x}}_{\mathrm{f}}$ can easily be constructed. Since ${\widehat{Y}}_{\mathrm{f}}$ will always incur a prediction error, it is desirable to accompany the point predictor by a prediction interval, say $C\left({\underline{x}}_{\mathrm{f}}\right)$, that will contain the target $Y_{\mathrm{f}}$ with a pre-specified high probability, e.g., 90%. An estimated prediction interval, say $\widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)$, is called pertinent if its construction incorporates the variability of all estimators that are employed in the prediction problem. So far, pertinent prediction intervals have only been constructed via some form of bootstrap. However, resampling can be quite computationally expensive since the estimation/prediction problem has to be re-calculated on a large number of pseudo-scatterplots, each having the same sample size as the original one. The paper at hand proposes a short-cut that directly employs the asymptotic normal distribution of relevant estimators—as opposed to a bootstrap histogram—in order to capture their variability. The resulting prediction interval achieves pertinence without full-scale resampling, thus offering computational savings of orders of magnitude.

1. Introduction

Consider regression data $\{(Y_1,{\underline{X}}_1^{\prime }),\dots ,(Y_n,{\underline{X}}_n^{\prime })\}$. The response $Y_i$ is assumed univariate while the regressor ${\underline{X}}_i$ is a $p\times 1$ vector. Note that the entries of ${\underline{X}}_i$ are in general real-valued but they can also be discrete, i.e., taking values on a countable subset of the real line; however, we will not consider categorical regressors as in [1].

Suppose that the $\{{\underline{X}}_1,\dots ,{\underline{X}}_n\}$ were chosen (or randomly generated—see Assumption 1 in what follows) as taking the values $\{{\underline{x}}_1,\dots ,{\underline{x}}_n\}$ respectively, and consider the usual linear regression model:

$$Y_i={\underline{x}}_i^{\prime }\underline{\beta }+Z_i,\mathrm{for}i=1,\dots ,n,$$

(1)

where the $Z_i$ are independent, identically distributed (i.i.d.) from some distribution F with $E{Z}_i=0$ and $E{Z}_i^2={\sigma }^2$.

The goal is to predict a future response $Y_{\mathrm{f}}$ associated with a particular regressor value ${\underline{x}}_{\mathrm{f}}$ of interest. Furthermore, we wish to construct a data-based prediction interval (PI), say $\widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)$, so that for some chosen $\alpha \in (0,1)$ we have

$$P(Y_{\mathrm{f}}\in \widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})\approx 1-\alpha$$

(2)

where the approximation is typically asymptotic as more data accrue. A review of the state-of-the-art of PI construction in different settings is given by [2].

At first glance, this exercise seems to be a straightforward application of employing the quantiles of the conditional distribution $F_{\mathrm{f}}\left(y\right)=P(Y_{\mathrm{f}}\le y|{\underline{X}}_{\mathrm{f}})$ which in many settings admits a consistent estimator, say ${\widehat{F}}_{\mathrm{f}}\left(y\right)$. Letting $G^{-1}\left(\gamma \right)=\inf \{x:G\left(x\right)\ge \gamma \}$ denote the quantile inverse of a distribution G, we can construct the ‘oracle’ equal-tailed PI $C\left({\underline{x}}_{\mathrm{f}}\right)=[F_{\mathrm{f}}^{-1}(\alpha /2),F_{\mathrm{f}}^{-1}(1-\alpha /2)]$, and its estimated version $\widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)=[{\widehat{F}}_{\mathrm{f}}^{-1}(\alpha /2),{\widehat{F}}_{\mathrm{f}}^{-1}(1-\alpha /2)]$. If $F_{\mathrm{f}}\left(y\right)$ is continuous and strictly increasing for y in the neighborhood of the two quantiles of interest, i.e., for $\gamma$ equal to $\alpha /2$ and $1-\alpha /2$, then

$$P(Y_{\mathrm{f}}\in C\left({\underline{x}}_{\mathrm{f}}\right)|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})=1-\alpha .$$

(3)

and

$$P(Y_{\mathrm{f}}\in \widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})\to 1-\alpha \mathrm{as}n\to \infty$$

(4)

since ${\widehat{F}}_{\mathrm{f}}^{-1}\left(\gamma \right)$ will be a consistent estimator of $F_{\mathrm{f}}^{-1}\left(\gamma \right)$; see e.g., Lemma 1.2.1 of [3].

Nevertheless, despite the asymptotic validity of Equation (4), the quantile-based PI $\widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)$ is well-known to suffer from finite-sample undercoverage, i.e.,

$$P(Y_{\mathrm{f}}\in \widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})<1-\alpha \mathrm{for} \mathrm{finite}n.$$

(5)

This undercoverage has been illustrated empirically time and again—see e.g., [4] and the references therein—but recently some theoretical explanations have also been given; see [5,6]. The implicit reason for the finite-sample undercoverage is that by plugging-in the estimated quantiles as if they were true, the PI $\widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)$ fails to capture the variability of these estimated quantiles, i.e., it is neglecting part of the variance of the problem at hand. In other words, the quantile-based PI $\widehat{C}\left({\underline{x}}_{\mathrm{f}}\right)$ is not pertinent; this term was coined by [7] to describe PIs that encapsulate the variability of all estimated quantities figuring in the PI construction.

Starting with the work of [8], there has been increased interest in PIs obtained via conformal prediction; see also [9]. Conformal prediction was originally developed for i.i.d. data. Modelling the regression pairs $(Y_1,{\underline{X}}_1^{\prime }),\dots ,(Y_n,{\underline{X}}_n^{\prime })$ as i.i.d., a conformal prediction PI $\tilde{C}\left({\underline{X}}_{\mathrm{f}}\right)$ can be constructed that satisfies

$$P(Y_{\mathrm{f}}\in \tilde{C}\left({\underline{X}}_{\mathrm{f}}\right))\ge 1-\alpha .$$

(6)

However, the above finite-sample coverage guarantee is on the average, over all possible regressor values ${\underline{X}}_{\mathrm{f}}$. If the practitioner desires $1-\alpha$ coverage conditionally on a particular regressor value ${\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}$, then the exchangeability of the pairs $(Y_i,{\underline{X}}_i^{\prime })$ breaks down and the finite-sample guarantee of (6) is lost; see [10,11]. There is an active body of literature developing variations of the conformal prediction theme to produce PIs that are asymptotically valid conditionally on ${\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}$ as in Equation (2); see e.g., [12,13,14], as well as [15] and the references therein. However, none of these variations yields pertinent PIs; see [6] for a discussion.

Note that if the regression errors $Z_i$ can be assumed to be Gaussian, then a pertinent PI with exact coverage is available; see Equation (A1) in the Appendix A. However, in the Big Data Era, the assumption of Gaussianity is typically seen to be unjustifiable, and often just plain wrong. Without having to resort to unnessesarily restrictive assumptions such as Gaussianity, the only general way so far to obtain pertinent PIs has been via some form of bootstrap. Furthermore, to achieve pertinence, the resampling procedure must be carefully crafted so that it captures the variability of all estimated quantities including the variability of estimating the point predictor; see e.g., Ch. 3.7 of [4].

Although the bootstrap approach gives generally good results in finite-samples (as well as asymptotically), it can be quite computationally expensive as it involves re-fitting the regression over M bootstrap pseudo-samples; note that each pseudo-sample entails a new scatterplot of n data points. In the paper at hand, we describe a shortcut that gives pertinent PIs based on a limited Monte Carlo simulation as opposed to full-scale resampling. The new method is introduced in Section 3 in the linear regression framework that is outlined in Section 2. Finally, Section 4 reports the results of a small numerical experiment, and provides some practical recommendations. A brief Conclusions section is also provided as well as an Appendix A to intuitively describe the notion of pertinence.

2. Linear Regression Framework

We can re-write the linear regression model of Equation (1) as

$${\underline{Y}}_n=X\underline{\beta }+{\underline{Z}}_n$$

where ${\underline{Y}}_n={(Y_1,\dots ,Y_n)}^{\prime }$ and ${\underline{Z}}_n={(Z_1,\dots ,Z_n)}^{\prime }$ are $n\times 1$ random vectors, $\underline{\beta }$ is a $p\times 1$ deterministic parameter vector with $p<n$, and X is an $n\times p$ design matrix with ith row given by vector ${\underline{X}}_i^{\prime }$. Throughout the paper we will also assume:

Assumption 1. 

Assume that the design matrix X is either deterministic or that inference is conducted conditionally on the ${\underline{X}}_i^{\prime }$ vectors that are assumed independent to the error vector ${\underline{Z}}_n$. Hence, without this being explicitly denoted, all probabilities and expectations will be tacitly understood to be conditional on the event $\{{\underline{X}}_1={\underline{x}}_1,\dots ,{\underline{X}}_n={\underline{x}}_n\}$ where $\{{\underline{x}}_1,\dots ,{\underline{x}}_n\}$ are some fixed values.

Consequently, the coverage of a PI will be evaluated conditionally on $\{{\underline{X}}_1={\underline{x}}_1,\dots ,{\underline{X}}_n={\underline{x}}_n\}$ as well as ${\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}$ as compared to the probability measure $P_2$ of [6].

Let $\underline{\widehat{\beta }}$ be an estimator of $\underline{\beta }$ that is linear in the data ${\underline{Y}}_n$, i.e., $\underline{\widehat{\beta }}=B{\underline{Y}}_n$ for some $p\times n$ matrix B; the matrix B will typically depend on X but this will not be explicitly denoted. For example, if X has full rank, we can let $B={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }$ so that $\underline{\widehat{\beta }}$ is the Least Squares (LS) estimator. If X is not of full rank, we can let $B={(X^{\prime }X+\lambda I_p)}^{-1}{X}^{\prime }$ so that $\underline{\widehat{\beta }}$ is a ridge regression estimator; here $I_p$ is the $p\times p$ identity matrix and $\lambda$ is a (small) positive constant. Other examples are readily available in the literature.

Let ${\underline{\widehat{\beta }}}^{\left(i\right)}$ be the same estimator based on the dataset that has the pair $(Y_i,{\underline{X}}_i^{\prime })$ omitted. In other words, ${\underline{\widehat{\beta }}}^{\left(i\right)}=B^{\left(i\right)}{\underline{Y}}_n^{\left(i\right)}$ where ${\underline{Y}}_n^{\left(i\right)}$ is ${\underline{Y}}_n$ with $Y_i$ omitted, and $B^{\left(i\right)}$ is B with its ith column omitted. The predictive and fitted residuals (${\tilde{z}}_i$ and $z_i$ respectively) corresponding to data point $Y_i$ are defined in the usual manner, i.e., ${\tilde{z}}_i=Y_i-{\underline{x}}_i^{\prime }{\underline{\widehat{\beta }}}^{\left(i\right)}$ and $z_i=Y_i-{\underline{x}}_i^{\prime }\underline{\widehat{\beta }}$.

Remark 1 

(LS residuals). In the LS case where $\underline{\widehat{\beta }}={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }{\underline{Y}}_n$ the predictive residuals can be easily obtained without re-fitting the regression since ${\tilde{z}}_i=z_i/(1-h_i)$ where $h_i={\underline{x}}_i^{\prime }{\left(X^{\prime }X\right)}^{-1}{\underline{x}}_i$ is the ith diagonal element of the ‘hat’ matrix $X{\left(X^{\prime }X\right)}^{-1}{X}^{\prime }$; see Theorem 10.1 of [16]. In this setting, we can also use the so-called studentized residuals ${\widehat{z}}_i=z_i/\sqrt{1-h_i}$ as defined by [17]. Assuming that the regression has an intercept term, Equation (10.12) of [16] further implies $1/n\le h_i\le 1$, yielding the ordering $|z_i| \le |{\widehat{z}}_i| \le |{\tilde{z}}_i|$ for all i, i.e., the predictive residuals have the largest scale while the fitted residuals have the smallest.

Recall that the $L_2$–optimal point predictor of $Y_{\mathrm{f}}$ given ${\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}$ is the conditional expectation $E\left(Y_{\mathrm{f}}\right|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\beta }$. Since $\underline{\beta }$ is unknown, our practical point predictor of $Y_{\mathrm{f}}$ will be ${\widehat{Y}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}$. In order to construct a pertinent PI, the goal is to approximate the distribution of the so-called ‘root’ $Y_{\mathrm{f}}-{\widehat{Y}}_{\mathrm{f}}$. If we were to do so via bootstrap, one would proceed by resampling the residuals and creating new pseudo-scatterplots via the (fitted) model (1). In this setting, the transformation-based approach of [4] suggested resampling the predictive residuals ${\tilde{z}}_i$—as opposed to the fitted residuals $z_i$—in order to alleviate the undercoverage of bootstrap PIs first pointed out by [18]. The predictive residuals will be found useful in what follows as well.

3. Pertinent Prediction Intervals

Consider again the root $Y_{\mathrm{f}}-{\widehat{Y}}_{\mathrm{f}}=Y_{\mathrm{f}}-{\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}$. Model (1) implies that $Y_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\beta }+Z_{\mathrm{f}}$ where $Z_{\mathrm{f}}\sim F.$ Hence,

$$Y_{\mathrm{f}}-{\widehat{Y}}_{\mathrm{f}}=Z_{\mathrm{f}}+{\underline{x}}_{\mathrm{f}}^{\prime }\underline{\beta }-{\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}=Z_{\mathrm{f}}+E_{\mathrm{f}}$$

where $E_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\beta }-{\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}$ is the estimation error. Standard regression conditions typically ensure that

$$E{\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\beta }+o(1/\sqrt{n})\mathrm{and}Var\left({\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}\right)=O(1/n)$$

(7)

as $n\to \infty$. For instance, if $\underline{\widehat{\beta }}$ is the LS estimator in a well-specified model, then $E\underline{\widehat{\beta }}=\underline{\widehat{\beta }}$. Further conditions on the design matrix are available to ensure the 2nd part of Equation (7); see [16].

Note that Equation (7) implies that $E_{\mathrm{f}}\to 0$ in probability as $n\to \infty$. It is tempting to treat $E_{\mathrm{f}}$ as negligible, and hence construct a simple quantile-based PI for $Y_{\mathrm{f}}$ as

$$[{\widehat{Y}}_{\mathrm{f}}+{\widehat{F}}^{-1}(\alpha /2),{\widehat{Y}}_{\mathrm{f}}+{\widehat{F}}^{-1}(1-\alpha /2)]$$

(8)

where $\widehat{F}$ is a (uniformly) consistent estimator of F. For example, $\widehat{F}$ can be the empirical distribution function (e.d.f.) of the residuals (fitted, predictive or studentized) after centering them to ensure that $\widehat{F}$ is a distribution with mean zero. If F is continuous and strictly increasing in the neighborhood of the two quantiles of interest, namely $F^{-1}(\alpha /2)$ and $F^{-1}(1-\alpha /2)$, then the latter are estimated consistently by ${\widehat{F}}^{-1}(\alpha /2)$ and ${\widehat{F}}^{-1}(1-\alpha /2)$ respectively, implying that the quantile-based PI (8) has asymptotic coverage $1-\alpha$. However, PI (8) will be referred to as naive in what follows because it ignores the term $E_{\mathrm{f}}$ and its variability, and typically exhibits finite-sample undercoverage.

Recall that $\underline{\widehat{\beta }}$ is a function of ${\underline{Z}}_n$ which is independent of $Z_{\mathrm{f}}$. Hence, the distribution of $Z_{\mathrm{f}}+E_{\mathrm{f}}$ is given by $G=F\ast F_E$ where ∗ denotes convolution and $F_E$ is the distribution of $E_{\mathrm{f}}$. Consequently,

$$P(G^{-1}(\alpha /2)\le Y_{\mathrm{f}}-{\widehat{Y}}_{\mathrm{f}}\le G^{-1}(1-\alpha /2)|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})=1-\alpha$$

(9)

assuming G is continuous and strictly increasing at its tail quantiles $G^{-1}(\alpha /2)$ and $G^{-1}(1-\alpha /2)$. Equation (9) then suggest the ‘oracle’ PI

$$[{\widehat{Y}}_{\mathrm{f}}+G^{-1}(\alpha /2),{\widehat{Y}}_{\mathrm{f}}+G^{-1}(1-\alpha /2)].$$

(10)

The above oracle PI has exact coverage $1-\alpha$ but it involves the two unknown distributions, F and $F_E$. To construct a practical pertinent PI we propose to replace F and $F_E$ by respective estimators $\widehat{F}$ and ${\widehat{F}}_E$. $\widehat{F}$ was already mentioned; as regards $F_E$, recall that regularity conditions typically ensure that $\underline{\widehat{\beta }}$ is asymptotically normal, and therefore

$${\displaystyle \frac{E_{\mathrm{f}}}{\sqrt{Var\left({\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}\right)}}}\Rightarrow N(0,1)\mathrm{as}n\to \infty .$$

(11)

Coupled with the fact that $Var\left({\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}\right)={\sigma }^2{\underline{x}}_{\mathrm{f}}^{\prime }B{B}^{\prime }{\underline{x}}_{\mathrm{f}}$ we are led to defining ${\widehat{F}}_E$ as a normal distribution with mean zero and variance ${\widehat{\sigma }}^2{\underline{x}}_{\mathrm{f}}^{\prime }B{B}^{\prime }{\underline{x}}_{\mathrm{f}}$; here, ${\widehat{\sigma }}^2$ is a consistent estimator of ${\sigma }^2$, e.g., the sample variance of the residuals (fitted, predictive or studentized). Although $\widehat{F}$ may be a discontinuous distribution, the continuity of ${\widehat{F}}_E$ implies that our estimator $\widehat{G}=\widehat{F}\ast {\widehat{F}}_E$ is a continuous, strictly increasing distribution that is (uniformly) consistent for G.

The following lemma puts it all together.

Lemma 1. 

Assume conditions ensuring (7) and (11), and the (uniform) consistency of $\widehat{F}$ and ${\widehat{\sigma }}^2$ for F and ${\sigma }^2$ respectively. If G is continuous and strictly increasing at its tail quantiles $G^{-1}(\alpha /2)$ and $G^{-1}(1-\alpha /2)$, then

$$P({\widehat{G}}^{-1}(\alpha /2)\le Y_{\mathrm{f}}-{\widehat{Y}}_{\mathrm{f}}\le {\widehat{G}}^{-1}(1-\alpha /2)|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})\to 1-\alpha$$

(12)

as $n\to \infty$.

Equation (12) leads us to construct our novel pertinent PI as

$$[{\widehat{Y}}_{\mathrm{f}}+{\widehat{G}}^{-1}(\alpha /2),{\widehat{Y}}_{\mathrm{f}}+{\widehat{G}}^{-1}(1-\alpha /2)].$$

(13)

Note that the Central Limit Theorem (CLT) of Equation (11), as well as the approximations of F and ${\sigma }^2$ by $\widehat{F}$ and ${\widehat{\sigma }}^2$, do require a reasonably large sample. But even with n as small as 100 these approximations can be good enough for practical application; see Section 4 for some numerical evidence. Moreover, recall that assumption (7) implies that $E_{\mathrm{f}}=O_P(1/\sqrt{n})$ which is not negligible with n about 100—therefore the importance of actually including it in the PI construction; the following remark elaborates.

Remark 2 

(Asymptotic validity). Lemma 1 implies that the pertinent PI (13) has asymptotic coverage $1-\alpha$ conditionally on ${\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}$. Although consistency is a sine qua non, it does not tell the whole story; recall that even the naive PI (8) is consistent. The PI (13) is called pertinent because it captures the variance of the estimation error $E_{\mathrm{f}}$ which the naive PI (8) treats as negligible. As shown in the simulation results of Section 4, the pertinent PI (13) alleviates the finite-sample undercoverage of the naive PI (8).

In principle, the convolution $\widehat{G}=\widehat{F}\ast {\widehat{F}}_E$ could be carried out numerically; however, the following Monte Carlo algorithm makes its implementation straightforward.

• Monte Carlo Algorithm for Pertinent Prediction Intervals

• Compute the estimator $\underline{\widehat{\beta }}=B{\underline{Y}}_n$, the point predictor ${\widehat{Y}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\widehat{\beta }}$, and the residuals of choice (fitted, predictive or studentized).
• Compute an estimator ${\widehat{\sigma }}^2$ of ${\sigma }^2$; e.g., ${\widehat{\sigma }}^2$ could be the sample variance of the chosen residuals. [In the case of Least Squares, the unbiased estimator ${(n-p)}^{-1}{\sum }_{i=1}^n{z}_i^2$ could also be entertained.]
• Compute an estimator $\widehat{F}$ of F; e.g., $\widehat{F}$ can be the e.d.f. of the residuals of choice after centering to ensure that $\widehat{F}$ is a distribution with mean zero.
• Choose a large integer M, e.g., $M\ge 1000$, and generate $Z_1^{*},\dots ,Z_M^{*}$ i.i.d. from $\widehat{F}$; the $Z_i^{*}$s can be thought of as pseudo-residuals. [If $\widehat{F}$ is an e.d.f. that puts mass $1/n$ on values $a_1,\dots ,a_n$ (say), then generating $Z_1^{*},\dots ,Z_M^{*}$ can be done by drawing randomly (with replacement) from the set $\{a_1,\dots ,a_n\}$.]
• Generate $E_1^{*},\dots ,E_M^{*}$ i.i.d. from a normal distribution with mean zero and variance ${\widehat{\sigma }}^2{\underline{x}}_{\mathrm{f}}^{\prime }B{B}^{\prime }{\underline{x}}_{\mathrm{f}}$.
• Define $W_i=Z_i^{*}+E_i^{*}$ for $i=1,\dots ,M$, and compute ${\widehat{G}}^{-1}\left(\gamma \right)$ as the $\gamma$ sample quantile of $W_1,\dots ,W_M$, i.e., ${\widehat{G}}^{-1}\left(\gamma \right)=W_{\left[\gamma M\right]}$ where $W_{\left[1\right]}\le W_{\left[2\right]}\le \dots \le W_{\left[M\right]}$ are the order statistics.
• Use the above ${\widehat{G}}^{-1}\left(\gamma \right)$ with $\gamma$ equal to $\alpha /2$ and $1-\alpha /2$ to construct the pertinent PI (13).

Remark 3 

(Choice of residuals). In the LS setting of Remark 1, note that our Equation (7) implies that $h_i\to 0$ as $n\to \infty$, and hence all residuals (fitted, predictive or studentized) are asymptotically equivalent. Therefore, asymptotic considerations can not help us in choosing which residuals to employ for the construction of the estimators ${\widehat{\sigma }}^2$ and $\widehat{F}$ in the above algorithm. Note, however, that the validity of Equation (7) hinges on either p being finite, or at least that $p<<n$. For example, if p is of the same order of magnitude as n, then the $h_i$s will not be close to 0 even for large n, and the fitted, predictive andstudentized residuals will remain distinct. The numerical experiment of Section 4 will provide some finite-sample guidance on our choice of residuals and the effect of the relative magnitude of p to n.

Remark 4 

(Computational cost as compared to bootstrap). Note that sampling from $\widehat{F}$ is effectively a bootstrap procedure. However, in the above Monte Carlo algorithm, sampling from $\widehat{F}$ is done only once, i.e., we are drawing just one bootstrap sample of size M. Suppose that the estimators $\underline{\widehat{\beta }}$, ${\widehat{\sigma }}^2$ and $\widehat{F}$ can be computed in $O\left(n^{\zeta }\right)$ operations for some $\zeta \ge 1.$ Then, the above algorithm has computational cost of $O\left(n^{\zeta }\right)+O\left(M\right)$ operations. By contrast, the pertinent version of the residual-based bootstrap requires drawing M resamples each of size n from $\widehat{F}$; using each of the M resamples as a set of pseudo-residuals, a new pseudo-scatterplot is created upon which $\underline{\widehat{\beta }}$, ${\widehat{\sigma }}^2$, etc. are recomputed—see the MF/MB procedure in Part II of [4] for more details. Hence, the pertinent version of the residual-based bootstrap requires $O\left(M{n}^{\zeta }\right)$ operations which is more expensive by orders of magnitude. To elaborate, consider the realistic case where $n^{\zeta }\ge 1000$; since M is also taken not less than 1000, the result is that the above Monte Carlo algorithm can be a thousand times faster than the pertinent version of the residual bootstrap.

To elaborate, consider the popular example of LASSO regression that can be computed in $O(p^3+p^{2}n)$ operations as long as $p<n$; see [19]. Suppose that $p\sim \sqrt{n}$ in which case LASSO regression is computable in $O\left(n^{\zeta }\right)$ operations with $\zeta =2$. Let $M=1000$, and that the sample size satisfies $n>33$. Then, we have $M<n^2$ and our Monte Carlo algorithm is computable in $O\left(n^2\right)$ operations, i.e., same order as computing a single LASSO regression. By contrast, the residual-based bootstrap requires $O\left(M{n}^2\right)$ operations, i.e., it is like fitting 1000 LASSO regressions.

4. Numerical Results

We now conduct a small numerical experiment with three goals: (a) to show that the pertinent PI (denoted PPI) of Equation (13) has better finite-sample coverage than the naive PI of Equation (8); (b) to provide some finite-sample guidance on which type of residuals to employ in the PPI algorithm; and (c) to see the effect of the magnitude of p (relative to n) in our choice of residuals, and the resulting PPI performance.

Throughout this section, we will focus on the LS estimator $\underline{\widehat{\beta }}={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }{\underline{Y}}_n$ where X is an $n\times p$ matrix having a column of 1s as its first column.

The structure of our numerical experiment is as follows:

• Choose p and n; for our main results we will employ the following combinations for $(p,n)$: (10,50), (20,50), (20,100), (40,100).
• Generate all the elements of X (other than its first column) as i.i.d. standard normal.
• Choose the error distribution F; for our purposes we will take F as either standard normal or Laplace (i.e., two-sided exponential) with mean zero and variance one.
• Choose $\underline{\beta }$, the $\alpha$ level, and the regressor of future interest ${\underline{x}}_{\mathrm{f}}$; for our purposes, $\underline{\beta }$ is generated as $3·{(U_1,\dots ,U_p)}^{\prime }$ where $U_1,\dots ,U_p$ are i.i.d. Uniform $[0,1]$, the $\alpha$ level is chosen as 0.10, and ${\underline{x}}_{\mathrm{f}}^{\prime }$ is a row of 1s.
• Generate $Z_1,\dots ,Z_n$ i.i.d. from F, and use them to generate $Y_1,\dots ,Y_n$ from model (1).
• Fit the data via LS and compute $\underline{\widehat{\beta }}$ and ${\widehat{Y}}_{\mathrm{f}}$; also compute the three sets of residuals: fitted, studentized and predictive.
• Calculate four PIs for $Y_{\mathrm{f}}$: PI.naive of Equation (8), and three versions of the PPI of Equation (13): using the fitted residuals (PPI.fitted), using the studentized residuals (PPI.stud), and using the predictive residuals (PPI.pred). The three PPIs employ the Monte Carlo algorithm using $M=1000$. Compute the lengths (LEN) of the four PIs.
• Further generate $Z_{n+1},\dots ,Z_{n+500}$ i.i.d. from F, and use them to generate 500 realizations of $Y_{\mathrm{f}}$ to be used to test the PI coverage. The kth realization $Y_{\mathrm{f}}^{\left(k\right)}$ is generated as $Y_{\mathrm{f}}^{\left(k\right)}={\underline{x}}_{\mathrm{f}}^{\prime }\underline{\beta }+Z_{n+k}.$
• Calculate the coverage (CVR) of each PI using the 500 realizations of $Y_{\mathrm{f}}$, i.e., CVR of a PI is ${500}^{-1}{\sum }_{k=1}^{500}\mathbf{1}\{Y_{\mathrm{f}}\in \mathrm{PI}\}$ where $\mathbf{1}$ denotes the indicator function.
• Repeat steps 5 to 9 many times—in our case 200. Each of these $j=1,\dots ,200$ replications has its own CVR[j] and LEN[j] associated with each PI.
  Finally:

  (a)

  Calculate the average CVR of each method over the 200 replications.

  (b)

  Plot a histogram of CVR[j] for $j=1,\dots ,200$ to portray the CVR variability, and compute what percentage of these CVRs fall below the target level of $1-\alpha$.

  (c)

  Calculate the average LEN of each method over the 200 replications; also calculate the variability of LEN via its sample standard deviation.

To elaborate on the PI constructions:

• PI.naive and PPI.fitted employ as $\widehat{F}$ the e.d.f. of the fitted residuals. The latter further uses ${\widehat{\sigma }}^2$ to be the unbiased estimator ${(n-p)}^{-1}{\sum }_{k=1}^n{z}_k^2$. Recall that having a column of 1s in X ensures ${\sum }_{k=1}^n{z}_k=0$.
• PPI.stud employs $\widehat{F}$ the e.d.f. of the centered studentized residuals, i.e.,

  $$\widehat{F}\left(a\right)=n^{-1}{\displaystyle \sum _{i=1}^n}\mathbf{1}\{{\widehat{z}}_i-n^{-1}{\displaystyle \sum _{k=1}^n}{\widehat{z}}_k\le a\}.$$
  It also employs ${\widehat{\sigma }}^2=n^{-1}{\sum }_{i=1}^n{[{\widehat{z}}_i-n^{-1}{\sum }_{k=1}^n{\widehat{z}}_k]}^2$.
• Finally, PPI.pred employs $\widehat{F}$ the e.d.f. of the centered predictive residuals, i.e.,

  $$\widehat{F}\left(a\right)=n^{-1}{\displaystyle \sum _{i=1}^n}\mathbf{1}\{{\tilde{z}}_i-n^{-1}{\displaystyle \sum _{k=1}^n}{\tilde{z}}_k\le a\}.$$
  It also employs ${\widehat{\sigma }}^2=n^{-1}{\sum }_{i=1}^n{[{\tilde{z}}_i-n^{-1}{\sum }_{k=1}^n{\tilde{z}}_k]}^2$.

The main results of our experiment are presented in

Table 1. Performance of 90% PI constructions: mean coverage levels (CVR), % of CVRs that are <0.9, mean PI length (with standard deviation in parentheses); case of Normal errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.765	97%	2.79 (0.37)
PPI.fitted	0.837	80.5%	3.31 (0.42)
PPI.stud	0.874	60%	3.62 (0.46)
PPI.pred	0.913	29%	4.06 (0.51)

,

Table 2. Entries as in Table 1; case of Laplace errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.812	90%	2.74 (0.48)
PPI.fitted	0.865	66%	3.24 (0.55)
PPI.stud	0.889	48%	3.52 (0.60)
PPI.pred	0.917	27%	4.00 (0.68)

,

Table 3. Entries as in Table 1; case of Normal errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.680	100%	2.45 (0.32)
PPI.fitted	0.845	72%	3.41 (0.43)
PPI.stud	0.899	39%	4.01 (0.53)
PPI.pred	0.960	8%	5.27 (0.70)

,

Table 4. Entries as in Table 1; case of Laplace errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.639	99%	2.40 (0.41)
PPI.fitted	0.842	60%	3.71 (0.59)
PPI.stud	0.888	37%	4.35 (0.69)
PPI.pred	0.950	10%	5.71 (0.99)

,

Table 5. Entries as in Table 1; case of Normal errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.791	97.5%	2.87 (0.27)
PPI.fitted	0.860	73%	3.38 (0.28)
PPI.stud	0.893	44%	3.71 (0.32)
PPI.pred	0.925	16%	4.13 (0.35)

,

Table 6. Entries as in Table 1; case of Laplace errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.832	87%	2.77 (0.35)
PPI.fitted	0.868	55%	3.37 (0.40)
PPI.stud	0.895	31.5%	3.66 (0.41)
PPI.pred	0.927	16.5%	4.10 (0.51)

,

Table 7. Entries as in Table 1; case of Normal errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.676	100%	2.55 (0.26)
PPI.fitted	0.854	61%	3.82 (0.36)
PPI.stud	0.890	44.5%	4.34 (0.41)
PPI.pred	0.963	9.5%	5.60 (0.53)

,

Table 8. Entries as in Table 1; case of Laplace errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.667	99%	2.47 (0.35)
PPI.fitted	0.845	57%	3.75 (0.53)
PPI.stud	0.888	39%	4.26 (0.59)
PPI.pred	0.950	10%	5.54 (0.79)

and

Table 9. Entries as in Table 1; case of Normal errors.

$\mathit{p}=10,\mathit{n}=50$	CVR	% < 0.9	Length (St.Dev.)
PI.naive	0.866	73%	3.12 (0.29)
PPI.fitted	0.887	60.5%	3.28 (0.29)
PPI.stud	0.897	51%	3.36 (0.30)
PPI.pred	0.908	38%	3.45 (0.31)

. There were also nine pages of CVR histograms associated with the results of each of the nine tables. To save space, we only present five representative ones in Figure 1, Figure 2, Figure 3, Figure 4 and Figure 5; the omitted histograms were very similar to the ones included here. Note that each of these histograms is an approximation of the probability distribution of the random variable

$$P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}},{\underline{Y}}_n).$$

(14)

Recall that per Assumption 1 the above probability is also conditional on the event $\{{\underline{X}}_1={\underline{x}}_1,\dots ,{\underline{X}}_n={\underline{x}}_n\}$ without it being explicitly denoted.

Being a function of the sample ${\underline{Y}}_n$, the quantity in Equation (14) is a random variable that varies across samples. Our numerical experiment tries to encapsulate this variability in an analogous way as in Figure 2 of [6]. Note that the mean CVR reported in the first column of Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8 and Table 9 gives the center of each of these histograms, namely

$$P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})$$

(15)

which is the expected value of the random variable (14) averaged over all possible ${\underline{Y}}_n$ samples.

From the numerical findings of Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8 and Table 9 we can draw several interesting conclusions:

–

The undercoverage of PI.naive is strongly illustrated in all Tables and Figures.

–

All three of our proposed PPIs have larger coverage than PI.naive; furthermore, we observe the ordering:

CVR[PI.fitted] < CVR[PI.stud] < CVR[PI.pred]

which is not unexpected in view of Remark 1 that compares the scales of the three types of residuals.

–

It is clear that PI.fitted falls short of achieving acceptable coverage for finite samples; this can be attributed to the known tendency of the fitted residuals to underestimate the scale of the errors—see [20].

–

The preferable methods seems to be either PPI.stud or PPI.pred; the latter appears to overcover in several instances so it can be thought of as conservative.

–

The variability of the CVRs that is illustrated by the histograms in Figure 1, Figure 2, Figure 3 and Figure 4 is striking. Also interesting is the high proportion of CVRs (conditional on the sample) that are below the nominal level over the 200 replications. PPI.pred is more conservative in that respect as well, i.e., having the smallest proportion of PIs that undercover.

–

In our Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8, the ratio $p/n$ was either 0.2 or 0.4 which can not be considered small; neither a sample size of 50 or 100 can be considered particularly large. Despite the fact that these setups are quite challenging, both PPI.stud and PPI.pred provided reliable performances (with the latter being more conservative).

–

Interestingly, there is no appreciable difference of PI performance going from normal errors to Laplace even though in the latter case the LS estimator does not coincide with the minimum maximum likelihood estimator. The only side-effect of the heavier tails of the Laplace distribution appears to be a slightly heavier left tail in some of our histograms.

As discussed in Remark 3, the standard asymptotics kick in when $p/n\to 0$ in which case the fitted, studentized and predictive residuals become approximately equal to each other. In that case the performance of all three PPIs is expected to be good, although PPI.stud and PPI.pred will still be preferable over the other two PIs. To corroborate this claim, we offer the results of an additional simulation in the ‘easy’ case of $p=5$ and $n=100$, i.e., when $p/n$ is small; as Table 9 shows, PPI.stud and PPI.pred have mean CVRs of 0.897 and 0.908 respectively that are practically equivalent to the nominal 90%. So are two methods PPI.stud and PPI.pred equivalent in this ‘easy’ case?

The above question allows us to think harder on our design goals for a successful PI construction. Until now, our target was to have CVR (15), i.e., $P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})$ close to the nominal 90% (and ideally not below the nominal). However, Figure 5 shows the histogram of the CVR (14) that has Equation (15) as its center. In order for the center to be about 90%, the CVR (14) should take values both below and above 90%. Indeed, as shown in Corollary 4.1 of [21], PIs based on the residual bootstrap will exhibit large-sample histograms with their median at the nominal level, i.e., asymptotically the histogram will have 50% of occurences below the nominal and 50% above.

By the Law of Large Numbers, such a histogram will have progressively smaller and smaller range since asymptotically

$$P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}},{\underline{Y}}_n)\to E\left(P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}},{\underline{Y}}_n)\right)=P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}})$$

where the above expectation is taken over the distribution of ${\underline{Y}}_n$. In spite of the above, it seems preferable that a desirable finite-sample histogram of $P(Y_{\mathrm{f}}\in \mathrm{PI}|{\underline{X}}_{\mathrm{f}}={\underline{x}}_{\mathrm{f}},{\underline{Y}}_n)$ would have less mass below the nominal 90% (as compared to above the nominal), i.e., only a minority of the CVRs (14) would be below the nominal 90%. In this sense, PPI.pred appears preferable to PPI.stud even here since, as shown in Table 9, PPI.stud and PPI.pred have respectively 51% and 38% of occurences below the nominal.

5. Conclusions

An interesting problem in statistics is the construction of prediction intervals (PI) in linear regression without assuming Gaussianity which is often not justifiable. We desire PIs that have (a) asymptotic conditional validity, and (b) are ‘pertinent’, i.e., are able to capture/incorporate the variability associated with estimated quantities employed in the PI construction. Such PIs in the past have been traditionally constructed via resampling.

The paper at hand proposes a short-cut that directly employs the asymptotic normal distribution of relevant estimators—as opposed to a bootstrap histogram—in order to capture their variability. The resulting prediction interval achieves the property of pertinence without full-scale resampling, thus offering computational savings of orders of magnitude over the bootstrap.

Asymptotic conditional validity of the new approach is shown under general conditions using either fitted, studentized or predictive residuals. Numerical work confirms that the new method has good finite-sample performance as well, and suggests the use of predictive residuals in order to construct conservative PIs, i.e., resulting into the smallest proportion of PIs that undercover.