Consistency of the OLS Bootstrap for Independently but Not-Identically Distributed Data: A Permutation Perspective

Abstract

This paper introduces a new approach to proving bootstrap consistency based upon the distribution of permutation statistics, using it to derive results covering fundamentally not-identically distributed groups of data, in which average moments do not converge to anything, with moment conditions that are less demanding than earlier results for either identically distributed or not-identically distributed data.

1. Introduction

Data are often drawn from dissimilar environments which render the independent and identically distributed (iid) assumption that underlies many results on the bootstrap suspect.1 This paper extends results concerning the consistency of the pairs and wild OLS bootstraps, which have mostly been derived for iid data, to general regression frameworks with independently but not-necessarily identically distributed (inid) data. Instead of considering the sampling distribution of the bootstraps, the usual approach, it notes that any permutation of the pairs bootstrap vector of sampling frequencies or the realization of the external variable used by the wild bootstrap to transform residuals is equally likely. Using results on the asymptotic distribution of permutation statistics by Wald and Wolfowitz (1944), Noether (1949), and Hoeffding (1951), these equally likely permutations can be used to characterize the bootstrap distributions conditional on the data as normal given restrictions on sample moments of the data. White’s (1980a) conditions for the asymptotic normality of OLS coefficients with clustered/heteroskedastic residuals and inid data guarantee these restrictions almost surely, ensuring that the asymptotic distribution of pairs and wild bootstrapped coefficients and Wald statistics conditional on the data matches the unconditional distribution of the original OLS estimates.

While proofs of bootstrap consistency typically require the existence of at least fourth moments of the regressors with iid data, the permutation distribution allows this paper to prove consistency with no more than second regressor moments and inid data. For iid data, Mammen (1993) proved consistency of the wild OLS bootstrap coefficient and homoskedasticity-based Wald test distributions with bounded expectations of the product of the fourth power of the regressors with the squared errors and an additional Lindeberg condition. Similarly, for the pairs OLS bootstrap with iid data Freedman (1981) showed that bounded fourth moments of both regressors and errors are sufficient for consistency of the pairs bootstrap coefficient distribution2 and that of the Wald statistic based upon the (potentially incorrect) assumption of homoskedastic errors. Stute (1990) tightened the result for the coefficient distribution alone, showing it is sufficient for the squared regressors and the product of the squared regressors with the squared errors to have finite expectation. This paper proves consistency of both the coefficient and clustered heteroskedasticity robust Wald statistic distribution in a broader inid environment for both the pairs and wild bootstraps with finite expectations of only slightly more than second powers of the regressors and of the product of the second powers of the regressors with the second power of the errors. These are much less demanding assumptions than those used by Freedman and Mammen, requiring only slightly higher moments than used by Stute for the proof of only the pairs bootstrap coefficient distribution in a narrower iid environment. Moreover, when residuals are heteroskedastic or interrelated within clusters, the homoskedasticity-based Wald test is not guaranteed to be asymptotically accurate, as recognized by Freedman (1981) and Mammen (1993). In such cases, practitioners are likely to prefer clustered/heteroskedasticity robust covariance estimates and Wald statistics as these are asymptotically accurate and pivotal, respectively, ensuring the asymptotic accuracy of the conventional test and higher order accuracy and faster convergence of rejection probabilities to the nominal value in the bootstrap (Singh, 1981; Hall, 1992).

For OLS models with inid data, the salient contribution is Liu (1988), who showed that the wild bootstrap provides consistent estimates of the second central moment of a linear combination of coefficients in an OLS regression model with bounded regressors, provided the first and second moments of the wild bootstrap external variable are 0 and 1, respectively. Liu’s result regarding the second central moment is easily extended to the case of the multivariate second central moments of coefficients for unbounded inid regressors without any additional restrictions on the moments of the external variable, as shown below. Our interest here, however, is in the full distribution of wild bootstrap coefficient and Wald statistic estimates, where our proof requires the existence of higher moments of the wild bootstrap external variable to ensure the convergence of higher moments to the normal. As the external variable is selected by the practitioner, and not an exogenous characteristic of the data, these additional moment conditions pose no obstacle. The two-point distribution proposed by Mammen (1993) and the Rademacher distribution, both often used in practical applications (e.g., Davidson & Flachaire, 2008), have moments of all orders.

Liu’s consideration of inid data has largely not been extended, as the OLS bootstrap literature has since focused on time series dependent data, where the absence of random sampling of independent observations raises different statistical issues and the use of different bootstrap methods (see the review in Hardle et al., 2003). Djogbenou et al. (2019), who prove consistency of the wild bootstrap t-statistic distribution for independently distributed cluster groupings of data, are a notable exception. With the moment assumptions used here, plus the additional requirement of bounded slightly higher than fourth moments of the regressors, their proof allows for heterogeneity in the distribution of data across clusters. However, they limit that heterogeneity in requiring that the cross product of the regressors and the covariance matrix of coefficient estimates converge to matrices of constants, a condition that in other papers is typically motivated by an iid assumption.3 The data-generating process examined in this paper is more fully inid in that there is no restriction that such matrices converge to anything, and the proof requires only slightly higher than second moments of the regressors. In sum, by emphasizing the permutation distribution, this paper lowers typical fourth moment restrictions on regressors to second moments, allows a fully inid data process in which average moments do not converge, and highlights the conceptual similarity between the wild and pairs bootstraps, proving results for both in a unified framework.

The paper proceeds as follows: Section 2 reviews the OLS model, White’s assumptions and results regarding OLS with inid data, and pairs and wild bootstrap methods for clustered/heteroskedastic data. Section 3 presents the foundational theorems regarding the asymptotic normality of permutation distributions that motivate the results. Section 4 then combines these with White’s (1980a) result to derive sufficient conditions for pairs and wild OLS bootstrap consistency with inid data and potentially cluster interdependent heteroskedastic residuals. Section 5 more fully contrasts the assumptions and results herein with those found in the papers cited above. Section 6 provides Monte Carlo evidence of the consistency of the bootstrap in a challenging environment with an inid data process where average moments do not converge, regressors have barely second moments, and residuals are bounded, varyingly skewed, sometimes bi-modal, and otherwise generally highly non-normal. The Appendix provides proofs of the main theorems, while the on-line Appendix extends the pair results to sub-sampling and provides lengthy technical proofs of otherwise minor lemmas and extensions of the theorems.

2. Framework and Notation

Our interest is in inference for the linear model where, with i = 1 … N observations,

$$\mathit{y}=\mathit{X}\mathit{\beta }+\mathit{\epsilon }$$

(1)

where $\mathit{y}$ represents the N × 1 matrix of observations on the dependent (outcome) variable, X the N × K matrix of observations of independent variables, $\mathit{\beta }$ the K × 1 vector of unobserved parameters of interest, and $\mathit{\epsilon }$ the N × 1 matrix of unobserved disturbances. The ordinary least squares (OLS) estimates $\widehat{\mathit{\beta }}$ of $\mathit{\beta }$ minimize the sum of squared estimated residuals ${\widehat{\mathit{\epsilon }}}^{\prime }\widehat{\mathit{\epsilon }}$, where $\widehat{\mathit{\epsilon }}=\mathit{y}-\mathit{X}\widehat{\mathit{\beta }},$ producing the estimates

$$\widehat{\mathit{\beta }}=({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}{\mathit{X}}^{\prime }\mathit{y}.$$

(2)

If the disturbances ε_i are homoskedastic with common variance ${\sigma }_i^2={\sigma }^2$, one can use the homoskedastic variance estimate of $\widehat{\mathit{\beta }}$, $({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}{\widehat{\mathit{\epsilon }}}^{\prime }\widehat{\mathit{\epsilon }}/(N-K)$, but we focus on more general inference where the ε_i are heteroskedastic and possibly interdependent within C ≤ N “cluster” groupings of observations, using the clustered/heteroskedasticity robust covariance estimate

$$\widehat{\mathit{V}}(\widehat{\mathit{\beta }})=({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g\right)({\mathit{X}}^{\prime }\mathit{X}{)}^{-1},$$

(3)

where we use the subscript $g$ to denote the rows of matrices and vectors associated with the observations in cluster grouping $g$. As will be seen later, we assume that the regressors and disturbances $({\mathit{X}}_g,{\mathit{\epsilon }}_g)$ are independent across cluster groupings. When observations themselves are independent, each grouping $g$ equals an individual observation i, C = N, and (3) is White’s (1980a) heteroskedasticity robust covariance estimate. The clustered extension, however, is often used to allow for unspecified grouped dependence, and so we present the results within a more general framework. In describing limits, we use the subscript C, as in ${\widehat{\mathit{\beta }}}_C$ and $\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)$, to emphasize that the estimated coefficients and covariance estimates are functions of C realized observation groupings.

White (1980a) provided conditions for valid OLS inference when the row vector of random variables associated with each observation is independently but not necessarily identically distributed (inid). With ${\mathit{x}}_{gj}$ denoting the jth column of ${\mathit{X}}_g$, we extend these to allow for grouped dependence:

Theorem I (extending White, 1980a). 

If there exist strictly positive finite constants γ, Δ, and η, such that

• (Ia) $({\mathit{X}}_g,{\mathit{\epsilon }}_g)$ is a sequence of independent but not necessarily identically distributed random matrices, such that $E({\mathit{X}}_g^{\prime }{\mathit{\epsilon }}_g)={\mathbf{0}}_K$;
• (Ib) For all $g$ $E(|{{\mathit{x}}_{gj}^{\prime }\mathit{x}}_{gk}{|}^{1+\gamma })<\Delta$ for all j, k = 1 … K, and for all C sufficiently large ${\mathit{M}}_C=C^{-1}{\sum }_{g=1}^{C}E({{\mathit{X}}_g^{\prime }\mathit{X}}_g)$ is non-singular with determinant (M_C) > η;
• (Ic) For all i, $E(|{\mathit{x}}_{gj}^{\prime }{\mathit{\epsilon }}_g{\mathit{\epsilon }}_g^{\prime }{\mathit{x}}_{gk}{|}^{1+\gamma })<\Delta$ for all j, k = 1 … K, and for all C sufficiently large ${\mathit{V}}_C=C^{-1}{\sum }_{g=1}^{C}E({\mathit{X}}_g^{\prime }{\mathit{\epsilon }}_g{\mathit{\epsilon }}_g^{\prime }{\mathit{X}}_g)$ is non-singular with determinant (V_C) > η;

then

(i)

${\widehat{\mathit{\beta }}}_C\stackrel{as\left(\mathit{X},\mathit{\epsilon }\right)}{\to }\mathit{\beta }$;

(ii)

${\mathbf{V}}_C^{-½}{\mathit{M}}_C\sqrt{C}\left({\widehat{\mathit{\beta }}}_C-\mathit{\beta }\right)\hspace{0.33em}\stackrel{d\left(\mathit{X},\mathit{\epsilon }\right)}{\to }\hspace{0.33em}{\mathit{n}}_K$;

(iii)

${\mathit{M}}_C,{\mathit{V}}_c, \mathit{and} \mathit{their} \mathit{inverses} \mathit{are} \mathit{uniformly} \mathit{bounded} \mathit{for} \mathit{all} C \mathit{sufficiently} \mathit{large}$

(iv)

$C\widehat{\mathbf{V}}\left({\widehat{\mathit{\beta }}}_C\right)-{\mathit{M}}_C^{-1}{\mathit{V}}_C{\mathit{M}}_c^{-1}\stackrel{as\left(\mathit{X},\mathit{\epsilon }\right)}{\to }\hspace{0.33em}{\mathbf{0}}_{KxK}$;

(v)

$({\widehat{\mathit{\beta }}}_C-\mathit{\beta }{)}^{\prime }\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C{)}^{-1}({\widehat{\mathit{\beta }}}_C-\mathit{\beta })\hspace{0.33em}\stackrel{d(\mathit{X},\mathit{\epsilon })}{\to }\hspace{0.33em}{\chi }_K^2$;

where  $\stackrel{as(\mathit{X},\mathit{\epsilon })}{\to }$  and  $\stackrel{d(\mathit{X},\mathit{\epsilon })}{\to }$ denote convergence almost surely and in distribution across (X,ε), respectively, A^½ the “square root” of symmetric positive definite matrix A,4  n_K the K dimensional standard normal,  ${\chi }_K^2$  the central chi-squared with K degrees of freedom, and  0_K and  0_KxK vectors and matrices of zeros of the indicated dimensions.

Remark 1. 

White’s covariance estimate often motivates inference with heteroskedasticity or clustering in an otherwise iid setting where each observation or cluster grouping is a draw from a fixed distribution. However,  $\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)$  allows for asymptotically accurate inference in the much more general inid setting given the above, where M_C, V_C, and  $C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)$  do not necessarily converge to matrices of constants, as illustrated in Monte Carlos further below.

Remark 2. 

White (1980a) used (Ia)–(Ic) to prove (i), (ii), and parts of (iii) and added the assumption $E(|{\mathit{x}}_{gj}^{\prime }{\mathit{x}}_{gj}{\mathit{x}}_{gk}^{\prime }{\mathit{x}}_{gl}{|}^{1+\gamma })$ to prove (iv), (v), and other results. As reviewed below, a similar fourth moment condition on the regressors is also used in prior proofs of bootstrap consistency. However, (Ia)–(Ic), with only slightly higher than second regressor moments, suffice to prove (i)–(v) and ensure bootstrap consistency, as shown in proofs and Monte Carlos below.

Remark 3. 

In practical application, moment restrictions on the data-generating process can be tested using techniques suggested by Meerschaert and Scheffler (1998), Fedotenkov (2013), and Trapani (2016), among others.

In this paper, we examine two bootstrap techniques commonly used for OLS inference with heteroskedastic and clustered disturbances and prove the asymptotic consistency of their distributions for general inid data. Wu’s (1986) external bootstrap, now commonly known as the wild bootstrap, holds the design matrix $\mathit{X}$ constant and generates new realizations of the outcome vector $\mathit{y}$ by multiplying the estimated residuals of each cluster grouping by an independently and identically distributed external random variable ${\delta }_g^w$, so that the dependent variable for grouping $g$ is now given by ${\mathit{y}}_g^w={\mathit{X}}_g{\widehat{\mathit{\beta }}}_C+{\widehat{\mathit{\epsilon }}}_g{\delta }_g^w$. Selecting ${\widehat{\mathit{\beta }}}_C^w$ so as to minimize the sum of squared residuals for this new data yields coefficient and covariance estimates expressed in terms of the original data and its estimates as

$$\begin{gathered} {\widehat{\mathit{\beta }}}_C^w={\widehat{\mathit{\beta }}}_C+({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^w\right) \\ \mathrm{and} \hspace{0.33em}\widehat{\mathbf{V}}({\widehat{\mathit{\beta }}}_C^w)=({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g^w{\widehat{\mathit{\epsilon }}}_g^{w\prime }{\mathit{X}}_g\right)({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}, \\ \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} {\widehat{\mathit{\epsilon }}}_g^w={\mathit{X}}_g\left({\widehat{\mathit{\beta }}}_C-{\widehat{\mathit{\beta }}}_C^w\right)+{\widehat{\mathit{\epsilon }}}_g{\delta }_g^w. \end{gathered}$$

(4)

Repeated draws of the Cx1 vector δ^w of iid variables are made and the resulting distribution of coefficients ${\widehat{\mathit{\beta }}}_C^w-{\widehat{\mathit{\beta }}}_C$ and Wald statistics $({\widehat{\mathit{\beta }}}_C^w-{\widehat{\mathit{\beta }}}_C{)}^{\prime }\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C^w{)}^{-1}({\widehat{\mathit{\beta }}}_C^w-{\widehat{\mathit{\beta }}}_C)$ used to evaluate the statistical significance of corresponding measures for tests of the null hypothesis $\mathit{\beta }={\mathit{\beta }}_0$ in the original sample, i.e., ${\widehat{\mathit{\beta }}}_C-{\mathit{\beta }}_0$ and $({\widehat{\mathit{\beta }}}_C-{\mathit{\beta }}_0{)}^{\prime }\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C{)}^{-1}({\widehat{\mathit{\beta }}}_C-{\mathit{\beta }}_0)$. All permutations of any given realization of ${\mathit{\delta }}^w$ are equally likely, a fact that plays a prominent role in the results of this paper.

The pairs bootstrap samples with replacement C cluster groupings of “pairs” of dependent and independent variables $({\mathit{y}}_g,{\mathit{X}}_g$) from the rows of the original data $(\mathit{y},\mathit{X})$, producing a new data set composed of h = 1 … C cluster groups of observations $({\mathit{y}}_h,{\mathit{X}}_h$), with each h corresponding to one of the original $g$ groupings.5 Selecting ${\widehat{\mathit{\beta }}}_C^p$ so as to minimize the sum of squared residuals for this new data, the resulting coefficient and covariance estimates can be expressed in terms of the original data, its estimates, and its indices $g$ = 1 … C as

$$\begin{gathered} {\widehat{\mathit{\beta }}}_C^p={\widehat{\mathit{\beta }}}_C+{\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p\right)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^p\right) \\ \mathrm{a}\mathrm{n}\mathrm{d} \widehat{\mathbf{V}}\left({\widehat{\mathit{\beta }}}_C^p\right)={\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p\right)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g^p{\widehat{\mathit{\epsilon }}}_g^{p\prime }{\mathit{X}}_g{\delta }_g^p\right){\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p\right)}^{-1}, \\ \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} {\widehat{\mathit{\epsilon }}}_g^p={\mathit{X}}_g\left({\widehat{\mathit{\beta }}}_C-{\widehat{\mathit{\beta }}}_C^p\right)+{\widehat{\mathit{\epsilon }}}_g, \end{gathered}$$

(5)

where ${\delta }_g^p$ denotes the number of times (possibly 0) cluster grouping $g$ was drawn. Repeated bootstrap samples are made and the resulting distribution of coefficients ${\widehat{\mathit{\beta }}}_C^p-{\widehat{\mathit{\beta }}}_C$ and Wald statistics $({\widehat{\mathit{\beta }}}_C^p-{\widehat{\mathit{\beta }}}_C{)}^{\prime }\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C^p{)}^{-1}({\widehat{\mathit{\beta }}}_C^p-{\widehat{\mathit{\beta }}}_C)$ once again used to evaluate the statistical significance of corresponding measures for tests of the null hypothesis $\mathit{\beta }={\mathit{\beta }}_0$ in the original sample. As in the case of the wild bootstrap, all permutations of any given realization of the Cx1 sampling frequency vector ${\mathit{\delta }}^p$ are equally likely. Consequently, we use the common notation δ, distinguished by superscripted p or w, for seemingly dissimilar objects because these operate identically in the theorems and proofs below.

Our interest is in deriving sufficient conditions for the conditional consistency of the bootstrap distributions in an inid framework. Specifically, we show that White’s (1980a) assumptions are sufficient to ensure that for the bootstrapped coefficient and clustered/heteroskedasticity robust covariance estimates, with b (both) denoting p (pairs) or w (wild),

$$\begin{gathered} {\left(\sum _{g=1}^C{\displaystyle \frac{{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g}{C}}\right)}^{-½}\left({\displaystyle \frac{{\mathit{X}}^{\prime }\mathit{X}}{C}}\right)\sqrt{C}({\widehat{\mathit{\beta }}}_C^b-{\widehat{\mathit{\beta }}}_C)\hspace{0.33em}\stackrel{d({\mathit{\delta }}^b)|as(\mathit{X},\mathit{\epsilon })}{\to }{\mathit{n}}_K \\ \mathrm{and} \sqrt{C}({\widehat{\mathit{\beta }}}_C^b-{\widehat{\mathit{\beta }}}_C{)}^{\prime }[C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C^b){]}^{-1}({\widehat{\mathit{\beta }}}_C^b-{\widehat{\mathit{\beta }}}_C)\sqrt{C}\hspace{0.33em}\stackrel{d({\mathit{\delta }}^b)|as(\mathit{X},\mathit{\epsilon })}{\to }{\chi }_K^2, \end{gathered}$$

(6)

where $\stackrel{d(\mathit{\delta })|as(\mathit{X},\mathit{\epsilon })}{\to }$ denotes convergence in distribution across δ almost surely across realizations of (X,ε). These results show that the asymptotic conditional distribution given the data (X,ε) of the bootstrap equals the asymptotic distribution of the OLS estimates across (X,ε), allowing for valid inference using the percentiles of bootstrapped coefficient estimates or Wald statistics.6

The key characteristic exploited in the proofs below is that any of the row permutations of the vectors δ are equally likely. Consequently, the distribution of the bootstraps can be thought of as the distribution across permutations of δ integrated across the ordered realizations of δ. Permutation theorems characterize this permutation distribution as asymptotically normal with covariance matrix $C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)$ provided (X,ε) and δ have certain moment properties. White’s (1980a) assumptions ensure that these properties hold almost surely for (X,ε), while the properties of the multinomial sampling frequencies δ^p and moment assumptions on the iid elements of δ^w ensure the requisite conditions on δ also hold almost surely. Consequently, almost surely conditional on the data (X,ε), the distributions of the bootstraps across the draws δ that determine their coefficient estimates and Wald statistics converge to the distribution of their OLS counterparts for the original sample (X,ε) across its data-generating process.

3. Foundational Permutation Theorems

The proofs in this paper rely on a theorem first proven by Wald and Wolfowitz (1944) and later refined by Noether (1949) and Hoeffding (1951) concerning the asymptotic distribution of root-C times the correlation of a permuted sequence with another sequence:

Theorem II: 

Let z’ = (z₁, …, z_C) and δ’ = (δ₁, …, δ_C) denote sequences of real numbers, not all equal, and d’ = (d₁, …, d_C) denote any of the C! equally likely permutations of δ. Then, as C → ∞ the distribution across the realizations of d of the random variable

$$\begin{gathered} v_C=\sum _{g=1}^C{\displaystyle \frac{\left[z_g-m\left(z_g\right)\right]\left[d_g-m\left(d_g\right)\right]}{s\left(z_g\right)s\left(d_g\right)C^{½}}}, \\ \left[where for h=z or d, m(h_g)=\sum _{g=1}^C{\displaystyle \frac{h_g}{C}} \& s(h_g{)}^2={\sum }_{g=1}^C{\displaystyle \frac{[h_g-m(h_g){]}^2}{C}}\right] \end{gathered}$$

(IIa)

converges to that of the standard normal if for all integer τ > 2

$$\underset{C\to \infty }{lim}{\displaystyle \frac{C^{\frac{\tau }{2}-1}{\sum }_{g=1}^C[z_g-m(z_g){]}^{\tau }{\sum }_{g=1}^C[{\delta }_g-m({\delta }_g){]}^{\tau }}{{\left({\sum }_{g=1}^C[z_g-m(z_g){]}^2\right)}^{\tau /2}{\left({\sum }_{g=1}^C[{\delta }_g-m({\delta }_g){]}^2\right)}^{\tau /2}}}=0.$$

(IIb)

The proof is based on showing that the moments of v_C converge to those of the standard normal. A simple multivariate extension, proven in the on-line Appendix, is

Theorem IIm:

Let $\mathit{O}={\mathit{I}}_{CxC}-{\mathbf{1}}_C{\mathbf{1}}_C^{\prime }/C$ denote the centering matrix,7 ${\mathit{Z}}^{\prime }=(z_1,\dots ,z_C)$ a sequence of K x 1 vectors such that ${\mathit{Z}}^{\prime }\mathit{O}\mathit{Z}$ is positive definite, δ’ = (δ₁, …, δ_C) a sequence of real numbers not all equal, and d’ = (d₁, …, d_C) any of the C! equally likely permutations of δ. Then, as C → ∞ the distribution across the realizations of d of the random variable

$${\mathit{v}}_C={\left({\displaystyle \frac{{\mathit{Z}}^{\prime }\mathit{O}\mathit{Z}}{C}}{\displaystyle \frac{{\mathit{d}}^{\prime }\mathit{O}\mathit{d}}{C}}\right)}^{-½}{\displaystyle \frac{({\mathit{Z}}^{\prime }\mathit{O}\mathit{d})}{\sqrt{C}}}$$

(IIc)

converges to that of the multivariate iid standard normal if (IIb) holds for each element in the vector sequence ${\mathit{z}}_g$.

Theorem II is easily extended to a probabilistic environment by noting the following result due to Ghosh (1950) that translates the almost-sure or in probability characteristics of an infinite number of moment conditions into similar statements regarding a distribution:

Theorem III: 

If all the moments of the cumulative distribution function F_C(x) converge almost surely (in probability) to those of F(x), which possesses a density function, and for which, with ${\nu }_{k+1}$ denoting the absolute moment of order k + 1,

$$\underset{k\to \infty }{lim}{\displaystyle \frac{{\alpha }^{k+2}{\nu }_{k+1}}{k+2!}}=0 \mathrm{for} \mathrm{any} \mathrm{given} \mathrm{value} \mathrm{of} \alpha ,$$

(IIIa)

then F_C(x) converges almost surely (in probability) to F(x).

Condition (IIIa) is of course true for the normal distribution. Hoeffding (1952) generalized the result by showing that condition (IIIa) is not even needed for convergence in probability at all points of continuity of any F(x) that is uniquely determined by its moments. By virtue of the Cramér–Wold device, Theorem III covers the multivariate case given in (IIc) above, as for all $\mathit{\lambda }$, such that ${\mathit{\lambda }}^{\prime }\mathit{\lambda }=1$, all moments of ${\mathit{\lambda }}^{\prime }{\mathit{v}}_C$ converge to those of the standard normal. In light of Theorem III, in applying Theorem II below, we use the notation $\stackrel{d(\mathit{d})|as(\mathit{\delta },\mathit{X},\mathit{\epsilon })}{\to }$, i.e., almost surely across the realizations of (δ,X,ε), the distribution of ${\mathit{v}}_C$ across permutations d of δ converges to the multivariate standard normal. Theorems II and III are used to characterize the asymptotic distribution of $\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{d}_g^b/C$, which appears in the expressions for the bootstrapped coefficient estimates in (4) and (5) above.

A less demanding form of Theorem II, proven in Appendix B below, provides a weaker condition under which the mean of products converges in probability across permutations to the product of means:

Theorem IV:

Let z’ = (z₁, …, z_C) and δ’ = (δ₁, …, δ_C) denote sequences of real numbers, possibly all equal, and d’ = (d₁, …, d_C) any of the C! equally likely permutations of δ. Then, as C → ∞, across permutations d of δ,

$$m\left(z_g{d}_g\right)-m\left(z_g\right)m\left({\delta }_g\right)=\sum _{g=1}^C{\displaystyle \frac{z_g{d}_g}{C}}-\hspace{0.33em}\sum _{g=1}^C{\displaystyle \frac{z_g}{C}}\sum _{g=1}^C{\displaystyle \frac{{\delta }_g}{C}} \stackrel{p}{\to }\hspace{0.33em}0,$$

(IVa)

if

$$\underset{C\to \infty }{lim}{\displaystyle \frac{\sum _{g=1}^C\frac{[z_g-m(z_g){]}^2}{C}\sum _{g=1}^C\frac{[{\delta }_g-m({\delta }_g){]}^2}{C}}{C}}=0.$$

(IVb)

Theorem IV is used in proofs to make statements regarding the convergence in probability of terms such as $\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g{{(d}_g^w)}^2/C$, $\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g^p{\widehat{\mathit{\epsilon }}}_g^{p\prime }{\mathit{X}}_g{d}_g^p/C$, and $\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{d}_g^p/C,$which appear in (4) and (5) above. As the satisfaction of (IVb) depends on the realized sample moments of (X,ε) and δ, we use the notation $\stackrel{p(\mathit{d})|as(\mathit{\delta },\mathit{X},\mathit{\epsilon })}{\to }$, i.e., almost surely across the realizations of (δ,X,ε) $m{(z}_g{d}_g)$ converges in probability across the permutations d of δ to $m{(z}_g)m({\delta }_g)$.

4. Results: Bootstrap Consistency with INID Data

The following result is proven in Appendix C, further below:

Theorem V: 

Assume that for the wild bootstrap $E\left[{\delta }_g^w\right]=0$, $E[({\delta }_g^w{)}^2]=1$ and $E[({\delta }_g^w{)}^{2(1+{\theta }_1)}]<\Delta$ for some finite $\Delta$ and θ₁ > 1/γ, with γ as given in Theorem I earlier. Assumptions (Ia)–(Ic) given in Theorem I in combination with the properties of δ are sufficient to ensure that across the permutations d of δ^b, for b = p (pairs) or w (wild),

$${\left(\sum _{g=1}^C{\displaystyle \frac{{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g}{C}}\right)}^{-½}\left({\displaystyle \frac{{\mathit{X}}^{\prime }\mathit{X}}{C}}\right){\left({\displaystyle \frac{{\mathit{\delta }}^b\prime \mathit{O}{\mathit{\delta }}^b}{C}}\right)}^{-½}\sqrt{C}\left({\widehat{\mathit{\beta }}}_C^b-{\widehat{\mathit{\beta }}}_C\right)\hspace{0.33em}\stackrel{d(\mathit{d})|as({\mathit{\delta }}^b,\mathit{X},\mathit{\epsilon })}{\to }\hspace{0.33em}{\mathit{n}}_K,$$

(Va)

$$C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C^b)-C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)\stackrel{p(\mathit{d})|as({\mathit{\delta }}^b,\mathit{X},\mathit{\epsilon })}{\to }{\mathbf{0}}_{KxK}\hspace{0.33em}.$$

(Vb)

Bounded higher moments of ${\delta }_g^w$ are needed to ensure that conditions (IIb) and (IVb) in Theorems II and IV are satisfied.

Let δ^* denote the ordered values of δ. Across permutations d of δ^* (Va) and (Vb) hold. These permutations, integrated across the distribution of δ^*, characterize the entire distribution of δ. Adding the result8

$${\displaystyle \frac{{\mathit{\delta }}^p\prime \mathit{O}{\mathit{\delta }}^p}{C}}\stackrel{p({\mathit{\delta }}^p)}{\to }1 \mathrm{and} {\displaystyle \frac{{\mathit{\delta }}^w\prime \mathit{O}{\mathit{\delta }}^w}{C}}\stackrel{as({\mathit{\delta }}^w)}{\to }1,$$

(7)

implies that

$${\left(\sum _{g=1}^C{\displaystyle \frac{{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g}{C}}\right)}^{-½}\left({\displaystyle \frac{{\mathit{X}}^{\prime }\mathit{X}}{C}}\right)\sqrt{C}({\widehat{\mathit{\beta }}}_C^b-{\widehat{\mathit{\beta }}}_C)\hspace{0.33em}\stackrel{d({\mathit{\delta }}^b)|as(\mathit{X},\mathit{\epsilon })}{\to }\hspace{0.33em}{\mathit{n}}_K.$$

(8a)

$$C\widehat{\mathbf{V}}({\widehat{\mathit{\beta }}}_C^b)-C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)\stackrel{p({\mathit{\delta }}^b)|as(\mathit{X},\mathit{\epsilon })}{\to }{\mathbf{0}}_{KxK},$$

(8b)

where the convergence in distribution in this case is across the bootstrap realizations of δ^b that determine the bootstrap coefficient and covariance estimates, as in (4) and (5) above. When combined with White’s (1980a) result in Theorem I regarding the asymptotic distribution of OLS coefficient and covariance estimates, this establishes that almost surely the conditional (on the data) distributions of the bootstrapped coefficients and Wald statistics converge to the unconditional distributions of their OLS regression counterparts.

5. Comparison of Bootstrap Consistency Results

This section contrasts the assumptions and results of this paper with other papers on bootstrap consistency. These usually assume independent observations, with moment conditions given at that level. To simplify the comparison of moment conditions, where possible I use the i = 1 … N notation, taking each cluster $\mathit{g}$ as composed of one observation and using the implied observational level assumptions in the theorems given above.

Table 1. Comparison of assumptions and results.

	Mammen (1993)	Freedman (1981)	Stute (1990)	Liu (1988)	Djogbenou et al. (2019)	This Paper
type of bootstrap	wild	pairs	pairs	wild	wild	both
type of data	iid	iid	iid	inid	inid	inid
bounded moments	${{\mathit{x}}_{\mathit{i}\mathit{j}}^4\mathit{\epsilon }}_{\mathit{i}}^2$ $\& {\mathit{x}}_{\mathit{i}\mathit{j}}^4$	${{\mathit{x}}_{\mathit{i}\mathit{j}}^{\mathit{a}}\mathit{\epsilon }}_{\mathit{i}}^{\mathit{b}}$ $\mathit{a}+\mathit{b}=4$	${{\mathit{x}}_{\mathit{i}\mathit{j}}^2\mathit{\epsilon }}_{\mathit{i}}^2$ $\& {\mathit{x}}_{\mathit{i}\mathit{j}}^2$	${\mathit{\epsilon }}_{\mathit{i}}^2$ & ${\mathit{a}\mathit{l}\mathit{l} \mathit{x}}_{\mathit{i}\mathit{j}}^{\mathit{n}}$	${({{\mathit{x}}_{\mathit{i}\mathit{j}}^2\mathit{\epsilon }}_{\mathit{i}}^2)}^{1+\mathit{\gamma }}$ $\& {({\mathit{x}}_{\mathit{i}\mathit{j}}^4)}^{1+\mathit{\gamma }}$	${({{\mathit{x}}_{\mathit{i}\mathit{j}}^2\mathit{\epsilon }}_{\mathit{i}}^2)}^{1+\mathit{\gamma }}$ $\& {({\mathit{x}}_{\mathit{i}\mathit{j}}^2)}^{1+\mathit{\gamma }}$
avg. moments converge	yes	yes	yes	no	yes	no
maximum cluster size					unbounded	bounded
distribution of coefficients … and Wald statistics	yes homo.	yes homo.	yes no	no no	yes cl/hetero.	yes cl/hetero.
sub-sampling M of N		$\mathit{M}\to \mathit{\infty }$	$\mathit{M}\to \mathit{\infty }$			$\mathit{M}/\mathit{N}\to 0$ $\underset{\inf }{\lim }{\displaystyle \frac{\mathit{M}}{{\mathit{N}}^{\mathit{\gamma }\mathit{*}}}}>0$
moments of coefficients				yes		yes (wild)

Notes: Wald statistics based upon homoskedastic (homo.) or clustered/heteroskedasticity robust (cl/hetero.) covariance estimates.

below summarizes key elements of the discussion that follows.

• Assumptions on regressors and errors

For an OLS model with iid data and potentially heteroskedastic residuals, Mammen (1993) showed that for a fixed number of regressors the wild bootstrap distributions of linear combinations of the coefficients and Wald statistics based upon the homoskedastic covariance estimate are in probability consistent, given ${\mathit{sup}}_{‖\mathit{c}‖=1}E[({\mathit{c}}^{\prime }{\mathit{x}}_i{)}^4(1+{\epsilon }_i^2)]$ < ∞ and the Lindeberg type condition $E[({\mathit{c}}^{\prime }{\mathit{x}}_i{)}^2{\epsilon }_i^{2}I[({\mathit{c}}^{\prime }{\mathit{x}}_i{)}^2{\epsilon }_i^2\ge \gamma N]\to 0$ for every fixed γ > 0. For the same model, Freedman (1981) proved almost-sure consistency of pairs bootstrap coefficients and homoskedastic-based Wald tests if the row vectors $({\mathit{x}}_i^{\prime },y_i)$ are independently and identically distributed and $E[(({\mathit{x}}_i^{\prime },y_i)({\mathit{x}}_i^{\prime },y_i{)}^{\prime }{)}^2]$ < ∞. Stute (1990) tightened part of the result, showing that almost-sure convergence of the pairs bootstrap coefficients alone for iid data only requires $E(x_{ij}{x}_{ik})$ and $E(x_{ij}{x}_{ik}{\epsilon }_i^2)$ to be finite. By adopting a permutation approach, this paper proves almost-sure consistency of both coefficients and Wald statistics based upon the clustered/heteroskedasticity robust covariance estimate with inid observations for both the pairs and wild bootstrap with the existence of only slightly higher moments than required by Stute (1990), i.e., $E|x_{ij}{x}_{ik}{|}^{1+\gamma }<\infty$ and $E|x_{ij}{x}_{ik}{\epsilon }_i^2{|}^{1+\gamma }<\infty$ for some γ > 0. It should be noted, however, that Mammen’s result was part of a broader framework that allowed for a growing number of regressors.

For inid data, Liu (1988) proved consistency in probability of the second central moment of the wild OLS bootstrap coefficient distribution with bounded regressors (with all moments) and finite second moments of ε_i. This paper proves almost-sure consistency of the wild bootstrap distribution for inid data with unbounded regressors using the additional moment conditions described above.

Djogbenou et al. (2019) prove consistency in probability of the distribution of the wild bootstrap t-statistic for within-cluster correlated but cross-cluster independent but not identically distributed data. Their assumptions on the existence of moments are those used in this paper, plus the addition of the fourth moment restriction $E|x_{ij}^4{|}^{1+\gamma }<\infty$ for some γ > 0. They also impose asymptotic homogeneity of the data-generating process in the form of assuming that ${\mathit{X}}^{\prime }\mathit{X}/N$ converges to a matrix of constants, while for any vector α, such that ${\mathit{\alpha }}^{\prime }\mathit{\alpha }=1$, there exists a finite scalar v_α > 0 and non-random sequence μ_N → ∞, such that ${\mu }_N{\mathit{\alpha }}^{\prime }({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\sum _{g=1}^{C}E({\mathit{X}}_g^{\prime }{\mathit{\epsilon }}_g{\mathit{\epsilon }}_g^{\prime }{\mathit{X}}_g)({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\mathit{\alpha }\to v_{\mathit{\alpha }}$. Thus, while papers usually use the iid assumption to motivate the convergence of key matrices to matrices of constants, Djogbenou et al. (2019) avoid the iid assumption but assume that the data nevertheless converge to such matrices. This paper, using clustered versions of White’s (1980a) assumptions, requires no such convergence of the asymptotic regressor cross product and covariance matrix of coefficient estimates and as such covers more fundamentally inid data without the addition of the fourth moment condition $E|x_{ij}^4{|}^{1+\gamma }<\infty$.

This paper makes no explicit assumptions regarding maximum cluster size, but in practice the assumption that the expectation of vector products of the regressors are uniformly bounded for all $g$, i.e., $E(|{{\mathit{x}}_{gj}^{\prime }\mathit{x}}_{gk}{|}^{1+\gamma })<\Delta$, implies that either the maximum cluster size is bounded or, as seems less likely, the expectation of individual observations shrinks with cluster size. In contrast, Djogbenou et al.’s (2019) proof of consistency allows the maximum cluster size to increase with the sample size in an unbounded fashion at a rate determined by the form of dependency (albeit unknown) within clusters. All proofs of consistency necessarily require that asymptotically individual observations or clusters exert a negligible influence on coefficient and variance estimates, although ironically it is often the strong influence of outlier observations or groupings in finite samples that makes conventional tests less accurate relative to the bootstrap (c.f. Davidson and Flachaire, 2008; Young, 2019, and the simulations below).

2.

Type of consistency proven

Aside from consistency of the coefficient distribution, Freedman (1981) and Mammen (1993) prove consistency of the Wald statistic for the pairs and wild bootstrap, respectively, based upon the covariance estimate with homoskedastic errors. Djogbenou et al. (2019) prove consistency of the Wald statistic using the cluster/heteroskedasticity robust covariance estimate, which is also asymptotically accurate with homoskedastic errors. This test statistic is asymptotically pivotal and hence provides higher-order asymptotic bootstrap accuracy (Singh, 1981; Hall, 1992). This paper does the same for both the pairs and wild bootstrap using weaker moment conditions and a unified permutation framework that highlights a similarity between the two methods.

Freedman and Stute allowed for sub-sampling M < N observations in the pairs bootstrap and proved convergence in distribution if M and N both go to infinity. As shown in the on-line Appendix, at the expense of complicating the proofs, the permutation-based pairs bootstrap consistency results can be extended to sub-sampling, with and without replacement, if M/N → 0 and for some γ^* > (1 + γ)⁻¹, M is such that lim_inf $M/N^{\gamma *}$ > 0. The requirement that M not fall too rapidly relative to N is needed to ensure the existence and convergence of higher moments to the normal, as the proof of Theorem II is based upon the method of moments.

Liu (1988) proves consistency of the wild bootstrap second central moment with bounded regressors. Proving such consistency with the unbounded regressors of this paper is trivial. If we assume, as did Liu (1988), that $E\left[{\mathit{\delta }}^w\right]={\mathbf{0}}_C$ and $E\left[{\mathit{\delta }}^w{\mathit{\delta }}^{\prime w}\right]={\mathit{I}}_{CxC}$ (the identity matrix), then taking the expectation with respect to this variable for a given realization of X and ε, we have

$$\begin{gathered} E[\hspace{0.33em}{\widehat{\mathit{\beta }}}_C^w|\mathit{X},\mathit{\epsilon }]={\widehat{\mathit{\beta }}}_C+({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_{g}E[{\delta }_g^w]={\widehat{\mathit{\beta }}}_C \left[\mathrm{a}\mathrm{s} \sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g={\mathbf{0}}_K\right] \\ E[({\widehat{\mathit{\beta }}}_C^w-E[{\widehat{\mathit{\beta }}}_C^w])({\widehat{\mathit{\beta }}}_C^w-E[{\widehat{\mathit{\beta }}}_C^w]{)}^{\prime }|\mathit{X},\mathit{\epsilon }]= \\ ({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\left(\sum _{g=1}^C\sum _{h=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_h^{\prime }{\mathit{X}}_{h}E[{\delta }_g^w{\delta }_h^w]\right)({\mathit{X}}^{\prime }\mathit{X}{)}^{-1} \\ =({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g\right)({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}=\mathit{V}\left({\widehat{\mathit{\beta }}}_C\right), \end{gathered}$$

(9)

where we make use of the fact that ${\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g=\mathit{X}}^{\prime }\widehat{\mathit{\epsilon }}={\mathbf{0}}_K$ as the OLS estimates ${\widehat{\mathit{\beta }}}_C$ in (2) above minimize ${\widehat{\mathit{\epsilon }}}_C^{\prime }{\widehat{\mathit{\epsilon }}}_C.$ Thus, for any sample size the variance of wild bootstrap coefficient estimates equals White’s clustered/heteroskedasticity robust covariance estimate for the sample. Since under White’s conditions given in Theorem I, $C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)$ is a consistent estimator of the asymptotic variance of $\sqrt{C}({\widehat{\mathit{\beta }}}_C-\mathit{\beta })$, it follows that for such general inid data the wild bootstrap coefficient variance is a consistent estimator as well, reproducing Liu’s result in a more general framework.

A similar result for the pairs bootstrap is more problematic. The first two moments of the multinomial sampling frequencies (${\mathit{\delta }}^p$) for C draws with replacement from C cluster groups are $E\left[{\mathit{\delta }}^p\right]={\mathbf{1}}_C$ (a vector of ones) and $E\left[{\mathit{\delta }}^p{\mathit{\delta }}^{\prime p}\right]={\mathit{I}}_{CxC}-C^{-1}{\mathbf{1}}_C{\mathbf{1}}_C^{\prime }.$ Examining the moments of the latter half of ${\widehat{\mathit{\beta }}}_C^p-{\widehat{\mathit{\beta }}}_C={\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p\right)}^{-1}\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^p\right)$, we see:

$$\begin{gathered} E\left[\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^p|\mathit{X},\mathit{\epsilon }\right]=\hspace{0.33em}\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_{g}E\left[{\delta }_g^p\right]=\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g={\mathbf{0}}_K, \\ \&\hspace{0.33em}\hspace{0.33em}E\left[\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^p\right){\left(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^p\right)}^{\prime }|\mathit{X},\mathit{\epsilon }\right]=\sum _{g=1}^C\sum _{h=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_h^{\prime }{\mathit{X}}_{h}E[{\delta }_g^p{\delta }_h^p] \\ =\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g-C^{-1}\sum _{g=1}^C\sum _{h=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_h^{\prime }{\mathit{X}}_h=\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g \\ =\left({\mathit{X}}^{\prime }\mathit{X}\right)\mathit{V}\left({\widehat{\mathit{\beta }}}_C\right)\left({\mathit{X}}^{\prime }\mathit{X}\right). \end{gathered}$$

(10)

Were $\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^p$multiplied by $({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}$, this would prove consistency of the second central moment of pairs bootstrap coefficients, but unfortunately it is multiplied by $(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p{)}^{-1}$. However, it is easy to show that $(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p{)}^{-1}$ converges in probability to $({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}$ (see Appendix C below). Using this fact, Shao and Tu (1995) prove consistency of the second central moment using the artifice of assuming that when the minimum eigenvalue of $(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p{)}^{-1}$ is less than ½ of the minimum eigenvalue of $({\mathit{X}}^{\prime }\mathit{X}{)}^{-1}$, an event whose probability converges to zero, ${\widehat{\mathit{\beta }}}_C^p$ is set equal to ${\widehat{\mathit{\beta }}}_C$.

It is well known that convergence in distribution does not imply convergence of moments, but the fact that the proof of Theorem II regarding the asymptotic permutation distribution of root-C correlation coefficients is based upon the method of moments (see Hoeffding, 1951 and the on-line Appendix of this paper) might lead to the erroneous conclusion that the results here imply consistency of all moments. They do not, as already implied by the discussion of the second moment of the pairs bootstrap. In Appendix C below, Theorem II is used to prove that across the equally likely permutations d of a given δ^b, for b (both) = p (pairs) or w (wild)

$${\left(\sum _{g=1}^C{\displaystyle \frac{{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g}{C}}\right)}^{-½}{\left({\displaystyle \frac{{\mathit{\delta }}^{\prime b}\mathit{O}{\mathit{\delta }}^b}{C}}\right)}^{-½}{\displaystyle \frac{\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\delta }_g^b}{\sqrt{C}}}\stackrel{d(\mathit{d})|as({\mathit{\delta }}^b,\mathit{X},\mathit{\epsilon })}{\to }{\mathit{n}}_K,$$

(11)

signifying, by the method of proof, that the moments across permutations d of δ of the left-hand side converge to those of the multivariate standard normal. Since this is true for all δ, such that ${\mathit{\delta }}^{\prime b}\mathit{O}{\mathit{\delta }}^b>0$, which almost surely holds (see (L2) in Appendix C), we can equally say that across the distribution of δ, the moments of (11) converge to those of the multivariate standard normal. For the wild bootstrap $\sqrt{C}({\widehat{\mathit{\beta }}}_C^w-{\widehat{\mathit{\beta }}}_C)$ consists of (11) multiplied by $({\mathit{X}}^{\prime }\mathit{X}/C{)}^{-1}({\mathit{\delta }}^{\prime w}\mathit{O}{\mathit{\delta }}^w/C{)}^{½}(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g/C{)}^{½}$, and as ${\mathit{\delta }}^{\prime w}\mathit{O}{\mathit{\delta }}^w/C$ $\stackrel{as({\mathit{\delta }}^w)}{\to }$ 1, we can say that all the moments of $\sqrt{C}({\widehat{\mathit{\beta }}}_C^w-{\widehat{\mathit{\beta }}}_C)$ converge to those of the multivariate normal with covariance matrix $C\widehat{\mathit{V}}({\widehat{\mathit{\beta }}}_C)$, although these need not be the asymptotic moments of the sample coefficients $\sqrt{C}({\widehat{\mathit{\beta }}}_C-\mathit{\beta })$. In the case of the pairs bootstrap, $\sqrt{C}({\widehat{\mathit{\beta }}}_C^p-{\widehat{\mathit{\beta }}}_C)$ equals (11) multiplied by $(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p/C{)}^{-1}({\mathit{\delta }}^{\prime b}\mathit{O}{\mathit{\delta }}^b/C{)}^{½}(\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\widehat{\mathit{\epsilon }}}_g{\widehat{\mathit{\epsilon }}}_g^{\prime }{\mathit{X}}_g/C{)}^{½}$, and as both $\sum _{g=1}^C{\mathit{X}}_g^{\prime }{\mathit{X}}_g{\delta }_g^p/C$ and ${\mathit{\delta }}^{\prime b}\mathit{O}{\mathit{\delta }}^b/C$ are only shown to converge in probability, nothing can be said about the asymptotic moments of $\sqrt{C}({\widehat{\mathit{\beta }}}_C^p-{\widehat{\mathit{\beta }}}_C)$ without the use of an artifice such as that of Shao and Tu (1995) mentioned above.

3.

Assumptions on the wild bootstrap external variable

Liu (1988) proves the consistency of the second central moment of the wild bootstrap coefficients, assuming that the first and second moments of the wild bootstrap external variable ${\delta }_i^w$ are 0 and 1, respectively.9 This paper extends the proof to consistency in distribution by additionally requiring that $E[({\delta }_i^w{)}^{2(1+{\theta }_1)}]<\infty$ for θ₁ > 1/γ, where γ > 0 is such that $E|x_{ij}{x}_{ik}{|}^{1+\gamma }<\infty$ and $E|x_{ij}{x}_{ik}{\epsilon }_i^2{|}^{1+\gamma }<\infty$. As the proof of Theorem II is based on the method of moments, depending upon the existence of higher moments for the regressors higher moments on ${\delta }_i^w$ are needed to ensure that all moments of (11) above exist and converge to the normal. Proofs of the consistency of wild bootstrap distributions typically assume that the external variable ${\delta }_i^w$ comes from a particular distribution, such as the Rademacher, with moments of all orders (e.g., Mammen, 1993; Canay et al., 2021). A notable exception is Djogbenou et al. (2019), where the proof of convergence in distribution merely requires that $|{\delta }_i^w{|}^{2+\lambda }<\infty$ for some λ > 0. The wild bootstrap external variable, however, is under the control of the practitioner (i.e., not a characteristic of the given data) and, at this time, there appear to be no known advantages to using an external variable without higher moments.

6. Monte Carlo Illustration with INID Data

To illustrate the properties and consistency of the bootstraps with fully inid data, I use a data-generating process that departs strongly from the independently and identically distributed ideal. To ensure average moments do not even begin to converge in finite samples, I model underlying distributional parameters as following a random walk across the data. To stress test the theorems above, I use regressors with heavy-tailed distributions that barely satisfy the specified moment conditions. Finally, to further hinder convergence to the normal, I choose an error distribution that departs strongly from the shape of that ideal.

To begin with unclustered data, for i = 1 … N independent observations, I assume that:

$$\begin{gathered} y_i={\epsilon }_i, {\epsilon }_i=B\left(\left|a_{\epsilon i}\right|,\left|b_{\epsilon i}\right|\right)-{\displaystyle \frac{\left|a_{\epsilon i}\right|}{\left|a_{\epsilon i}\right|+\left|b_{\epsilon i}\right|}}, x_i=t\left(2.01+B\left(\left|a_{xi}\right|,\left|b_{xi}\right|\right)\right) \\ a_{\epsilon i}=a_{\epsilon i-1}+\mathrm{U}\left[-.5,.5\right], { b}_{\epsilon i}=b_{\epsilon i-1}+\mathrm{U}\left[-.5,.5\right], { a}_{\epsilon 0}=b_{\epsilon 0}=\mathrm{U}\left[-.5,.5\right], \\ a_{xi}=a_{xi-1}+\mathrm{U}\left[-.5,.5\right], b_{xi}=b_{xi-1}+\mathrm{U}\left[-.5,.5\right],\& a_{x0}=b_{x0}=\mathrm{U}\left[-.5,.5\right], \end{gathered}$$

(12)

where $B\left(a,b\right)$ denotes an independent draw from the Beta distribution with parameters a and b (and expectation a/(a + b)), t(v) an independent draw from the t-distribution with v degrees of freedom, and $\mathrm{U}\left[-.5,.5\right]$ an independent draw from the uniform distribution on [−0.5, 0.5]. The random walks a and b (separate for ε and x) with their expanding variances ensure that the moments of the data do not meaningfully converge in simulation.10 These random walks can be thought of as an underlying population characteristic that develops, say, geographically or intertemporally. Observations, however, are drawn independently from these otherwise related distributions. Beta random variables are bounded on [0, 1] and, depending upon a and b, can be heavily skewed toward 0 or 1, symmetric unimodally around 0.5, or bimodally concentrated at both 0 and 1, to name just a few possibilities. This departs strongly from the symmetric unimodal normal distribution on the real line. Random variables drawn from a t-distribution only have finite moments up to their degrees of freedom. Thus, the regressors x_i only have moments between 2.01 and 3.01, approaching the limits of the assumptions in Theorem I.

For a clustered data-generating process, with dependence within clusters, I generate $g=1 \dots C$ clusters with cluster effects that follow (12) above (substituting $g$for i everywhere in those equations), and observation level data:

$$\begin{gathered} y_i={\epsilon }_i, {\epsilon }_i={\epsilon }_{g\left(i\right)}+B\left(\left|a_{\epsilon g(i)}\right|,\left|b_{\epsilon g(i)}\right|\right)-{\displaystyle \frac{\left|a_{\epsilon g(i)}\right|}{\left|a_{\epsilon g(i)}\right|+\left|b_{\epsilon g(i)}\right|}}, \\ \mathrm{and} x_i=x_{g\left(i\right)}+t(2.01+B\left(\left|a_{xg(i)}\right|,\left|b_{xg(i)}\right|\right)), \end{gathered}$$

(13)

where$g(i)$ denotes the cluster to which observation i belongs. Thus, each regressor and disturbance observation within a cluster is composed of a common cluster effect plus a similarly, but independently, distributed observation effect. The estimated regression model is:

$$y_i=\alpha +\beta x_i+{\epsilon }_i, \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} \alpha =\beta =0.$$

(14)

Table 2. Monte Carlo’s illustrating consistency (1000 data sets per data-generating process, 99 bootstrap draws per data set).

	Conventional						Pairs						Pairs						Wild						Wild
	Chi-Squared						Bootstrap-c						Bootstrap-t						Bootstrap-c						Bootstrap-t
	(A) empirical rejection rates at .01, .05 and .10 levels
	.01		.05		.10		.01		.05		.10		.01		.05		.10		.01		.05		.10		.01		.05		.10
observations	independent observations
10 100 1000 10,000 100,000 1,000,000	.108 .043 .022 .015 .016 .012		.200 .100 .072 .067 .063 .065		.272 .173 .137 .124 .127 .113		.003 .012 .008 .006 .017 .009		.038 .047 .051 .050 .059 .050		.098 .105 .108 .103 .123 .104		.020 .033 .018 .020 .017 .012		.069 .082 .067 .062 .060 .056		.126 .142 .125 .116 .126 .111		.203 .053 .021 .020 .011 .013		.268 .110 .075 .073 .070 .068		.308 .178 .141 .122 .124 .113		.084 .062 .030 .024 .017 .014		.146 .108 .076 .067 .068 .069		.205 .159 .135 .115 .121 .110
clusters	independent clusters of observations
10 100 1000 10,000 100,000 1,000,000	.096 .030 .015 .023 .015 .018		.169 .076 .062 .070 .058 .053		.227 .136 .116 .125 .110 .101		.022 .007 .005 .005 .012 .016		.073 .045 .045 .050 .048 .048		.126 .095 .094 .103 .104 .093		.023 .018 .012 .016 .014 .018		.081 .059 .061 .062 .060 .053		.139 .104 .109 .119 .109 .094		.149 .037 .018 .023 .014 .015		.208 .088 .060 .076 .055 .059		.245 .131 .118 .131 .116 .098		.083 .037 .020 .026 .019 .014		.127 .084 .053 .069 .057 .055		.171 .118 .108 .124 .109 .095
	(B) Kolmogorov–Smirnov test statistics and p-values (1,000,000 observations or clusters)
	obs.			clusters			obs.			clusters			obs.			clusters			obs.			clusters			obs.			clusters
statistic	.021			.024			.023			.028			.020			.027			.025			.021			.022			.022
p-value	.744			.616			.678			.413			.815			.437			.573			.747			.697			.720
(C) Correlation of p-values with conventional and p-values of said correlations (1,000,000 obs. or cl.)
							obs.			clusters			obs.			clusters			obs.			clusters			obs.			clusters
correlation							.9907			.9889			.9905			.9895			.9901			.9910			.9891			.9900
p-value1 p-value2							.466 .870			.001 .045			.287 .778			.027 .243			.092 .573			.849 .940			.000 .087			.162 .502

Notes: Author’s calculations using Stata version 18.0 code provided in on-line materials. Conventional test incorporates Stata’s HC1 correction of the covariance estimate, which substantially reduces the rejection rate in samples with 10 observations. Kolmogorov–Smirnov tests are of the null that each set of 1000 p-values is drawn from the uniform distribution, with the distribution under the null calculated using 100,000 draws of 1000 uniform random variables. p-values of correlations in panel (C) are the likelihood of a smaller correlation in 100,000 instances of using 99 independent draws from the chi-squared to evaluate each of the 1000 chi-squared statistics vs. using the chi-squared distribution itself. As explained in the text, p-value1 calculates the correlation distribution conditional on the realized conventional squared t-statistics, while p-value2 calculates the correlation distribution based on conventional test statistics which are random draws from the chi-squared.

reports Monte Carlo results for tests of the true null of β = 0 in the OLS regression (14) for the data-generating processes described in (12) and (13) with 10, 100, 1000, 10,000, 100,000, and 1,000,000 observations or clusters (and five observations per cluster). For the conventional test, I report the p-value of the two-sided test using the squared sample t-statistic based upon the clustered/heteroskedasticity robust covariance estimate evaluated using its asymptotic chi-squared distribution. For the bootstraps, I report p-values based upon the bootstrap-c, evaluating the squared coefficient deviation from the null using the percentiles of the squared coefficient deviations of the bootstraps from the mean of their data-generating processes, and the bootstrap-t, evaluating the sample squared t-statistic using the corresponding squared bootstrap test statistics, i.e.,

$$\begin{gathered} \mathrm{B}\mathrm{o}\mathrm{o}\mathrm{t}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{p}-\mathrm{c}:{\left(\widehat{\beta }-\beta \right)}^2 \mathrm{e}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{u}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g} {\left({\widehat{\beta }}^b-\widehat{\beta }\right)}^2, \\ \mathrm{B}\mathrm{o}\mathrm{o}\mathrm{t}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{p}-\mathrm{t}:{\displaystyle \frac{{\left(\widehat{\beta }-\beta \right)}^2}{\widehat{V}\left(\widehat{\beta }\right)}} \mathrm{e}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{u}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g} {\displaystyle \frac{{\left({\widehat{\beta }}^b-\widehat{\beta }\right)}^2}{\widehat{V}\left({\widehat{\beta }}^b\right)}}. \end{gathered}$$

(15)

99 draws are used for each bootstrap, and an exact test relative to the bootstrap distribution is achieved using a p-value given by (G + (T + 1) U[0,1])/100, where G and T are the number of bootstrap test statistics greater than and equal to, respectively, that of the sample and U[0,1] is a draw from the uniform distribution on [0.1].11 For the wild bootstrap, ${\delta }_c^w$ is drawn from the Rademacher distribution, which equals $\pm$1 with equal probability and appears to perform better than alternatives (Davidson & Flachaire, 2008). 1000 realizations of the data-generating process are used for each specification.

As can be seen in panel (A) of the table, rejection rates using both the conventional chi-squared distribution and those of the bootstraps differ substantially from nominal value in small samples, but converge to the 0.01, 0.05, and 0.10 levels as the number of observations or clusters increases. The central 95 percentiles of the binomially distributed Monte Carlo rejection probability in 1000 independent draws are 0.004 to 0.016 at the 0.01 level, 0.037 to 0.063 at the 0.05 level, and 0.082 to 0.118 at the 0.1 level. With 1,000,000 independent observations or clusters, most rejection rates lie within those bounds. As shown in panel (B) of the table, the Kolmogorov–Smirnov test statistic for the null that the p-value distributions are uniform, i.e., the maximum absolute difference between the cumulative distribution function of each set of 1000 p-values from that of the uniform distribution, is less than or equal to 0.028 in all cases, with the p-value of the null that the distributions are uniform on [0,1] exceeding 0.41 in each instance.

Theorem V asserts conditional consistency, i.e., the asymptotic distribution of the bootstraps is normal with a covariance matrix equal to that of the conventional estimate. If so, evaluating the conventional test statistic with the full distribution of each bootstrap should asymptotically yield a p-value identical to that found by evaluating the same using the chi-squared distribution.12 Panel (C) of Table 2 reports the correlation between the bootstrap and conventional p-values with 1,000,000 observations or clusters. As can be seen, this is at least 0.9889 in all cases. As the bootstrap p-values are based upon only a sample from their distribution, while exact relative to that distribution, they cannot be expected to equal the conventional chi-squared p-value. Their correlation with the conventional p-value, however, should be the same as that found when evaluating the conventional test statistic using 99 independent draws from the chi-squared distribution and the same formula p = (G + (T + 1) U[0,1])/100.

Panel (C) of Table 2 reports that the probability that a correlation less than or equal to that found between the bootstraps and the conventional p-value would be found when evaluating a squared t-statistic using 99 draws from the chi-squared vs. the full chi-squared distribution itself. “p-value1” evaluates the correlation using the distribution conditional on the realized conventional squared t-statistics, i.e., is a test of conditional consistency alone and does not assume consistency of the conventional test. While most p-values are very large, two of the eight are near zero, indicating that the correlation is not yet quite up to the level that would be expected from completed conditional convergence.13 “p-value2” evaluates the correlations using the distribution across random draws of the initial conventional test statistics from the chi-squared, i.e., a joint test of convergence of the conventional test statistic and conditional consistency of the bootstraps. Here the smallest p-value is 0.045, which, given any adjustment for multiple testing, can be taken as indicating that the tests do not reject the joint null implied by the theorems above at the 0.05 level.

Table 2 illustrates the consistency of bootstrap procedures with a highly challenging data-generating process. While previous results cited above require the existence of at least fourth moments of the regressors for convergence of both coefficients and Wald statistics in environments with iid data or inid data with convergent average moments, no more than slightly higher than second regressor moments are actually sufficient for fully inid data whose moments need not converge to anything, as specified in the theorems above and illustrated in these Monte Carlos.

As can also be seen in Table 2, in small samples with heavy-tailed regressors the conventional clustered/heteroskedasticity robust test statistic performs poorly, with rejection probabilities far above the nominal value. In such environments, the bootstraps often perform better (Davidson & Flachaire, 2008; Young, 2019). In the simulations of Table 2, this is clearly the case for the pairs bootstrap and, to a lesser degree, with the wild bootstrap using the asymptotically pivotal t-statistic.14 Thus, while providing the same asymptotic assurances as conventional inference methods, the bootstraps often provide a better approximation to the distribution of test statistics in small finite sample environments. It should be noted, however, that other methods also exist for improving the finite sample performance of the conventional test, such as the HC bias corrections of MacKinnon and White (1985) and the effective degrees of freedom corrections of Bell and McCaffrey (2002), Pustejovsky and Tipton (2018), and Young (2016).15

7. Conclusions

This paper characterizes the pairs and wild bootstraps as realizations of a permutation distribution and uses previously unexploited permutation theorems to derive less restrictive moment conditions for their conditional consistency. While prior work requires at least fourth moments of the regressors for consistency of distributions in an iid framework or inid framework where average moments converge to constants, only slightly more than second moments on the regressors are actually needed for consistency in a fully inid environment where average moments need never converge. The use of the same permutation theorems to characterize and derive new results for the asymptotic distributions of other techniques, such as bootstraps for time series, the jackknife, randomization inference, and conventional estimates on exchangeable data, is the subject of ongoing research.