Fast Cluster Bootstrap Methods for Spatial Error Models

Abstract

Typically, the traditional bootstrap methods for parameter inference of spatial error models suffer from high computational costs, so this study proposes fast cluster bootstrap methods for spatial error models to deal with the dilemma. The key idea is to calculate the sufficient statistics for each cluster before performing the bootstrap loop of the spatial error model, and based on these sufficient statistics, all quantities needed for bootstrap inference can be computed. Furthermore, this study performed Monte Carlo simulations, and the result reveals that compared with traditional bootstrap methods, our proposed methods can reduce the computational cost substantially and improve the reliability for obtaining the bootstrap test statistics and confidence intervals of the parameters for spatial error models.

1. Introduction

Traditionally, maximum likelihood estimation is typically used to perform parameter inference for spatial regression models. Lim et al. [1] proposed the maximum block independent likelihood estimator based on independent sub-blocks in spatial models. As for the spatial autoregressive model, Wang and Song [2] developed a quasi-maximum likelihood method with a penalty to achieve parameter estimation in the presence of missing responses. Al-Momani and Arashi [3] constructed the maximum likelihood estimator for the parameters of the spatial error model. However, the computational cost of parameter inference for spatial regression models based on maximum likelihood estimation is relatively high, especially for high-dimensional data. Furthermore, spatial dependencies generally increase the complexity of the computation.

Typically, the objects or observations collected from the same geographic region are more comparable than those collected from distant regions, which can be regarded as a cluster, and the objects or observations within the same cluster are more comparable to each other than to those in other clusters. Spatial regression models, incorporating various spatial dependencies, provide an opportunity to analyze the objects or observations collected from the same geographic location, known as a site [3], which is similar to the idea of clusters described above.

Moreover, inference based on cluster-robust variance estimators (CRVEs) performs well when large-sample theory serves as a good guide to the finite-sample properties of the CRVE. Nevertheless, when the number of clusters is small, the size of the clusters and the characteristics of the regressors and regressand are not fairly homogeneous, the CRVE becomes seriously unreliable, and there is a great deal of theoretical evidence and simulation results in the latest papers about this problem [4,5,6,7,8,9].

In fact, one of the alternative methods for spatial regression models is the bootstrap method, which involves generating a number of bootstrap samples mimicking the actual sample distribution, and the bootstrap test statistics and confidence intervals for each sample are acquired with the same test procedure as for the original sample [10,11,12,13,14]. There are two bootstrap methods based on clustered data, including pairs cluster bootstrap and the wild cluster bootstrap, which were originally proposed by Freedman [15], Liu [16], and Mammen [17], respectively, and further extended by Davidson and Flachaire [18] and Cameron et al. [19] and have been proved to be asymptotically valid [8,20]. Furthermore, these bootstrap methods are widely applied to various problems. Bouzebda et al. [21] extended the existing theory on the bootstrap of the M-estimators, and they proposed an exchangeably weighted bootstrap for function-valued estimators defined as a zero point of a function-valued random criterion function subsequently [22].

However, applying these traditional bootstrap methods to perform bootstrapping on linear regression models for clustered data still suffers from high computational costs. MacKinnon [23] has proposed an efficient computational algorithm for bootstrapping linear regression models based on clustered data. Inspired by MacKinnon [23], this study uses a similar idea to deal with the high computational costs of parameter inference for spatial regression models.

In addition, one of the most widely used spatial regression models is the spatial error model (SEM), which models the mean of the spatial response variable by using a linear regression with a spatially lagged autoregressive error component [24].

Considering the notable gap in the literature, this study proposes fast cluster bootstrap methods for SEM to reduce the computational cost and improve the reliability for obtaining the bootstrap test statistics and confidence intervals of SEM. Specifically, before performing the bootstrap loop of SEM, the sufficient statistics are computed for each of the clusters, and based on these sufficient statistics, it is possible to calculate all of the quantities needed for bootstrap inference. In addition, the test statistics and the bootstrap confidence intervals are associated with the samples solely by these sufficient statistics.

The key contributions of this study are presented as follows:

Firstly, this is the first study to propose fast cluster bootstrap methods to overcome the high computational cost of parameter inference for SEM. Secondly, this study simulates the computational cost, two-sided equal-tailed coverage frequencies, and the precision of the bootstrap confidence intervals of the parameter with various bootstrap methods, respectively. The result reveals that the computational cost of our proposed methods is substantially reduced, and the optimal coverage of the parameter is higher than the nominal frequency. Finally, compared with traditional methods, the methods proposed in this study are conceptually simple, easier to understand, and can also be used in other spatial models.

The remainder of this paper is organized as follows. In Section 2, we present an overview of the SEM. In Section 3, we describe bootstrap computations for maximum likelihood estimation of SEM, including the fast pairs cluster bootstrap method for SEM and the fast unrestricted and restricted wild cluster bootstrap methods for SEM, respectively. In Section 4, we perform simulation experiments and show that using these methods can yield substantial computational savings. In Section 5, we investigate the finite-sample performance of the two-sided equal-tailed coverage frequencies and average widths of the bootstrap confidence intervals of the parameter for SEM based on various bootstrap methods. In Section 6, we give the conclusions. Finally, in Section 7, we present the future work of this study. In addition, in order to improve clarity, we provide a summary of key notations in Table 1.

Table 1. Summary of key notations.

Notation	Description	Notation	Description
$\mathit{L}$	Number of clusters	${s\left(\gamma \right)}_l$	$\mathrm{Score} \mathrm{vector} \mathrm{of} \mathrm{the} l^{th}$ cluster
$\mathit{\gamma }$	Spatial dependence parameter	$\widehat{\mathit{s}}{\left(\mathit{\gamma }\right)}_{\mathit{l}}$	$\mathrm{MLE} \mathrm{of} {\mathit{s}\left(\mathit{\gamma }\right)}_{\mathit{l}}$
$\mathit{W}$	Spatial weight matrix	${{\widehat{\mathit{s}}\left(\mathit{\gamma }\right)}_{\mathit{l}}^{\mathit{*}}}^{\mathit{j}}$	Bootstrap $\mathrm{version} \mathrm{of} {\mathit{s}\left(\mathit{\gamma }\right)}_{\mathit{l}}$
$\mathit{n}$	Sample size	$\mathit{\beta }$	Vector of unknown regression parameters
$\mathit{B}$	Bootstrap replications	${\mathit{\beta }}_0$	True value of $\mathit{\beta }$
${\mathit{n}}_{\mathit{l}}$	Number of observations for the ${\mathit{l}}^{\mathit{t}\mathit{h}}$ cluster	$\widehat{\mathit{\beta }}$	$\mathrm{MLE} \mathrm{of} \mathit{\beta }$
${\mathit{X}}_{\mathit{l}}$	Matrix of exogenous regressors	$\tilde{\mathit{\beta }}$	$\mathrm{Restricted} \mathrm{MLE} \mathrm{of} \mathit{\beta }$
${\mathit{y}}_{\mathit{l}}$	$\mathrm{Vector} \mathrm{of} \mathrm{observations} \mathrm{collected} \mathrm{for} \mathrm{the} {\mathit{l}}^{\mathit{t}\mathit{h}}$ cluster	${\widehat{\mathit{\beta }}\left(\mathit{\gamma }\right)}^{\mathit{*}\mathit{j}}$	Bootstrap $\mathrm{version} \mathrm{of} \widehat{\mathit{\beta }}$
${\mathit{u}}_{\mathit{l}}$	$\mathrm{Vector} \mathrm{of} \mathrm{spatially} \mathrm{autocorrelated} \mathrm{remainder} \mathrm{error} \mathrm{for} \mathrm{the} {\mathit{l}}^{\mathit{t}\mathit{h}}$ cluster	${\mathit{t}}^{\mathit{*}\mathit{j}}$	Bootstrap t-test statistic
${\mathit{\epsilon }}_{\mathit{l}}$	$\mathrm{Vector} \mathrm{of} \mathrm{the} \mathrm{noise} \mathrm{for} \mathrm{the} {\mathit{l}}^{\mathit{t}\mathit{h}}$ cluster

2. Spatial Error Model

Let $\mathcal{L}=\{\mathrm{1,2},\dots \dots ,L\}$ represent a set of $L$ disjoint clusters, which are often referred to as locations, regions, etc. The $l^{th}$ cluster is assumed to have $n_l$ observations, and the sample size is $n=\sum _{l=1}^L{n}_l$. The SEM can be written as

$$y_l=X_l\beta +u_l$$

with

$$u_l=\gamma W{u}_l+{\epsilon }_l, l=\mathrm{1,2},\dots \dots ,L,$$

(1)

where $X_l$ is an $n_l\times r$ matrix of exogenous regressors, $\beta$ represents a $r\times 1$ vector of unknown regression parameters, $y_l$ is an $n_l\times 1$ vector of observations collected for the $l^{th}$ cluster, and $u_l$ denotes an $n_l\times 1$ vector of spatially autocorrelated remainder error. The error term involves the spatial dependence parameter $\gamma$, the $n_l\times n_l$ matrix of spatial weight matrix $W$, and an $n_l\times 1$ vector of the noise ${\epsilon }_l$.

Furthermore, we make the following assumptions:

Assumption 1. 

the noise ${\epsilon }_l$ has a normal distribution $N(0, {\sigma }^{2}I)$, where $I$ is the $n_l\times n_l$ identity matrix.

Assumption 2. 

the elements of the main diagonal of spatial weight matrix $W$ are zeros, and if position $j$ is adjacent to position $i$, the off-diagonal element $w_{ij}=1$; otherwise, $w_{ij}=0$ for $i\ne j$, and the normalization of $W$ is row normalized as $W=\{{\displaystyle \frac{w_{ij}}{\sum _{j=1}^{n_l}{w}_{ij}}}{\}}_{i=1}^{n_l}$.

Equivalently, under these assumptions, the SEM in (1) can be rewritten as follows,

$$\left(I-\gamma W\right)y_l=\left(I-\gamma W\right)X_l\beta +{\epsilon }_l, l=\mathrm{1,2},\dots \dots ,L.$$

(2)

Let ${y\left(\gamma \right)}_l=\left(I-\gamma W\right)y_l,{X\left(\gamma \right)}_l=\left(I-\gamma W\right)X_l$, Equation (2) is equivalent to

$${y\left(\gamma \right)}_l={X\left(\gamma \right)}_l\beta +{\epsilon }_l,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}l=\mathrm{1,2},\dots \dots ,L,$$

(3)

and it can be rewritten in matrix form, as follows,

$$y\left(\gamma \right)=X\left(\gamma \right)\beta +\epsilon .$$

(4)

Let $\mathsf{\Theta }=(\beta ,{\sigma }^2,\gamma )$. The maximum likelihood estimator (MLE) of $\mathsf{\Theta }$ can be obtained by maximizing the concentrated likelihood function [25]. Specifically, we first fix $\gamma$ and find the MLEs of $\beta and {\sigma }^2$ as a function of $\gamma$ as follows,

$$\widehat{\beta }\left(\gamma \right)={\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{X\left(\gamma \right)}^{T}y\left(\gamma \right)$$

(5)

$${\widehat{\sigma }}^2\left(\gamma \right)={\displaystyle \frac{1}{n}}{\left(y\left(\gamma \right)-X\left(\gamma \right)\widehat{\beta }\left(\gamma \right)\right)}^T\left(y\left(\gamma \right)-X\left(\gamma \right)\widehat{\beta }\left(\gamma \right)\right).$$

(6)

Secondly, we plug $\widehat{\beta }$ and ${\widehat{\sigma }}^2$ into the log-likelihood to obtain the MLE of $\gamma$. Finally, the MLEs for $\widehat{\beta }$ and ${\widehat{\sigma }}^2$ are acquired by substituting $\widehat{\gamma }$ for $\gamma$ in Equations (5) and (6), respectively.

Thus, when the data generation process (DGP) is a special case of (1), the following formular holds:

$$\widehat{\beta }\left(\gamma \right)-{\beta }_0={\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{\sum }_{l=1}^L{X\left(\gamma \right)}_l^T{\epsilon }_l={\left({\sum }_{l=1}^L{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\right)}^{-1}{\sum }_{l=1}^L{s\left(\gamma \right)}_l$$

(7)

where ${\beta }_0$ is the true value of $\beta$, and ${s\left(\gamma \right)}_l={X\left(\gamma \right)}_l^T{\epsilon }_l$ represents the $r\times 1$ score vector corresponding to the $l^{th}$ cluster. Assuming that

$$\begin{gathered} E\left({{{s\left(\gamma \right)}_{l}s\left(\gamma \right)}_l}^T\right)={\mathsf{\Sigma }}_l \\ E\left({{{s\left(\gamma \right)}_{l}s\left(\gamma \right)}_{l^{\prime }}}^T\right)=0\hspace{1em}\hspace{1em}l,l^{\prime }=\mathrm{1,2},\dots \dots ,L,l\ne l^{\prime }, \end{gathered}$$

(8)

where the expectations here are conditional on the ${X\left(\gamma \right)}_l$, the matrix ${\mathsf{\Sigma }}_l$ denotes a $r\times r$ positive semi-definite and symmetric matrix, which is the (conditional) covariance matrix of the score vector corresponding to the $l^{th}$ cluster. Based on Equations (7) and (8), we provide a more informative representation of the (conditional) covariance matrix of $\widehat{\beta }\left(\gamma \right)$, which makes it more explicit that the key to estimate is ${\mathsf{\Sigma }}_l$ in terms of the covariance matrix of the score vectors, and it follows

$$Var\left(\widehat{\beta }\left(\gamma \right)\right)={\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\left({\sum }_{l=1}^L{\mathsf{\Sigma }}_l\right){\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}.$$

(9)

When estimating ${\mathsf{\Sigma }}_l$, it is natural to apply the outer products of the empirical score vectors:

$$\widehat{s}{\left(\gamma \right)}_l={X\left(\gamma \right)}_l^T{\widehat{\epsilon }}_l={X\left(\gamma \right)}_l^T\left({y\left(\gamma \right)}_l-{X\left(\gamma \right)}_l\widehat{\beta }\left(\gamma \right)\right)={X\left(\gamma \right)}_l^T{y\left(\gamma \right)}_l-{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\widehat{\beta }\left(\gamma \right)$$

(10)

By correcting for degrees of freedom, we acquire the most widely used CRVE [24]:

$$\widehat{Var}\left(\widehat{\beta }\left(\gamma \right)\right)={\displaystyle \frac{L(n-1)}{(L-1)(n-r)}}{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\left({\sum }_{l=1}^L\widehat{s}{\left(\gamma \right)}_l{\widehat{s}{\left(\gamma \right)}_l}^T\right){\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}.$$

(11)

Note that given the matrix ${X\left(\gamma \right)}^{T}X\left(\gamma \right)$ and the vectors ${X\left(\gamma \right)}_l^T{y\left(\gamma \right)}_l$, $\widehat{Var}\left(\widehat{\beta }\left(\gamma \right)\right)$ can be acquired without computing the residual subvector ${\widehat{\epsilon }}_l$, which can reduce the computational costs. Since each of the $\widehat{s}{\left(\gamma \right)}_l{\widehat{s}{\left(\gamma \right)}_l}^T$ matrices has rank at most 1, then (11) has rank at most $L$ (it has rank $L-1$ in many situations), which indicates that asymptotic inference derived from the above (11) may not be reliable when $L$ is not large, and particularly when there are several restrictions. Consequently, the bootstrap methods are essential.

3. Bootstrap Computations for Maximum Likelihood Estimation of SEM

Bootstrapping is an alternative method for inferring statistical parameters where computational complexity arises [10]. In addition, different cluster bootstrap methods (to illustrate the asymptotic properties of the estimator proposed in our paper, we cite an existing theorem in Appendix A and briefly explain why this theorem is applicable to our estimator.) for SEM yield bootstrap samples in different ways, which incur different computational costs. In this part, we discuss the fast pairs cluster bootstrap method for SEM and the fast unrestricted and restricted wild cluster bootstrap methods for SEM, respectively.

3.1. The Fast Pairs Cluster Bootstrap Method for SEM

Bootstrapping is an alternative method for inferring statistical parameters where computational complexity arises [10]. In addition, different cluster bootstrap methods for SEM yield bootstrap samples in different ways, which incur different computational costs. In this part, we discuss the fast pairs cluster bootstrap method for SEM and the fast unrestricted and restricted wild cluster bootstrap methods for SEM, respectively.

The original pairs cluster bootstrap method for SEM is to group the data for each cluster into a $\left[{X\left(\gamma \right)}_l, {y\left(\gamma \right)}_l\right]$ pair, and then to resample the data from the original $L$ pairs with replacement, such that each bootstrap sample is composed of the original $L$ pairs with equal probability $1/L$. The big difference between the fast pairs cluster bootstrap method (pcb) and the original method is the first step. As for pcb, firstly, we group the data for each cluster into a $\left[{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l, {X\left(\gamma \right)}_l^T{y\left(\gamma \right)}_l\right]$ pair, and then we resample from the $L$ pairs. Consequently, each bootstrap sample is constructed by randomly choosing $L$ pairs with equal probability $1/L$, and the $j^{th} \left(j=\mathrm{1,2},\dots \dots B\right)$ bootstrap sample is represented as follows,

$$\left[{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}, {{X\left(\gamma \right)}_l^{*j}}^T{y\left(\gamma \right)}_l^{*j}\right], l=\mathrm{1,2},\dots \dots L.$$

(12)

Thus, the fast pairs cluster bootstrap estimates of $\beta$ and ${s\left(\gamma \right)}_l$ are given by the following, respectively:

$${\widehat{\beta }\left(\gamma \right)}^{*j}={\left({\sum }_{l=1}^L{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}\right)}^{-1}{\sum }_{l=1}^L{{X\left(\gamma \right)}_l^{*j}}^T{y\left(\gamma \right)}_l^{*j}$$

(13)

$${\widehat{s}{\left(\gamma \right)}_l}^{*j}={{X\left(\gamma \right)}_l^{*j}}^T{y\left(\gamma \right)}_l^{*j}-{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}{\widehat{\beta }\left(\gamma \right)}^{*j} .$$

(14)

Typically, $r$ and $L$ are small, and the computational cost in both (13) and (14) is $O\left(r^{2}L\right)$. Furthermore, the CRVE for ${\widehat{\beta }\left(\gamma \right)}^{*j}$ is

$$\begin{array}{c}{\widehat{Var}\left({\widehat{\beta }\left(\gamma \right)}^{*j}\right)}_{pcb}=\hfill \\ {\displaystyle \frac{L(n-1)}{(L-1)(n-r)}}{\left({\sum }_{l=1}^L{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}\right)}^{-1}\left({\sum }_{l=1}^L{\widehat{s}{\left(\gamma \right)}_l}^{*j}{\left({{\widehat{s}\left(\gamma \right)}_l^{*}}^j\right)}^T\right){\left({\sum }_{l=1}^L{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}\right)}^{-1},\hfill \end{array}$$

(15)

and the bootstrap test statistic for $R^T\beta =0$, where $R$ is a known $r$-vector:

$${t_{pcb}}^{*j}={\displaystyle \frac{R^T({\widehat{\beta }(\gamma )}^{*j}-\widehat{\beta }(\gamma ))}{{(R^T{\widehat{Var}\left({\widehat{\beta }(\gamma )}^{*j}\right)}_{pcb}R)}^{1/2}}} .$$

(16)

Let $c_{{\displaystyle \frac{\alpha }{2}}}^{*}$ and $c_{1-{\displaystyle \frac{\alpha }{2}}}^{*}$ represent the ${\displaystyle \frac{\alpha }{2}}$ and $1-{\displaystyle \frac{\alpha }{2}}$ quantiles of the ${t_{pcb}}^{*j}$, respectively, and based on these statistics, the bootstrap percentile-t confidence interval for $\beta$ is constructed as follows,

$$\left[\widehat{\beta }\left(\gamma \right)-s\left(\widehat{\beta }\left(\gamma \right)\right)c_{1-{\displaystyle \frac{\alpha }{2}}}^{*} , \widehat{\beta }\left(\gamma \right)-s\left(\widehat{\beta }\left(\gamma \right)\right)c_{{\displaystyle \frac{\alpha }{2}}}^{*} \right] ,$$

(17)

where $s\left(\widehat{\beta }\left(\gamma \right)\right)$ is the cluster-robust standard error of $\widehat{\beta }\left(\gamma \right)$.

In terms of the pcb, there are several limitations. Firstly, the pcb generally involves higher computational costs; specifically, the matrix $\sum _{l=1}^L{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}$ in the above calculation is different for each bootstrap sample and requires to be generated and inverted $B$ times. Secondly, a technical problem that is typically ignored is that it is possible for $\sum _{l=1}^L{{X\left(\gamma \right)}_l^{*j}}^T{X\left(\gamma \right)}_l^{*j}$ to be singular in a simulated bootstrap sample, in which case the least squares estimator ${\widehat{\beta }\left(\gamma \right)}^{*j}$ cannot be defined, especially when our problem involves treatment at the level of the cluster and few clusters are treated, and ${\widehat{\beta }\left(\gamma \right)}^{*j}$ obtained based on this method may be inaccurate [5,6,26,27]. Finally, since the null hypothesis is not imposed on bootstrap samples, many of the existing studies indicate that pcb may not yield satisfactory results, especially for samples with different sizes [19].

3.2. The Fast Wild Cluster Bootstrap Methods for SEM

The original wild cluster bootstrap methods use auxiliary random variables $v_l^{*}$ for each cluster, which are i.i.d. with zero mean and unit variance, and the most popular choice for ${ v}_l^{*}$ is the Rademacher random variables, which satisfy $P\left(v_l^{*}=1\right)={\displaystyle \frac{1}{2}}$ and $P\left(v_l^{*}=-1\right)={\displaystyle \frac{1}{2}}$ [8,16,18,28]. The bootstrap observations are then generated as ${{y\left(\gamma \right)}_l}^{*}={X\left(\gamma \right)}_l\widehat{\beta }\left(\gamma \right)+{{\epsilon }_l }^{*}$ with ${{\epsilon }_l }^{*}={\widehat{\epsilon }}_l{v}_l^{*} (l=\mathrm{1,2},\dots \dots ,L)$, where the regressors ${X\left(\gamma \right)}_l$ are held fixed at their sample values, $\widehat{\beta }\left(\gamma \right)$ denotes the sample estimator, and ${\widehat{\epsilon }}_l$ are the least squares residuals, which are also held fixed at their sample values. The big difference between fast wild cluster bootstrap methods (WCB) and the original method is the first step. As for WCB, the bootstrap disturbances affect the estimates only through scores with ${s\left(\gamma \right)}_l^{*}=\widehat{s}{\left(\gamma \right)}_l{v}_l^{*}$, and based on whether generating bootstrap samples depends on the restrictions to be tested, WCB is divided into the fast unrestricted and restricted wild cluster bootstrap methods.

3.2.1. The Fast Unrestricted Wild Cluster Bootstrap Method for SEM

The fast unrestricted wild cluster bootstrap method (UWC) generates bootstrap samples that do not depend on the restrictions to be tested; furthermore, it is based on different steps of the computation, which is divided into UWC1, UWC2, and UWC3.

The detailed steps for applying UWC1 to generate the $j^{th}$ bootstrap sample and statistics are described below:

Step 1: Generate ${v_l^{*}}^j$ from Rademacher distribution, and furthermore, generate the bootstrap score vectors as follows,

$${{s\left(\gamma \right)}_l^{*}}^j={v_l^{*}}^j\widehat{s}{\left(\gamma \right)}_l, l=\mathrm{1,2},\dots \dots L,$$

(18)

where $\widehat{s}{\left(\gamma \right)}_l={X\left(\gamma \right)}_l^T{y\left(\gamma \right)}_l-{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\widehat{\beta }\left(\gamma \right)$.

Step 2: Based on the bootstrap scores from (18), we can easily obtain

$${\widehat{\beta }\left(\gamma \right)}^{*j}-\widehat{\beta }\left(\gamma \right)={\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{\sum }_{l=1}^L{{s\left(\gamma \right)}_l^{*}}^j.$$

(19)

And Equation (10) is equivalent to

$$\widehat{s}{\left(\gamma \right)}_l={s\left(\gamma \right)}_l-{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\left(\widehat{\beta }\left(\gamma \right)-{\beta }_0\right) , l=\mathrm{1,2},\dots \dots L$$

(20)

Similarly, the $j^{th}$ empirical bootstrap score ${{\widehat{s}\left(\gamma \right)}_l^{*}}^j$ are given as follows,

$${{\widehat{s}\left(\gamma \right)}_l^{*}}^j={{s\left(\gamma \right)}_l^{*}}^j-{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\left({\widehat{\beta }\left(\gamma \right)}^{*j}-\widehat{\beta }\left(\gamma \right)\right), l=\mathrm{1,2},\dots \dots L.$$

(21)

Step 3: We can obtain the CRVE for ${\widehat{\beta }\left(\gamma \right)}^{*j}$ and the bootstrap test statistic for $R^T\beta =0$, respectively:

$${\widehat{Var}\left({\widehat{\beta }\left(\gamma \right)}^{*j}\right)}_{UWC1}={\displaystyle \frac{L(n-1)}{(L-1)(n-r)}}{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\left({\sum }_{l=1}^L{{\widehat{s}\left(\gamma \right)}_l^{*}}^j{\left({{\widehat{s}\left(\gamma \right)}_l^{*}}^j\right)}^T\right){\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}$$

(22)

$${t_{UWC1}}^{*j}={\displaystyle \frac{R^T({\widehat{\beta }(\gamma )}^{*j}-\widehat{\beta }(\gamma ))}{{(R^T{\widehat{Var}\left({\widehat{\beta }(\gamma )}^{*j}\right)}_{UWC1}R)}^{1/2}}} .$$

(23)

Similar to the bootstrap percentile-t confidence interval constructed by pcb in Section 3.1, we can acquire the bootstrap percentile-t confidence interval constructed by UWC1.

UWC1 involves some unnecessary work in order to obtain t-statistics and further construct confidence intervals, which can be improved by UWC2.

In fact, when we calculate the numerator of the t-statistic, it is possible to avoid calculating ${\widehat{\beta }\left(\gamma \right)}^{*j}$. To this end, before the bootstrap loop, form the matrices

$${G\left(\gamma \right)}_l={X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}, l=\mathrm{1,2},\dots \dots L.$$

(24)

generate the vector

$$v_j^{*}=\left({v_1^{*}}^j,\dots \dots ,{v_L^{*}}^j\right) ,$$

(25)

and compute the numerator of the t-statistic as follows,

$$R^T\left({\widehat{\beta }\left(\gamma \right)}^{*j}-\widehat{\beta }\left(\gamma \right)\right)\begin{array}{c}={\displaystyle \sum _{l=1}^L}{R}^T{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{{s\left(\gamma \right)}_l^{*}}^j={\displaystyle \sum _{l=1}^L}{R}^T{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\widehat{s}{\left(\gamma \right)}_l{v_l^{*}}^j \hfill \\ ={\displaystyle \sum _{l=1}^L}{f\left(\gamma \right)}_l{v_l^{*}}^j={f\left(\gamma \right)}^T{v}_j^{*} ,\hfill \end{array}$$

(26)

where

$${f\left(\gamma \right)}_l{\equiv R^T\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\widehat{s}{\left(\gamma \right)}_l$$

and

$$f\left(\gamma \right)\equiv \left({f\left(\gamma \right)}_1,\dots \dots ,{f\left(\gamma \right)}_L\right).$$

(27)

The detailed steps for applying UWC2 to generate the $j^{th}$ bootstrap sample and statistics are described below.

Step 1: Generate the bootstrap score vectors as follows,

$${{s\left(\gamma \right)}_l^{*}}^j={v_l^{*}}^j\widehat{s}{\left(\gamma \right)}_l, l=\mathrm{1,2},\dots \dots L,$$

(28)

where $\widehat{s}{\left(\gamma \right)}_l={X\left(\gamma \right)}_l^T{y\left(\gamma \right)}_l-{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\widehat{\beta }\left(\gamma \right)$.

Step 2: Based on Equation (24), we can obtain the $j^{th}$ empirical bootstrap scores ${{\widehat{s}\left(\gamma \right)}_l^{*}}^j$ as follows,

$${{\widehat{s}\left(\gamma \right)}_l^{*}}^j={{s\left(\gamma \right)}_l^{*}}^j-{G\left(\gamma \right)}_l{\sum }_{g=1}^L{{s\left(\gamma \right)}_g^{*}}^j, l=\mathrm{1,2},\dots \dots L.$$

(29)

Step 3: We can calculate the CRVE for ${\widehat{\beta }\left(\gamma \right)}^{*j}$ and the bootstrap t-statistic, respectively:

$$\begin{array}{c}{\widehat{Var}\left({\widehat{\beta }(\gamma )}^{*j}\right)}_{UWC2}\hfill \\ ={\displaystyle \frac{L(n-1)}{(L-1)(n-r)}}{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\left({\displaystyle \sum _{l=1}^L}{{\widehat{s}\left(\gamma \right)}_l^{*}}^j{\left({{\widehat{s}\left(\gamma \right)}_l^{*}}^j\right)}^T\right){\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\hfill \end{array}$$

(30)

$${t_{UWC2}}^{*j}={\displaystyle \frac{R^T({\widehat{\beta }(\gamma )}^{*j}-\widehat{\beta }(\gamma ))}{{(R^T{\widehat{Var}\left({\widehat{\beta }(\gamma )}^{*j}\right)}_{UWC2}R)}^{1/2}}} .$$

(31)

In fact, in terms of calculating the denominator of the bootstrap t-statistic, a trick proposed by Roodman et al. [29] can further reduce the computational cost, and based on this trick, a method called UWC3 is derived. Specifically, before the bootstrap loop, in addition to (24), (25), (26), and (27), we can define the matrix $C\left(\gamma \right)$ with the typical element:

$${ C\left(\gamma \right)}_{lg}=R^T{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\widehat{s}{\left(\gamma \right)}_g.$$

(32)

The detailed steps for applying UWC3 to generate the $j^{th}$ bootstrap sample and statistics are described below.

Step 1: For each bootstrap replication, based on (25) and (27), we can calculate the ${q_l^{*}}^j$ instead of ${{\widehat{s}\left(\gamma \right)}_l^{*}}^j$ in the denominator of the bootstrap t-statistic for UWC2 as follows,

$${q_l^{*}}^j={f\left(\gamma \right)}_l{v_l^{*}}^j-{\sum }_{g=1}^L{C\left(\gamma \right)}_{lg}{v_g^{*}}^j,\hspace{1em}\hspace{1em}\hspace{1em}l=\mathrm{1,2},\dots \dots L.$$

(33)

Step 2: We can obtain the denominator of the bootstrap t-statistic and the bootstrap t-statistic, respectively, as follows,

$$R^T\widehat{Var}\left({\widehat{\beta }(\gamma )}^{*j}\right)R={\displaystyle \frac{L(n-1)}{(L-1)(n-r)}}\sum _{l=1}^L{\left({q_l^{*}}^j\right)}^2$$

(34)

$${t_{UWC3}}^{*j}={\displaystyle \frac{R^T({\widehat{\beta }(\gamma )}^{*j}-\widehat{\beta }(\gamma ))}{{(R^T\widehat{Var}\left({\widehat{\beta }(\gamma )}^{*j}\right)R)}^{1/2}}} .$$

(35)

Furthermore, we can obtain the bootstrap percentile-t confidence interval constructed by UWC2 and UWC3, respectively, which are similar to that of UWC1.

Obviously, by applying Equation (18), UWCs have two advantages over pcb; firstly, the computational cost of applying Equation (18) is $O \left(rL\right)$, rather than $O\left(\right(r+ 1) n)$ that would be required by the traditional methods [30]. Secondly, Equation (18) can preserve the covariance matrix of the scores evidently; specifically, ${\widehat{\beta }\left(\gamma \right)}^{*j}$ is determined by the bootstrap scores, while the covariance matrix is determined by the empirical bootstrap scores, which captures the key feature of WCB.

Moreover, when calculating the denominator of the bootstrap t-statistic, the computational cost required to form ${{\widehat{s}\left(\gamma \right)}_l^{*}}^j$ in UWC1 or UWC2 is $O \left(r^{2}L\right)$. Meanwhile, when the initial work of UWC3 is completed, the effort required for each bootstrap sample is $O \left(L^2\right)$ and does not rely on $n$ or $r$. Therefore, in general, UWC3 is less computationally expensive than UWC2, except for very large $L$.

3.2.2. The Fast Restricted Wild Cluster Bootstrap Method for SEM

The difference between the restricted wild cluster bootstrap method (RWC) and UWC is that the former uses the restricted empirical score vectors ${\tilde{s}\left(\gamma \right)}_l$ instead of unrestricted ones $\widehat{s}{\left(\gamma \right)}_l$.

Let us partition $\beta$ as [${\beta }_1, {\beta }_2]$, where ${\beta }_1$ is a $(r -1)$-vector, and ${\beta }_2$ is a scalar, and consider a restriction of the form ${\beta }_2=0$. When ${X\left(\gamma \right)}_l$ is partitioned conformably with $\beta$, the SEM can be rewritten as

$${y\left(\gamma \right)}_l={X\left(\gamma \right)}_{1l}{\beta }_1+{\beta }_2{x\left(\gamma \right)}_{2l}+{\epsilon }_l , l=\mathrm{1,2},\dots \dots L,$$

(36)

where ${X\left(\gamma \right)}_{1l}$ represents the $n_l\times (r-1)$ matrix, and ${x\left(\gamma \right)}_{2l}$ denotes the $n_l$-vector, with ${X\left(\gamma \right)}_l=[{X\left(\gamma \right)}_{1l},{x\left(\gamma \right)}_{2l}]$. It can also be rewritten in matrix form as follows,

$$y\left(\gamma \right)={X\left(\gamma \right)}_1{\beta }_1+{{\beta }_{2}x\left(\gamma \right)}_2+\epsilon ,$$

and the restrict MLEs of $\beta$ is given by

$$\tilde{\beta }\left(\gamma \right)=\left[{\displaystyle \genfrac{}{}{0pt}{}{{\left({{X\left(\gamma \right)}_1}^T{X\left(\gamma \right)}_1\right)}^{-1}{{X\left(\gamma \right)}_1}^{T}y\left(\gamma \right) }{0}}\right] ,$$

where ${{X\left(\gamma \right)}_1}^T{X\left(\gamma \right)}_1$ consists of the $(r -1) \times (r-1)$ upper-left matrix block of ${X\left(\gamma \right)}^{T}X\left(\gamma \right)$, and ${{X\left(\gamma \right)}_1}^{T}y\left(\gamma \right)$ consists of the first $(r-1)$ elements of ${X\left(\gamma \right)}^{T}y\left(\gamma \right)$.

Before the bootstrap loop, form the matrix $\tilde{C}\left(\gamma \right)$ with the typical element,

$${\tilde{C}\left(\gamma \right)}_{lg}=R^T{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}\tilde{s}{\left(\gamma \right)}_g ,$$

(37)

generate the vector

$${\tilde{v}}_j^{*}\equiv \left({{\tilde{v}}_1^{*}}^j,\dots \dots ,{{\tilde{v}}_L^{*}}^j\right) ,$$

(38)

and compute the numerator of the t-statistic as follows,

$$R^T\left({\tilde{\beta }\left(\gamma \right)}^{*j}-\tilde{\beta }\left(\gamma \right)\right)=\sum _{l=1}^L{R}^T{\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{\tilde{s}\left(\gamma \right)}_l{{\tilde{v}}_l^{*}}^j=\sum _{l=1}^L{\tilde{f}\left(\gamma \right)}_l{{\tilde{v}}_l^{*}}^j={\tilde{f}\left(\gamma \right)}^T{\tilde{v}}_j^{*} ,$$

(39)

where

$$\begin{gathered} {\tilde{f}\left(\gamma \right)}_l{\equiv R^T\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{\tilde{s}\left(\gamma \right)}_l, \\ \tilde{f}\left(\gamma \right)\equiv \left({\tilde{f}\left(\gamma \right)}_1,\dots \dots ,{\tilde{f}\left(\gamma \right)}_L\right) , \end{gathered}$$

(40)

and ${\tilde{s}\left(\gamma \right)}_l={X\left(\gamma \right)}_l^T{y\left(\gamma \right)}_l-{X\left(\gamma \right)}_l^T{X\left(\gamma \right)}_l\tilde{\beta }\left(\gamma \right)$.

The detailed steps for applying RWC to generate the $j^{th}$ bootstrap sample and statistics are described below.

Step 1: For each bootstrap replication, we can calculate the ${{\tilde{q}}_l^{*}}^j$ in the denominator of the bootstrap t-statistic as follows,

$${{\tilde{q}}_l^{*}}^j={\tilde{f}\left(\gamma \right)}_l{{\tilde{v}}_l^{*}}^j-\sum _{g=1}^L{\tilde{C}\left(\gamma \right)}_{lg}{{\tilde{v}}_g^{*}}^j, l=\mathrm{1,2},\dots \dots L.$$

(41)

Step 2: We can obtain the denominator of the bootstrap t-statistic and the bootstrap t-statistic, respectively:

$$R^T\widehat{Var}\left({\tilde{\beta }\left(\gamma \right)}^{*j}\right)R={\displaystyle \frac{L(n-1)}{(L-1)(n-r)}}{\sum }_{l=1}^L{\left({{\tilde{q}}_l^{*}}^j\right)}^2$$

(42)

$${t_{RWC}}^{*j}={\displaystyle \frac{R^T\left({\tilde{\beta }\left(\gamma \right)}^{*j}-\tilde{\beta }\left(\gamma \right)\right)}{{(R^T\widehat{Var}\left({\tilde{\beta }(\gamma )}^{*j}\right)R)}^{1/2}}} .$$

(43)

However, the bootstrap percentile-t confidence interval constructed by RWC differs from the one by UWC. Typically, inverting a RWC bootstrap test is a good way to obtain a confidence interval for one of the parameters in a linear model. In specific, the confidence interval of the parameter is obtained by performing two RWC bootstrap tests: one for the upper limit of the interval, and the other for the lower limit of the interval [30,31]. For example, if we intend to form a bootstrap confidence interval for the parameter ${\beta }_2$ based on RWC, it is necessary to acquire the upper and lower limits of the interval through an iterative process. Specifically, let ${\beta \left(\gamma \right)}_2^0$ represent one of the limits of the confidence interval, the upper limit ${\beta \left(\gamma \right)}_2^u$ or the lower limit ${\beta \left(\gamma \right)}_2^l$, and in such a case, we could apply the equal-tail bootstrap p-value as follows,

$${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^0\right)=2min\left({\displaystyle \frac{1}{B}}\sum _{j=1}^B\mathbb{I}\left({t_2^{*}}^j>t_2\right),{\displaystyle \frac{1}{B}}\sum _{j=1}^B\mathbb{I}\left({t_2^{*}}^j\le t_2\right)\right),$$

where

$$t_2={\displaystyle \frac{{\tilde{\beta }\left(\gamma \right)}_2-{\beta \left(\gamma \right)}_2^0}{{\left(\widehat{Var}\left({\tilde{\beta }\left(\gamma \right)}_2\right)\right)}^{1/2}}},\hspace{1em}\hspace{1em}{t_2^{*}}^j={\displaystyle \frac{{{\tilde{\beta }\left(\gamma \right)}_2^{*}}^j-{\beta \left(\gamma \right)}_2^0}{{\left(\widehat{Var}\left({{\tilde{\beta }\left(\gamma \right)}_2^{*}}^j\right)\right)}^{1/2}}},\hspace{1em}\hspace{1em}j=1,\dots \dots ,B,$$

(44)

and $\mathbb{I}(·)$ is an indicator function that takes 1 if its argument is true and 0 otherwise.

Let $\alpha$ denotes the significance level. Obviously, the lower limit ${\beta \left(\gamma \right)}_2^l$ is virtually certain to be less than ${\tilde{\beta }\left(\gamma \right)}_2$; thus, ${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^l\right)$ is less than $\alpha$ for ${\beta \left(\gamma \right)}_2^l$ $>{\beta }_2$ and greater than $\alpha$ for ${\beta \left(\gamma \right)}_2^l$ $<$ ${\beta }_2$. Similarly, it is certain that ${\beta \left(\gamma \right)}_2^u$, which is greater than ${\tilde{\beta }\left(\gamma \right)}_2$, also satisfies these two inequalities.

During the iterative process, to determine the bootstrap confidence interval of ${\beta }_2$ based on RWC, ${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^0\right)$ is needed to be evaluated several times with the same set of realized values of the auxiliary random variables. Specifically, before generating the bootstrap samples and statistics for any value ${\beta \left(\gamma \right)}_2^0$, we need to form the vector $\tilde{f}\left(\gamma \right)$ and the matrix $\tilde{C}\left(\gamma \right)$ for testing ${{\beta }_2=\beta \left(\gamma \right)}_2^0$, respectively, which both depend on ${\beta \left(\gamma \right)}_2^0$. Let $h\left(\gamma \right)={\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}R$ and $h^{\prime }\left(\gamma \right)={\left({X\left(\gamma \right)}^{T}X\left(\gamma \right)\right)}^{-1}{{X\left(\gamma \right)}_l}^T{X\left(\gamma \right)}_{l}h\left(\gamma \right)$, where $R$ is a vector with 1 for the $r^{th}$ element and 0 for other elements, then ${\tilde{f}\left(\gamma \right)}_l$ and ${\tilde{C}\left(\gamma \right)}_{lg}$ can be rewritten as

$$\begin{gathered} {\tilde{f}\left(\gamma \right)}_l\left\{{\beta \left(\gamma \right)}_2^0\right\}={{y\left(\gamma \right)}_l}^T{X\left(\gamma \right)}_{l}h\left(\gamma \right)-{y\left(\gamma \right)}^T{X\left(\gamma \right)}_1{\left({{X\left(\gamma \right)}_1}^T{X\left(\gamma \right)}_1\right)}^{-1}{{X\left(\gamma \right)}_{1l}}^T{X\left(\gamma \right)}_{l}h\left(\gamma \right)- \\ {\beta \left(\gamma \right)}_2^0\left({{x\left(\gamma \right)}_{2l}}^T{X\left(\gamma \right)}_{l}h\left(\gamma \right)-{{x\left(\gamma \right)}_2}^T{X\left(\gamma \right)}_1{\left({{X\left(\gamma \right)}_1}^T{X\left(\gamma \right)}_1\right)}^{-1}{{X\left(\gamma \right)}_{1l}}^T{X\left(\gamma \right)}_{l}h\left(\gamma \right)\right) \end{gathered}$$

(45)

and

$${\tilde{C}\left(\gamma \right)}_{lg}\left\{{\beta \left(\gamma \right)}_2^0\right\}=\begin{array}{c}{{y\left(\gamma \right)}_l}^T{X\left(\gamma \right)}_l{h}^{\prime }\left(\gamma \right)\hfill \\ -{y\left(\gamma \right)}^T{X\left(\gamma \right)}_1{\left({{X\left(\gamma \right)}_1}^T{X\left(\gamma \right)}_1\right)}^{-1}{{X\left(\gamma \right)}_{1l}}^T{X\left(\gamma \right)}_l{h}^{\prime }\left(\gamma \right)\hfill \\ -{\beta \left(\gamma \right)}_2^0({{x\left(\gamma \right)}_{2l}}^T{X\left(\gamma \right)}_l{h}^{\prime }\left(\gamma \right)\hfill \\ -{{x\left(\gamma \right)}_2}^T{X\left(\gamma \right)}_1{\left({{X\left(\gamma \right)}_1}^T{X\left(\gamma \right)}_1\right)}^{-1}{{X\left(\gamma \right)}_{1l}}^T{X\left(\gamma \right)}_l{h}^{\prime }\left(\gamma \right)) .\hfill \end{array}$$

(46)

Moreover, for each bootstrap sample, the bootstrap t-statistic for testing ${{\beta }_2=\beta \left(\gamma \right)}_2^0$ is available based on (39), (41), and (42), and it involves less computational cost. However, since ${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^0\right)$ is not a continuous function of its parameter, it is not possible to obtain the optimal value using the methods that rely on derivatives. Based on the properties discussed above, the bisection method is a reliable, easy to implement, and guaranteed convergence method to determine the real roots of such problems, and the steps for applying the bisection method to determine the limits of the confidence interval are as follows,

Step1: Define $f\left({\beta \left(\gamma \right)}_2^0\right)={\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^0\right)-\alpha ,$ choose initial values of ${\beta }_2$, including ${\beta \left(\gamma \right)}_2^a$ and ${\beta \left(\gamma \right)}_2^b$, such that $f\left({\beta \left(\gamma \right)}_2^b\right)f\left({\beta \left(\gamma \right)}_2^a\right)<0$, and tolerance rate $e$.

Step 2: If $f\left({\beta \left(\gamma \right)}_2^b\right)f\left({\beta \left(\gamma \right)}_2^a\right) \ge 0$, then the root does not lie in this interval.

Step 3: Find the midpoint, set ${\beta \left(\gamma \right)}_2^m= ({\beta \left(\gamma \right)}_2^b+{\beta \left(\gamma \right)}_2^a)/2$,

(i) if the function value of the midpoint $f\left({\beta \left(\gamma \right)}_2^m\right) = 0$, then ${\beta \left(\gamma \right)}_2^m$ is the root. Go to Step 5.

(ii) if $f({\beta \left(\gamma \right)}_2^b$) $f$(${\beta \left(\gamma \right)}_2^m$) $< 0$, the root lies between ${\beta \left(\gamma \right)}_2^b$ and ${\beta \left(\gamma \right)}_2^m$, then set ${\beta \left(\gamma \right)}_2^b$ $={\beta \left(\gamma \right)}_2^b$, ${\beta \left(\gamma \right)}_2^a=$ ${\beta \left(\gamma \right)}_2^m$.

(iii) or else set ${\beta \left(\gamma \right)}_2^b$ $=$ ${\beta \left(\gamma \right)}_2^m$, ${\beta \left(\gamma \right)}_2^a$ $=$ ${\beta \left(\gamma \right)}_2^a$.

Step 4: If the $\left|{\beta \left(\gamma \right)}_2^b-{\beta \left(\gamma \right)}_2^a\right|$ is higher than $e$, go to step 3, otherwise display the interval $\left[{\beta \left(\gamma \right)}_2^a,{\beta \left(\gamma \right)}_2^b\right]$, where ${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^a\right)\cong \alpha$ and ${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^b\right)\cong \alpha$.

Step 5: Display ${\beta \left(\gamma \right)}_2^m$ as the approximate root, which satisfies ${\widehat{P}}_{et}^{*}\left({\beta \left(\gamma \right)}_2^m\right)=\alpha$.

The bisection process is repeated, and finally, the interval $\left[{\beta \left(\gamma \right)}_2^l,{\beta \left(\gamma \right)}_2^u\right]$ is available.

4. Computing Costs

Obviously, the methods presented in Section 3 are much faster than the bootstrap method that directly generates the full bootstrap samples. Specifically, if we perform the full maximum likelihood estimation of SEM for each bootstrap sample, the cost of each bootstrap replication is $O (r^2 n)$. In contrast, by pre-computing some sufficient statistics, we can calculate all of the quantities needed for bootstrap inference without performing a computationally expensive process of $O \left(n\right)$. Thus, all calculations with computational cost $O \left(n\right)$ are performed only once, not for each bootstrap sample, and the computational cost of each bootstrap replication depends solely on $L$ and $r$, but not on $n$.

In this section, we support the above claims by simulating the computational cost of calculating the bootstrap t-statistics with pcb, UWC1, UWC2, UWC3, and RWC, as proposed in Section 3, respectively. In addition, for comparative purposes, this study also reports a benchmark number that is $B + 1$ times the computing time of a single test statistic. Table 2 illustrates the average computing times (in seconds) of 1000 Monte Carlo simulations for various bootstrap methods with $n=1000$ or $\mathrm{100,000}$, $L= 10$ or 20, $B=999$ or 9999, $r=10$, $\gamma$ varying in the set $\left\{0.2, 0.4, 0.6, 0.8\right\}$, and $W$ having the rook-based contiguity neighborhood. In addition, the consuming time to generate the data (which is typically larger than the computing time of bootstrapping) is not included in all experimental results.

Table 2. Computing times (in seconds) for various bootstrap methods.

$n=1000$
$L, B$	10, 999	20, 999	10, 9999	20, 9999
Benchmark	0.1844	0.2003	1.8440	2.0030
pcb	0.0895	0.0899	0.8850	0.8889
UWC1	0.0057	0.0068	0.0560	0.0645
UWC2	0.0046	0.0052	0.0450	0.0517
UWC3	0.0017	0.0023	0.0169	0.0184
RWC	0.0012	0.0016	0.0103	0.0110
$n=\mathrm{100,000}$
$L, B$	10, 999	20, 999	10, 9999	20, 9999
Benchmark	20.3400	21.5200	203.4000	215.2000
pcb	0.8610	0.8740	1.0103	1.0216
UWC1	0.0614	0.0620	0.1052	0.1083
UWC2	0.0429	0.0435	0.0856	0.0867
UWC3	0.0174	0.0187	0.0291	0.0293
RWC	0.0102	0.0108	0.0194	0.0195

Firstly, it is obvious that pcb is much less computationally expensive than the baseline method; nevertheless, pcb is more computationally expensive than UWCs since UWCs pre-compute the same ${X\left(\gamma \right)}^{T}X\left(\gamma \right)$ matrix for all bootstrap samples, while pcb involves constructing the ${{X\left(\gamma \right)}^{*j}}^T{X\left(\gamma \right)}^{*j}$ matrix for each bootstrap sample. Secondly, compared to UWC1, UWC2 is faster, which may be attributed to the fact that the ${G\left(\gamma \right)}_l$ matrices and $f\left(\gamma \right)$ and $v_j^{*}$ vectors are formed before the bootstrap loop, which makes computing the t-statistic less expensive. Moreover, UWC3 applies a trick to compute the denominator of the bootstrap t-statistic, which further reduces computational cost. Thirdly, applying RWC is marginally faster than UWC3, and it performs the fastest; therefore, it can be used for large samples in applications. Finally, when $n=\mathrm{100,000}$, increasing the times of $B$ has less impact on the computational cost, which is different from the performance when $n=1000$, and this is because that the initial computation takes up a large proportion of the total time for large samples.

5. Monte Carlo Simulations

To demonstrate our theoretical findings, we perform 1000 Monte Carlo simulations and 999 bootstrap iterations to investigate the empirical coverage frequencies of nominal 95% confidence intervals constructed by pcb, UWC1, UWC2, UWC3, and RWC, respectively. Specifically, the Monte Carlo analysis was performed with a simple SEM, which includes one explanatory variable and a constant:

$$\left(I-\gamma W\right)y=\left(I-\gamma W\right)X{\beta }_1+{\beta }_0+\epsilon ,$$

where ${\beta }_1=0.2$, ${\beta }_0=0.5$, $\epsilon$ is generated from a normal distribution with zero mean and unit variance, and $X$ is drawn from another normal distribution with zero mean and variance 2. We consider a rook-based contiguity neighborhood for the spatial weight matrix $W$, which has been normalized, and the parameter $\gamma$ varies over the set $\left\{0.2,0.4,0.6,0.8\right\}$.

In each of these experiments, there are a total of $n=100L$ observations, and these observations are grouped into $L$ clusters given by the following equations,

$$\begin{gathered} n_l=\left[n{\displaystyle \frac{e^{\frac{\delta l}{L}}}{\sum _{h=1}^L{e}^{\frac{\delta h}{L}}}}\right],l=\mathrm{1,2},\dots \dots ,L-1, \\ {n}_L=n-{\sum }_{l=1}^{L-1}{n}_l, \end{gathered}$$

(47)

where $\left[·\right]$ denotes the integer of its argument. The parameter $\delta$ determines the heterogeneity of the cluster size; specifically, when $\delta =0$, for all $l$ there holds $n_l=n/L$, while the difference in cluster size grows as $\delta$ increases. Table 3 reports the two-sided equal-tailed coverage frequencies of ${\beta }_1$ with the precisions of the bootstrap confidence intervals (in parentheses) when $L=10 \mathrm{a}\mathrm{n}\mathrm{d} \delta =0$ or $3$.

Table 3. Two-sided equal-tailed coverage frequencies of ${\beta }_1$ with 10 clusters.

	pcb		UWC1		UWC2		UWC3		RWC
$\mathit{\delta }$	0	3	0	3	0	3	0	3	0	3
$\gamma =0.2$	0.8159	0.7855	0.8835	0.8119	0.8874	0.8224	0.8907	0.8523	0.8978	0.8728
	(0.9016)	(0.9184)	(0.4686)	(0.6031)	(0.4652)	(0.5601)	(0.4285)	(0.4916)	(0.2017)	(0.2210)
$\gamma =0.4$	0.8231	0.7922	0.8948	0.8605	0.8980	0.8561	0.9032	0.8702	0.9302	0.8939
	(0.8550)	(0.8657)	(0.3322)	(0.4037)	(0.3370)	(0.3928)	(0.2355)	(0.3831)	(0.1911)	(0.2146)
$\gamma =0.6$	0.8796	0.8509	0.9064	0.8829	0.9183	0.8834	0.9299	0.9194	0.9345	0.9255
	(0.7283)	(0.7502)	(0.2817)	(0.3932)	(0.2756)	(0.3898)	(0.2284)	(0.3205)	(0.1814)	(0.2036)
$\gamma =0.8$	0.8398	0.8059	0.8829	0.8635	0.8972	0.8727	0.9030	0.8979	0.9296	0.9107
	(0.8345)	(0.8414)	(0.3247)	(0.4750)	(0.3173)	(0.4622)	(0.3031)	(0.4313)	(0.2269)	(0.2261)

From Table 3, we can find that firstly, when $\delta =0$, (i.e., all clusters contain 100 observations), the two-sided equal-tailed coverage frequencies and the precisions of the bootstrap confidence intervals of ${\beta }_1$ obtained with each method perform better than $\delta =3$ (i.e., the cluster sizes vary from 18 to 277). Secondly, the bootstrap confidence intervals of ${\beta }_1$ constructed by pcb severely under-cover for all values of $\gamma$, and the UWCs improve on the pcb but still under-cover for all $\gamma$. Thirdly, the RWC outperforms all UWCs but still gives coverage less than the nominal frequency for all values of $\gamma$, which is consistent with the findings that the under-coverage of bootstrap confidence interval based on UWCs is more severe than that of bootstrap confidence interval based on RWC in some cases [21,29]. Finally, it is obvious that the RWC obtains the optimal performance with $\delta =0$ and $\gamma$ = 0.6; in particular, the two-sided equal-tailed coverage frequency of ${\beta }_1$ is 0.9345.

Table 4 shows the two-sided equal-tailed coverage frequencies of ${\beta }_1$ with the precisions of the bootstrap confidence intervals (in parentheses) when $L=20$ and $\delta =0$ or $3$.

Table 4. Two-sided equal-tailed coverage frequencies of ${\beta }_1$ with 20 clusters.

	pcb		UWC1		UWC2		UWC3		RWC
$\mathit{\delta }$	0	3	0	3	0	3	0	3	0	3
$\gamma =0.2$	0.8782	0.8660	0.9042	0.8959	0.9231	0.9162	0.9356	0.9324	0.9404	0.9387
	(0.7248)	(0.7690)	(0.2249)	(0.2417)	(0.2149)	(0.2244)	(0.2279)	(0.2286)	(0.1896)	(0.1920)
$\gamma =0.4$	0.9125	0.8801	0.9226	0.9180	0.9492	0.9275	0.9494	0.9398	0.9612	0.9574
	(0.5845)	(0.6149)	(0.2057)	(0.2243)	(0.1942)	(0.2188)	(0.1986)	(0.2049)	(0.1414)	(0.1482)
$\gamma =0.6$	0.9073	0.8944	0.9479	0.9351	0.9524	0.9460	0.9575	0.9451	0.9595	0.9498
	(0.4742)	(0.5015)	(0.1860)	(0.2004)	(0.1805)	(0.1904)	(0.1811)	(0.1968)	(0.1515)	(0.1546)
$\gamma =0.8$	0.8961	0.8826	0.9207	0.9196	0.9398	0.9342	0.9446	0.9316	0.9523	0.9406
	(0.4894)	(0.5120)	(0.2037)	(0.2139)	(0.1987)	(0.2105)	(0.2003)	(0.2106)	(0.1602)	(0.1639)

Firstly, with a bigger cluster size, the two-sided equal-tailed coverage frequencies and the precisions of the bootstrap confidence intervals of ${\beta }_1$ based on each method with $\delta =0$ perform better than those with $\delta =3$ (i.e., the cluster sizes vary from 16 to 304). Secondly, when $L=20$, the results of all experiments largely outperform those when $L=10$; in particular, the optimal coverage of ${\beta }_1$ based on RWC is around 0.96 with $\delta =0$ and $\gamma =0.4$, which is higher than the nominal frequency of 0.95. Finally, the best performances of ${\beta }_1$ based on UWCs and RWC are obtained with $\gamma =0.6$ and $\gamma =0.4$, respectively, which suggests that choosing an optimal value of $\gamma$ for different methods may result in coverage closer to the nominal frequency.

6. Conclusions

This study proposes fast cluster bootstrap methods for SEM, which significantly reduce the computational cost and improve the reliability of parameter inference for SEM with large samples compared with traditional bootstrap methods. Specifically, before performing the bootstrap loop of SEM, the sufficient statistics are computed for each of the clusters, and based on these sufficient statistics, we can calculate all of the quantities needed for bootstrap inference. In addition, the test statistics and the bootstrap confidence intervals are associated with the samples solely by these sufficient statistics.

This study simulates the computational cost of calculating bootstrap t-statistics with various bootstrap methods, and the result reveals that the computational cost of our proposed methods is substantially reduced compared with traditional bootstrap methods, particularly with a large sample size.

Moreover, an extensive Monte Carlo simulation study indicates that, firstly, when all clusters have the same size, the two-sided equal-tailed coverage frequencies and the precisions of the bootstrap confidence intervals of the parameter obtained with the various methods outperform those of the clusters with varied sizes. Secondly, when $L=20$, the results of all experiments largely outperform those when $L=10$, and the optimal coverage of the parameter based on RWC is higher than the nominal frequency of 0.95. Finally, choosing an optimal value of $\gamma$ for different methods may result in coverage closer to the nominal frequency. Table 5 shows the main empirical findings of this study.

Table 5. The main empirical findings of this study.

The Main Empirical Findings of This Study
Finding 1:	The computational cost of our proposed methods is substantially reduced compared with traditional bootstrap methods.
Finding 2:	The optimal coverage of the parameter based on RWC is higher than the nominal frequency.

7. Future Work

This study proposes fast cluster bootstrap methods for SEM and has the future research work as follows: firstly, our methods can be extended to other spatial models such as the Spatial Lag Model (SLM). In specific, the SLM can be written as $y_l=\theta W{y}_l+X_l\beta +{\epsilon }_l,\left(l=\mathrm{1,2},\dots \dots ,L\right).$ Equivalently, the SLM can be expressed as $\left(I-\theta W\right)y_l=X_l\beta +{\epsilon }_l,\left(l=\mathrm{1,2},\dots \dots ,L\right).$ Furthermore, let ${y\left(\theta \right)}_l=\left(I-\theta W\right)y_l,$ and the SLM is equivalent to ${y\left(\theta \right)}_l=X_l\beta +{\epsilon }_l, \left(l=\mathrm{1,2},\dots \dots ,L\right).$ Similarly, it can be rewritten in matrix form with $y\left(\theta \right)=X\beta +\epsilon .$ Obviously, the bootstrap computations for maximum likelihood estimation of SLM, including the fast pairs cluster bootstrap method for SLM and the fast unrestricted and restricted wild cluster bootstrap methods for SLM, are similar to the calculation process used for SEM in the above sections.

Secondly, our experiments are limited in scope. We suppose that the error of the model with the classical assumption has a normal distribution. In fact, more realistic data-generating processes (e.g., non-normal errors, spatial heteroscedasticity) have more challenging problems, which will be our focus of research in the future.