Sparse Boosting for Additive Spatial Autoregressive Model with High Dimensionality

Abstract

Variable selection methods have been a focus in the context of econometrics and statistics literature. In this paper, we consider additive spatial autoregressive model with high-dimensional covariates. Instead of adopting the traditional regularization approaches, we offer a novel multi-step sparse boosting algorithm to conduct model-based prediction and variable selection. One main advantage of this new method is that we do not need to perform the time-consuming selection of tuning parameters. Extensive numerical examples illustrate the advantage of the proposed methodology. An application of Boston housing price data is further provided to demonstrate the proposed methodology.

1. Introduction

Spatial data, also known as geospatial data, are widely used in fields such as environmental health, economics, and epidemiology [1]. These data are typically represented by numerical values within a geographic coordinate system. Among the various models for analyzing spatial data, the spatial autoregressive (SAR) model has gained significant attention in econometrics and statistics. The SAR model incorporates a spatially lagged term of the response variable to account for spatial dependence among spatial units. Key developments in the estimation and inference of SAR models can be traced to foundational works by Cliff and Ord [2], Anselin [3], and Robert [4], among others.

The linear SAR model, an extension of ordinary linear regression, has been extensively studied [5,6,7,8,9]. However, its reliance on strict assumptions can lead to inefficiencies if these assumptions are violated. To address this limitation, nonparametric SAR models have been developed, allowing for more flexible relationships between the response variable and predictors without assuming a specific functional form. Among these, the partial linear SAR model has emerged as a powerful tool, combining linear and nonparametric components. For instance, Su and Jin [10] proposed a profile quasi-maximum likelihood estimation method, while Koch and Krisztin [11] introduced an approach based on B-splines and genetic algorithms. Other notable contributions include Bayesian methods by Chen et al. [12] and penalized splines by Krisztin [13], as well as testing methods by Li and Mei [14].

Beyond the partial linear SAR model, researchers have explored more complex structures. Su [15] studied a SAR model with a linear spatially lagged response and fully nonparametric predictors, proposing semiparametric GMM estimation. Wei et al. [16] introduced SAR models with varying coefficients to capture heterogeneous covariate effects, while Du et al. [17] developed generalized method of moments estimators for partially linear additive SAR models. Cheng and Chen [18] investigated partially linear single-index SAR models, and Yang, Song, and Yu [19] proposed nonparametric tests for model specification. In this paper, we focus on the additive SAR model, which offers greater flexibility than previous models by addressing the curse of dimensionality through a single nonparametric component. We specifically address high-dimensional settings where the number of predictors may exceed the sample size.

High-dimensional data analysis has become a central focus in statistics and data science. In such settings, the inclusion of irrelevant features can lead to computational challenges and unstable estimates. To mitigate these issues, variable screening and selection techniques have been developed. Penalized approaches, such as lasso [20], SCAD [21], and MCP [22], have been widely studied for their ability to simultaneously select important covariates and estimate coefficients. Extensions like group lasso [23] and adaptive lasso [24] have further enhanced these methods. For example, Wei et al. [16] applied local linear shrinkage estimators to SAR models with varying coefficients, while Yu, Yao, and Liu [25] used elastic net penalties in Bayesian spatial quantile panel autoregressive models. Nevertheless, these regularization methods have one major disadvantage: they all involve tuning parameters that must be chosen using computationally intensive methods such as cross-validation.

Boosting methods have emerged as a powerful alternative for high-dimensional data analysis, offering advantages such as lower computational costs, reduced overfitting risk, and flexibility in incorporating constraints. Boosting was initially conceptualized as a machine learning algorithm that constructs better base learners to minimize the loss function with every iteration. The original version of the boosting algorithm proposed by Schapire [26] and Freund [27] did not fully exploit the potential of base learners. This changed when Freund and Schapire [28] proposed the AdaBoost algorithm, which could adapt to base learners more effectively. Following the development of this algorithm, many other boosting algorithms have been formulated, with major variations in their loss functions. For instance, AdaBoost with the exponential loss, L2Boosting [29] with the squared error loss, SparseL2Boosting [30] with the penalized loss, and HingeBoost [31] with the weighted hinge loss. Other versions of boosting algorithms proposed recently include pAUCBoost [32], Twin Boosting [33], Twin HingeBoost [31], ER-Boost [34], GSBoosting [35], among others. Recently, the rapidly developed boosting algorithms have been widely applied in the analysis of high-dimensional data in fields such as economics and health. Specifically, Yousuf and Ng [36] proposed two boosting algorithms: one uses componentwise local constant estimators and the other employs componentwise local linear estimators, testing the economic series. Meanwhile, Huang et al. [37] utilized eXtreme Gradient Boosting (XGBoost) to develop a dose prediction model that addresses variability in vancomycin dosing among individuals. As demonstrated by Yue et al. [38,39,40,41], it’s worth mentioning that sparse boosting achieves better variable selection performance than L2Boosting as well as some penalized methods such as lasso.

Despite these advancements, research on SAR models in high-dimensional settings has predominantly focused on penalized likelihood methods. For instance, Liu et al. [42] developed a penalized quasi-likelihood approach for simultaneous model selection and parameter estimation, while Liu and Chen [43] extended this to models with autoregressive disturbances. Xia et al. [44] addressed variable selection in high-dimensional spatial autoregressive panel models using penalized quasi-likelihood methods. However, the application of boosting techniques, particularly sparse boosting, remains underexplored in this context, presenting a significant opportunity for further research.

This paper proposes a novel multi-step sparse boosting algorithm for additive spatial autoregressive models with high-dimensional covariates. Unlike traditional regularization methods, our approach eliminates the need for time-consuming tuning parameter selection, enhancing modeling efficiency. By extending boosting techniques to complex spatial autoregressive structures, we address a critical gap in the literature. Through simulation studies and real-world data analysis, we demonstrate the superior performance of our method compared to existing alternatives. Our results provide a foundation for future research and highlight the potential of sparse boosting as a powerful tool for high-dimensional spatial data analysis.

The remainder of the paper is organized as follows. Section 2 introduces the additive autoregressive model and presents the multi-step sparse boosting algorithm. Section 3 evaluates the method through simulation studies. Section 4 applies the method to Boston housing price data. Finally, Section 5 provides discussion and concluding remarks.

2. Materials and Methods

2.1. Model and Estimation

Consider the following additive spatial autoregressive model:

$$Y=\rho WY+\sum _{j=1}^p{g}_j\left(X_j\right)+ϵ$$

(1)

where $Y={(y_1,\cdots ,y_n)}^T$ is the $n\times 1$ vector of the observations of the response variable with n being the number of spatial units, W is an $n\times n$ spatial weight matrix of known constants with zero diagonal elements and $WY$ is referred to as spatial lag of Y, $\rho$ is the spatial autoregressive parameter with $\left|\rho \right|<1$, $X_j={(X_{1j},\cdots ,X_{nj})}^T$ is the $n\times 1$ vector of the j-th regressor and $X=(X_1,\cdots ,X_p)$ is the $n\times p$ observed matrix of regressors. For simplicity, the covariates $X_1,\cdots ,X_p$ are assumed to be distributed on compact intervals $[a_1,b_1],\cdots ,[a_p,b_p]$ respectively. $g_j(.)s$ are unknown smooth functions on $[a_j,b_j]$ with the assumption that $E\left[g_j\left(X_j\right)\right]=0$ for identifiability purpose. $ϵ={({ϵ}_1,\cdots ,{ϵ}_n)}^T$ is an $n\times 1$ vector of i.i.d disturbances with zero mean and finite variance ${\sigma }^2$.

The smoothing splines technique is typically used to estimate the unknown functions, demonstrating steady performance in practice. In this paper, we will use B-spline basis functions to approximate the unknown coefficient functions $g_j(.),j=1,\cdots ,p$. Let $B(.)=(B_1(.),\cdots ,B_L(.))$ be an equally spaced B-spline basis, where L is the dimension of the basis. Similar to Huang et al. [45], let ${\overline{B}}_{lj}={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{B}_l\left(X_{nj}\right)$, ${\pi }_{lj}(.)=B_l(.)-{\overline{B}}_{lj}$, ${\pi }_j(.)=({\pi }_{1j}(.),\cdots ,{\pi }_{Lj}(.))$. Then under appropriate smoothness assumptions, $g_j(.)\approx {\sum }_{l=1}^L{\pi }_{lj}(.){\alpha }_{lj}={\pi }_j(.){\alpha }_j$ for $j=1,\cdots ,p$. Then the spline estimator of $g(.)={\sum }_{j=1}^p{g}_j(.)$ is $\widehat{g}(.)=\pi (.)\alpha$, where $\pi (.)=({\pi }_1(.),\cdots ,{\pi }_p(.))$, $\alpha ={({\alpha }_1^T,\cdots ,{\alpha }_p^T)}^T$. It is obvious that ${\sum }_{i=1}^n{\widehat{g}}_j\left(X_{ij}\right)=0$ for $j=1,\cdots ,p$.

Then the model (1) can be rewritten as follows

$$\begin{array}{cc}\hfill Y& \approx \rho WY+\sum _{j=1}^p{\pi }_j\left(X_j\right){\alpha }_j+ϵ\hfill \\ \hfill & =\rho WY+\pi \alpha +ϵ,\hfill \end{array}$$

(2)

where $\pi =({\pi }_1\left(X_1\right),\cdots ,{\pi }_p\left(X_d\right)$. Let $P=\pi {\left({\pi }^T\pi \right)}^{-1}{\pi }^T$ denote the $n\times n$ projection matrix onto the space spanned by $\pi$. Similar to [46], partialing out the B-spline approximation, we obtain

$$(I-P)Y\approx (I-P)\rho WY+(I-P)ϵ.$$

(3)

In the above equation, a problem of endogeneity emerges because the spatially lagged value of Y is correlated with the stochastic disturbance. Suppose H is a relevant vector instrument to eliminate the endogeneity of $WY$. Let $D=WY$. Regressing D (a $n\times 1$ vector) on instrument H via OLS produces the fitted value for the endogenous variable D:

$$\widehat{D}=H{\left(H^{T}H\right)}^{-1}{H}^{T}D.$$

(4)

Substituting D with its predicted value in model (3), it becomes

$$\begin{array}{cc}\hfill (I-P)Y& \approx (I-P)\rho H{\left(H^{T}H\right)}^{-1}{H}^{T}D+(I-P)ϵ\hfill \\ \hfill & =(I-P)\rho MD+(I-P)ϵ,\hfill \end{array}$$

(5)

where $M=H{\left(H^{T}H\right)}^{-1}{H}^T$ is a $n\times n$ matrix. Then by OLS,

$$\widehat{\rho }={\left(D^{T}M(I-P)MD\right)}^{-1}{D}^{T}M(I-P)Y.$$

(6)

Substituting D with $\widehat{D}$ and $\rho$ with $\widehat{\rho }$ in model (2), it becomes

$$Y\approx \widehat{\rho }D+\pi \alpha +ϵ.$$

(7)

Then the least squares loss function is close to

$${(Y-\widehat{\rho }D-\pi \alpha )}^T(Y-\widehat{\rho }D-\pi \alpha ).$$

(8)

However, when dimensionality of $\pi$ is larger than sample size n, least square estimation fails. In this case, we will adopt sparse boosting approach to estimate $\alpha$. Denote $\widehat{\alpha }$ as the estimator of $\alpha$ obtained through sparse boosting, using the squared loss function (8) as the loss function. The detailed sparse boosting algorithm will be given in the next subsection.

Consequently, $g_j(.)$ can be estimated by ${\widehat{g}}_j(.)={\pi }_j(.){\widehat{\alpha }}_j$ for $j=1,\cdots ,p$. The variance ${\sigma }^2$ can be estimated by

$${\widehat{\sigma }}^2={\displaystyle \frac{1}{n}}{(Y-\widehat{\rho }WY-\pi \widehat{\alpha })}^T(Y-\widehat{\rho }WY-\pi \widehat{\alpha }).$$

(9)

Similar to [46], we use an analogous rationality for the construction of instrument variables. In the first step, we simply regress Y on pseudo regressors variables $WY$ and $\pi$. Then the least squares loss function is

$${(Y-\rho WY-\pi \alpha )}^T(Y-\rho WY-\pi \alpha )$$

(10)

Denote $(\tilde{\rho },\tilde{\alpha })$ as the estimators of $(\rho ,\alpha )$ obtained through sparse boosting, using the squared loss function (10) as the loss function. Then, we can use the following instrumental variable:

$$\tilde{H}=W{(I-\tilde{\rho }W)}^{-1}\pi \tilde{\alpha }.$$

(11)

In the second step, instrumental variable $\tilde{H}$ is used to obtain the estimators $\overline{\alpha }$ and $\overline{\rho }$, which are then used to construct the instrumental variable

$$H=W{(I-\overline{\rho }W)}^{-1}\pi \overline{\alpha }.$$

(12)

Finally, use the instrumental variables H to obtain the final estimators $\widehat{\rho }$ and $\widehat{\alpha }={({\widehat{\alpha }}_1^T,\cdots ,{\widehat{\alpha }}_p^T)}^T$. The function $g_j(.)$ can be estimated by ${\widehat{g}}_j(.)={\widehat{\pi }}_j(.){\widehat{\alpha }}_j$ for $j=1,\cdots ,p$ and the response Y can be estimated by $\widehat{Y}={(I-\widehat{\rho }W)}^{-1}\pi \widehat{\alpha }$.

2.2. Sparse Boosting Techniques

Sparse boosting can be viewed as iteratively pursuing gradient descending in function space using a penalized empirical risk function that integrates squared loss and the complexity of the boosting measure. Similar to Yue et al. ([38,39,40,41]), we will adopt the g-prior minimum description length (gMDL) [47], a combination of squared loss and the trace of boosting operator, as the penalized empirical risk function to estimate the update criterion in each iteration and the stopping criterion. We use it because it has a data driven penalty to avoid the selection of the tuning parameter. To facilitate presentation, suppose the vector $Y^{*}$ is regressed on the $p^{*}$-dimensional matrix $X^{*}=(X_1^{*},\cdots ,X_p^{*})$. Then the least squares loss function involved in sparse boosting is ${(Y^{*}-X^{*}{\theta }^{*})}^T(Y^{*}-X^{*}{\theta }^{*})$, where ${\theta }^{*}={({\left({\theta }_1^{*}\right)}^T,\cdots ,{\left({\theta }_p^{*}\right)}^T)}^T$. The gMDL takes the form:

$$\begin{array}{cc}\hfill \mathrm{gMDL}(Y^{*},RSS,\mathrm{trace}\left(\mathcal{B}\right))=& \log \left(F\right)+{\displaystyle \frac{\mathrm{trace}\left(\mathcal{B}\right)}{n}}\log \left({\displaystyle \frac{Y^{*T}{Y}^{*}-RSS}{\mathrm{trace}\left(\mathcal{B}\right)\times F}}\right),\hfill \\ \hfill & F={\displaystyle \frac{RSS}{n-\mathrm{trace}\left(\mathcal{B}\right)}},\hfill \end{array}$$

(13)

where $RSS$ is the residual sum of squares and $\mathcal{B}$ is the boosting operator. The model achieve shortest description of data will be chosen.

We present the sparse boosting approach more specifically. The initial value of ${\theta }^{*}$ is set to the zero vector, i.e. ${\theta }^{*\left[0\right]}=0$. In each of the kth iteration ($0<k\le K$, and K is the maximum number of iterations considered in the first step), we use the residual $R^{\left[k\right]}=Y^{*}-X^{*}{\theta }^{*[k-1]}$ from the current iteration to fit each of the j-th component $X_j^{*},j=1,\cdots ,p^{*}$. The fit, denoted by ${\widehat{\lambda }}_j^{\left[k\right]}$, is calculated by minimizing the squared loss function ${(R^{\left[k\right]}-X_j^{*}\lambda )}^T(R^{\left[k\right]}-X_j^{*}\lambda )$ with respect to $\lambda$. Therefore, the least squares estimate is ${\widehat{\lambda }}_j^{\left[k\right]}={\left[{\left(X_j^{*}\right)}^T\left(X_j^{*}\right)\right]}^{-1}{\left(X_j^{*}\right)}^T{R}^{\left[k\right]}$, the corresponding hat matrix is ${\mathcal{H}}_d=\left(X_j^{*}\right){\left[{\left(X_j^{*}\right)}^T\left(X_j^{*}\right)\right]}^{-1}{\left(X_j^{*}\right)}^T$ and the residual sum of squares is $RS{S}_j^{\left[k\right]}={(R^{\left[k\right]}-X_j^{*}{\widehat{\lambda }}_j^{\left[k\right]})}^T(R^{\left[k\right]}-X_j^{*}{\widehat{\lambda }}_j^{\left[k\right]})$. The chosen element ${\widehat{s}}_k$ is attained by:

$$\begin{array}{c}\hfill {\widehat{s}}_k={\mathrm{argmin}}_{1\le j\le p^{*}}\mathrm{gMDL}(Y^{*},RS{S}_j^{\left[k\right]},\mathrm{trace}\left({\mathcal{B}}_j^{\left[k\right]}\right)),\end{array}$$

(14)

where ${\mathcal{B}}_j^{\left[1\right]}={\mathcal{H}}_j$ and ${\mathcal{B}}_j^{\left[k\right]}=I-(I-{\mathcal{H}}_j)(I-\nu {\mathcal{H}}_{{\widehat{s}}_{k-1}}).\cdots .(I-\nu {\mathcal{H}}_{{\widehat{s}}_1})$ for $k>1$ is the first step boosting operator for choosing jth element in the kth iteration. Hence, there is an unique element $X_{{\widehat{s}}_k}^{*}$ to be selected at each iteration, and only the corresponding coefficient vector ${\theta }_{{\widehat{s}}_k}^{*\left[k\right]}$ changes, i.e., ${\theta }_{{\widehat{s}}_k}^{*\left[k\right]}={\theta }_{{\widehat{s}}_k}^{*[k-1]}+\nu {\widehat{\lambda }}_{{\widehat{s}}_k}^{\left[k\right]}$, where $\nu$ is the pre-specified step-size parameter. All the other ${\theta }_j^{*\left[k\right]}$ for $j\ne {\widehat{s}}_k$ keep unchanged. We repeat this procedure for K times and the number of iterations K can be estimated by

$$\widehat{K}={\mathrm{argmin}}_{1\le k\le K}\mathrm{gMDL}(Y^{*},RS{S}_{{\widehat{s}}_k}^{\left[k\right]},\mathrm{trace}\left({\mathcal{B}}^{\left[k\right]}\right)),$$

(15)

where ${\mathcal{B}}^{\left[k\right]}=I-(I-\nu {\mathcal{H}}_{{\widehat{s}}_k}).\cdots .(I-\nu {\mathcal{H}}_{{\widehat{s}}_1})$.

From the sparse boosting, we get the estimator of ${\theta }^{*}$ by ${\theta }^{*\left[\widehat{K}\right]}={({({\theta }_1^{*\left[\widehat{K}\right]})}^T,\cdots ,{({\theta }_p^{*\left[\widehat{K}\right]})}^T)}^T$.

For least squares loss function (8), $Y^{*}=Y-\widehat{\rho }D$ and $X^{*}=\pi$, and for least squares loss function (10), $Y^{*}=Y$ and $X^{*}=(WY,\pi )$.

Overall, as illustrated in Figure 1, the flowchart visually summarizes the methodologies and steps articulated in this paper, thereby enhancing the understanding of our proposed multi-step sparse boosting algorithm for high-dimensional additive spatial autoregressive models.

In Step I, the algorithm starts with parameter initialization, followed by iterations to identify optimal variables using the g-prior minimum description length (gMDL) criterion. Coefficients are updated until a stopping criterion is met, resulting in covariance matrix estimates.

Step II applies the sparse boosting algorithm again to refine variable selection and update coefficients. This approach enhances model accuracy and selection efficiency, making it well-suited for high-dimensional data analysis.

2.3. Discussion

In summary, sparse boosting has the following advantages (

Table 1. Specifications, advantages, and limitations of different methods.

Method	Specifications	Advantages	Limitations
Lasso	L1 penalty on coefficients;minimizes L1-penalized loss function.	Encourages sparsity; computationally efficient.	Biased estimates for large coefficients; struggles with correlated predictors.
Elastic net	Combines L1 and L2 penalties; minimizes a weighted sum of L1 and L2 penalties.	Handles correlated predictors better than LASSO; more flexible regularization.	Requires tuning of two parameters ($\lambda$ and $\alpha$); can still introduce bias.
Sparse boosting	Iteratively selects variables and updates coefficients; no explicit regularization.	Adaptive sparsity; handles non-convex loss functions; less sensitive to tuning.	Computationally intensive for very high dimensions; requires careful stopping.

). Firstly, sparse boosting adaptively selects variables and controls sparsity without requiring explicit regularization parameters (unlike lasso and elastic net, which rely on tuning parameters like $\lambda$). Secondly, sparse boosting can handle non-convex loss functions, which is particularly useful for complex models like the additive spatial autoregressive model. Lasso and elastic net, on the other hand, are limited to convex optimization. Thirdly, sparse boosting iteratively reduces bias while controlling variance, whereas lasso and elastic net may introduce bias due to the L1 penalty. Last but not least, sparse boosting provides a more interpretable model by sequentially adding the most relevant variables, while lasso and elastic net may shrink coefficients indiscriminately.

3. Simulation

In this section, we investigate the finite sample performance of the proposed methodology with Monte Carlo simulation studies. The data is generated from the following model:

$$Y=\rho WY+\sum _{j=1}^{1000}{g}_j\left(X_j\right)+ϵ,$$

(16)

where $X_1,\cdots ,X_{1000}$ are all i.i.d on $[-1,1]$. $g_1\left(x\right)=g_3\left(x\right)=2tan\left(x\right)$, $g_2\left(x\right)=g_4\left(x\right)=2sin\left(x\right)$ and $g_5\left(x\right)=\cdots g_{1000}\left(x\right)=0$. The error term $ϵ$ follows a normal distribution with mean 0 and variance ${\sigma }^2$. Similar to [48], the weight matrix is set to be $W=I_R\otimes E_m$, where $E_m=(l_m^T{l}_m-l_m)/(m-1)$, $l_m$ is the m-dimensional unit vector, and ⊗ is Kronecker product. Thus, the number of spatial units is $n=R\times m$. For comparison, we consider $(R,m)=(10,10)$ and $(20,20)$, corresponding to $n=100$ and 400 respectively. We evaluate three different values of $\rho =0.2,0.5,0.8$, representing weak to strong spatial dependence of the responses. Additionally, we consider ${\sigma }^2=0.5$ and 1. The degree of freedom for B-splines basis is set to be $\lfloor (N/\lfloor N/log(N)\rfloor \rfloor$, where $\lfloor x\rfloor$ is the integer part of x.

In face of ultra-high dimensional data, pre-screening can be adopted to reduce the dimensionality to a moderate size. In particular, we adopt marginal screening by $l^2$-norm [49] to screen out irrelevant covariates. More specifically, we regress Y on each covariate to construct the marginal model $Y={\rho }_{0}WY+g_{0j}\left(X_j\right)+{ϵ}_{0j},j=1,\cdots ,p$. The empirical $l^2$-norm of an estimated function ${\widehat{g}}_{0j}\left(X_j\right)={({\widehat{g}}_{0j}\left(X_{1j}\right),\cdots ,{\widehat{g}}_{0j}\left(X_{nj}\right))}^T$ is defined as:

$$\parallel {\widehat{g}}_{0j}\left(X_j\right){\parallel }^2={\displaystyle \frac{1}{n}}\left({\widehat{g}}_{0j}\left(X_{1j}\right),\cdots ,{\widehat{g}}_{0j}\left(X_{nj}\right)\right){\left({\widehat{g}}_{0j}\left(X_{1j}\right),\cdots ,{\widehat{g}}_{0j}\left(X_{nj}\right)\right)}^T.$$

(17)

A greater value of this measure suggest a stronger association between the covariate and the response. Following the recommendation by [49,50,51,52], the covariates with the largest $\lfloor {\displaystyle \frac{n}{log\left(n\right)}}\rfloor$$l^2$-norm are selected. Thus, for sample size 100 and 400, the first 21 and 66 variables in the ranked list are selected to conduct downstream analysis, respectively. Thereafter, we proceed to use our proposed multi-step sparse boosting approach to build up the final parsimonious model. For comparison, besides the proposed method using sparse boosting in each step, we also examine other approaches such as $l_2$-boosting, lasso regression and elastic net regression in each step.

In our implementation, we set the maximum number of boosting iterations to $K=500$ and the elastic net mixing parameter to $0.5$. Penalized methods were executed using the R package glmnet 4.1.8., with tuning parameters selected via 5-fold cross-validation. To assess the performance of our approach, we analyze results from 500 replications, reporting the following metrics:

• S: coverage probability that the top $\lfloor {\displaystyle \frac{n}{log\left(n\right)}}\rfloor$ covariates after screening includes all important covariates;
• TP: the median of true positives;
• FP: the median of false positives;
• Size: the median of model sizes;
• ISPE: the average of in-sample prediction errors defined as ${(Y-\widehat{Y})}^T((Y-\widehat{Y})/n$;
• RMISE: the average of root mean integrated squared errors defined as
• $\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{j=1}^4{(g_j\left(X_j\right)-{\widehat{g}}_j\left(X_j\right))}^T(g_j\left(X_j\right)-{\widehat{g}}_j\left(X_j\right))}$;
• Bias($\rho$): the mean bias of $\rho$;
• Bias($\sigma$): the mean bias of $\sigma$.

The simulation results presented in

Table 2. Results of variable screening, variable selection, and estimation for simulation when $\sigma =0.5$. The values in the parentheses are the robust standard deviations. M1: our proposed multi-step sparse boosting method; M2: except sparse boosting, $l_2$-boosting are used in each step; M3: except sparse boosting, lasso regression are used in each step; M4: except sparse boosting, elastic net regression are used in each step. S: coverage probability that the top $\lfloor {\displaystyle \frac{n}{log\left(n\right)}}\rfloor$ covariates after screening includes all important covariates; TP: the median of true positive; FP: the median of false positive; Size: the median of model sizes; ISPE: the average of mean square error; RMISE: the average of root mean integrated squared error; Bias($\rho$): the mean bias of $\rho$; Bias($\sigma$): the mean bias of $\sigma$. The values in the parentheses are the robust standard deviations; Simulation based on 500 replicates.

n	$\mathit{\rho }$	Method	S	TP	FP	Size	ISPE	RMISE	Bias ($\mathit{\rho }$)	Bias ($\mathit{\sigma }$)
100	0.2	M1	0.88 (0.32)	4 (0.45)	4 (2.31)	8 (2.31)	1.123 (1.640)	0.754 (0.538)	−0.025 (0.240)	0.393 (0.538)
		M2	0.88 (0.32)	4 (0.20)	12 (2.19)	16 (2.18)	1.212 (1.578)	0.782 (0.496)	−0.032 (0.216)	0.461 (0.507)
		M3	0.88 (0.32)	4 (0.20)	15 (2.42)	19 (2.49)	1.235 (1.500)	0.802 (0.499)	−0.031 (0.228)	0.486 (0.496)
		M4	0.88 (0.32)	4 (0.16)	16 (1.66)	20 (1.70)	1.336 (1.527)	0.843 (0.495)	−0.029  (0.216)	0.535 (0.486)
	0.5	M1	0.92 (0.28)	4 (0.28)	4 (2.50)	8 (2.43)	1.320 (1.657)	0.694 (0.444)	−0.002 (0.112)	0.331 (0.447)
		M2	0.92 (0.28)	4 (0.04)	12 (2.43)	16 (2.43)	1.495 (1.764)	0.744 (0.442)	0.004 (0.118)	0.420 (0.453)
		M3	0.92 (0.28)	4 (0.06)	15 (2.36)	19 (2.36)	1.567 (1.796)	0.773 (0.449)	0.001 (0.109)	0.451 (0.450)
		M4	0.92 (0.28)	4 (0.05)	16 (1.82)	20  (1.82)	1.609 (1.688)	0.797 (0.428)	0.001 (0.107)	0.486 (0.426)
	0.8	M1	0.99 (0.12)	4 (0.08)	4 (1.94)	8  (1.93)	3.127 (2.182)	0.578 (0.197)	0.005 (0.049)	0.204 (0.191)
		M2	0.99 (0.12)	4 (0)	12 (2.29)	16 (2.29)	3.405 (2.099)	0.623 (0.175)	0.006 (0.039)	0.284 (0.158)
		M3	0.99 (0.12)	4 (0)	14 (2.24)	18 (2.24)	3.443 (2.061)	0.641 (0.165)	0.006 (0.038)	0.310 (0.147)
		M4	0.99 (0.12)	4 (0)	16 (1.69)	20 (1.69)	3.64 (2.096)	0.670 (0.167)	0.006 (0.038)	0.354 (0.151)
400	0.2	M1	1 (0)	4 (0.75)	5 (3.10)	9 (3.38)	1.116 (1.674)	0.630 (0.450)	−0.018 (0.691)	0.402 (0.512)
		M2	1 (0)	4 (0)	33 (4.28)	37 (4.28)	1.023 (1.199)	0.589 (0.068)	−0.038 (0.649)	0.376 (0.294)
		M3	1 (0)	4 (0)	52 (8.40)	56 (8.40)	0.841 (1.199)	0.410 (0.153)	−0.039  (0.635)	0.276 (0.329)
		M4	1 (0)	4 (0)	55 (8.24)	59 (8.24)	0.827 (1.008)	0.443 (0.135)	−0.055 (0.604)	0.299 (0.287)
	0.5	M1	1 (0)	4 (0.62)	5 (2.91)	9 (3.12)	1.370 (1.892)	0.596 (0.376)	−0.009 (0.416)	0.360 (0.433)
		M2	1 (0)	4 (0)	33 (4.19)	37 (4.19)	1.261 (1.510)	0.580 (0.050)	−0.026 (0.308)	0.338 (0.166)
		M3	1  (0)	4 (0)	52 (8.55)	56 (8.55)	0.982 (1.156)	0.389 (0.124)	−0.020 (0.331)	0.237 (0.236)
		M4	1 (0)	4 (0)	55 (8.72)	59 (8.72)	1.068 (1.273)	0.427 (0.122)	−0.022 (0.328)	0.268 (0.219)
	0.8	M1	1 (0)	4 (0)	5 (2.25)	9 (2.25)	2.437 (1.593)	0.534 (0.049)	−0.046 (0.134)	0.284 (0.204)
		M2	1 (0)	4 (0)	33 (4.34)	37 (4.34)	2.720 (1.892)	0.579 (0.046)	−0.021 (0.134)	0.331 (0.200)
		M3	1 (0)	4 (0)	52 (8.62)	56 (8.62)	2.299 (1.820)	0.388 (0.116)	−0.023 (0.157)	0.233 (0.279)
		M4	1 (0)	4 (0)	56 (8.83)	60 (8.83)	2.377 (1.832)	0.417 (0.095)	−0.028 (0.109)	0.245 (0.180)

and

Table 3. Results of variable screening, variable selection, and estimation for simulation when $\sigma =1$. The values in the parentheses are the robust standard deviations. M1: our proposed multi-step sparse boosting method; M2: except sparse boosting, $l_2$-boosting are used in each step; M3: except sparse boosting, lasso regression are used in each step; M4: except sparse boosting, elastic net regression are used in each step. S: coverage probability that the top $\lfloor {\displaystyle \frac{n}{log\left(n\right)}}\rfloor$ covariates after screening includes all important covariates; TP: the median of true positive; FP: the median of false positive; Size: the median of model sizes; ISPE: the average of mean square error; RMISE: the average of root mean integrated squared error; Bias ($\rho$): the mean bias of $\rho$; Bias ($\sigma$): the mean bias of $\sigma$. The values in the parentheses are the robust standard deviations; Simulation based on 500 replicates.

n	$\mathit{\rho }$	Method	S	TP	FP	Size	ISPE	RMISE	Bias ($\mathit{\rho }$)	Bias ($\mathit{\sigma }$)
100	0.2	M1	0.85 (0.36)	4 (0.38)	6 (2.65)	10 (2.67)	2.280 (1.853)	1.000 (0.537)	−0.079 (0.329)	0.402 (0.503)
		M2	0.85 (0.36)	4 (0.15)	14 (1.91)	17 (1.92)	2.586 (1.790)	1.037 (0.493)	−0.072 (0.309)	0.523 (0.470)
		M3	0.85 (0.36)	4 (0.17)	15 (2.14)	19 (2.20)	2.796 (1.752)	1.128 (0.489)	−0.074 (0.319)	0.598 (0.462)
		M4	0.85 (0.36)	4 (0.15)	16 (1.73)	20 (1.79)	2.948 (1.726)	1.173 (0.479)	−0.074 (0.316)	0.650 (0.452)
	0.5	M1	0.90 (0.30)	4 (0.19)	6 (2.49)	10 (2.44)	2.656 (1.762)	0.921 (0.429)	−0.004 (0.186)	0.335 (0.387)
		M2	0.90 (0.30)	4 (0.10)	14 (1.85)	18 (1.84)	3.067 (1.799)	0.980 (0.413)	0.003 (0.174)	0.471 (0.371)
		M3	0.90 (0.30)	4 (0.08)	16 (1.80)	20 (1.80)	3.323 (1.865)	1.070 (0.418)	0.003 (0.184)	0.544 (0.374)
		M4	0.90 (0.30)	4 (0.05)	16 (1.44)	20 (1.44)	3.446 (1.726)	1.116 (0.408)	0.001 (0.181)	0.594 (0.360)
	0.8	M1	1 (0.06)	4 (0)	6 (2.16)	10 (2.16)	5.771 (2.019)	0.782 (0.186)	0.002 (0.058)	0.190 (0.164)
		M2	1 (0.06)	4 (0)	13 (1.76)	17 (1.76)	6.277 (1.939)	0.824 (0.162)	0.004 (0.056)	0.313 (0.159)
		M3	1 (0.06)	4 (0)	15 (1.92)	19 (1.92)	6.567 (1.921)	0.906 (0.172)	0.008 (0.056)	0.373 (0.162)
		M4	1 (0.06)	4 (0)	16 (1.37)	20 (1.37)	6.681 (1.848)	0.943 (0.179)	0.006 (0.055)	0.424 (0.168)
400	0.2	M1	1 (0)	4 (0.65)	12 (4.03)	16 (4.38)	2.327 (1.895)	0.715 (0.386)	−0.051 (1.052)	0.440 (0.526)
		M2	1 (0)	4 (0)	42 (3.43)	46 (3.43)	2.367 (1.498)	0.715 (0.089)	−0.012 (0.939)	0.451 (0.390)
		M3	1 (0)	4 (0)	45 (12.00)	49 (12.00)	2.245 (1.363)	0.727 (0.157)	−0.005 (1.020)	0.460 (0.418)
		M4	1 (0)	4 (0)	47 (10.79)	51 (10.79)	2.304 (1.337)	0.762 (0.151)	−0.036 (1.047)	0.494 (0.432)
	0.5	M1	1 (0)	4 (0.43)	12 (3.86)	16 (4.01)	2.484 (1.819)	0.665 (0.258)	−0.080 (0.594)	0.361 (0.416)
		M2	1 (0)	4 (0)	41 (3.60)	45 (3.60)	2.635 (1.733)	0.714 (0.092)	−0.069 (0.599)	0.444 (0.411)
		M3	1 (0)	4 (0)	44 (11.66)	48 (11.66)	2.579 (1.601)	0.715 (0.154)	−0.085 (0.648)	0.432 (0.407)
		M4	1 (0)	4 (0)	46 (10.75)	50 (10.75)	2.650 (1.606)	0.751 (0.148)	−0.039 (0.613)	0.453 (0.387)
	0.8	M1	1 (0)	4 (0)	12 (3.38)	16 (3.38)	4.800 (1.832)	0.628 (0.081)	−0.107 (0.227)	0.330 (0.382)
		M2	1 (0)	4 (0)	41 (3.81)	45 (3.81)	5.042 (1.664)	0.687 (0.056)	−0.111 (0.195)	0.412 (0.347)
		M3	1 (0)	4 (0)	44 (12.22)	48 (12.22)	5.101 (1.827)	0.682 (0.120)	−0.109 (0.236)	0.415 (0.401)
		M4	1 (0)	4 (0)	45 (10.98)	49 (10.98)	5.152 (1.787)	0.716 (0.102)	−0.105 (0.229)	0.432 (0.376)

, based on 500 replications, provide comprehensive insights into the performance of various methods for variable screening and selection under two conditions ($\sigma =0.5$ and $\sigma =1.0$).

At $\sigma =0.5$, M1 consistently achieves high coverage probabilities, reaching 0.99 when $\rho =0.8$, while maintaining a true positive count of 4 and significantly reducing the false positive rates compared to other methods. For example, M2 has a maximum of 12 false positives when $n=100$ and $\rho =0.2$, while M3 and M4 peak at 15 and 16 false positives, respectively. In terms of in-sample prediction error (ISPE), M1 demonstrates robust performance with values ranging from 1.123 to 3.127. M2, M3, and M4 show higher ISPE values, indicating a tendency toward overfitting, with M2 reaching 3.405 at $n=100$ and $\rho =0.8$, M3 reaching 3.443, and M4 reaching 3.64. M1 also excels in root mean integrated squared error (RMISE), with values between 0.534 and 0.754 at $n=100$ and $n=400$, while M2 ranges from 0.579 to 0.782. At $n=100$, M3 and M4 have even higher values, further demonstrating M1’s effectiveness in making accurate predictions.

At $\sigma =1.0$, similar trends are observed with the performance of M1 being even more pronounced. For instance, M1 maintains a coverage probability of 1.00 when $\rho =0.8$. The ISPE for M1 ranges from 2.280 to 5.771, while M4’s ISPE peaks at 6.681, highlighting M1’s advantage in prediction accuracy. Additionally, the false positive rates for M2, M3, and M4 increase, reaching more than 40 at $n=400$, further affirming M1’s superior performance. Despite maintaining a median true positive count of 4, M1 shows lower bias in parameter estimation, with Bias($\rho$) ranging from −0.107 to 0.002 compared to higher biases in the other methods.

In summary, the multi-step sparse boosting method (M1) outperforms alternative approaches in variable selection and parameter estimation, excelling in maintaining lower false positive rates and superior estimation accuracy. This establishes the multi-step sparse boosting method as a highly effective method for high-dimensional data analysis, particularly in scenarios involving complex relationships among variables.

4. Real Data Analysis

In this section, we apply the proposed method to the Boston housing price data, originally collected by Harrison and Rubinfield [53] and integrated by Gilley and Pace [54]. The dataset is available for download from https://lib.stat.cmu.edu/datasets/boston (accessed on 23 February 2025). It comprises median house prices observed in 506 census tracts in the Boston area in 1970, alongside a set of variables presumed to influence house prices.

Table 4. Description of the variables in Boston housing data.

Variable	Varaible Description
MEDV	Median value of owner-occupied housing expressed in USD 1000’s
CRIM	Per capita murder rate by town
ZN	Proportion of residential land zoned for lots over 25,000 square feet
B	Proportion of Black residents by town
RM	Average number of rooms per dwelling
DIS	Weighted distances to five Boston employment centers
NOX	Nitric oxides concentration (parts per 10 millions) per town
AGE	Proportion of owner-occupied units built before 1940
INDUS	Proportion of non-retail business acres per town
RAD	Index of accessibility to radial highways per town
PTRATIO	Pupil-teacher ratio by town
LSTAT	Percentage of lower status population
TAX	Full-value property tax rate per USD 10,000
CHAS	Charles River dummy variable (1 if tract bounds river; 0 otherwise)

provides a detailed description of these variables:

The Boston housing price dataset is widely used in spatial econometrics and has been extensively studied in the literature. For the same data set, Xie et al. [55] adopted spatial autoregressive model, Li and Mei [14] considered partially linear spatial autoregressive model and Du et al. [17] used partially linear additive spatial autoregressive models. In the following, we will consider the additive spatial autoregressive model

$$\begin{array}{cc}\hfill {\mathrm{MEDV}}_i=& \rho \sum _{j=1}^{506}{w}_{ij}{\mathrm{MEDV}}_j+g_1\left(\mathrm{CRIM}\right)+g_2\left(\mathrm{ZN}\right)+g_3\left(\mathrm{B}\right)+g_4\left(\mathrm{RM}\right)+g_5\left(\mathrm{DIS}\right)\hfill \\ \hfill & +g_6\left(\mathrm{NOX}\right)+g_7\left(\mathrm{AGE}\right)+g_8\left(\mathrm{INDUS}\right)+g_9\left(\mathrm{RAD}\right)+g_{10}\left(\mathrm{PTRATIO}\right)\hfill \\ \hfill & +g_{11}\left(\mathrm{LSTAT}\right)+g_{12}\left(\mathrm{TAX}\right)+{ϵ}_i,i=1,\cdots ,506.\hfill \end{array}$$

(18)

where ${\mathrm{MEDV}}_i$ represents the median house value for census tract i, $\rho$ is the spatial autoregressive parameter, $w_{ij}$ denotes the spatial weight matrix with entries set as $\max (1-{\displaystyle \frac{d_{ij}}{d_0}},0)$, where $d_{ij}$ is the Euclidean distance based on longitude and latitude coordinates of any two houses, $d_0$ is the threshold distance (set to $0.05$ following Su and Yang [5]), resulting in a weight matrix with 19.1% nonzero elements. For comparative analysis, we consider the following methods explored in our simulation study: multi-step sparse boosting, multi-step $l_2$-boosting, multi-step lasso and multi-step elastic net. These methods will be used to estimate the functions $g_j(.)$ and $\rho$, aiming to identify the significant determinants of housing prices while accounting for spatial dependencies among census tracts.

The results of variable selection, estimation and prediction are summarized in

Table 5. Boston housing price example: variable selection, estimation and prediction performance. No.: Number of variables selected; Variables: Names of selected variables; ISPE: In-sample prediction error ${\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{(Y_i-{\widehat{Y}}_i)}^2$; OSPE: 5-fold cross-validation out-of-sample prediction error.

Method	No.	Variables	ISPE	OSPE
multi-step sparse boosting	2	RM (3), LSTAT (9)	0.665	0.951
multi-step $l_2$ boosting	2	RM (3), LSTAT (9)	0.665	0.951
multi-step lasso	12	CRIM (1), B (2), RM (3), DIS (4), NOX (5), AGE (6), INDUS (7), PTRATIO (8), LSTAT (9), ZN (10), RAD (11), TAX (12)	0.172	1.197
multi-step elastic net	12	CRIM (1), B (2), RM (3), DIS (4), NOX (5), AGE (6), INDUS (7), PTRATIO (8), LSTAT (9), ZN (10), RAD (11), TAX (12)	0.160	1.232

. In addition to the number of selected variables, we also report the in-sample prediction error (ISPE) ${\sum }_{i=1}^n{(Y_i-{\widehat{Y}}_i)}^2$ and out-of-sample prediction error (OSPE) measured by 5-fold cross-validation.

From Table 5, we observe that, compared to traditional lasso and elastic net methods, the $l_2$ Boosting and Sparse $l_2$ Boosting methods for the varying-coefficient model exhibit smaller OSPE, while also producing relatively sparser models. The smaller ISPE may be due to overfitting in the traditional models which choosing all of the variables. These conventional models exhibit errors that increase by a factor of 6 to 7 during out-of-sample predictions, whereas our proposed model demonstrates greater stability with smaller fluctuations when comparing ISPE to OSPE. Our Sparse $L_2$ Boosting method for the varying-coefficient AFT model performs remarkably well in terms of sparsity, estimation, and prediction.

Additionally, we present a Venn diagram in Figure 2 showing the overlapping genes identified by all different methods. We observe that Sparse $L_2$ Boosting and $l_2$ Boosting select 2 variables (RM and LSTAT) which mostly affect the Boston housing prices in common, while lasso and elastic net methods identify all the 12 variables in common. However, Sparse$L_2$Boosting and $l_2$Boosting produce distinct lists of selected variables compared to the other methods, with only 2 variables selected by all methods.

We plot the estimated curves of varying coefficients for RM and LSTAT, which are the two selected variables by sparse boosting method, with their 95% confidence bands constructed by 500 bootstrap resamples in Figure 3. All of the functions are quite different from a straight line and this suggest that the varying-coefficient model is more appropriate to describe the covariate effects on boston housing prices in our data. Using more sophisticated semiparametric model specification may provide more accurate model for the high-dimensional analysis. Furthermore, we observe different functional forms for the two impact variables.

5. Discussion and Concluding Remarks

This paper introduces a useful multi-step sparse boosting algorithm specifically designed for additive spatial autoregressive models with high-dimensional covariates, addressing critical aspects of variable selection and parameter estimation. Our methodology effectively facilitates model-based prediction without the burden of time-consuming tuning parameter selection, thereby streamlining the modeling process.

Using B-spline basis functions for approximating varying coefficients is a critical component of our approach, which allows for smooth and accurate representations even when the underlying assumptions regarding smoothness may be violated, enhancing the robustness of the model. Moreover, a noteworthy aspect of our finding is the two-step sparse boosting approach employing the generalized Minimum Description Length (gMDL) model selection criterion. The gMDL criterion utilizes a data-driven penalty for squared loss, promoting model parsimony while effectively filtering out irrelevant covariates.

The simulation results illustrate the efficacy of our proposed approach. Our multi-step sparse boosting method consistently exhibits fewer false positives than alternative methods, while true positive rates accurately reflect the actual number of relevant variables. In contrast, traditional methods utilizing boosting, lasso, or elastic net tend to suffer from overfitting, as evidenced by lower in-sample prediction errors (ISPE) and larger model sizes. Meanwhile, the real-world application of our methodology to the Boston housing price data further confirms its effectiveness and stability in identifying significant predictors of housing prices. By accurately selecting key variables, our approach enhances the model’s predictive performance and interpretability.

In summary, sparse boosting has the following advantages. Firstly, sparse boosting adaptively selects variables and controls sparsity without requiring explicit regularization parameters (unlike LASSO and Elastic Net, which rely on tuning parameters). Secondly, sparse boosting can handle non-convex loss functions, which is particularly useful for complex models like the additive spatial autoregressive model. LASSO and Elastic Net, on the other hand, are limited to convex optimization. Thirdly, sparse boosting iteratively reduces bias while controlling variance, whereas LASSO and Elastic Net may introduce bias due to the L1 penalty. Last but not least, sparse boosting provides a more interpretable model by sequentially adding the most relevant variables, while LASSO and Elastic Net may shrink coefficients indiscriminately.

However, several limitations of our study must be acknowledged. Firstly, although B-spline basis functions offer significant advantages, they may not fully capture the intricate complexities of all data relationships, potentially limiting the model’s applicability to more diverse or nuanced datasets. Additionally, while our method maintains interpretability through iterative variable selection and additive decomposition of spatial effects, scenarios with highly complex spatial structures—such as those involving nonlinearities or hierarchical dependencies—may pose challenges. In such cases, advanced visualization techniques and uncertainty quantification methods could further enhance interpretability. Future research should explore these directions to improve the model’s applicability in more intricate spatial settings.

This method can be extended to various fields, not limited to environmental studies, healthcare, and social sciences. Additionally, it is applicable to both nonparametric and semiparametric models, although our focus here has been on nonparametric approaches. This broader analysis is crucial to comprehensively validate the model’s versatility and effectiveness across various contexts within the disciplines of econometrics and statistics.