Variable Selection of Heterogeneous Spatial Autoregressive Models via Double-Penalized Likelihood

Abstract

Heteroscedasticity is often encountered in spatial-data analysis, so a new class of heterogeneous spatial autoregressive models is introduced in this paper, where the variance parameters are allowed to depend on some explanatory variables. Here, we are interested in the problem of parameter estimation and the variable selection for both the mean and variance models. Then, a unified procedure via double-penalized quasi-maximum likelihood is proposed, to simultaneously select important variables. Under certain regular conditions, the consistency and oracle property of the resulting estimators are established. Finally, both simulation studies and a real data analysis of the Boston housing data are carried to illustrate the developed methodology.

1. Introduction

In the field of modern economics and statistics, the spatial autoregressive (SAR) model has always been a hot topic for economists and statisticians. Theories and methods for estimation as well as other inferences based on linear SAR models and their extensions have been deeply studied. Among them, there is a lot of wonderful literature based on linear SAR models, such as Cliff and Ord [1], Anselin [2], and Anselin and Bera [3]. Other research results for linear SAR models can also be referred to: Xu and Lee [4], Liu et al. [5], Xie et al. [6], Xie et al. [7], and so on. In order to capture the nonlinear relationship between response variables and some explanatory variables, a variety of semiparametric SAR models have recently been proposed and studied in depth. For example, based on a partially linear spatial autoregressive model, Su and Jin [8] developed a profile quasi-maximum likelihood-estimation method and discussed the asymptotic properties of the obtained estimators. Du et al. [9] developed partially linear additive spatial autoregressive models and proposed an estimation method via combining the spline approximations with an instrumental-variables-estimation method. Cheng and Chen [10] discussed partially linear single-index spatial autoregressive model as well as obtained the consistency and asymptotic normality of the proposed estimators, under some mild conditions. Other research results of semiparametric SAR models can also be seen in Wei et al. [11], Hu et al. [12], and so on. Previous research on various SAR models focused mainly on the homoskedasticity assumption, that the variance of the unobservable error, conditional on the explanatory variables, is constant. It is well-known that if the innovations are heteroskedastic, most existing statistical inference methods under homoskedasticity assumption could lead to incorrect inference, see Lin and Lee [13]. Therefore, many researchers want to relax the homoskedasticity assumption for spatial autoregressive models, by allowing different variances for each unobservable error. For example, Dai et al. [14] developed a Bayesian local-influence analysis for heterogeneous spatial autoregressive models. However, the variance terms in these above-mentioned heterogeneous spatial autoregressive models are assumed fixed and do not depend on the regression variables. Furthermore, in many application fields, such as economics and quality management, it is a topic of interest to model the variance itself, which is helpful to identify the factors that affect the variability in the observations. Thus, we intend to propose a class of heterogeneous spatial autoregressive models where the variance parameters are modelled in terms of covariates.

In addition, many authors have done a lot of research on joint mean and variance models. For example, Wu and Li [15] proposed a variable-selection procedure via a penalized maximum likelihood for the joint mean and dispersion models of the inverse Gaussian distribution. Xu and Zhang [16] discussed a Bayesian estimation for semiparametric joint mean and variance models, based on B-spline approximations of nonparametric components. Zhao et al. [17] studied variable selection for beta-regression models with varying dispersion, where both the mean and the dispersion are modeled by explanatory variables. Li et al. [18] proposed an efficient unified variable selection of the joint location, scale, and skewness models, based on a penalized likelihood method. Zhang et al. [19] developed a Bayesian quantile-regression analysis for semiparametric mixed-effects double-regression models, on the basis of the asymmetric Laplace distribution for the errors.

As we all know, variable selection is the most important problem to be considered in regression analysis. In practice, a large number of variables should be used in the initial analysis, but many of them may not be important and should be excluded from the final model, in order to improve the accuracy of the prediction. When the number of predictive variables is large, traditional variable selection methods such as stepwise regression and best-subset selection are not computationally feasible. Therefore, various shrinkage methods have been proposed in recent years and have gained much attention, such as the LASSO (Tibshirani [20]), the adaptive LASSO (Zou [21]), and the SCAD (Fan and Li [22]). Based on these shrinkage methods, the variable selection for SAR models (see Liu et al. [5]; Xie et al. [6]; Xie et al. [7]; Luo and Wu [23]) and the variable selection for other models without spatial dependence (see, for example, Li and Liang [24]; Zhao and Xue [25]; Tian et al. [26]) have been studied extensively in recent years. To the best of our knowledge, most existing variable-selection methods in spatial-data analysis are limited, to only select the mean explanatory variables in the literature, and little work has been done to select the variance explanatory variables.

Therefore, in this paper we aim to make variable selection of heterogeneous spatial autoregressive models (heterogeneous SAR models), based on penalized quasi-maximum likelihood, using different penalty functions. The proposed variable-selection method simultaneously selects important explanatory variables in a mean model and a variance model. Furthermore, it can be proven that this variable selection procedure is consistent, and the obtained estimators of regression coefficients have oracle property under certain regular conditions. This indicates that the penalized estimators work as well as if the subset of true zero coefficients were already known. A simulation and a real data analysis of a Boston housing price are used to illustrate the proposed variable selection method.

The outline of the paper is organized as follows. Section 2 introduces new heterogeneous spatial autoregressive models. Then, we propose a unified variable-selection procedure for the joint models via the double-penalized quasi-maximum likelihood method. Section 3 gives the theoretical results of the resulting estimators. The computation of the penalized quasi-maximum likelihood estimator as well as the choice of the tuning parameters are presented in Section 4. The finite-sample performance of the method is investigated, based on some simulation studies, in Section 5. Section 6 gives a real data analysis of a Boston housing price to illustrate the proposed method. Some conclusions as well as a brief discussion are given in Section 7. Some assumptions and the technical proofs of all the asymptotic results are provided in Appendix A.

2. Variable Selection via Penalized Quasi-Maximum Likelihood

2.1. Heterogeneous SAR Models

The classical spatial autoregressive models have the following form:

$$Y=\rho WY+X\beta +\epsilon ,$$

(1)

where $Y={(y_1,y_2,\cdots ,y_n)}^T$ is an n-dimensional observation vector on the dependent variable; $\left|\rho \right|<1$ is an unknown spatial parameter; and W is a specified spatial weight matrix of known constants with zero diagonal elements. Let $X={(x_1,x_2,\cdots ,x_n)}^T$ is an $n\times p$ matrix whose ith row $x_i^T=(x_{i1},\cdots ,x_{ip})$ is the observation of the explanatory variables and $\beta ={({\beta }_1,\cdots ,{\beta }_p)}^T$ be a $p\times 1$ vector of unknown regression parameters in the mean model; $\epsilon$ is an n-dimensional vector of independent identically distributed disturbances with zero mean and finite variance ${\sigma }^2$.

Furthermore, similar to Xu and Zhang [16], we consider variance heterogeneity in the models and assume an explicit variance modeling related to other explanatory variables, that is:

$${\sigma }_i^2=h\left(z_i^T\gamma \right),$$

(2)

where $z_i^T=(z_{i1},\cdots ,z_{iq})$ is the observation of explanatory variables associated with the variance of $y_i$ and $\gamma ={({\gamma }_1,\cdots ,{\gamma }_q)}^T$ is a $q\times 1$ vector of regression parameters in the variance model. There might be some components of $z_i$ which coincide with some $x_i$’s. In addition, $h(·)$ is a known function, for the identifiability of the models, so we always assume that $h(·)$ is a monotone function and $h(·)>0$, which takes into account the positiveness of the variance. For example, the researcher can take $h\left(x\right)=exp\left(x\right)$ in general. So, this paper considers the following heterogeneous SAR models:

$$\left\{\begin{array}{c}Y=\rho WY+X\beta +\epsilon ,\hfill \\ \Sigma =\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_n^2)\hfill \\ {\sigma }_i^2=exp\left(z_i^T\gamma \right),\hfill \\ i=1,2,\cdots ,n.\hfill \end{array}\right.$$

(3)

According to the idea of the quasi-maximum likelihood estimators (Lee [27]), the log-likelihood function of the model (3) is

$${\ell }_n(\rho ,\beta ,\gamma |Y,X,Z)=-\frac{n}{2}ln\left(2\pi \right)-\frac{1}{2}\sum _{i=1}^n{z}_i^T\gamma +ln\left|S\right|-\frac{1}{2}{e}^T{\Sigma }^{-1}e,$$

(4)

where $e=SY-X\beta ,S=I_n-\rho W$, and $I_n$ is an $(n\times n)$ identity matrix.

2.2. Penalized Quasi-Maximum Likelihood

In order to obtain the desired sparsity in the resulting estimators, we propose the penalized quasi-maximum likelihood

$$\mathcal{L}(\rho ,\beta ,\gamma )={\ell }_n(\rho ,\beta ,\gamma )-n\sum _{j=1}^p{p}_{{\lambda }_j^{\left(1\right)}}\left(\right|{\beta }_j\left|\right)-n\sum _{k=1}^q{p}_{{\lambda }_k^{\left(2\right)}}\left(\right|{\gamma }_k\left|\right).$$

(5)

For notational simplicity, we rewrite (5) in the following

$$\mathcal{L}\left(\theta \right)={\ell }_n\left(\theta \right)-n\sum _{j=2}^s{p}_{{\lambda }_n}\left(\right|{\theta }_j\left|\right),$$

(6)

where $\theta ={({\theta }_1,\cdots ,{\theta }_s)}^T={(\rho ,{\beta }_1,\cdots ,{\beta }_p;{\gamma }_1,\cdots ,{\gamma }_q)}^T$ with $s=p+q+1$, and $p_{{\lambda }_n}(·)$ is a given-penalty function with the tuning parameters ${\lambda }_n$. The data-driven criteria, such as cross validation (CV), generalized cross-validation (GCV), or the BIC-type tuning-parameter selector (see Wang et al. [28]), can be used to choose the tuning parameters, which is described in Section 4. Here we use the same penalty function $p(·)$ for all the regression coefficients but with different tuning parameters ${\lambda }_n$, for the mean parameters and the variance parameters, respectively. Note that the penalty functions and tuning parameters are not necessarily the same for all the parameters. For example, we wish to keep some important variables in the final model and, therefore, do not want to penalize their coefficients. In this paper, we mainly use the smoothly clipped absolute deviation (SCAD) penalty, with a first derivative that satisfies

$$p_{\lambda }^{\prime }\left(t\right)=\lambda \left\{I(t\le \lambda )+\frac{{(a\lambda -t)}_{+}}{(a-1)\lambda }I(t>\lambda )\right\}$$

in which $a=3.7$ is taken in our work. The details about SCAD can be seen in Fan and Li [22].

The penalized quasi-maximum likelihood estimator of $\theta$ is denoted by $\widehat{\theta }$, which maximizes the function $\mathcal{L}\left(\theta \right)$ in (6) with respect to $\theta$. The technical details and an algorithm for calculating the penalized quasi-maximum likelihood estimator $\widehat{\theta }$ are provided in Section 4.

3. Asymptotic Properties

We next study the asymptotic properties of the resulting penalized quasi-maximum likelihood estimators. We first introduce some notations. Let ${\theta }_0$ denote the true value of $\theta$. Furthermore, let ${\theta }_0={({\theta }_{01},\cdots ,{\theta }_{0s})}^T={({\left({\theta }_0^{\left(1\right)}\right)}^T,{\left({\theta }_0^{\left(2\right)}\right)}^T)}^T.$ For ease of presentation and without loss of generality, it is assumed that ${\theta }_0^{\left(1\right)}$ is an $s_1\times 1$ nonzero regression coefficient and that ${\theta }_0^{\left(2\right)}=0$ is an $(s-s_1)\times 1$ vector of the zero-valued regression coefficient. Let

$$a_n=\underset{2\le j\le s}{max}\left\{\right|p_{{\lambda }_n}^{\prime }\left(\right|{\theta }_{0j}\left|\right)|,{\theta }_{0j}\ne 0\}$$

and

$$b_n=\underset{2\le j\le s}{max}\{p_{{\lambda }_n}^{″}\left(\right|{\theta }_{0j}\left|\right):{\theta }_{0j}\ne 0\}.$$

Theorem 1.

Suppose that $a_n=O_p\left(n^{-\frac{1}{2}}\right),b_n\to 0$, and ${\lambda }_n\to 0$ as $n\to \infty .$ Under the conditions $\left(C1\right)$–$\left(C10\right)$ in Appendix A, with probability tending to 1 there must exist a local maximizer $\widehat{\theta }$ of the penalized quasi-maximum likelihood function $\mathcal{L}\left(\theta \right)$ in (6), such that $\widehat{\theta }$ is a $\sqrt{n}$-consistent estimator of ${\theta }_0$.

The following theorem gives the asymptotic normality of $\widehat{\theta }$. Let

$$A_n=\mathrm{diag}(0,p_{{\lambda }_n}^{″}\left(\right|{\theta }_{01}^{\left(1\right)}\left|\right),\cdots ,p_{{\lambda }_n}^{″}\left(\right|{\theta }_{0s_1}^{\left(1\right)}\left|\right)),$$

$$B_n=(0,p_{{\lambda }_n}^{\prime }\left(\right|{\theta }_{01}^{\left(1\right)}\left|\right)\mathrm{sgn}\left({\theta }_{01}^{\left(1\right)}\right),\cdots ,p_{{\lambda }_n}^{\prime }\left(\right|{\theta }_{0s_1}^{\left(1\right)}\left|\right)\mathrm{sgn}\left({\theta }_{0s_1}^{\left(1\right)}\right){)}^T,$$

where ${\theta }_{0j}^{\left(1\right)}$ is the jth component of ${\theta }_0^{\left(1\right)}$$(1\le j\le s_1)$.

Furthermore, denote

$$\begin{array}{ccc}\hfill {\mathcal{I}}_n\left({\theta }_0\right)& =& -E\left(\frac{1}{n}\frac{{\partial }^2{\ell }_n\left({\theta }_0\right)}{\partial \theta \partial {\theta }^T}\right)\hfill \\ & =& \left(\begin{array}{ccc}\frac{1}{n}{\left[G_{n}X{\beta }_0\right)}^T{\Sigma }^{-1}\left(G_{n}X{\beta }_0\right)+tr\left(G_n^s{G}_n\right)]& \ast & \ast \\ \frac{1}{n}{X}^T{\Sigma }^{-1}\left(G_{n}X{\beta }_0\right)& \frac{1}{n}{X}^T{\Sigma }^{-1}X& 0\\ \frac{1}{n}\sum _{i=1}^n{g}_{n,ii}{Z}_i^T& 0& \frac{1}{2n}{Z}^{T}Z\end{array}\right.,\hfill \end{array}$$

$$\begin{array}{c}\hfill {\mathcal{J}}_n\left({\theta }_0\right)=\left(\begin{array}{ccc}{\mathcal{J}}_1& \ast & \ast \\ \frac{1}{n}{X}^T{\Sigma }^{-2}{G}_n{\mu }_3& 0& \frac{1}{2n}{X}^T\mathrm{diag}\left({\mu }_3\right){\Sigma }^{-2}Z\\ {\mathcal{J}}_2& \ast & \frac{1}{4n}[Z^T{\Sigma }^{-2}\mathrm{diag}\left({\mu }_4\right)Z-Z^{T}Z]\end{array}\right.,\end{array}$$

where ${\mathcal{J}}_1=\frac{1}{n}[{\mu }_4^T{\Sigma }^{-2}{1}_{n}tr\left(G_n^2\right)+2{\left(G_{n}X{\beta }_0\right)}^T{G}_n{\Sigma }^{-2}{\mu }_3-3tr\left(G_n^2\right)]$,

${\mathcal{J}}_2=\frac{1}{2n}[Z^T{\Sigma }^{-2}\mathrm{diag}\left({\mu }_3\right)\left(G_{n}X{\beta }_0\right)+Z^T{\Sigma }^{-2}{G}_n{\mu }_4-3{\sum }_{i=1}^n{g}_{n,ii}{Z}_i^T]$,

$G_n=W{A}^{-1}$, and $G_n^s=G_n+G_n^T$, thus, ${\mathcal{I}}_n\left({\theta }_0\right)$ is called as the average Hessian matrix (the information matrix when $\left\{e_i\right\}$ are normal). In addition, ${\mathcal{J}}_n\left({\theta }_0\right)$ is a symmetric matrix, $1_n$ is an n-dimensional column vector of ones, ${\mu }_j=E\left(e_i^j\right),j=2,3,4$, $G_{in}$ is the ith row of $G_n$, and $g_{n,ij}$ is the $(i,j)$th entry of $G_n$. Besides, $\mathrm{diag}\left(\mathrm{C}\right)$ represents a diagonal matrix with C, where C is an arbitrary vector.

Theorem 2.

Suppose that the penalty function $p_{\lambda }\left(t\right)$ satisfies

$$\underset{n\to \infty }{lim\; inf}\underset{t\to 0^{+}}{lim\; inf}\frac{p_{{\lambda }_n}^{\prime }\left(t\right)}{{\lambda }_n}>0,$$

and under the same mild conditions as these given in Theorem 1, if ${\lambda }_n\to 0$ and $\sqrt{n}{\lambda }_n\to \infty$ as $n\to \infty ,$ and $\mathcal{J}\left({\theta }_0\right)={lim}_{n\to \infty }{\mathcal{J}}_n\left({\theta }_0\right)$, then the $\sqrt{n}$-consistent estimator $\widehat{\theta }={({\left({\widehat{\theta }}^{\left(1\right)}\right)}^T,{\left({\widehat{\theta }}^{\left(2\right)}\right)}^T)}^T$ in Theorem 1 must satisfy

(i)

${\widehat{\theta }}^{\left(2\right)}=0$ with probability tending to 1.

(ii)

$\sqrt{n}({\mathcal{I}}_n^{\left(1\right)}\left({\theta }_0^{\left(1\right)}\right)+A_n)\{({\widehat{\theta }}^{\left(1\right)}-{\theta }_0^{\left(1\right)})+{({\mathcal{I}}_n^{\left(1\right)}\left({\theta }_0^{\left(1\right)}\right)+A_n)}^{-1}{B}_n\}\stackrel{\mathcal{L}}{⟶}{\mathcal{N}}_{s_1}(0,{\mathcal{I}}_1\left({\theta }_0^{\left(1\right)}\right)+{\mathcal{J}}_1\left({\theta }_0^{\left(1\right)}\right)),$

where ${\mathcal{I}}_n^{\left(1\right)}\left({\theta }_0^{\left(1\right)}\right)$, ${\mathcal{I}}_1\left({\theta }_0^{\left(1\right)}\right)$, and ${\mathcal{J}}_1\left({\theta }_0^{\left(1\right)}\right)$ are the first super-left submatrix of ${\mathcal{I}}_n\left({\theta }_0\right)$, $\mathcal{I}\left({\theta }_0\right)={lim}_{n\to \infty }{\mathcal{I}}_n\left({\theta }_0\right)$ and $\mathcal{J}\left({\theta }_0\right)$. In addition, “$\stackrel{\mathcal{L}}{⟶}$” denotes convergence in distribution.

4. Computation

4.1. Algorithm

Since $\mathcal{L}\left(\theta \right)$ is irregular at the origin, the commonly used gradient method is not applicable. Now, an iterative algorithm is developed based on the local quadratic approximation of the penalty function $p_{{\lambda }_n}(·)$, as in Fan and Li [22].

Firstly, note that the first two derivatives of the log-likelihood function ${\ell }_n\left(\theta \right)$ are continuous. Around a fixed point ${\theta }_0={({\rho }_0,{\beta }_0^T,{\gamma }_0^T)}^T$, we approximate the log-likelihood function by

$${\ell }_n\left(\theta \right)\approx {\ell }_n\left({\theta }_0\right)+{\left[\frac{\partial {\ell }_n\left({\theta }_0\right)}{\partial \theta }\right]}^T(\theta -{\theta }_0)+\frac{1}{2}{(\theta -{\theta }_0)}^T\left[\frac{{\partial }^2{\ell }_n\left({\theta }_0\right)}{\partial \theta \partial {\theta }^T}\right](\theta -{\theta }_0).$$

Moreover, given an initial value $t_0$, the penalty function $p_{{\lambda }_n}^{\prime }\left(t\right)$ can be approximated by a quadratic function

$$p_{{\lambda }_n}\left(\right|t\left|\right)\approx p_{{\lambda }_n}\left(\right|t_0\left|\right)+\frac{1}{2}\frac{p_{{\lambda }_n}^{\prime }\left(\right|t_0\left|\right)}{|t_0|}(t^2-t_0^2),\mathrm{for}t\approx t_0.$$

Therefore, we can approximate the penalized quasi-maximum likelihood function (6) by the following formula

$$\mathcal{L}\left(\theta \right)\approx {\ell }_n\left({\theta }_0\right)+{\left[\frac{\partial {\ell }_n\left({\theta }_0\right)}{\partial \theta }\right]}^T(\theta -{\theta }_0)+\frac{1}{2}{(\theta -{\theta }_0)}^T\left[\frac{{\partial }^2{\ell }_n\left({\theta }_0\right)}{\partial \theta \partial {\theta }^T}\right](\theta -{\theta }_0)-\frac{n}{2}{\theta }^T{A}_{n\lambda }\left({\theta }_0\right)\theta ,$$

where

$$\begin{array}{c}{A}_{n\lambda }\left({\theta }_0\right)=\mathrm{diag}\left\{0,\frac{p_{{\lambda }_{n1}^{\left(1\right)}}^{\prime }\left(\right|{\beta }_{01}\left|\right)}{|{\beta }_{01}|},\cdots ,\frac{p_{{\lambda }_{np}^{\left(1\right)}}^{\prime }\left(\right|{\beta }_{0p}\left|\right)}{|{\beta }_{0p}|},\frac{p_{{\lambda }_{n1}^{\left(2\right)}}^{\prime }\left(\right|{\gamma }_{01}\left|\right)}{|{\gamma }_{01}|},\cdots ,\frac{p_{{\lambda }_{nq}^{\left(2\right)}}^{\prime }\left(\right|{\gamma }_{0q}\left|\right)}{|{\gamma }_{0q}|}\right\},\hfill \end{array}$$

$\theta ={({\theta }_1,\cdots ,{\theta }_s)}^T={(\rho ,{\beta }_1,\cdots ,{\beta }_p;{\gamma }_1,\cdots ,{\gamma }_q)}^T$ and ${\theta }_0={({\theta }_{01},\cdots ,{\theta }_{0s})}^T={({\rho }_0,{\beta }_{01},\cdots ,{\beta }_{0p};{\gamma }_{01},\cdots ,{\gamma }_{0q})}^T.$ Accordingly, the quadratic maximization problem for $\mathcal{L}\left(\theta \right)$ leads to a solution iterated by

$${\theta }_{new}\approx {\theta }_0+{\left\{\frac{{\partial }^2{\ell }_n({\rho }_0,{\beta }_0,{\gamma }_0)}{\partial \theta \partial {\theta }^T}-n{A}_{n\lambda }\left({\theta }_0\right)\right\}}^{-1}\left\{n{A}_{n\lambda }\left({\theta }_0\right){\theta }_0-\frac{\partial {\ell }_n({\rho }_0,{\beta }_0,{\gamma }_0)}{\partial \theta }\right\}.$$

Secondly, based on the log-likelihood function (4) we can obtain the score functions

$$U\left(\theta \right)=\frac{\partial {\ell }_n(\rho ,\beta ,\gamma )}{\partial \theta }={(U_1^T\left(\rho \right),U_2^T\left(\beta \right),U_3^T\left(\gamma \right))}^T,$$

where ${\displaystyle U_1\left(\rho \right)=\frac{\partial {\ell }_n(\rho ,\beta ,\gamma )}{\partial \rho }=Y^T{W}^T{\Sigma }^{-1}(SY-X\beta )-tr\left(G\right)}$, ${\displaystyle U_2\left(\beta \right)=\frac{\partial {\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta }=X{\Sigma }^{-1}(SY-X^T\beta )}$, ${\displaystyle U_3\left(\gamma \right)=\frac{\partial {\ell }_n(\rho ,\beta ,\gamma )}{\partial \gamma }=-\frac{1}{2}\sum _{i=1}^n{z}_i+\frac{1}{2}\sum _{i=1}^n\frac{{({\overline{y}}_i-x_i^T\beta )}^2}{{\sigma }_i^2}{z}_i,}$ $SY={({\overline{y}}_1,{\overline{y}}_2,\dots ,{\overline{y}}_n)}^T$. Denote

$$H\left(\theta \right)=\frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \theta \partial {\theta }^T}=\left(\begin{array}{ccc}{\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \rho \partial {\rho }^T}}& {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \rho \partial {\beta }^T}}& {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \rho \partial {\gamma }^T}}\\ {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta \partial {\rho }^T}}& {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta \partial {\beta }^T}}& {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta \partial {\gamma }^T}}\\ {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \gamma \partial {\rho }^T}}& {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \gamma \partial {\beta }^T}}& {\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \gamma \partial {\gamma }^T}}\end{array}\right),$$

where ${\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \rho \partial {\rho }^T}=-Y^T{W}^T{\Sigma }^{-1}WY-tr\left(G^2\right)}$, ${\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta \partial {\beta }^T}=-X{\Sigma }^{-1}{X}^T}$, ${\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \gamma \partial {\gamma }^T}=-\frac{1}{2}\sum _{i=1}^n\frac{{({\overline{y}}_i-x_i^T\beta )}^2}{{\sigma }_i^2}{z}_i{z}_i^T}$, ${\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta \partial {\rho }^T}=-X^T{\Sigma }^{-1}WY}$,

${\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \beta \partial {\gamma }^T}=-\sum _{i=1}^n\frac{({\overline{y}}_i-x_i^T\beta )}{{\sigma }_i^2}{x}_i{z}_i^T}$, ${\displaystyle \frac{{\partial }^2{\ell }_n(\rho ,\beta ,\gamma )}{\partial \gamma \partial {\rho }^T}=-\sum _{i=1}^n{y}_{w_i}{e}_i{z}_i/{\sigma }_i^2}$, here, $Y_w=WY={(y_{w_1},y_{w_2},\dots ,y_{w_n})}^T$. Finally, we give the following Algorithm 1, which can summarize the computation of penalized quasi-maximum likelihood estimators of the parameters in the heterogeneous SAR Models.

Algorithm 1

Step 1. The ordinary quasi-maximum likelihood estimators (without penalty) ${\beta }^{\left(0\right)},{\gamma }^{\left(0\right)},{\rho }^{\left(0\right)}$ of $\beta ,\gamma ,\rho$ are taken as their initial values. Step 2. ${\theta }^{\left(m\right)}={\{{\rho }^{\left(m\right)},{{\beta }^{\left(m\right)}}^T,{{\gamma }^{\left(m\right)}}^T\}}^T$ are given as current values, then update them by $${\theta }^{(m+1)}={\theta }^{\left(m\right)}+{\left\{H\left({\theta }^{\left(m\right)}\right)-n{A}_{n\lambda }\left({\theta }^{\left(m\right)}\right)\right\}}^{-1}\left\{n{A}_{n\lambda }\left({\theta }^{\left(m\right)}\right){\theta }^{\left(m\right)}-U\left({\theta }^{\left(m\right)}\right)\right\}.$$ Step 3. Repeat Step 2 above until $|{\theta }^{(m+1)}-{\theta }^{\left(m\right)}|<ϵ$, where $ϵ$ is a given small number, such as ${10}^{-5}$.

4.2. Choosing the Tuning Parameters

The penalty function $p_{{\lambda }_n^{\left(l\right)}}(·)$ involves the tuning parameters ${\lambda }_n^{\left(l\right)}$ ($l=1,2$) that control the amount of the penalty. Many selection criteria, such as CV, GCV, and BIC selection can be used to select the tuning parameters. Wang et al. [28] suggested using a BIC for the SCAD estimator in linear models and partially linear models and proved that its model selection consistency property, i.e., the optimal parameter chosen by the BIC, can identify the true model with the probability tending to one. Hence, their suggestion will be adopted in this paper. Nevertheless, in real application, how to simultaneously select a total of $p+q$ shrinkage parameters $\{{\lambda }_{ni},i=1,\cdots ,p+q\}$ is challenging. To bypass this difficulty, we follow the idea of Li et al. [18], and simplify the tuning parameters as follows,

(i)

${\lambda }_{nj}^{\left(1\right)}=\frac{{\lambda }_n}{|{\tilde{\beta }}_j^{\left(0\right)}|},j=1,\cdots ,p;$

(ii)

${\lambda }_{nk}^{\left(2\right)}=\frac{{\lambda }_n}{|{\tilde{\gamma }}_k^{\left(0\right)}|},k=1,\cdots ,q$

where ${\tilde{\beta }}_j^{\left(0\right)}$ and ${\tilde{\gamma }}_k^{\left(0\right)}$ are, respectively, the jth element and kth element of the unpenalized estimates ${\tilde{\beta }}^{\left(0\right)}$ and ${\tilde{\gamma }}^{\left(0\right)}$. Consequently, the original $p+q$ dimensional problem about ${\lambda }_{ni}$ becomes a one-dimensional problem about ${\lambda }_n$. ${\lambda }_n$ can be selected according to the following BIC-type criterion

$$BI{C}_{{\lambda }_n}=-\frac{2}{n}{\ell }_n(\widehat{\rho },\widehat{\beta },\widehat{\gamma })+d{f}_{\lambda }\times \frac{log\left(n\right)}{n},$$

where $0\le d{f}_{\lambda }\le s$ is simply the number of nonzero coefficients of ${\widehat{\theta }}_{{\lambda }_n}$, and, here, ${\widehat{\theta }}_{{\lambda }_n}={(\widehat{\rho },{\widehat{\beta }}^T,{\widehat{\gamma }}^T)}^T$ is the estimate of $\theta$ for a given ${\lambda }_n$.

The tuning parameter can be obtained as

$${\widehat{\lambda }}_n=arg\underset{{\lambda }_n}{min}BI{C}_{{\lambda }_n}.$$

From our simulation studies, we found that this method works well.

5. Simulation Study

In this section we conduct a simulation study to assess the small sample performance of the proposed procedure. In this simulation, we choose m to be 5 and R to be 30, 40, 60, thus $n=R\times m$ to be 150, 200, and 300. X and Z are, respectively, generated from multivariate normal distribution with zero mean vector and covariance matrix being ${\Sigma }_0=\left(c_{ij}\right)$ where $c_{ij}={0.5}^{|i-j|},i=1,\cdots ,8,j=1,\cdots ,8$. Besides, let the spatial parameter $\rho =0.5,0,-0.5$, which represents different spatial dependencies; $\beta ={(3,1.8,2.2,0,0,0,0,0)}^T$ and the structure of the variance model is $log\left({\sigma }_i^2\right)=z_i^T\gamma$ with $\gamma ={(1,1,1,0,0,0,0,0)}^T$. In these simulation, the weight matrix is taken to be $W=I_R\otimes H_m,H_m=(l_m{l}_m^T-I_m)/(m-1),$ where $l_m$ is an m-dimensional vector with all elements being 1,⊗ means Kronecker product.

We generated $M=500$ random samples of size n = 150, 200, and 300, respectively. For each random sample, the proposed variable-selection method, based on penalized quasi-maximum likelihood with SCAD and ALASSO penalty functions, is considered. The unknown tuning parameters ${\lambda }^{\left(l\right)},(l=1,2)$ for the penalty function are chosen by the BIC criterion in the simulation. The average number of the estimated zero coefficients with 500 simulation runs is reported in Table 1, Table 2 and Table 3. Note that “C” in the tables means the average number of zero regression coefficients that are correctly estimated as zero, and “IC” depicts the average number of non-zero regression coefficients that are erroneously set to zero. The performance of estimators $\widehat{\beta }$, and $\widehat{\gamma }$ is assessed by the mean square error(MSE), defined as

$$\mathrm{MSE}\left(\widehat{\beta }\right)=E{(\widehat{\beta }-{\beta }_0)}^T(\widehat{\beta }-{\beta }_0),\mathrm{MSE}\left(\widehat{\gamma }\right)=E{(\widehat{\gamma }-{\gamma }_0)}^T(\widehat{\gamma }-{\gamma }_0).$$

Table 1. Variable selections for $\beta$ and $\gamma$ under different sample sizes when $\rho =0.5$.

		SCAD			ALASSO
$\mathit{\beta }$	n	MSE	C	IC	MSE	C	IC
	150	0.0043	4.9360	0	0.0041	4.9760	0
	200	0.0029	4.9480	0	0.0027	4.9900	0
	300	0.0016	4.9760	0	0.0015	4.9980	0
$\gamma$	n	MSE	C	IC	MSE	C	IC
	150	0.0903	4.8960	0.0040	0.0970	4.9000	0.0040
	200	0.0633	4.9040	0	0.0631	4.9400	0
	300	0.0342	4.9580	0	0.0352	4.9900	0

Table 2. Variable selections for $\beta$ and $\gamma$ under different sample sizes when $\rho =0$.

		SCAD			ALASSO
$\mathit{\beta }$	n	MSE	C	IC	MSE	C	IC
	150	0.0041	4.9180	0	0.0038	4.9700	0
	200	0.0027	4.9520	0	0.0026	4.9900	0
	300	0.0016	4.9760	0	0.0016	5.0000	0
$\gamma$	n	MSE	C	IC	MSE	C	IC
	150	0.0903	4.9040	0	0.0955	4.9080	0
	200	0.0632	4.9200	0	0.0645	4.9460	0
	300	0.0387	4.9260	0	0.0383	4.9760	0

Table 3. Variable selections for $\beta$ and $\gamma$ under different sample sizes when $\rho =-0.5$.

		SCAD			ALASSO
$\mathit{\beta }$	n	MSE	C	IC	MSE	C	IC
	150	0.0046	4.9080	0	0.0044	4.9560	0
	200	0.0032	4.9240	0	0.0030	4.9800	0
	300	0.0017	4.9720	0	0.0017	5.0000	0
$\gamma$	n	MSE	C	IC	MSE	C	IC
	150	0.0919	4.8780	0	0.1033	4.8860	0
	200	0.0600	4.9220	0	0.0636	4.9500	0
	300	0.0363	4.9420	0	0.0372	4.9800	0

The simulation results are reported in Table 1, Table 2 and Table 3.

From Table 1, Table 2 and Table 3, we can make the following observations: (i) as n increases, the performances of variable-selection procedures become better and better. For example, the values in the column labeled ‘MSE’ of $\beta$ and $\gamma$ become smaller and smaller; the values in the column labeled ‘C’ become closer and closer to the true number of zero regression coefficients in the models. (ii) Based on different spatial parameters, the results of variable selection are similar. (iii) Two different penalty functions (SCAD and ALASSO) are used in this paper, and both methods perform almost equally well. (iv) Under the same circumstances, the ‘MSE’ of the mean parameters is smaller than the variance parameters’, which are very common in parametric estimation due to the fact that lower-order moments are easier to estimate than higher-order moments’.

6. Real Data Analysis

In this section, the proposed variable selection method is used to analyze the Boston housing price data that has been analyzed by many authors, for example, Pace and Gilley [29], Su and Yang [30], and so on. The database can be found in the spdep library of R. The dataset contains 14 variables with 506 observations, and a detailed description of these variables is listed in Table 4.

Table 4. Description of the variables in Boston housing data.

Variables	Description
CRIM$\left(X_1\right)$	Per capita crime rate by town
ZN$\left(X_2\right)$	Proportion of residential land zoned for lots over 25,000 sq. ft.
INDUS$\left(X_3\right)$	Proportion of non-retail business acres per town
CHAS$\left(X_4\right)$	Charles River dummy variable (=1 if tract bounds river; 0 otherwise)
NOX$\left(X_5\right)$	Nitric-oxides concentration (parts per 10 million)
RM$\left(X_6\right)$	Average number of rooms per dwelling
AGE$\left(X_7\right)$	Proportion of owner-occupied units built prior to 1940
DIS$\left(X_8\right)$	Weighted distances to five Boston employment centres
RAD$\left(X_9\right)$	Index of accessibility to radial highways
TAX$\left(X_{10}\right)$	Full-value property-tax rate per USD 10,000
PTRATIO$\left(X_{11}\right)$	Pupil–teacher ratio by town
B$\left(X_{12}\right)$	$1000{(Bk-0.63)}^2$, where Bk is the proportion of blacks by town
LSTAT$\left(X_{13}\right)$	% of the population with lower status
MEDV$\left(Y\right)$	Median value of owner-occupied homes in USD 1000s

This dataset has been used by Pace and Gilley [29] on the basis of spatial econometric models, and longitude–latitude coordinates for the census tracts are added to the dataset. In this paper, we take MEDV as the response variable, and the other 13 variables in Table 4 are treated as explanatory variables. Similar with Pace and Gilley [29] as well as Su and Yang [30], we use the Euclidean distances in terms of longitude and latitude to generate weight matrix $W=\left(w_{ij}\right)$, where

$$w_{ij}=\max \left(1-\frac{d_{ij}}{d_0},0\right),$$

$d_{ij}$ is the Euclidean distance and $d_0$ is the threshold distance, which is set to be 0.05 as in Su and Yang [30]. Thus, a spatial weight matrix is used with 19.1% nonzero elements. In addition, Z-variables in the variance model are taken to be the same as X-variables in the mean model. Then, the heterogeneous SAR models are considered here as follows:

$$\left\{\begin{array}{c}{Y}_i=\rho {\displaystyle \sum _{j=1}^n}{w}_{ij}{Y}_j+{\displaystyle \sum _{k=1}^{13}}{X}_{ik}{\beta }_k+{\epsilon }_i,\hfill \\ {\sigma }_i^2=exp\left({\displaystyle \sum _{k=1}^{13}}{Z}_{ik}{\gamma }_k\right),\hfill \\ i=1,2,\cdots ,506.\hfill \end{array}\right.$$

(7)

The ordinary quasi-maximum likelihood estimators (QMLE) and the penalized quasi-maximum likelihood estimators using the SCAD and ALASSO penalty functions are all considered. The tuning parameter was selected by the BIC. The estimated spatial parameter and the regression coefficients by different penalty-estimation methods are presented in Table 5. From Table 5, we can clearly see the following facts. (i) As expected, the spatial parameters are estimated very close to each other, and the other estimates are slightly different among different methods. (ii) Both the SCAD and ALASSO methods can eliminate many unimportant variables in joint mean and variance models. Concretely, the ALASSO method can select the same variables with the SCAD method in the mean model, and the SCAD method can select two more variables than the ALASSO method in the variance model. (iii) The important explanatory variables, selected by the proposed methods, are basically consistent with existing research results, for example, the regression coefficient of $X_{11}$ is negative in the mean model, which reveals that the housing price would decrease as the pupil–teacher ratio increases. In addition, based on the estimate of $\gamma$’s, we can obtain the estimates of ${\sigma }_i^2$ based on different methods and list the scatter plot of ${\widehat{\sigma }}_i^2$ in Figure 1, Figure 2 and Figure 3, which shows that the heteroscedasticity modeling for this dataset is reasonable.

Table 5. Penalized quasi-maximum likelihood estimators for $\beta$ and $\gamma$.

$\mathit{Coefficient}$	QMLE	SCAD	ALASSO
${\beta }_1\left(X_1\right)$	−0.0882	0	0
${\beta }_2\left(X_2\right)$	0.0699	0	0
${\beta }_3\left(X_3\right)$	−0.0427	0	0
${\beta }_4\left(X_4\right)$	0.0038	0	0
${\beta }_5\left(X_5\right)$	−0.0145	0	0
${\beta }_6\left(X_6\right)$	0.4641	0.5709	0.6023
${\beta }_7\left(X_7\right)$	−0.1509	−0.1788	−0.1489
${\beta }_8\left(X_8\right)$	−0.2076	−0.1287	−0.1175
${\beta }_9\left(X_9\right)$	0.2008	0.1440	0.0502
${\beta }_{10}\left(X_{10}\right)$	−0.2031	−0.2049	−0.1502
${\beta }_{11}\left(X_{11}\right)$	−0.1073	−0.0750	−0.0829
${\beta }_{12}\left(X_{12}\right)$	0.1440	0.1638	0.1033
${\beta }_{13}\left(X_{13}\right)$	−0.0597	0	0
${\gamma }_1\left(Z_1\right)$	0.1265	0	0
${\gamma }_2\left(Z_2\right)$	0.1020	0	0
${\gamma }_3\left(Z_3\right)$	−0.0830	0	0
${\gamma }_4\left(Z_4\right)$	0.0823	0	0
${\gamma }_5\left(Z_5\right)$	−0.4000	−0.3673	0
${\gamma }_6\left(Z_6\right)$	0.1113	0	0
${\gamma }_7\left(Z_7\right)$	0.3637	0.3461	0.2357
${\gamma }_8\left(Z_8\right)$	−0.4004	−0.3877	−0.2381
${\gamma }_9\left(Z_9\right)$	0.7366	0.8952	0.9143
${\gamma }_{10}\left(Z_{10}\right)$	0.2920	0.2105	0
${\gamma }_{11}\left(Z_{11}\right)$	−0.3533	−0.4085	−0.2723
${\gamma }_{12}\left(Z_{12}\right)$	−0.0141	0	0
${\gamma }_{13}\left(Z_{13}\right)$	−0.4262	−0.4586	−0.3929
$\rho$	0.2531	0.2881	0.2809

Figure 1. The scatter plot of ${\widehat{\sigma }}_i^2$ based on the ordinary quasi-maximum likelihood estimators (QMLE).

Figure 2. The scatter plot of ${\widehat{\sigma }}_i^2$ based on the penalized quasi-maximum likelihood using SCAD.

Figure 3. The scatter plot of ${\widehat{\sigma }}_i^2$ based on the penalized quasi-maximum likelihood using ALASSO.

7. Conclusions and Discussion

Within the framework of heterogeneous spatial autoregressive models, we proposed a variable-selection method on the basis of a penalized quasi-maximum likelihood approach. Like the mean, the variance parameters may be dependent on various explanatory variables of interest, so that simultaneous variable selection to the mean and variance models becomes important to avoid the modeling biases and reduce the model complexities. We have proven that the proposed penalized quasi-maximum likelihood estimators of the parameters in the mean and variance models are asymptotically consistent and normally distributed under some mild conditions. Simulation studies and a real data analysis of the Boston housing data are conducted to illustrate the proposed methodology. The results show that the proposed variable selection method is highly efficient and computationally fast.

Furthermore, there are several interesting issues that merit further research. For example, (i) it is interesting to increase the model flexibility by introducing nonparametric functions in the context of spatial autoregressive model and to study the variable selection for both the parametric component and the nonparametric component; (ii) a possible extension of the heterogeneous spatial autoregressive models is considered when the response variables are missing under a different missingness mechanism; and (iii) in the penalized estimation, we can also penalize the spatial parameter just like regression coefficients, which can help us directly judge whether the analyzed data have a spatial structure. These are all topics of interest and worthy of further study.