High-Dimensional Variable Selection for Quantile Regression Based on Variational Bayesian Method

Abstract

The quantile regression model is widely used in variable relationship research of moderate sized data, due to its strong robustness and more comprehensive description of response variable characteristics. With the increase of data size and data dimensions, there have been some studies on high-dimensional quantile regression under the classical statistical framework, including a high-efficiency frequency perspective; however, this comes at the cost of randomness quantification, or the use of a lower efficiency Bayesian method based on MCMC sampling. To overcome these problems, we propose high-dimensional quantile regression with a spike-and-slab lasso penalty based on variational Bayesian (VBSSLQR), which can, not only improve the computational efficiency, but also measure the randomness via variational distributions. Simulation studies and real data analysis illustrated that the proposed VBSSLQR method was superior or equivalent to other quantile and nonquantile regression methods (including Bayesian and non-Bayesian methods), and its efficiency was higher than any other method.

1. Introduction

Quantile regression, as introduced by Koenker and Bassett (1978) [1], is an important statistical inference about the relationship between quantiles of the response distribution and available covariates, and can offer a practically significant alternative to the traditional mean regression, because it provides a more comprehensive description of the response distribution than the mean. Moreover, quantile regression can capture the heterogeneous impact of regressors on different parts of the distribution [2], has excellent computational properties [3], exhibits robustness to outliers, and has a wide applicability [4]. For these reasons, quantile regression has attracted extensive attention in the literature. For example, see [5] for a Bayesian quantile regression with asymmetric Laplace distribution used to specify the likelihood, [6] for a Bayesian nonparametric approach to inference for quantile regression, ref. [7] for a mechanism of Bayesian inference of quantile regression models, and [8] for model selection in quantile regression, among others.

Although there is a growing literature on quantile regression, to the best of our knowledge, little of the existing quantile regression models have focused on high-dimensional data with variable numbers and larger sample sizes. In practice, a large number of variables may be collected and some of these will be insignificant and should be excluded from the final model. In the past two decades, there has been active methodological research on penalized methods for significant variable selection in linear parametric models. For example, see [9] for ridge regression, ref. [10] for a least absolute shrinkage and selection operator (Lasso), ref. [11] for smoothly clipped absolute deviation penalty (SCAD), ref. [12] for elastic net penalty, and ref. [13] for adaptive lasso methods. These methods have been extended to quantile regression; for example, see [14] for a $L_1$-regularization method for quantile regression, and ref. [15] for variable selection of quantile regression using SCAD and adaptive-Lasso penalties. Nevertheless, the aforementioned regularization methods discussed are both computationally complex and unstable. Additionally, they fail to account for prior information about parameters, which can lead to an unsatisfactory parametric estimation accuracy. In recent decades, Bayesian approaches to variable selection and parameter estimation have garnered significant attention. This is because they can substantially enhance the accuracy and efficiency of parametric estimation, by imposing various priors on model parameters, consistently selecting crucial variables, and providing more information for variable selection than penalization methods for highly nonconvex optimization problems. For example, see [16] for a Bayesian Lasso where the $L_1$ penalty is involved in Laplace prior, [17] for a Bayesian form of adaptive Lasso, ref. [18] for Bayesian Lasso quantile regression (BLQR), and [19] for Bayesian adaptive Lsso quantile regression (BALQR). The literature mentioned above implemented the standard Gibbs sampler for posterior computation, which is not easily scalable for high-dimensional data, where the number of variables is large compared with the sample size [20].

To address this issue, the Bayesian variable selection method with a spike-and-slab prior [20] has been favored by researchers, and this can be applied to high-dimensional data at the cost of a heavy computation burden. As a computationally efficient alternative to Markov chain Monte Carlo (MCMC) simulation, variational Bayes (VB) methods are gaining traction in machine learning and statistics for approximating posterior distributions in Bayesian inference. High-efficiency variational Bayesian spike-and-slab lasso(VBSSL) has been explored for certain high-dimensional models. Ray and Szabo (2022) [21] used a VBSSL method in a high-dimensional linear model, with regression coefficient’s prior specified as a mixture of Laplace distribution and Dirac mass. Yi and Tang (2022) [22] used VBSSL technology in high-dimensional linear mixed models, with a interesting prior parameter as a mixture of two Laplace distributions. However, to the best of our knowledge, there has been little work done on a VBSSL method for quantile regression. Xi et al. (2016) [23] considered Bayesian variable selection for nonparametric quantile regression with a small variable dimension, in which a spike-and-slab prior was chosen as a mixture of the point mass at zero and normal distribution. In this paper, to reduce the computational burden and quantify the parametric uncertainty, we propose quantile regression with spike-and-slab lasso penalty based on variational Bayesian (VBSSLQR), in which the prior is a mixture of two Laplace distributions, with a smaller or larger variance, respectively.

The main contributions of this paper are as follows: First, our proposed VBSSLQR method can perform variable selection for high-dimensional quantile regression at a relatively low computational cost, and without the need for nonconvex optimization, while also avoiding the curse of dimensionality problem. Second, in contrast to mean regressions, our proposed quantile approach offers a more systematic strategy for analyzing how covariates impact the various quantiles of the response distribution. Third, in ultra-high-dimensional regression, the mean regression errors are frequently presumed to be subGaussian, which is not required in our setting.

The rest of the paper is organized as follows: In Section 2, for high-dimensional data, we propose an efficient quantile regression with a spike-and-slab lasso penalty based on variational Bayes (VBSSLQR). In Section 3, we randomly generate high-dimensional data with n = 200 and p = 500 (excluding intercept items), and perform 500 simulation experiments, to explore the performance of our algorithm and compare it with other quantile regression methods (Bayesian and non-Bayesian) and nonquantile regression methods. The results show that our method is superior to other approaches in the case of high-dimensional data. We applied VBSSLQR to a real dataset that contained information about crime in various cities in the United States, and compared it with other quantile regression methods. The results showed that our method also had a good performance and excellent efficiency with real data, and the relevant results are shown in Section 4. Some concluding remarks are given in Section 5. Technical details are presented in the Appendix A, Appendix B, Appendix C and Appendix D.

2. Models and Methods

2.1. Quantile Regression

Consider a dataset of n independent subjects. For the ith subject, let $y_i$ be the response, while ${\mathit{x}}_i={(1,x_{i1},...,x_{ir})}^{\top }$ is an $(r+1)\times 1$ predictor vector, a simple linear regression model is defined as follows:

$$\begin{array}{c}\hfill y_i={\mathit{x}}_i^{\top }\mathbf{\beta }+{\epsilon }_i,i=1,2,\cdots ,n,\end{array}$$

(1)

where $\mathbf{\beta }={({\beta }_0,{\beta }_1,\cdots ,{\beta }_r)}^{\top }$ is the regression coefficient vector with ${\beta }_0$ corresponding to the intercept terms, and ${\epsilon }_i$ represents the error term with unknown distribution. It is usual to assume that the $\tau$th quantile of the random error term is 0; that is, $Q_{\tau }\left({\epsilon }_i\right)=0$ for $0<\tau <1$. According to this assumption, the form of $\tau$th quantile regression of model (1) is specified as follows:

$$\begin{array}{c}\hfill Q_{y_i}\left(\tau \right|{\mathit{x}}_i)={\mathit{x}}_i^{\top }\mathbf{\beta },\end{array}$$

(2)

where $Q_{y_i}\left(\tau \right|{\mathit{x}}_i)$ is the inverse cumulative distribution function of $y_i$, given ${\mathit{x}}_i$ evaluated at $\tau$. The estimate of the regression coefficient vector $\mathbf{\beta }$ in Equation (2) is

$$\begin{array}{cc}\hfill \widehat{\mathbf{\beta }}& =\underset{\mathbf{\beta }\in {\mathit{R}}^{r+1}}{\mathrm{argmin}}\left[\sum _{i:y_i\ge {\mathit{x}}_i\mathbf{\beta }}\tau |y_i-{\mathit{x}}_i^{\top }\mathbf{\beta }|+\sum _{i:y_i<{\mathit{x}}_i\beta }(1-\tau )|y_i-{\mathit{x}}_i^{\top }\mathbf{\beta }|\right]\hfill \\ \hfill & =\underset{\mathbf{\beta }\in {\mathit{R}}^{r+1}}{\mathrm{argmin}}\sum _{i=1}^n{\rho }_{\tau }(y_i-{\mathit{x}}_i^{\top }\mathbf{\beta }),\hfill \end{array}$$

(3)

where the loss function ${\rho }_{\tau }\left(\mu \right)=\mu \times (\tau -\mathrm{I}(\mu <0))$ with the indicator function $\mathrm{I}(·)$.

In light of [5,24], minimizing Equation (3) is equivalent to maximizing the likelihood of n independent individuals with the ith one distributed as an asymmetric Laplace distribution (ALD) specified as

$$\begin{array}{c}\hfill p\left(y_i\right|{\mathit{x}}_i,\mathbf{\beta },\sigma ,\tau )=\frac{\tau (1-\tau )}{\sigma }\exp \left\{-{\rho }_{\tau }\left(\frac{y_i-{\mu }_i}{\sigma }\right)\right\},\end{array}$$

(4)

where the local parameter ${\mu }_i={\mathit{x}}_i^{\top }\mathbf{\beta }$, the scale parameter $\sigma >0$, and the skewness parameter $\tau$ is between 0 and 1; obviously, ALD is a Laplace distribution when $\tau =0.5$ and ${\mu }_i=0$. However, it is computationally infeasible to carry out statistical inference based directly on Equation (4) involving the nondifferentiable point ${\mu }_i$. Following [25], Equation (4) can be rewritten in the following hierarchical fashion:

$$\left\{\begin{array}{c}{y}_i={\mathit{x}}_i^{\top }\mathbf{\beta }+k_1{z}_i+\sqrt{k_2\sigma z_i}{\xi }_i,\hfill \\ z_i|\sigma \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Exp}\left(\frac{1}{\sigma }\right),\hfill \\ {\xi }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }N(0,1),\hfill \\ z_i\mathrm{is}\mathrm{independent}\mathrm{of}{\xi }_i,\hfill \end{array}\right.$$

(5)

where $k_1=\frac{1-2\tau }{\tau (1-\tau )}$, $k_2=\frac{2}{\tau (1-\tau )}$ and $\mathrm{Exp}\left(\frac{1}{\sigma }\right)$ denotes the exponential distribution with mean $\sigma$, whose specific density function is $p\left(z_i\right|\sigma )=\frac{1}{\sigma }exp\left(-\frac{1}{\sigma }{z}_i\right)\mathrm{I}(z_i>0)$. Equation (5) illustrates that an asymmetric Laplace distribution can also be represented as a mixture of exponential and standard normal distributions, which allows us to express a quantile regression model as a normal regression model, in which the response has the following conditional distribution:

$$y_i|{\mathit{x}}_i,\mathbf{\beta },z_i,\sigma \stackrel{\mathrm{ind}}{\sim }N\left({\mathit{x}}_i^{\top }\mathbf{\beta }+k_1{z}_i,k_2\sigma z_i\right).$$

For the above-defined quantile regression model with high-dimensional covariate vector (r is large enough), it is of interest to estimate the parameter vector $\mathbf{\beta }$ and to identify the critical covariates. To this end, we considered Bayesian quantile regression based on spike-and-slab lasso, as follows:

2.2. Bayesian Quantile Regression Based on a Spike-and-Slab Lasso

As early as 2016, Xi et al. [23] applied a spike-and-slab prior to Bayesian quantile regression, but their proposed prior was a mixture of zero particle and normal distribution with large variance, and the estimate of posterior density was obtained using a Gibbs sampler. To offer novel theoretical insights into a class of continuous spike-and-slab priors, Rockova (2018) [26] introduced a novel family of spike-and-slab priors, which are a mixture of two density functions with spike or slab probability. In this paper, we adopt a spike-and-slab lasso prior with a mixture of two Laplace distributions with large or small variance, respectively [26], which facilitates the variational Bayesian technique for approximating the posterior density of parameters and for improving the efficiency of the algorithm. In light of ref. [26], given the indicator ${\gamma }_j=0$ or 1, the prior of $\mathbf{\beta }$ in the Bayesian quantile regression model (5) can be written as

$$\begin{array}{c}\hfill \pi \left(\mathbf{\beta }\right|\mathbf{\gamma })=\prod _{j=0}^r\pi \left({\beta }_j\right|{\gamma }_j)=\prod _{j=0}^r\left[{\gamma }_j{\Psi }_0\left({\beta }_j\right|{\lambda }_0)+(1-{\gamma }_j){\Psi }_1\left({\beta }_j\right|{\lambda }_1)\right],\end{array}$$

(6)

where the Laplace density ${\Psi }_0\left({\beta }_j\right|{\lambda }_0)=\frac{{\lambda }_0}{2}exp(-{\lambda }_0|{\beta }_j\left|\right)$ and ${\Psi }_1\left({\beta }_j\right|{\lambda }_1)=\frac{{\lambda }_1}{2}exp(-{\lambda }_1|{\beta }_j\left|\right)$ with precision parameters ${\lambda }_0$ and ${\lambda }_1$ satisfying that ${\lambda }_0\gg {\lambda }_1$, the indicator variable set $\mathbf{\gamma }=\left\{{\gamma }_j\right|j=0,1,\cdots ,r\}$, the jth variable is active when ${\gamma }_j=0$, and inactive otherwise. Similarly to [27], the Laplace distribution for the regression coefficient ${\beta }_j$ can be represented as a mixture of a normal distribution and an exponential distribution, specifically the distribution of ${\beta }_j$ can be expressed as a hierarchical structure, as follows:

$$\begin{array}{cc}\hfill {\beta }_j|h_{0j}^2,h_{1j}^2,{\gamma }_j& \stackrel{\mathrm{ind}.}{\sim }{\gamma }_{j}N\left(0,h_{0j}^2\right)+\left(1-{\gamma }_j\right)N\left(0,h_{1j}^2\right),\hfill \\ \hfill h_{0j}^2|{\lambda }_0^2& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Exp}\left(\frac{{\lambda }_0^2}{2}\right),\hfill \\ \hfill h_{1j}^2|{\lambda }_1^2& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Exp}\left(\frac{{\lambda }_1^2}{2}\right),\hfill \\ \hfill {\gamma }_j& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }B\left({\pi }_{\gamma }\right),\hfill \end{array}$$

(7)

where $B\left({\pi }_{\gamma }\right)$ denotes the Bernoulli distribution, with ${\pi }_{\gamma }$ being the probability that indicator variable ${\gamma }_j$ equals one for $j=0,1,\cdots ,r$, and specifies the prior of ${\pi }_{\gamma }$ as a Beta distribution $\mathrm{Be}\left(a_{{\pi }_{\gamma }},b_{{\pi }_{\gamma }}\right)$, where hyperparameters $a_{{\pi }_{\gamma }}$ and $b_{{\pi }_{\gamma }}$, ${\lambda }_0^2$, and ${\lambda }_1^2$ are regularization parameters to identify important variables, for which we consider the following conjugate priors:

$$\begin{array}{ccc}{\lambda }_0^2\hfill & \sim \mathrm{Ga}\left({\nu }_{{\lambda }_0},1\right),{\lambda }_1^2\hfill & \sim \mathrm{Ga}\left({\nu }_{{\lambda }_1},1\right),\hfill \end{array}$$

where $\mathrm{Ga}\left(a,b\right)$ denotes the gamma distribution with shape parameter a and scale parameter b. As mentioned above, ${\lambda }_0$ and ${\lambda }_1$ should satisfy that ${\lambda }_0\gg {\lambda }_1$, to this end, we select hyperparmeters ${\nu }_{{\lambda }_0}$ and ${\nu }_{{\lambda }_1}$ to satisfy that ${\nu }_{{\lambda }_0}\gg {\nu }_{{\lambda }_1}$. The prior of scale parameter $\sigma$ in (5) is an inverse gamma distribution $\mathrm{IG}\left(a_{\sigma },b_{\sigma }\right)$ of the hyperparameters $a_{\sigma }=1$ and $b_{\sigma }=0.01$ in the paper leading to almost non-informative prior.

Under a Bayesian statistical paradise, based on the above priors and likelihood of quantile regression, it is required to induce the following posterior distribution $\pi \left(\mathbf{\theta }\right|\mathit{D})\propto p(\mathbf{\theta },\mathit{D})$, where $\mathbf{\theta }=\{\mathbf{\beta },\mathit{z},\sigma ,\mathbf{\gamma },{\mathit{h}}_0^2,{\mathit{h}}_1^2,{\lambda }_0^2,{\lambda }_1^2,{\pi }_{\gamma }\}$, the latent variable set $\mathit{z}=\left\{z_i\right|i=1,\cdots ,n\}$, ${\mathit{h}}_0^2=\left\{h_{0j}^2\right|j=0,1,\cdots ,r\}$, ${\mathit{h}}_1^2=\left\{h_{1j}^2\right|j=0,1,\cdots ,r\}$, observing set $\mathit{D}=\{\mathit{y},\mathit{x}\}$ with the response set $\mathit{y}=\left\{y_i\right|i=1,\cdots ,n\}$ and covariate set $\mathit{x}=\left\{x_i\right|i=1,\cdots ,n\}$. Based on the hierarchical structure (5) of the quantile regression likelihood $p\left(y_i\right|{\mathit{x}}_i,\mathbf{\beta })$ and the hierarchical structure (7) of the spike-and-slab prior to regression coefficient vector $\mathbf{\beta }$, we derive the joint density

$$\begin{array}{cc}\hfill p(\mathbf{\theta },\mathit{D})& =\prod _{i=1}^{n}N\left(y_i\right|{\mathit{x}}_i^{\top }\mathbf{\beta }+k_1{z}_i,k_2\sigma z_i)\mathrm{Exp}\left(z_i\right|{\sigma }^{-1})\mathrm{Ga}\left({\lambda }_0^2\right|{\nu }_{{\lambda }_0},1)\mathrm{Ga}\left({\lambda }_1^2\right|{\nu }_{{\lambda }_1},1)\hfill \\ \hfill & \times \prod _{j=0}^r{\left[N\left({\beta }_j\right|0,h_{0j}^2)\right]}^{{\gamma }_j}{\left[N\left({\beta }_j\right|0,h_{1j}^2)\right]}^{(1-{\gamma }_j)}\mathrm{Exp}\left(h_{0j}^2\right|\frac{{\lambda }_0^2}{2})\mathrm{Exp}\left(h_{1j}^2\right|\frac{{\lambda }_1^2}{2})\hfill \\ \hfill & \times \prod _{j=0}^{r}B\left({\gamma }_j\right|1,{\pi }_{\gamma })\mathrm{Be}\left({\pi }_{\gamma }\right|a_{{\pi }_{\gamma }},b_{{\pi }_{\gamma }})\mathrm{IG}\left(\sigma \right|a_{\sigma },b_{\sigma })\hfill \\ \hfill & \propto \prod _{i=1}^n{\left(k_2\sigma z_i\right)}^{-\frac{1}{2}}\exp \left(-\frac{{(y_i-{\mathit{x}}_i^{\top }\mathbf{\beta }-k_1{z}_i)}^2}{2k_2\sigma z_i}\right)\frac{1}{\sigma }\exp \left(-\frac{z_i}{\sigma }\right){\left({\lambda }_0^2\right)}^{v_{{\lambda }_0}-1}\exp (-{\lambda }_0^2)\hfill \\ \hfill & \times {\left({\lambda }_1^2\right)}^{(v_{{\lambda }_1}-1)}\exp (-{\lambda }_1^2)\prod _{j=0}^r{\left[\frac{1}{\sqrt{h_{0j}^2}}\exp \left(-\frac{{\beta }_j^2}{2h_{0j}^2}\right)\right]}^{{\gamma }_j}{\left[\frac{1}{\sqrt{h_{1j}^2}}\exp \left(-\frac{{\beta }_j^2}{2h_{1j}^2}\right)\right]}^{(1-{\gamma }_j)}\hfill \\ \hfill & \times \frac{{\lambda }_0^2}{2}\exp \left(-\frac{{\lambda }_0^2}{2}{h}_{0j}^2\right)\frac{{\lambda }_1^2}{2}\exp \left(-\frac{{\lambda }_1^2}{2}{h}_{1j}^2\right)\prod _{j=0}^r{\pi }_{\gamma }^{{\gamma }_j}{(1-{\pi }_{\gamma })}^{1-{\gamma }_j}{\pi }_{\gamma }^{a_{{\pi }_{\gamma }}-1}{(1-{\pi }_{\gamma })}^{b_{{\pi }_{\gamma }}-1}\hfill \\ \hfill & \times \frac{b_{\sigma }^{a_{\sigma }}}{\Gamma \left(a_{\sigma }\right)}{\sigma }^{-a_{\sigma }-1}\exp \left(-\frac{b_{\sigma }}{\sigma }\right).\hfill \end{array}$$

(8)

Although sampling from the aforementioned posterior is simple, it becomes increasingly time-consuming for higher dimension quantile models. To tackle this issue, we developed a faster and more efficient alternative method based on variational Bayesian.

2.3. Quantile Regression with a Spike-and-Slab Lasso Penalty Based on Variational BAYESIAN

At present, the most commonly variational Bayesian approximation posterior distribution methods use mean field approximation theory [28], which has the highest efficiency among variational methods, especially for those parameters or parameter block with conjugate priors. Bayesian quantile regression needs to take into account that the variance of each observation value is different, and each $y_i$ corresponds to a potential variable $z_i$, which will result in the algorithm efficiency of quantile regression being lower than that of general mean regression. Therefore, in this paper, we use the variational Bayesian algorithm of the mean field approximation, which is the most efficient algorithm, to derive the quantile regression model with the spike-and-slab lasso penalty.

Based on variational theory, we choose a densities for random variables $\mathbf{\theta }$ from variational family $\mathcal{F}$, which having the same support $\Theta$ as the posterior density $\pi \left(\mathbf{\theta }\right|\mathit{D})$. We approximate the posterior density $\pi \left(\mathbf{\theta }\right|\mathit{D})$ by any variational density $\mathrm{q}\left(\mathbf{\theta }\right)\in \mathcal{F}$. The variational Bayesian method is to seek the optimal approximation to $\pi \left(\mathbf{\theta }\right|\mathit{D})$ by minimizing the Kullback-Leibler divergence between $\mathrm{q}\left(\mathbf{\theta }\right)$ and $\pi \left(\mathbf{\theta }\right|\mathit{D})$, which is an optimization problem that can be expressed as:

$${\mathrm{q}}^{*}\left(\mathbf{\theta }\right)=\underset{\mathrm{q}\left(\mathbf{\theta }\right)\in \mathcal{F}}{\mathrm{argmin}}\mathrm{KL}\left[\mathrm{q}\left(\mathbf{\theta }\right)\Vert \pi \left(\mathbf{\theta }\right|\mathit{D})\right],$$

where $\mathrm{KL}\left[\mathrm{q}\left(\mathbf{\theta }\right)\Vert \pi \left(\mathbf{\theta }\right|\mathit{D})\right]={\int }_{\Theta }\mathrm{q}\left(\mathbf{\theta }\right)\log \frac{\mathrm{q}\left(\mathbf{\theta }\right)}{\pi \left(\mathbf{\theta }\right|\mathit{D})}d\mathbf{\theta }$, which is not less than zero and equal to zero if, and only if, $\mathrm{q}\left(\mathbf{\theta }\right)\equiv \pi \left(\mathbf{\theta }\right|\mathit{D})$. The posterior density $\pi \left(\mathbf{\theta }\right|\mathit{D})=\frac{p(\mathbf{\theta },\mathit{D})}{p\left(\mathit{D}\right)}$ with the joint distribution $p(\mathbf{\theta },\mathit{D})$ of parameter $\mathbf{\theta }$ and data $\mathit{D}$ and the marginal distribution $p\left(\mathit{D}\right)$ data $\mathit{D}$. Since $p\left(\mathit{D}\right)={\int }_{\Theta }p(\mathbf{\theta },\mathit{D})d\mathbf{\theta }$, which does not have an analytic expression for our considered model, it is rather difficult to implement the optimization problem presented above. It is easy to induce that

$$\log p\left(\mathit{D}\right)=\mathrm{KL}\left[\mathrm{q}\left(\mathbf{\theta }\right)\Vert \pi \left(\mathbf{\theta }\right|\mathit{D})\right]+\mathcal{L}\left\{\mathrm{q}\left(\mathbf{\theta }\right)\right\}\ge \mathcal{L}\left\{\mathrm{q}\left(\mathbf{\theta }\right)\right\},$$

in which the evidence lower bound (ELOB) $\mathcal{L}\left\{\mathrm{q}\left(\mathbf{\theta }\right)\right\}={\mathrm{E}}_{\mathrm{q}\left(\mathbf{\theta }\right)}\left[\log p(\mathbf{\theta },\mathit{D})\right]-{\mathrm{E}}_{\mathrm{q}\left(\mathbf{\theta }\right)}\left[\log \mathrm{q}\left(\mathbf{\theta }\right)\right]$ with ${\mathrm{E}}_{\mathrm{q}\left(\mathbf{\theta }\right)}\left[·\right]$ representing the expectation taken with respect to the variational density $\mathrm{q}\left(\mathbf{\theta }\right)$. Thus, minimizing $\mathrm{KL}\left[\mathrm{q}\left(\mathbf{\theta }\right)\Vert \pi \left(\mathbf{\theta }\right|\mathit{D})\right]$ is equivalent to maximizing $\mathcal{L}\left\{\mathrm{q}\right(\mathbf{\theta }\left)\right\}$ because log$p\left(\mathit{D}\right)$ does not depend on $\mathbf{\theta }$. That is,

$${\mathrm{q}}^{*}\left(\mathbf{\theta }\right)=\underset{\mathrm{q}\left(\mathbf{\theta }\right)\in \mathcal{F}}{\mathrm{argmin}}\mathrm{KL}\left[\mathrm{q}\left(\mathbf{\theta }\right)\Vert \pi \left(\mathbf{\theta }\right|\mathit{D})\right]=\underset{\mathrm{q}\left(\mathbf{\theta }\right)\in \mathcal{F}}{\mathrm{argmax}}\mathcal{L}\left\{\mathrm{q}\left(\mathbf{\theta }\right)\right\},$$

which indicates that seeking the optimal approximation problem of $\pi \left(\mathbf{\theta }\right|\mathit{D})$ becomes maximizing $\mathcal{L}\left\{\mathrm{q}\right(\mathbf{\theta }\left)\right\}$ under the variational family $\mathcal{F}$. The complexity of the approximation problem heavily is related to the variational family $\mathcal{F}$. Therefore, choosing a comparatively simple variational family $\mathcal{F}$ to optimize the objective function $\mathcal{L}\left\{\mathrm{q}\right(\mathbf{\theta }\left)\right\}$ with respect to $q\left(\mathbf{\theta }\right)$ is fascinating.

Following the commonly used approach to choosing a tractable variational family $\mathcal{F}$ in the variational studies, we consider the frequently-used mean-field theory, which assumes that blocks of $\mathbf{\theta }$ are mutually independent and each is measured by the parameters of the variational density. Obviously, the variational density $\mathrm{q}\left(\mathbf{\theta }\right)$ is assumed to be factorized across the blocks of $\mathbf{\theta }$:

$$\mathrm{q}\left(\mathbf{\theta }\right)=\prod _{j=0}^r{\mathrm{q}}_1({\beta }_j,{\gamma }_j){\mathrm{q}}_2\left(h_{0j}^2\right){\mathrm{q}}_3\left(h_{1j}^2\right)\prod _{i=1}^n{\mathrm{q}}_4\left(z_i\right){\mathrm{q}}_5\left({\lambda }_0^2\right){\mathrm{q}}_6\left({\lambda }_1^2\right){\mathrm{q}}_7\left({\pi }_{\gamma }\right){\mathrm{q}}_8\left(\sigma \right)\equiv \prod _s^8{\mathrm{q}}_s\left({\mathbf{\theta }}_s\right),$$

in which form of each variational densities ${\mathrm{q}}_s\left({\mathbf{\theta }}_s\right)$’s is unknown, but the above assumed factorization across components is predetermined. Moreover, the best solutions for ${\mathrm{q}}_s\left({\mathbf{\theta }}_s\right)$’s are to be achieved by maximizing $\mathcal{L}\left\{\mathrm{q}({\mathbf{\theta }}_1,\cdots ,{\mathbf{\theta }}_8)\right\}$ with respect to variational densities ${\mathrm{q}}_1\left({\mathbf{\theta }}_1\right),\cdots ,{\mathrm{q}}_8\left({\mathbf{\theta }}_8\right)$ by the coordinate ascent method, where $\mathbf{\theta }=\{{\mathbf{\theta }}_1,\cdots ,{\mathbf{\theta }}_8\}$ where ${\mathbf{\theta }}_s$ can be either a scalar or a vector. This means that when the correlation between several unknown parameters or potential variables cannot be ignored, they should be put in the same block and merged into ${\mathbf{\theta }}_s$. Following the idea of the coordinate ascent method given in ref. [29], when fixing other variational factors ${\mathrm{q}}_k\left({\mathbf{\theta }}_k\right)$ for $k\ne s$, i.e., the optimal density ${\mathrm{q}}_s^{*}\left({\mathbf{\theta }}_s\right)$, which maximizes $\mathcal{L}\left\{\mathrm{q}\right(\mathbf{\theta }\left)\right\}$ with respect to ${\mathrm{q}}_s\left({\mathbf{\theta }}_s\right)$, is shown to take the form

$$\begin{array}{c}\hfill {\mathrm{q}}_s^{*}\left({\mathbf{\theta }}_s\right)\propto \exp \left\{{\mathrm{E}}_{-{\theta }_s}\left[\log p(\mathbf{\theta },\mathit{D})\right]\right\},\end{array}$$

(9)

where $\log p(\mathbf{\theta },\mathit{D})$ is the logarithm of the joint density function and ${\mathrm{E}}_{-s}[·]$ is the expectation taken with respect to the density ${\prod }_{k\ne s}{\mathrm{q}}_k\left({\mathbf{\theta }}_k\right)$ for $s=1,\cdots ,8$.

According to Equations (8) and (9), we can derive the variational posterior for each parameter as follows (see Appendix A for the details):

$$\begin{array}{cc}\hfill {\beta }_j& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{\mu }_{{\gamma }_j}N\left({\mu }_{0j},{\sigma }_{0j}^2\right)+(1-{\mu }_{{\gamma }_j})N\left({\mu }_{1j},{\sigma }_{1j}^2\right),\mathrm{for}j=1,\cdots ,r,\hfill \\ \hfill h_{0j}^2& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{\mu }_{{\gamma }_j}\mathrm{GIG}\left(\frac{1}{2},{\mathrm{E}}_{{\lambda }_0^2}\left({\lambda }_0^2\right),{\sigma }_{0j}^2+{\mu }_{0j}^2\right)+(1-{\mu }_{{\gamma }_j})\mathrm{Exp}\left(\frac{1}{2}{\mathrm{E}}_{{\lambda }_0^2}\left({\lambda }_0^2\right)\right),\hfill \\ \hfill h_{1j}^2& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }(1-{\mu }_{{\gamma }_j})\mathrm{GIG}\left(\frac{1}{2},{\mathrm{E}}_{{\lambda }_1^2}\left({\lambda }_1^2\right),{\sigma }_{1j}^2+{\mu }_{1j}^2\right)+{\mu }_{{\gamma }_j}\mathrm{Exp}\left(\frac{1}{2}{\mathrm{E}}_{{\lambda }_1^2}\left({\lambda }_1^2\right)\right),\hfill \\ \hfill {\lambda }_0^2& \sim \mathrm{Ga}\left(r+1+{\nu }_{{\lambda }_0},1+\frac{1}{2}\sum _{j=1}^r{\mathrm{E}}_{h_{0j}^2}\left(h_{0j}^2\right)\right),\hfill \\ \hfill {\lambda }_1^2& \sim \mathrm{Ga}\left(r+1+{\nu }_{{\lambda }_1},1+\frac{1}{2}\sum _{j=1}^r{\mathrm{E}}_{h_{1j}^2}\left(h_{1j}^2\right)\right),\hfill \\ \hfill {\gamma }_j& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }B\left(1,{(1+{\mathrm{e}}^{{\zeta }_i})}^{-1}\right),\mathrm{for}j=0,1,\cdots ,r,\hfill \\ \hfill {\pi }_{\gamma }& \sim \mathrm{Be}\left(a+\sum _{j=0}^r{\mu }_{{\gamma }_j},r+1+b-\sum _{j=0}^r{\mu }_{{\gamma }_j}\right),\hfill \\ \hfill z_i& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{GIG}\left(\frac{1}{2},a_{z_i},b_{z_i}\right),\mathrm{for}i=1,2,\cdots ,n,\hfill \\ \hfill \sigma & \sim \mathrm{IG}\left(\frac{3}{2}n+a_{\sigma },c_{\sigma }\right),\hfill \end{array}$$

where $\mathrm{GIG}(·,·,·)$ denotes generalized inverse Gaussian distribution,

$$\begin{array}{cc}\hfill & {\mu }_{0j}={\sigma }_{oj}^2\frac{{\mathrm{E}}_{\sigma }\left({\sigma }^{-1}\right)}{k_2}\sum _{i=1}^n\left[{\mathrm{E}}_{z_i}\left(z_i^{-1}\right)\left(y_i-{\mathit{x}}_{i,(-j)}^{\top }{\mathbf{\mu }}_{{\mathbf{\beta }}_{(-j)}}\right)-k_1\right]x_{ij},\hfill \\ \hfill & {\sigma }_{0j}^2={\left[{\mathrm{E}}_{h_{0j}^2|{\gamma }_j=1}\left(h_{0j}^{-2}\right)+\frac{{\mathrm{E}}_{\sigma }\left({\sigma }^{-1}\right)}{k_2}\sum _{i=1}^n{x}_{ij}^2{\mathrm{E}}_{z_i}\left(z_i^{-1}\right)\right]}^{-1},\hfill \\ \hfill & {\mu }_{1j}={\sigma }_{1j}^2\frac{{\mathrm{E}}_{\sigma }\left({\sigma }^{-1}\right)}{k_2}\sum _{i=1}^n\left[{\mathrm{E}}_{z_i}\left(z_i^{-1}\right)\left(y_i-{\mathit{x}}_{i,(-j)}^{\top }{\mathbf{\mu }}_{{\mathbf{\beta }}_{(-j)}}\right)-k_1\right]x_{ij},\hfill \\ \hfill & {\sigma }_{1j}^2={\left[{\mathrm{E}}_{h_{1j}^2|{\gamma }_j=0}\left(h_{1j}^{-2}\right)+\frac{{\mathrm{E}}_{\sigma }\left({\sigma }^{-1}\right)}{k_2}\sum _{i=1}^n{x}_{ij}^2{\mathrm{E}}_{z_i}\left(z_i^{-1}\right)\right]}^{-1},\hfill \end{array}$$

(10)

and

$$\begin{array}{cc}\hfill & {\zeta }_j={\mathrm{E}}_{{\pi }_{\gamma }}\left(\log (1-{\pi }_{\gamma })\right)-{\mathrm{E}}_{{\pi }_{\gamma }}\left(\log {\pi }_{\gamma }\right)+\frac{1}{2}{\mathrm{E}}_{h_{0j}^2|{\gamma }_j=1}\left(\log h_{0j}^2\right)-\frac{1}{2}{\mathrm{E}}_{h_{1j}^2|{\gamma }_j=0}\left(\log h_{1j}^2\right)\hfill \\ \hfill & +\frac{1}{2}\{\log {\sigma }_{1j}^2-\log {\sigma }_{0j}^2\}+\frac{1}{2}(\log {\sigma }_{1j}^2-\log {\sigma }_{0j}^2)+\frac{1}{2}({\mu }_{1j}^2{\left({\sigma }_{1j}^2\right)}^{-1}-{\mu }_{0j}^2{\left({\sigma }_{0j}^2\right)}^{-1}),\hfill \\ \hfill & a_{z_i}=\frac{{\mathrm{E}}_{\sigma }\left({\sigma }^{-1}\right)}{k_2}\left(k_1^2+2k_2\right),\hfill \\ \hfill & b_{z_i}=\frac{{\mathrm{E}}_{\sigma }\left({\sigma }^{-1}\right)}{k_2}[{(y_i-{\mathit{x}}_i^{\top }{\mathbf{\mu }}_{\mathbf{\beta }})}^2+{\mathit{x}}_i^{\top }{\mathsf{\Sigma }}_{\mathbf{\beta }}{\mathit{x}}_i],\hfill \\ \hfill & c_{\sigma }=\frac{1}{2k_2}\sum _{i=1}^n\{k_1^2{\mu }_{z_i}+{\mathrm{E}}_{z_i}\left(z_i^{-1}\right)[{(y_i-{\mathit{x}}_i^{\top }{\mathbf{\mu }}_{\mathbf{\beta }})}^2+{\mathit{x}}_i^{\top }{\mathsf{\Sigma }}_{\mathbf{\beta }}{\mathit{x}}_i]\hfill \\ \hfill & -2k_1(y_i-{\mathit{x}}_i^{\top }{\mathbf{\mu }}_{\mathbf{\beta }})\}+\sum _{i=1}^n{\mu }_{z_i}+b_{\sigma }.\hfill \end{array}$$

In the above equation, ${\mu }_{{\lambda }_0^2}={\mathrm{E}}_{{\lambda }_0^2}\left({\lambda }_0^2\right)$, and ${\mu }_{{\lambda }_1^2}$, ${\mu }_{{\beta }_j}$, ${\mu }_{{\gamma }_j}$, ${\mu }_{z_i}$ are similar to ${\mu }_{{\lambda }_0^2}$, ${\mathbf{\mu }}_{\mathbf{\beta }}={({\mu }_{{\beta }_0},{\mu }_{{\beta }_1},\cdots ,{\mu }_{{\beta }_p})}^{\top }$ with ${\mu }_{{\beta }_j}={\mu }_{{\gamma }_j}{\mu }_{0j}+(1-{\mu }_{{\gamma }_j}){\mu }_{1j}$, and ${\mathbf{\mu }}_{{\mathbf{\beta }}_{(-j)}}$ is a $p\times 1$ vector with the jth component of vector ${\mathbf{\mu }}_{\mathbf{\beta }}$ deleted, ${\mathsf{\Sigma }}_{\mathbf{\beta }}=\mathrm{diag}\left({\sigma }_{{\beta }_0}^2{\sigma }_{{\beta }_1}^2,\cdots {\sigma }_{{\beta }_p}^2\right)$ with ${\sigma }_{{\beta }_j}^2={\mu }_{{\gamma }_j}{\sigma }_{0j}^2+(1-{\mu }_{{\gamma }_j}){\sigma }_{1j}^2+{\mu }_{{\gamma }_j}(1-{\mu }_{{\gamma }_j}){({\mu }_{0j}-{\mu }_{1j})}^2$.

In the section above, we derived the variational posterior of each parameter. Using the idea of coordinate axis optimization, we can update each variational distribution iteratively until it converges.

For this reason, we list the variational Bayesian spike-and-slab lasso quantile regression (VBSSLQR) algorithm as shown in Algorithm 1:

Algorithm 1 Variational Bayesian spike-and-slab lasso quantile regression (VBSSLQR).

Input:

Data $\mathit{y}$, predictors $\mathit{x}$, prior parameters ${\nu }_{{\lambda }_0}={10}^4$,${\nu }_{{\lambda }_1}=1$, $a_{\sigma }=1$, $b_{\sigma }=0.01$, $a=b=1$,  precision $ϵ=0.01$ and quantile $\tau$;

Output:

Optimized variational parameters ${\mathrm{E}}_{{\beta }_j}\left({\beta }_j\right)$, ${\mathrm{Var}}_{{\beta }_j}\left({\beta }_j\right)$ for $j=0,1,\cdots ,r$ and the corresponding Bayesian confidence interval.

Initialize: ${\mathbf{\mu }}_{\mathbf{\beta }}^{\left(0\right)}$; ${\mathrm{E}}_{h_{0j}^2|{\gamma }_j=1}^{\left(0\right)}\left(\frac{1}{h_{0j}^2}\right)=0.01$; ${\mathrm{E}}_{h_{1j}^2|{\gamma }_j=0}^{\left(0\right)}\left(\frac{1}{h_{1j}^2}\right)=1$;${\mathbf{\mu }}_{\mathbf{\gamma }}^{\left(0\right)}=\mathbf{0.5}$;

${\mathrm{E}}_{{\lambda }_0^2}^{\left(0\right)}\left({\lambda }_0^2\right)=100$; ${\mathrm{E}}_{{\lambda }_1^2}^{\left(0\right)}\left({\lambda }_1^2\right)=1$; ${\mathrm{E}}_{{\pi }_{\gamma }}^{\left(0\right)}\left(\log {\pi }_{\gamma }\right)=0$;${\mathrm{E}}_{\sigma }^{\left(0\right)}\left({\sigma }^{-1}\right)=1$;

$E_{{\pi }_{\gamma }}^{\left(0\right)}\left(\log (1-{\pi }_{\gamma })\right)=-1$;${\mathbf{\gamma }}^{\left(0\right)}=\mathbf{0}$; ${\mathrm{E}}_{z_i}^{\left(0\right)}\left(z_i\right)={\mathrm{E}}_{z_i}^{\left(0\right)}\left(z_i^{-1}\right)$;

while $\left|d^{\left(t\right)}\right|>\epsilon$ do

for $j=0$ to r do

Update ${{\sigma }^2}_{0j}^{(t+1)}$, ${\mu }_{0j}^{(t+1)}$, ${{\sigma }^2}_{1j}^{(t+1)}$ and ${\mu }_{1j}^{(t+1)}$ according to Equation (10).

Update ${\mathrm{E}}_{h_{0j}^2|{\gamma }_j=1}^{(t+1)}\left(h_{0j}^2\right)$, ${\mathrm{E}}_{h_{0j}^2}^{(t+1)}\left(h_{0j}^2\right)$, ${\mathrm{E}}_{h_{0j}^2|{\gamma }_j=1}^{(t+1)}\left(h_{0j}^{-2}\right)$, $E_{h_{0j}^2|{\gamma }_j=1}^{(t+1)}\left(\log h_{0j}^2\right)$ according to $\mathrm{q}\left(h_{0j}^2\right)$,

Update ${\mathrm{E}}_{h_{1j}^2|{\gamma }_j=0}^{(t+1)}\left(h_{1j}^2\right)$, ${\mathrm{E}}_{h_{1j}^2}^{(t+1)}\left(h_{1j}^2\right)$, ${\mathrm{E}}_{h_{1j}^2|{\gamma }_j=0}^{(t+1)}\left(h_{1j}^{-2}\right)$, $E_{h_{1j}^2|{\gamma }_j=0}^{(t+1)}\left(\log h_{1j}^2\right)$ according to $\mathrm{q}\left(h_{1j}^2\right)$,

Update ${\mathrm{E}}_{{\gamma }_j}^{(t+1)}\left({\gamma }_j\right)$ according to the variational posterior $\mathrm{q}\left({\gamma }_j\right)$,

Update ${\mathrm{E}}_{{\beta }_j}^{(t+1)}\left({\beta }_j\right)$ and ${\mathrm{Var}}_{{\beta }_j}^{(t+1)}\left({\beta }_j\right)$,

end for

Update ${\mathrm{E}}_{{\lambda }_0^2}^{(t+1)}\left({\lambda }_0^2\right)$ according to the variational posterior $\mathrm{q}\left({\lambda }_0^2\right)$,

Update ${\mathrm{E}}_{{\lambda }_0^{2}1}^{(t+1)}\left({\lambda }_1^2\right)$ according to the variational posterior $\mathrm{q}\left({\lambda }_1^2\right)$,

Update ${\mathrm{E}}_{{\pi }_{\gamma }}^{(t+1)}\left({\pi }_{\gamma }\right)$, ${\mathrm{E}}_{{\pi }_{\gamma }}^{(t+1)}\left(\log {\pi }_{\gamma }\right)$ and ${\mathrm{E}}_{{\pi }_{\gamma }}^{(t+1)}\left(\log (1-{\pi }_{\gamma })\right)$ according to $\mathrm{q}\left({\pi }_{\gamma }\right)$,

for $i=1$ to n do

Update ${\mathrm{E}}_{z_i}^{(t+1)}\left(z_i\right)$ and ${\mathrm{E}}_{z_i}^{(t+1)}\left(z_i^{-1}\right)$ according to the variational posterior $\mathrm{q}\left(z_i\right)$,

end for

Update ${\mathrm{E}}_{\sigma }^{(t+1)}\left(\sigma \right)$ and ${\mathrm{E}}_{\sigma }^{(t+1)}\left({\sigma }^{-1}\right)$ according to the variational posterior $\mathrm{q}\left(\sigma \right)$,

$\left|d^{(t+1)}\right|=\max \left\{|{\mathbf{\theta }}_{q_1}^{(t+1)}-{\mathbf{\theta }}_{q_1}^{\left(t\right)}|,...,|{\mathbf{\theta }}_{q_m}^{(t+1)}-{\mathbf{\theta }}_{q_m}^{\left(t\right)}|\right\}$,

end while

In Algorithm 1 above, $\Psi (·)$ is the digamma function and the expectation ${\mathrm{E}}_{h_{0j}^2|{\gamma }_j=1}^{(t+1)}\left(\log h_{0j}^2\right)$ of $\log h_{0j}^2$ with respect to generalized inverse Gaussian distribution. Thus, we assume that $x\sim \mathrm{GIG}(p,a,b)$, then:

$$\begin{array}{c}\hfill {\mathrm{E}}_x\left(\log x\right)=\frac{\frac{\mathrm{d}{\mathrm{K}}_p\left(\sqrt{ab}\right)}{\mathrm{d}p}}{{\mathrm{K}}_p\left(\sqrt{ab}\right)}-\frac{1}{2}\ln \left(\frac{a}{b}\right),\end{array}$$

(11)

where ${\mathrm{K}}_p(·)$ represents the Bessel function of the second kind. Note that there is no analytic solution or function to the differential of the modified Bessel function. Therefore, we approximate ${\mathrm{E}}_x\left(\log x\right)$ using the second-order Taylor expansion of $\log x$. This paper lists the expectations of some parameter functions about variational posteriors involved in Algorithm 1; see Appendix B for details. Based on our proposed VBSSLQR algorithm, in the next section, we randomly generate high-dimensional data, conduct simulation research, and compare the performance with other methods. Notably, the asymptotic variance of the quantile regression is reciprocal proportional to the density of the errors at the quantile point. In cases where n is small and we estimate extreme quantiles, the correlating asymptotic variance will be large, resulting in less precise estimates [23]. Therefore, the regression coefficient is difficult to estimate at an extreme quantile and this is feasible when the sample sizes needs to be increased appropriately.

3. Simulation Studies

In this section, we used simulated high-dimensional data with a sample size $n=200$ and variable number $r=500$, in order to study the performance of VBSSLQR and compare it with existing methods, including linear regression with a lasso penalty (Lasso), linear regression with an adaptive lasso penalty (ALasso), quantile regression with a lasso penalty (QRL), quantile regression with an adaptive lasso penalty (QRAL), Bayesian regularized quantile regression with a lasso penalty (BLQR), and Bayesian regularized quantile regression with an adaptive lasso penalty (BALQR). The data in the simulation studies were generated using Equation (1), in which the covariate vector ${\mathit{x}}_i$ was randomly generated from the multivariate normal distribution $N(\mathbf{0},\mathsf{\Sigma })$ with the $(k,l)$th of $\mathsf{\Sigma }$ being $0.5^{|k-l|}$. Among these covariates, we only considered the ten important explanatory variables that have significant impact on the dependent variable. We set the 1, 51, 101, 151, 201, 251, 301, 351, 401, and 451 predictors to be active, and their regression coefficients are −3, −2.5, −2, −1.5, −1, 1, 1.5, 2, 2.5, 3, and the rest are zero. In addition, we discuss the performance of various approaches in the case of two types of random error; namely, independent and identically distributed (i.i.d.) random errors and heterogeneous random errors.

3.1. Independent and Identically Distributed Random Errors Random

In this subsection, with reference to [19,23], we set the random errors ${\epsilon }_i$’s in Equation (1) to be independently and identically distributed and consider the following five different distributions with $\tau$ quantile being zero:

• The error ${\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }N(-\mu ,1)$ with $\mu$ being the $\tau$ quantile of $N(0,1)$, for $i=1,\cdots ,n$;
• The error ${\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Laplace}(-\mu ,1)$ with $\mu$ being the $\tau$ quantile of $\mathrm{Laplace}(0,1)$, where $\mathrm{Laplace}(a,b)$ denotes the Laplace distribution with location parameter a and scale parameter b;
• The error ${\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }0.1N(-{\mu }_1,9)+0.9N(-{\mu }_2,1)$ with ${\mu }_1$ and ${\mu }_2$ respectively being the $\tau$ quantile of $N(0,9)$ and $N(0,1)$;
• The error ${\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }0.1\mathrm{Laplace}$$(-{\mu }_1,9)+0.9\mathrm{Laplace}(-{\mu }_2,1)$ with ${\mu }_1$ and ${\mu }_2$ respectively being the $\tau$ quantile of $\mathrm{Laplace}(0,9)$ and $\mathrm{Laplace}(0,1)$;
• The error ${\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Cauchy}(-\mu ,0.2)$ with $\mu$ being the $\tau$ quantile of $\mathrm{Cauchy}(0,0.2)$, where $\mathrm{Cauchy}(a,b)$ denotes the Cauchy distribution with location parameter a and scale parameter b;

For all of the above error distributions with any $\tau \in (0.3,0.5,0.7)$, we ran 500 replications for each method, and evaluated the performance using two criteria. The first criterion was the median of mean absolute deviations (MMAD), which quantifies the general distance between the estimated conditional quantile and the true conditional quantile. Specifically, the mean absolute deviation (MAD) in any replication is defined as $\frac{1}{n}{\sum }_{i=1}^n|{\mathit{x}}_i\mathbf{\beta }-{\mathit{x}}_i{\widehat{\mathbf{\beta }}}_{\tau }\left)\right|,$ where ${\widehat{\mathbf{\beta }}}_{\tau }$ is the estimate of regression coefficient $\mathbf{\beta }$ given $\tau$. The second criterion of the mean of true positives (TP) and false positives (FP) was selected for each method.

Table 1 shows the median and standard deviation (SD) of MADs estimated using each method for simulations with homogeneous errors. It is clear that our method was either optimal (bold) in all cases, especially in quantile $\tau =0.3,0.7$, or that our approach was significantly superior to the other six methods. When the error distribution was normal, a Laplace and normal mixture, the MMAD of the BALQR method was suboptimal. When the error distribution was Cauchy, the MMAD of the QRL approach was suboptimal, but Table 2 shows that this sacrificed the complexity of the model. When the error distribution was a Laplace mixture, the MMAD of the lasso approach was suboptimal; however, it selected an overfitting quantile regression model with about 30 FP variables, as can be seen from Table 2. It is particularly important to note that, in the case of high dimensional data, the MMAD for the quantile regression model with the lasso penalty or adaptive lasso penalty is not less than the MMAD for general linear models with a lasso penalty. Therefore, it is inappropriate to use a lasso penalty or adaptive lasso in this case.

Table 1. The median and standard deviation of 500 MADs estimated using various methods in simulations with i.i.d. errors.

Quantile	Error Distribution	Method
		Lasso	ALasso	QRL	QRAL	BLQR	BALQR	VBSSLQR
$\tau =0.3$	normal	0.60 (0.06)	0.89 (0.14)	0.88 (0.07)	3.90 (0.20)	1.28 (0.17)	0.55 (0.09)	$\mathbf{0.21}$ (0.05)
	Laplace	0.72 (0.08)	0.97 (0.15)	1.06 (0.15)	3.95 (0.20)	1.44 (0.21)	0.68 (0.20)	$\mathbf{0.24}$ (0.06)
	normal mixture	0.76 (0.09)	1.04 (0.14)	1.02 (0.15)	3.95 (0.21)	1.42 (0.19)	0.71 (0.16)	$\mathbf{0.23}$ (0.05)
	Laplace mixture	0.89 (0.13)	1.13 (0.18)	1.24 (0.33)	4.06 (0.23)	1.69 (0.32)	1.07 (0.31)	$\mathbf{0.26}$ (0.07)
	Cauchy	1.23 (4.59)	1.46 (12.07)	0.59 (0.62)	4.24 (12.40)	1.92 (1.07)	1.98 (4.95)	$\mathbf{0.11}$ (0.60)
$\tau =0.5$	normal	0.40 (0.05)	0.81 (0.16)	0.80 (0.07)	3.90 (0.21)	1.21 (0.19)	0.29 (0.07)	$\mathbf{0.20}$ (0.05)
	Laplace	0.56 (0.08)	0.90 (0.17)	1.01 (0.16)	3.96 (0.21)	1.35 (0.22)	0.46 (0.23)	$\mathbf{0.20}$ (0.05)
	normal mixture	0.53 (0.07)	0.89 (0.17)	0.96 (0.13)	3.95 (0.20)	1.36 (0.25)	0.41 (0.21)	$\mathbf{0.21}$ (0.05)
	Laplace mixture	0.74 (0.13)	1.02 (0.20)	1.20 (0.22)	4.01 (0.23)	1.72 (0.32)	0.83 (0.32)	$\mathbf{0.21}$ (0.06)
	Cauchy	1.31 (28.87)	1.44 (10.15)	0.48 (0.16)	4.40 (32.20)	2.026 (1.16)	4.28 (4.47)	$\mathbf{0.07}$ (0.71)
$\tau =0.7$	normal	0.61 (0.06)	0.93 (0.16)	0.88 (0.07)	3.88 (0.21)	1.22 (0.19)	0.56 (0.11)	$\mathbf{0.20}$ (0.05)
	Laplace	0.71 (0.08)	1.02 (0.16)	1.06 (0.16)	3.97 (0.20)	1.36 (0.27)	0.67 (0.20)	$\mathbf{0.25}$ (0.07)
	normal mixture	0.75 (0.09)	1.06 (0.17)	1.01 (0.14)	3.94 (0.21)	1.32 (0.23)	0.75 (0.19)	$\mathbf{0.22}$ (0.05)
	Laplace mixture	0.89 (0.13)	1.20 (0.21)	1.24 (0.32)	4.01 (0.22)	1.65 (0.33)	0.99 (0.36)	$\mathbf{0.26}$ (0.09)
	Cauchy	1.22 (8.78)	1.60 (8.43)	0.60 (0.60)	4.27 (13.65)	2.28 (1.25)	3.02 (2.91)	$\mathbf{0.12}$ (1.14)

The bold represents the optimal result in each scenario.

Table 2. Mean TP/FP of various methods for simulation with i.i.d. errors.

Quantile	Error Distribution	Method
		Lasso	ALasso	QRL	QRAL	BLQR	BALQR	VBSSLQR
		TP/FP	TP/FP	TP/FP	TP/FP	TP/FP	TP/FP	TP/FP
$\tau =0.3$	normal	10.00/38.24	9.80/0.20	10.00/175.01	5.00/95.33	8.48/0.00	10.00/0.02	$\mathbf{10.00}/\mathbf{0.02}$
	Laplace	10.00/40.43	9.75/0.34	9.99/169.28	5.00/94.88	8.06/0.00	9.86/0.13	$\mathbf{10.00}/\mathbf{0.01}$
	normal mixture	10.00/38.84	9.73/0.31	9.99/163.96	5.00/95.39	8.19/0.00	9.90/0.10	$\mathbf{10.00}/\mathbf{0.00}$
	Laplace mixture	10.00/37.99	9.62/0.71	9.87/146.31	5.00/95.25	7.43/0.00	9.50/0.70	$\mathbf{10.00}/\mathbf{0.01}$
	Cauchy	8.21/29.05	7.95/7.09	9.87/97.34	4.66/94.49	6.11/0.00	7.35/13.24	$\mathbf{9.83}/\mathbf{0.00}$
$\tau =0.5$	normal	10.00/39.00	9.75/0.29	10.00/156.04	5.00/94.73	8.37/0.00	10.00/0.01	$\mathbf{10.00}/\mathbf{0.01}$
	Laplace	10.00/38.06	9.69/0.47	9.99/107.53	5.00/95.30	7.97/0.00	9.81/0.09	$\mathbf{10.00}/\mathbf{0.00}$
	normal mixture	10.00/39.03	9.69/0.57	10.00/112.74	5.00/95.33	8.10/0.00	9.86/0.06	$\mathbf{10.00}/\mathbf{0.00}$
	Laplace mixture	10.00/39.71	9.55/0.83	9.94/71.54	5.00/94.77	7.40/0.00	9.53/0.47	$\mathbf{10.00}/\mathbf{0.00}$
	Cauchy	8.19/29.31	8.04/7.06	10.00/8.50	4.55/95.67	5.53/0.00	6.03/22.94	$\mathbf{9.77}/\mathbf{0.00}$
$\tau =0.7$	normal	10.00/38.85	9.70/0.41	10.00/174.87	5.00/95.40	8.41/0.00	9.99/0.03	$\mathbf{10.00}/\mathbf{0.01}$
	Laplace	10.00/37.68	9.64/0.67	9.99/168.15	5.00/95.05	7.91/0.00	9.77/0.13	$\mathbf{10.00}/\mathbf{0.01}$
	normal mixture	10.00/38.38	9.63/0.63	9.99/165.65	5.00/95.76	8.11/0.00	9.88/0.08	$\mathbf{10.00}/\mathbf{0.01}$
	Laplace mixture	10.00/39.64	9.41/1.22	9.88/150.33	5.00/95.56	7.32/0.00	9.50/1.03	$\mathbf{9.99}/\mathbf{0.01}$
	Cauchy	8.35/30.24	7.54/8.17	9.89/97.85	4.65/95.19	5.73/0.00	6.81/17.95	$\mathbf{9.36}/\mathbf{0.00}$

The bold represents the optimal result in each scenario.

In order to show the results of variable selection more intuitively, we introduced TP (true positives) and FP (false positives), to calculate the mean of TP and FP of 500 repeated simulations, respectively. Detailed results are shown in Table 2 below:

It is possible to conclude from the results in Table 2 that the lasso method can generally select all true active variables, but it fits many false active variables; and when the random error is a Cauchy distribution, it cannot select all true active variables; A lasso approach can identify true active variables, but there are also some misjudged behaviors, especially when the random error distribution is a Cauchy distribution; Although the QRL approach can identify real active variables, it still contains many false active variables, and the model it identifies has a high complexity; the QRAL method cannot identify all true active variables, and incorrectly identifies some inactive variables; In the case of high-dimensional data, the BLQR approach cannot select all the true active variables, but it also does not incorrectly select inactive variables. The true active variable selected using the BALQR method is better than using the BLQR method, but some inactive variables are incorrectly selected, especially when the random error distribution is the Cauchy distribution. Our VBSSLQR method not only had the smallest MMAD, but also could select true active variables and eliminate most false active variables. As a whole, our method was superior to the other six methods for variable selection, especially in quantile $\tau =0.3,0.7$, performing significantly better than BLQR, BALQR, and QRL.

3.2. Heterogeneous Random Errors

Now we consider the case of heterogeneous random errors, to demonstrate the performance of our method. In this subsection, the data were generated from the following model:

$$\begin{array}{c}\hfill y_i={\mathit{x}}_i^{\top }\mathbf{\beta }+(1+u_i){\epsilon }_i,\end{array}$$

(12)

where $u_i\stackrel{i.i.d.}{\sim }U(0,1)$, in which $U(a,b)$ represents the uniform distribution with support set $(a,b)$. The design matrix ${\mathit{x}}_i$ was generated in the same way as above, and the regression coefficient $\mathbf{\beta }$ was set as before. Furthermore, in the simulation study, we combined ${\mathit{x}}_i$ and $u_i$; that is, $u_i$ was also a covariate. Finally, random error ${\epsilon }_i$ was also generated from the five different distributions defined in Section 3.1.

We also studied the performance of the quantile $\tau \in (0.3,0.5,0.7)$ under the different methods and simulated 500 times, to calculate the MMAD and mean of TP/FP. We list the experimental results in Table 3 and Table 4.

Table 3. The median and standard deviation of 500 MADs were estimated via various methods for simulations with heterogeneous random errors.

Quantile	Error Distribution	Method
		Lasso	ALasso	QRL	QRAL	BLQR	BALQR	VBSSLQR
$\tau =0.3$	normal	0.91 (0.10)	1.13 (0.15)	1.31 (0.14)	3.96 (0.22)	1.57 (0.22)	0.90 (0.20)	$\mathbf{0.30}$ (0.07)
	Laplace	1.08 (0.13)	1.24 (0.17)	1.61 (0.39)	4.09 (0.23)	1.93 (0.29)	1.23 (0.35)	$\mathbf{0.36}$ (0.13)
	normal mixture	1.14 (0.14)	1.32 (0.17)	1.55 (0.35)	4.06 (0.24)	1.92 (0.31)	1.31 (0.35)	$\mathbf{0.33}$ (0.11)
	Laplace mixture	1.36 (0.21)	1.51 (0.23)	2.34 (0.60)	4.27 (0.26)	2.30 (0.42)	2.05 (0.52)	$\mathbf{0.39}$ (0.24)
	Cauchy	1.91 (102.51)	1.94 (33.45)	1.17 (0.80)	4.80 (19.32)	2.49 (1.11)	3.90 (3.39)	$\mathbf{0.15}$ (0.98)
$\tau =0.5$	normal	0.61 (0.08)	0.93 (0.17)	1.21 (0.12)	3.96/0.21	1.39 (0.27)	0.50 (0.22)	$\mathbf{0.30}$ (0.07)
	Laplace	0.87 (0.13)	1.12 (0.20)	1.53 (0.25)	4.09 (0.23)	1.84 (0.33)	1.06 (0.35)	$\mathbf{0.29}$ (0.08)
	normal mixture	0.81 (0.12)	1.08 (0.19)	1.44 (0.21)	4.05 (0.23)	1.77 (0.31)	1.08 (0.37)	$\mathbf{0.31}$ (0.08)
	Laplace mixture	1.12 (0.21)	1.31 (0.25)	1.84 (0.32)	4.28 (0.29)	2.25 (0.41)	1.65 (0.56)	$\mathbf{0.32}$ (0.12)
	Cauchy	1.93 (2.23)	1.93 (107.59)	0.74 (0.27)	4.93 (15.90)	2.88 (1.23)	3.72 (4.80)	$\mathbf{0.09}$ (0.88)
$\tau =0.7$	normal	0.92 (0.09)	1.20 (0.17)	1.32 (0.14)	3.97/0.20	1.38 (0.26)	0.86 (0.21)	$\mathbf{0.31}$ (0.09)
	Laplace	1.07 (0.13)	1.31 (0.18)	1.61 (0.37)	4.09 (0.22)	1.95 (0.33)	1.29 (0.31)	$\mathbf{0.37}$ (0.15)
	normal mixture	1.14 (0.14)	1.40 (0.19)	1.53 (0.33)	4.08 (0.23)	1.85 (0.36)	1.31 (0.29)	$\mathbf{0.34}$ (0.10)
	Laplace mixture	1.38 (0.20)	1.58 (0.23)	2.16 (0.61)	4.32 (0.28)	2.37 (0.52)	1.86 (0.50)	$\mathbf{0.40}$ (0.25)
	Cauchy	1.76 (3.76)	2.01 (17.31)	1.24 (0.76)	4.65 (31.22)	4.01 (1.30)	4.31 (18.21)	$\mathbf{0.16}$ (1.04)

The bold represents the optimal result in each scenario.

Table 4. Mean TP/FP of the various methods for simulations with heterogeneous random errors.

Quantile	Error Distribution	Method
		Lasso	ALasso	QRL	QRAL	BLQR	BALQR	VBSSLQR
		TP/FP	TP/FP	TP/FP	TP/FP	TP/FP	TP/FP	TP/FP
$\tau =0.3$	normal	10.00/40.76	9.72/0.35	9.98/170.29	5.00/95.23	7.97/0.00	9.75/0.13	$\mathbf{10.00}/\mathbf{0.03}$
	Laplace	10.00/38.80	9.54/0.83	9.73/140.09	4.99/94.59	7.23/0.00	9.31/1.20	$\mathbf{9.97}/\mathbf{0.05}$
	normal mixture	10.00/38.79	9.59/0.74	9.74/136.63	5.00/95.82	7.29/0.00	9.35/1.23	$\mathbf{9.98}/\mathbf{0.04}$
	Laplace mixture	9.95/37.96	9.17/2.07	8.82/76.98	4.96/95.42	6.53/0.00	8.51/4.48	$\mathbf{9.83}/\mathbf{0.05}$
	Cauchy	7.46/26.86	7.15/10.23	9.56/66.04	4.43/95.56	4.86/0.00	5.73/24.57	$\mathbf{9.52}/\mathbf{0.00}$
$\tau =0.5$	normal	10.00/39.21	9.67/0.44	9.99/142.56	5.00/95.56	7.93/0.00	9.86/0.16	$\mathbf{10.00}/\mathbf{0.05}$
	Laplace	10.00/39.03	9.49/0.93	9.81/89.76	5.00/95.33	7.07/0.00	9.32/1.09	$\mathbf{9.99}/\mathbf{0.01}$
	normal mixture	10.00/38.44	9.51/0.86	9.83/85.73	4.99/95.27	7.41/0.00	9.28/0.96	$\mathbf{9.99}/\mathbf{0.02}$
	Laplace mixture	9.96/38.91	9.26/2.08	9.33/31.13	4.95/95.76	6.23/0.00	8.75/4.13	$\mathbf{9.97}/\mathbf{0.02}$
	Cauchy	7.24/25.47	7.23/9.37	9.99/4.30	4.26/95.95	4.54/0.00	5.92/24.88	$\mathbf{9.60}/\mathbf{0.00}$
$\tau =0.7$	normal	10.00/40.58	9.59/0.52	9.98/171.43	5.00/95.53	7.88/0.00	9.76/0.16	$\mathbf{10.00}/\mathbf{0.07}$
	Laplace	10.00/39.20	9.44/1.14	9.74/144.66	4.99/95.20	6.87/0.00	9.26/0.97	$\mathbf{9.95}/\mathbf{0.06}$
	normal mixture	10.00/38.29	9.38/1.14	9.76/138.14	5.00/95.49	7.04/0.00	9.30/1.30	$\mathbf{9.99}/\mathbf{0.02}$
	Laplace mixture	9.96/38.88	9.08/2.37	8.93/85.78	4.96/95.02	5.95/0.00	8.69/3.50	$\mathbf{9.81}/\mathbf{0.05}$
	Cauchy	7.64/27.34	7.16/10.19	9.61/68.75	4.52/96.36	3.59/0.00	5.79/24.92	9.46/0.00

The bold represents the optimal result in each scenario.

For heterogeneous random errors, our approach was still the best, similarly to for i.i.d. random errors. It is noteworthy that the VBSSLQR method was more robust than the other methods, and our method had the smallest MMAD change compared to the case of i.i.d. random errors. We can see that the MMAD of our method remained basically unchanged, while the MMAD of the other six methods differed by more than 0.5 compared to the i.i.d. random errors in some states.

We also investigated the mean of TP and FP for heterogeneous random errors under different quantiles $\tau$. The results are listed in Table 4, which shows that the effect of variable selection of heterogeneous random errors was slightly lower than the effect of variable selection of i.i.d. random errors under the same sample size, but our method still provided the best selection results.

We also calculated the mean execution times of various Bayesian quantile regressions under different quantile $\tau$ for different distributions of random errors, and list the results in Table A1 of Appendix C, which illustrates that our proposed VBSSLQR approach was a lot more efficient than BLQR and BALQR, for which we sampled MCMC 1000 times and discarded the first 500 times, and made a statistical inference based on 500 samples (the experimental study shows that the algorithm converged after 500 samples). In order to illustrate the feasibility of applying our proposed VBSSLQR to cases with smaller effect sizes, we changed the above active predictors to 1, while the other settings remained unchanged. The performance of our proposed method is shown in Table A2, which illustrates that the results were not significantly different from those of Table 1 and Table 2 when random errors had an independent identical distribution, and slightly worse than those of Table 3 and Table 4 when the random errors were heterogeneous.

4. Examples

In this section, we analyzed a real dataset containing information about crime in various cities across the United States. This dataset is accessible from the University of Irvine machine learning repository (http://archive.ics.uci.edu/ml/datasets/communities+and+crime, accessed on 1 May 2023). We calculated the per capital rate of violent crimes by dividing the total number of violent crimes by the population of each city. The violent crimes considered in our analysis were those classified as murder, rape, robbery, and assault, as per United States law. The observed individuals were communities. The dataset has 116 variables, where the first four columns are response, name of the community, code of county, and code of community; the middle features are demographic information about each community, such as population, age, race, and income; and the final columns are regions. According to the source: “the data combines socio-economic data from the 1990 US Census, law enforcement data from the 1990 US LEMAS survey, and crime data from the 1995 FBI UCR”. This dataset has been applied for quantile regression [30]. The dataset is available at

• train set: https://academics.hamilton.edu/mathematics/ckuruwit/Data/Crime/train.csv (accessed on 1 May 2023).
• test set: https://academics.hamilton.edu/mathematics/ckuruwit/Data/Crime/test.csv (accessed on 1 May 2023).

Our dependent variable of interest was the murder rate of each city, denoted as $y_i$ for the ith city. As the murder rate denotes the most dangerous violent crimes, we choose this variable. Studying factors correlated with the response dependent variable is of significant importance for the public and law enforcement agencies.

To adapt the data to our model, we preprocessed the data as follows:

• Delete columns from the data set that contain missing data.
• Delete the data when the response variable $y_i$ equals 0, because this is not an issue of interest to us.
• Transform $y_i$: $y_i^{\prime }=\log \frac{y_i}{1-y_i}$ and let $y_i^{\prime }$ be the new response variable.
• Convert some qualitative variables into quantitative variables.
• Standardized covariates.

After the above data preprocessing, we obtained 1060 observation objects and 95 covariates in the training set, and we obtained 122 observation objects and 95 covariates in the testing set. We implemented a quantile regression model between the 96 predictors (including intercept) and the response $y_i^{\prime }$ with different quantiles.

In this section, we compare QRL, QRAL, BLQR, and BALQR with our method VBSSLQR for real datasets, all with quantile penalty regression. Under different quantiles $\tau \in (0.1,0.3,0.5,0.7,0.9)$, we compared the performance of the different approaches. We counted the root mean squared error (RMSE) of each method under each quantile $\tau$ and the number of selected active variables, to evaluate the performance of each approach on the test set, where the RMSE was evaluated using $\mathrm{RMSE}=\sqrt{\frac{1}{n}{\sum }_{i=1}^n{(y_i-{\widehat{y}}_{\tau i})}^2}$ with ${\widehat{y}}_{\tau i}$ being the fitted value of response $y_i$ under quantile $\tau$. Finally, the results are listed in Table 5 below.

Table 5. RMSE of fitting test dataset and the number of active variables for the various methods for the real dataset.

Method	RMSE/Number of Active Variables
	$\mathbf{\tau }=\mathbf{0.1}$	$\mathbf{\tau }=\mathbf{0.3}$	$\mathbf{\tau }=\mathbf{0.5}$	$\mathbf{\tau }=\mathbf{0.7}$	$\mathbf{\tau }=\mathbf{0.9}$
QRL	1.36/64	0.94/65	1.09/66	0.93/57	1.21/53
QRAL	3.11/37	3.05/40	3.08/37	2.84/36	3.10/32
BLQR	1.34/1	1.08/1	1.03/1	1.08/1	1.30/1
BALQR	1.67/2	1.08/1	1.02/1	1.08/1	1.30/1
VBSSLQR	1.34/12	$\mathbf{0.94}$/9	$\mathbf{0.78}$/12	$\mathbf{0.83}$/7	$\mathbf{1.14}$/9

The bold represents the optimal result in each scenario.

To visually show the results listed in Table 4, the best results are in bold under each quantile $\tau$. Clearly, our method performed better than the other methods for all quantiles, and the active variables selected by our method were suitable, which means that our method was very competitive compared with the other methods. The BLQR and BALQR approaches could only identify the intercept, and they could not identify the variables that really affected the response quantile. The QRL and QRAL methods were prone to overfitting, because they recognized too many active variables. Finally, the efficiency of our method with real data was also significantly higher than that of the other approaches. Although BLQR, QRL, and our proposed method had the same RMSE performance at $\tau =0.1,0.3$, BLQR showed underfitting and QRL showed overfitting. Thus, we believe there is sufficient evidence to show that our method was very competitive with the other approaches.

Similarly to [30], we list the active variables selected under each quantile in Table 6. Thus, Table 6 shows the variable selection of our proposed method with real data.

Table 6. Crime data analysis: variable selection.

Quantile Level ($\mathit{\tau }$)	Quantile Specific Variables
0.1	racepctblack	pctUrban	PctLess9thGrade	TotalPctDiv
	PctNotHSGrad	PctOccupMgmtProf	FemalePctDiv	PctPersDenseHous
	NumInShelters	PctHousOccup	PctBornSameState
0.3	racepctblack	NumInShelters	FemalePctDiv	PctPersDenseHous
	TotalPctDiv	PctHousOwnOcc	RentLowQ	MedRent
0.5	racePctWhite	racePctHisp	FemalePctDiv	TotalPctDiv
	PctWorkMom	PctSpeakEnglOnly	HousVacant	PctVacantBoarded
	RentLowQ	MedRent	NumInShelters
0.7	racePctWhite	racePctAsian	FemalePctDiv	TotalPctDiv
	PctVacantBoarded		NumInShelters
0.9	racePctWhite	racePctAsian	indianPerCap	PctOccupManu
	MalePctDivorce	PctHousOccup	PctVacantBoarded	NumInShelters

In the above table, our method selected only a small number of predictors at each quantile level. Notably, only the variable “NumInShelters” had an impact on the response at all quantiles; the variables “FemalePctDiv” and “TotalPctDiv” had an impact on the response at $\tau =0.1,0.3,0.5,0.7$; the variable “PctVacantBoarded” had an impact on the response at $\tau =0.5,0.7,0.9$; the variable “racePctWhite” had an impact on the response at $\tau =0.5,0.7,0.9$; and the other variables affected a few quantiles of response. Therefore, the five variables selected from the quantile regression model were

• PctVacantBoarded: percentage of households that are vacant and boarded up to prevent vandalism.
• NumInShelters: number of shelters in the community.
• FemalePctDiv: percentage of females who are divorced.
• TotalPctDiv: percentage of people who are divorced.
• racePctWhite: percentage of people of white race.

In order to obtain a better understanding of these five common variables, we plot their correlation to the response.

Figure 1 depicts the relationship between the five common variables and the murder rate. Only the correlation between the NumInShelters and murder rate is not obvious, with MalePctDivorce, RentLowQ, racePctWhite, and PctVacantBoarded significantly affecting the murder rate. The variable racePctWhite and murder rate are negatively correlated, while FemalePctDiv, TotalPctDiv, PctVacantBoarded, and murder rate are positively correlated. This result was in accordance with the practical situation, and it can be seen that our results were basically consistent with that of [30]. Our method could more comprehensively select important variables under the same quantile. Therefore, the percentage of females who are divorced, percentage of persons who are divorced, the percentage of households that are vacant and boarded up to prevent from vandalism, and the percentage of people with white race affect the murder rate.

Figure 1. Correlation between murder rate and common variables.

5. Conclusions

In this paper, we propose variational Bayesian spike-and-slab lasso quantile regression variable selection and estimation. This method applies spike-and-slab lasso prior to each regression coefficient $\mathbf{\beta }$, thus punishing the regression coefficient $\mathbf{\beta }$. The spike prior distribution we choose is a small variance Laplace distribution, while the slab prior distribution is a large variance Laplace distribution [26]. It is precisely because it punishes each regression coefficient $\mathbf{\beta }$ that it has a powerful variable selection function, but this also brings a problem of low efficiency, especially when introducing a spike-and-slab prior into quantile regression (note, the algorithm efficiency of quantile regression is inherently lower than that of general regression approaches). In order to solve the problem of inefficiency (influenced by both the quantile regression and spike-and-slab lasso prior) and to make the algorithm feasible. We introduced a variational Bayesian method, to approximate the posterior distribution of each parameter. The simulation studies and real data analyses illustrated that the quantile regression with the spike-and-slab lasso penalty based on variational Bayesian method performed effectively and exhibited a robust competitiveness compared with other approaches (Bayesian method or non-Bayesian method, quantile regression, or nonquantile regression), especially in the case of high-dimensional data. In future research work, it would be significant to improve the interpretability and computational efficiency of ultra-high-dimensional quantile regression based on VB and dimension reduction techniques.