Variational Bayesian Inference in High-Dimensional Linear Mixed Models

Abstract

In high-dimensional regression models, the Bayesian lasso with the Gaussian spike and slab priors is widely adopted to select variables and estimate unknown parameters. However, it involves large matrix computations in a standard Gibbs sampler. To solve this issue, the Skinny Gibbs sampler is employed to draw observations required for Bayesian variable selection. However, when the sample size is much smaller than the number of variables, the computation is rather time-consuming. As an alternative to the Skinny Gibbs sampler, we develop a variational Bayesian approach to simultaneously select variables and estimate parameters in high-dimensional linear mixed models under the Gaussian spike and slab priors of population-specific fixed-effects regression coefficients, which are reformulated as a mixture of a normal distribution and an exponential distribution. The coordinate ascent algorithm, which can be implemented efficiently, is proposed to optimize the evidence lower bound. The Bayes factor, which can be computed with the path sampling technique, is presented to compare two competing models in the variational Bayesian framework. Simulation studies are conducted to assess the performance of the proposed variational Bayesian method. An empirical example is analyzed by the proposed methodologies.

1. Introduction

Linear mixed models are widely used to analyze longitudinal and correlated data by considering the between-subject and within-subject variations and incorporating the random effects to account for heterogeneity among the subjects in many fields, such as psychology, medicine, epidemiology and econometrics. Various methods have been developed to estimate fixed-effects parameters and variance–covariance matrices for unobservable random effects and noises or select fixed-effects and random-effects components, even if it is quite challenging for the problem of variable selection and parameter estimation in linear mixed models. For example, see [1] for restricted maximum likelihood estimation of parameters, ref [2] for EM algorithm of parameter estimation, refs [3,4] for Bayesian parameter estimation, ref [5] for Bayesian random effects selection and [6] for moment-based method for random effects selection. The aforementioned methods mainly focus on low-dimensional linear mixed models, while high-dimensional data have become increasingly common with the rapid development of modern information technologies that facilitate data collection. Thus, the aforementioned methods do not work well in high-dimensional linear mixed models, and so some penalized methods have developed to simultaneously estimate parameters and select variables in high-dimensional linear mixed models. For example, Bondell, Krishna and Ghosh [7] and Ibrahim et al. [8] proposed the penalized likelihood methods for joint selection of fixed and random effects; Schelldorfer, Buhlmann and van De Geer [9] proposed an ${\ell }_1$–penalized estimation procedure; Fan and Li [10] investigated the problem of fixed and random effects selection when the cluster sizes are balanced; Li et al. [11] presented a doubly regularized approach to simultaneously select fixed and random effects; Bradic, Claeskens and Gueuning [12] considered testing a single parameter of fixed effects in high-dimensional linear mixed models with fixed cluster sizes, fixed numbers of random effects and sub-Gaussian designs; Li, Cai and Li [13] proposed a penalized quasi-likelihood method for statistical inference on unknown parameters in high-dimensional linear mixed-effects models. However, the aforementioned regularization methods are computationally complex and unstable and they do not consider the prior information of fixed-effects parameters and variance–covariance matrices, which may lead to unsatisfactory estimation accuracy of parameters or variance–covariance matrices. Bayesian approaches for variable selection and parameter estimation have received much attention over the past years because they can largely improve the accuracy and efficiency of parameter estimation, consistently select important variables and provide more information for variable selection than the corresponding penalization method with a highly non-convex optimization problem by imposing various priors on model parameters. For example, see [14] for reference prior, ref [15] for normal mixture prior, ref [16] for spike and slab prior, ref [17] for horseshoe prior and [18] for shrinking and diffusing prior. In the high-dimensional setting, Bayesian lasso, Bayesian adaptive lasso or the indicator model method, together with the Markov chain Monte Carlo (MCMC) algorithm, are widely used to select important variables. For example, see [19] for Bayesian lasso, ref [20] for Bayesian adaptive lasso and [21,22] for the EM approach in the Bayesian framework. The above-mentioned literature involves the implementation of the standard Gibbs sampler for posterior computation, which is not so scalable for large numbers of fixed-effects components [23]. To address the issue, Narisetty, Shen and He [23] proposed a Skinny Gibbs algorithm by using a sparse matrix to replace the high-dimensional variance–covariance matrix, which avoids large matrix operations. However, implementing the above MCMC algorithm in the presence of high-dimensional data may still be subject to the well-known ill-posed problems, i.e., low efficiency, slow convergence and huge memory being required.

As an alternative to the MCMC, the variational Bayesian method, also called ensemble learning, is widely adopted to approximate intractable integrals involved in Bayesian inference or machine learning due to its good properties, such as high-speed computation. Its basic idea is to transform the high-dimensional integration problem into an optimization problem in making Bayesian inference and then optimize the evidence lower bound (ELB), which is efficiently computed, and finally utilize the ELB to obtain a variational approximation to the posterior distribution in Bayesian analysis. The variational Bayesian approach has been applied to some familiar models, for example, latent variable models [24], mixtures of factor analysis [25], graphical models [26] and partially linear mean shift models with high-dimensional data [27].

Motivated by the aforementioned variational Bayesian studies, we develop a novel variational Bayesian approach to estimate model parameters and select important variables under the Skinny Gibbs sampling framework in a linear mixed model with low-dimensional random effects and high-dimensional fixed effects. We specify the spike and slab priors for the population-specific fixed-effects regression coefficients with completely different shrinkage parameters, which overcomes the problem of selecting a high-dimensional vector of the shrinkage parameters. We reformulate the spike and slab priors of parameter as a mixture of a normal distribution and an exponential distribution, which avoids the high-dimensional integral problem. The coordinate ascent algorithm, which can be implemented efficiently, is proposed to optimize the ELB. The Bayes factor, which can be computed with the path sampling technique, is presented to compare two competing models in the variational Bayesian framework. The merits of the proposed variational Bayesian method are (i) simultaneously estimating parameters and variance–covariance matrices and select fixed- and random-effects components with quite a low computation cost, (ii) efficiently analyzing high-dimensional data without requiring the non-convex optimization and avoiding the curse of dimensionality problem, (iii) automatically incorporating the shrinkage parameters and (iv) avoiding large matrix computations.

The rest of the article is organized as follows: Section 2 introduces the linear mixed model setup, including the spike and slab priors. Section 3 describes the Skinny Gibbs sampler algorithm for selecting fixed- and random-effects components and estimating parameters in coefficients and variance–covariance matrices via the Bayesian Lasso method. Section 4 develops a variational Bayesian approach to approximate posterior distributions of parameters and random effects and presents the Bayes factor for model comparison. The coordinate ascent algorithm is adopted to optimize the ELB in Section 4. Simulation studies are considered in Section 5. An empirical example is illustrated by the proposed methodologies in Section 6. A simple discussion is given in Section 7. Technique details are presented in the Appendix A, Appendix B and Appendix C.

2. Model

Consider a dataset with n subjects. For the ith subject, let $y_{ij}$ be the observation of the response variable, $x_{ij}$ be a $p\times 1$ vector of covariates associated with the fixed effects and $z_{ij}$ be a $q\times 1$ vector of covariates associated with the random effects, which may be a subvector of $x_{ij}$ for $j=1,\dots ,n_i$, where $n_i$ is the number of times observed repeatedly for the ith subject. Generally, $n_i$ varies across subjects. For simplicity, we suppose that $y_{ij}$ has been centered at zero for avoiding the requirement of intercept and $n_1=\dots =n_n=m$, i.e., the balanced design. It is assumed that $p\gg n$ and only a small number of covariates $x_{ij}$ contribute to response variable $y_{ij}$, i.e., $x_{ij}$ has sparsity, while q is smaller than n.

For the dataset $\mathcal{D}=\{(y_{ij},x_{ij},z_{ij}):i=1,\dots ,n,j=1,\dots ,m\}$, we consider the following linear mixed model:

$$y_{ij}=x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i+{\epsilon }_{ij},i=1,\dots ,n,j=1,\dots ,m,$$

(1)

where $\mathbf{\beta }={({\beta }_1,\dots ,{\beta }_p)}^{\top }$ is a $p\times 1$ vector of population-specific fixed-effects regression coefficients, $b_i$ is a $q\times 1$ vector of subject-specific random effects and ${\epsilon }_{ij}$ is measurement error. Here, we assume that $b_1,\dots ,b_n$ are independent and identically distributed (i.i.d.) as the multivariate normal distribution with mean zero and covariance matrix Q and ${\epsilon }_{ij}$’s are independently distributed as $\mathcal{N}(0,{\sigma }_j^2)$, where $\mathcal{N}(·,·)$ represents the normal distribution. Here, ${\sigma }_1^2,\dots ,{\sigma }_m^2$ are not completely different but some of them may be identical.

Under the aforementioned assumptions, a penalized likelihood approach to estimate $\mathbf{\beta }$ is implemented by

$$\widehat{\mathbf{\beta }}=\arg \underset{\beta \in {\mathbb{R}}^p}{max}\left[-\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^m{\displaystyle \frac{{(y_{ij}-x_{ij}^{\top }\mathbf{\beta })}^2}{{\sigma }_j^2+z_{ij}^{\top }Q{z}_{ij}}}+f_{\lambda }\left(\mathbf{\beta }\right)\right],$$

(2)

where $f_{\lambda }\left(\mathbf{\beta }\right)$ is some appropriate penalty function indexed by the penalty parameter $\lambda$. In variable selection literature, it is usually assumed that $f_{\lambda }\left(\mathbf{\beta }\right)$ has the form: $f_{\lambda }\left(\mathbf{\beta }\right)={\sum }_{k=1}^p{f}_{{\lambda }_k}\left({\beta }_k\right)$, where $f_{{\lambda }_k}\left({\beta }_k\right)$ takes the ${\ell }_0$-norm, ${\ell }_1$-norm, MCP penalty [28], SCAD penalty [29] and Elastic-Net penalty [30]. Recently, it was widely recognized that $\widehat{\mathbf{\beta }}$ can be regarded as a posterior mode of $\mathbf{\beta }$ with some proper prior. Inspired by this idea, we consider Bayesian variable selection procedure based on some proper prior on $\mathbf{\beta }$.

Following [31], we consider the following spike and slab prior of $\mathbf{\beta }$:

$$f\left(\mathbf{\beta }\right|\mathbf{\gamma },{\lambda }_0,{\lambda }_1)=\prod _{k=1}^p\{{\gamma }_k{g}_1\left({\beta }_k\right|{\lambda }_1)+(1-{\gamma }_k)g_0\left({\beta }_k\right|{\lambda }_0)\},$$

(3)

where $\mathbf{\gamma }={({\gamma }_1,\dots ,{\gamma }_p)}^{\top }$, in which ${\gamma }_k$ is a binary latent variable and follows a  Bernoulli distribution with the probability ${\rho }_k=\mathrm{Pr}({\gamma }_k=1)$, i.e., ${\gamma }_k=1$ indicates that the kth covariate is active and ${\gamma }_k=0$ implies that the kth covariate is inactive and $g_1\left({\beta }_k\right|{\lambda }_1)$ is usually referred to as a diffuse slab prior reflecting the effect of an active covariate, while $g_0\left({\beta }_k\right|{\lambda }_0)$ is called a concentrated spike prior reflecting the negligibly unimportant effect of an inactive covariate for $k=1,\dots ,p$. Let $f\left(\mathbf{\gamma }\right|\mathbf{\rho })$ be the prior distribution of $\mathbf{\gamma }$ indexed by $\mathbf{\rho }$. It is assumed that $f\left(\mathbf{\gamma }\right|\mathbf{\rho })$ has the form

$$f\left(\mathbf{\gamma }\right|\mathbf{\rho })=\prod _{k=1}^p{\rho }_k^{{\gamma }_k}{(1-{\rho }_k)}^{1-{\gamma }_k},$$

(4)

where $\mathbf{\rho }={({\rho }_1,\dots ,{\rho }_p)}^{\top }$. For simplicity, we assume ${\rho }_1=\dots ={\rho }_p=\rho$, which is the expected proportion of the active covariates. Generally, one can take $g_0(·)$ and $g_1(·)$ as the normal distribution with a small and a large variance, respectively. However, for the spike and slab lasso, we take the following slab and spike priors

$$g_1\left({\beta }_k\right|{\lambda }_1)=\frac{{\lambda }_1}{2}{e}^{-{\lambda }_1\left|{\beta }_k\right|},g_0\left({\beta }_k\right|{\lambda }_0)=\frac{{\lambda }_0}{2}{e}^{-{\lambda }_0\left|{\beta }_k\right|},$$

(5)

respectively, where ${\lambda }_1$ should tend to zero and ${\lambda }_0$ should tend to ∞ as the sample size is sufficiently large, which implies that the inactive covariates will be detected as zeros in that small values of ${\beta }_k$ relative to $1/{\lambda }_0$ or ${\lambda }_1$ are truncated to zero. Following [32], the density $g_{\ell }\left({\beta }_k\right|{\lambda }_{\ell })=\frac{{\lambda }_{\ell }}{2}exp(-{\lambda }_{\ell }|{\beta }_k\left|\right)$ can be hierarchically written as a mixture of a normal distribution and an exponential distribution, i.e.,

$${\beta }_k|{\xi }_{\ell k}^2,{\gamma }_k=\ell \sim \mathcal{N}(0,{\xi }_{\ell k}^2),{\xi }_{\ell k}^2|{\lambda }_{\ell }^2\sim \mathrm{Exp}({\lambda }_{\ell }^2/2),\ell =0,1.$$

(6)

Incorporating the above idea shows that the posterior distributions of binary latent variables can be employed to distinguish active covariates from inactive ones in the considered model.

For covariance matrix Q, the proportion $\rho$, ${\lambda }_0^2$, ${\lambda }_1^2$ and ${\sigma }_j^2$, we consider the following priors:

$$Q\sim \mathrm{IW}(S_0,{\nu }_0),\rho \sim \mathrm{Beta}(a_{\gamma },b_{\gamma }),{\lambda }_0^2\sim \mathsf{\Gamma }(c_0,d_0),{\lambda }_1^2\sim \mathsf{\Gamma }(c_1,d_1),{\sigma }_j^{-2}\sim \mathsf{\Gamma }(c_2,d_2),$$

(7)

where $\mathrm{IW}(·,·)$ denotes the inverted Wishart distribution, $\mathrm{Beta}(·,·)$ represents the Beta distribution, $\mathsf{\Gamma }(·)$ is the gamma distribution, $\mathrm{IG}(·,·)$ is the inverse gamma distribution and $S_0$, ${\nu }_0$, $a_{\gamma }$, $b_{\gamma }$, $c_0$, $d_0$, $c_1$, $d_1$, $c_2$ and $d_2$ are the user-specified hyperparameters. As mentioned above, ${\lambda }_1$ should tend to zero and ${\lambda }_0$ should tend to ∞ as the sample size is sufficiently large, which implies that $c_1/d_1$ is smaller than $c_0/d_0$. To this end, we assume $c_1\ll c_0$ and $d_0\ll d_1$.

Based on the above discussion, we can rewrite the considered linear mixed model together with the spike and slab lasso prior as the following hierarchical models:

$$\left\{\begin{array}{c}{y}_{ij}|b_i\sim \mathcal{N}({\mu }_{ij},{\sigma }_j^2),{\mu }_{ij}=x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i,i=1,\dots ,n,j=1,\dots ,m,\hfill \\ b_i\sim {\mathcal{N}}_q(0,Q),i=1,\dots ,n,\hfill \\ {\beta }_k|{\xi }_{1k}^2,{\gamma }_k=1\sim \mathcal{N}(0,{\xi }_{1k}^2),{\xi }_{1k}^2|{\lambda }_1^2\sim \mathrm{Exp}({\lambda }_1^2/2),{\lambda }_1^2\sim \mathsf{\Gamma }(c_1,d_1),\hfill \\ {\beta }_k|{\xi }_{0k}^2,{\gamma }_k=0\sim \mathcal{N}(0,{\xi }_{0k}^2),{\xi }_{0k}^2|{\lambda }_0^2\sim \mathrm{Exp}({\lambda }_0^2/2),{\lambda }_0^2\sim \mathsf{\Gamma }(c_0,d_0),\hfill \\ {\gamma }_k\sim \mathrm{Bernoulli}\left(\rho \right),k=1,\dots ,p,\hfill \\ Q\sim \mathrm{IW}(S_0,{\nu }_0),\rho \sim \mathrm{Beta}(a_{\gamma },b_{\gamma }),{\sigma }_j^{-2}\sim \mathsf{\Gamma }(c_2,d_2),j=1,\dots ,m.\hfill \end{array}\right.$$

(8)

3. Skinny Gibbs Sampler for Bayesian Lasso

Let $Y=\{y_{ij}:i=1,\dots ,n,j=1,\dots ,m\}$, $X=\{x_{ij}:i=1,\dots ,n,j=1,\dots ,m\}$ and $Z=\{z_{ij}:i=1,\dots ,n,j=1,\dots ,m\}$. From Equation (8), the joint posterior density of parameters $\mathbf{\beta }$, Q, $\mathbf{\gamma }=\{{\gamma }_1,\dots ,{\gamma }_p\}$, ${\mathbf{\sigma }}^2={({\sigma }_1^2,\dots ,{\sigma }_m^2)}^{\top }$ and $\mathbf{\vartheta }=\{\rho ,{\lambda }_0,{\lambda }_1\}$ given the data $\mathcal{D}=\{Y,X,Z\}$ is given by

$$\begin{array}{ccc}f(\mathbf{\beta },Q,\mathbf{\gamma },{\mathbf{\sigma }}^2,\mathbf{\vartheta }|\mathcal{D})\hfill & \propto \hfill & \left\{{\displaystyle \prod _{i=1}^n}{\displaystyle \prod _{j=1}^m}\psi (y_{ij},x_{ij}^{\top }\mathbf{\beta },z_{ij}^{\top }{Q}^{-1}{z}_{ij}+{\sigma }_j^2)\right\}\left\{{\displaystyle \prod _{j=1}^m}f\left({\sigma }_j^{-2}\right)\right\}\hfill \\ \hfill & \hfill & \times {\displaystyle \prod _{k=1}^p}{\left\{\rho g_1\left({\beta }_k\right|{\lambda }_1)\right\}}^{{\gamma }_k}{\left\{(1-\rho )g_0\left({\beta }_k\right|{\lambda }_0)\right\}}^{1-{\gamma }_k}{f}_W\left(Q\right)f_{\vartheta }\left(\mathbf{\vartheta }\right),\hfill \end{array}$$

(9)

where $\psi (x,\mu ,{\varsigma }^2)$ is the probability density of normal random variable x with mean $\mu$ and variance ${\varsigma }^2$, $f\left({\sigma }_j^{-2}\right)$ denotes the probability density of random variable ${\sigma }_j^{-2}$, $f_W\left(Q\right)$ is the inverted Wishart density function of random matrix Q and $f_{\vartheta }\left(\mathbf{\vartheta }\right)$ represents the joint prior density function of random variable vector $\mathbf{\vartheta }$. It is rather difficult to sample observations from the joint posterior density given in Equation (9) in the presence of high-dimensional fixed effects because of some non-standard distributions and large matrix computations involved. In what follows, the Gibbs sampler is utilized to sample observations required for Bayesian inference.

To avoid expensive computation in running the Gibbs sampler, similarly to [23], at each Gibbs iteration, we divide parameter vector $\mathbf{\beta }$ into two subvectors corresponding to those active (i.e., ${\gamma }_k=1$) and inactive (i.e., ${\gamma }_k=0$) covariates, respectively. To wit, we define $\mathbf{\beta }={({\mathbf{\beta }}_A^{\top },{\mathbf{\beta }}_I^{\top })}^{\top }$, where ${\mathbf{\beta }}_A$ and ${\mathbf{\beta }}_I$ are the subvectors of $\mathbf{\beta }$ associated with ${\gamma }_k=1$ and ${\gamma }_k=0$, respectively. Suppose that the cardinality of the set A is r. Without loss of generality, it is assumed that the first r components of $\mathbf{\beta }$ correspond to ${\mathbf{\beta }}_A$ and the last $p-r$ components of $\mathbf{\beta }$ correspond to ${\mathbf{\beta }}_I$. Similarly, we decompose $x_{ij}$ as $x_{ij}={(x_{ijA}^{\top },x_{ijI}^{\top })}^{\top }$. Under the above assumptions, the Gibbs sampler is implemented as follows. Observations required at each Gibbs iteration are iteratively drawn from the following conditional distributions: $f_A\left({\mathbf{\beta }}_A\right|\mathcal{D},b,{\mathbf{\sigma }}^2)$, $f_I\left({\mathbf{\beta }}_I\right|\mathcal{D})$, $f\left(b_i\right|\mathcal{D},\mathbf{\beta },{\mathbf{\sigma }}^2,Q)$, $f\left({\xi }_{0k}^2\right|{\beta }_k,{\gamma }_k)$, $f\left({\xi }_{1k}^2\right|{\beta }_k,{\gamma }_k)$, $f_{\gamma }\left({\gamma }_k\right|\mathcal{D},b,{\mathbf{\xi }}_1,{\mathbf{\xi }}_0)$, $f\left(Q\right|b)$, $f\left({\sigma }_j^{-2}\right|\mathcal{D},b)$, $f\left(\rho \right|\mathbf{\gamma })$, $f\left({\lambda }_0^2\right|{\mathbf{\xi }}_0)$ and $f\left({\lambda }_1^2\right|{\mathbf{\xi }}_1)$, which are given in Appendix A, where $b=\{b_1,\dots ,b_n\}$, ${\mathbf{\xi }}_0=\{{\xi }_{01}^2,\dots ,{\xi }_{0p}^2\}$ and ${\mathbf{\xi }}_1=\{{\xi }_{11}^2,\dots ,{\xi }_{1p}^2\}$.

Although the Skinny Gibbs sampler introduced above can be easily conducted, it is rather time-consuming for a sufficiently large p. To address the issue, we investigate a fast yet efficient approach as follows, i.e., the variational Bayesian method.

4. Variational Bayesian Inference

4.1. Variational Bayes

It follows from the principle of variational inference that it is necessary to first construct a variational set $\mathfrak{F}$ of densities for random variables $\Xi$ having the same support as the posterior density $f(\Xi |\mathcal{D})$, where $\Xi =\{\mathbf{\beta },b,{\mathbf{\xi }}_0,{\mathbf{\xi }}_1,Q,\mathbf{\gamma },{\mathbf{\sigma }}^2,\mathbf{\vartheta }\}$. It is assumed that $q(\Xi )\in \mathfrak{F}$ is any variational density for approximating $f(\Xi |\mathcal{D})$. The variational Bayes aims to find the best approximation to $f(\Xi |\mathcal{D})$ in terms of the Kullback–Leibler divergence between $q(\Xi )$ and $f(\Xi |\mathcal{D})$, which is a solution to the optimization problem:

$$q^{\ast }(\Xi )=\underset{q(\Xi )\in \mathfrak{F}}{\mathrm{argmin}}\mathrm{KL}(q(\Xi )\Vert f(\Xi |\mathcal{D})),$$

(10)

where

$$\begin{array}{ccc}\mathrm{KL}\left(q\right(\Xi )\Vert f(\Xi \left|\mathcal{D}\right))\hfill & =\hfill & \int log\left\{{\displaystyle \frac{q(\Xi )}{f(\Xi |\mathcal{D})}}\right\}q(\Xi )d\Xi \hfill \\ \hfill & =\hfill & \int log\left\{{\displaystyle \frac{q(\Xi )f\left(Y\right|X,Z)}{f(\Xi ,Y|X,Z)}}\right\}q(\Xi )d\Xi \hfill \\ \hfill & =\hfill & {\mathrm{E}}_{q(\Xi )}\left\{logq(\Xi )\right\}-{\mathrm{E}}_{q(\Xi )}\left\{logf(\Xi ,Y|X,Z)\right\}\hfill \\ \hfill & \hfill & +logf\left(Y\right|X,Z)\ge 0,\hfill \end{array}$$

(11)

in which $E_{q(\Xi )}(·)$ is the expectation taken with respect to $q(\Xi )$. Here, $\mathrm{KL}\left(q\right(\Xi )\Vert f(\Xi \left|\mathcal{D}\right))$ equals zero if and only if $q(\Xi )\equiv f(\Xi |\mathcal{D})$. Due to the intractable high-dimensional integral involved, it is quite troublesome to conduct the above optimization problem.

However, it follows from $\mathbb{L}\left\{q(\Xi )\right\}={\mathrm{E}}_{q(\Xi )}\left\{logf(\Xi ,Y|X,Z)\right\}-{\mathrm{E}}_{q(\Xi )}\left\{logq(\Xi )\right\}$ that

$$logf\left(Y\right|X,Z)=\mathrm{KL}(q(\Xi )\Vert f(\Xi |\mathcal{D}\left)\right)+\mathbb{L}\left\{q\right(\Xi \left)\right\}\ge \mathbb{L}\left\{q\right(\Xi \left)\right\}.$$

(12)

Thus, $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ might be regarded as a lower bound of $logf\left(Y\right|X,Z)$ and is usually referred to as the evidence lower bound (ELB). Then, minimizing $\mathrm{KL}\left(q\right(\Xi )\Vert f(\Xi \left|\mathcal{D}\right))$ is equivalent to maximizing $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ in that $logf\left(Y\right|X,Z)$ is not related to $\Xi$. That is,

$$q^{\ast }(\Xi )=\underset{q(\Xi )\in \mathfrak{F}}{\mathrm{argmin}}\mathrm{KL}(q(\Xi )\Vert f(\Xi |\mathcal{D}))=\underset{q(\Xi )\in \mathfrak{F}}{\mathrm{argmax}}\mathbb{L}\left\{q(\Xi )\right\}.$$

(13)

Finding the problem of the best approximation to $f(\Xi |\mathcal{D})$ is transformed into an optimization problem of maximizing $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ over the variational family $\mathfrak{F}$. The complexity of the optimization problem is associated with that of the variational set $\mathfrak{F}$. Thus, it is rather desirable to implement the optimization problem over a relatively simple variational set $\mathfrak{F}$.

Following the widely used methods for constructing a relatively simple variational set, we take $\mathfrak{F}$ as the mean-field variational family in which components of  $\Xi$ are mutually independent and each has a distinct factor in the variational density. Thus, we can assume that the variational density $q(\Xi )$ has the form

$$q(\Xi )=q\left(\mathbf{\beta }\right)q\left(b\right)q\left({\mathbf{\sigma }}^{-2}\right)q\left(\mathbf{\gamma }\right)q\left(Q\right)q\left(\mathbf{\vartheta }\right)\prod _{k=1}^p\left\{q\left({\xi }_{0k}^2\right)q\left({\xi }_{1k}^2\right)\right\}\equiv \prod _{s=1}^{\mathcal{S}}{q}_s\left({\mathbf{\zeta }}_s\right),$$

(14)

where $q_s\left({\mathbf{\zeta }}_s\right)$s are unspecified but the above assumed factorization across components is pre-specified. Similarly to considerable variational literature, the optimal solutions of $q_s\left({\mathbf{\zeta }}_s\right)$s can be obtained by maximizing $\mathbb{L}\left\{q({\mathbf{\zeta }}_1,\dots ,{\mathbf{\zeta }}_{\mathcal{S}})\right\}$ via the coordinate ascent method, where $\Xi =\{{\mathbf{\zeta }}_1,\dots ,{\mathbf{\zeta }}_{\mathcal{S}}\}$.

Following the idea of the coordinate ascent method given in [33,34,35], when fixing other variational factors $q_j\left({\mathbf{\zeta }}_j\right)$ for $j\ne s$, i.e., ${\mathbf{\zeta }}_{-s}=\{{\mathbf{\zeta }}_j:j\ne s,j=1,\dots ,\mathcal{S}\}$, the optimal variational density $q_s^{\ast }\left({\mathbf{\zeta }}_s\right)$ maximizing $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ with respect to $q_s\left({\mathbf{\zeta }}_s\right)$ has the form

$$\begin{array}{ccc}{q}_s^{\ast }\left({\mathbf{\zeta }}_s\right)\hfill & \propto \hfill & exp\left[{\mathrm{E}}_{-s}\left\{logf\left({\mathbf{\zeta }}_s\right|{\mathbf{\zeta }}_{-s},\mathcal{D})\right\}\right]\hfill \\ \hfill & \propto \hfill & exp\left[{\mathrm{E}}_{-s}\left\{logf(Y,\Xi |X,Z)\right\}\right],\hfill \end{array}$$

(15)

where $f\left({\mathbf{\zeta }}_s\right|{\mathbf{\zeta }}_{-s},\mathcal{D})$ is the conditional density for ${\mathbf{\zeta }}_s$ given (${\mathbf{\zeta }}_{-s}$$\mathcal{D}$) and ${\mathrm{E}}_{-s}(·)$ represents the expectation evaluated for $q_{-s}\left({\mathbf{\zeta }}_{-s}\right)={\prod }_{j\ne s}{q}_j\left({\mathbf{\zeta }}_j\right)$. Equation (15) implies that ${\mathrm{E}}_{-s}(·)$ is not associated with the sth variational factor $q_s\left({\mathbf{\zeta }}_s\right)$ and the optimal variational density $q_s^{\ast }\left({\mathbf{\zeta }}_s\right)$ cannot be obtained in that the $q_{-s}\left({\mathbf{\zeta }}_{-s}\right)$ on the right-hand side are not the optimal ones. To address this issue, the coordinate updating algorithm is employed to iteratively update $q_s^{\ast }\left({\mathbf{\zeta }}_s\right)$ via Equation (15). After the coordinate updating algorithm converges, we can take mean or mode of the optimal variational density $q_s^{\ast }\left({\mathbf{\zeta }}_s\right)$ as a variational Bayesian estimate of parameter vector ${\mathbf{\zeta }}_s$ and regard the covariate as active if its corresponding variational Bayesian estimate deviates from zero.

It is easily shown from Equation (15) that the optimal density $q_{\beta }^{\ast }\left(\mathbf{\beta }\right)$ has the form

$$q_{{\beta }_A}^{\ast }\left({\mathbf{\beta }}_A\right)\sim {\mathcal{N}}_r({\mathbf{\mu }}_A,{\mathsf{\Sigma }}_A),q_{{\beta }_I}^{\ast }\left({\mathbf{\beta }}_I\right)\sim {\mathcal{N}}_{p-r}(0,{\mathsf{\Sigma }}_I),$$

(16)

respectively, where ${\mathsf{\Sigma }}_A^{-1}={\sum }_{i=1}^n{\sum }_{j=1}^m{x}_{ijA}{x}_{ijA}^{\top }{E}_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)+\mathrm{diag}\left({\mathbf{\xi }}_A\right)$ with ${\mathbf{\xi }}_A=\{E_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right),k\in A\}$, ${\mathbf{\mu }}_A={\mathsf{\Sigma }}_A\left[{\sum }_{i=1}^n{\sum }_{j=1}^m{x}_{ijA}\{y_{ij}-z_{ij}^{\top }{E}_{b_i}^{\ast }\left(b_i\right)\}{E}_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)\right]$ and ${\mathsf{\Sigma }}_I^{-1}=\mathrm{diag}\left({\sum }_{i=1}^n{\sum }_{j=1}^m{x}_{ijI}{x}_{ijI}^{\top }\right)$$+\mathrm{diag}\left({\mathbf{\xi }}_I^0\right)=nm{I}_{p-r}+\mathrm{diag}\left({\mathbf{\xi }}_I^0\right)$ with ${\mathbf{\xi }}_I^0=\{E_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right),k\in I\}$, in which ${\mathrm{E}}_{{\sigma }_j^2}^{\ast }(·)$, ${\mathrm{E}}_{{\xi }_{1k}}^{\ast }(·)$, ${\mathrm{E}}_{{\xi }_{0k}}^{\ast }(·)$ and ${\mathrm{E}}_{b_i}^{\ast }(·)$ are the expectations taken with respect to $q_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)$, $q_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)$, $q_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)$ and $q_{b_i}^{\ast }\left(b_i\right)$, respectively. Then, the estimated posterior means and variance matrices of ${\mathbf{\beta }}_A$ and ${\mathbf{\beta }}_I$ for the variational densities $q_{{\beta }_A}^{\ast }\left({\mathbf{\beta }}_A\right)$ and $q_{{\beta }_I}^{\ast }\left({\mathbf{\beta }}_I\right)$ are ${\mathrm{E}}_A^{\ast }\left({\mathbf{\beta }}_A\right)={\mathbf{\mu }}_A$, ${\mathrm{var}}_A^{\ast }\left({\mathbf{\beta }}_A\right)={\mathsf{\Sigma }}_A$, ${\mathrm{E}}_I^{\ast }\left({\mathbf{\beta }}_I\right)=0$ and ${\mathrm{var}}_I^{\ast }\left({\mathbf{\beta }}_I\right)={\mathsf{\Sigma }}_I$, respectively. Moreover, the mode estimator ${\mathbf{\beta }}_A^q$ of ${\mathbf{\beta }}_A$ for the variational density $q_{{\beta }_A}^{\ast }\left({\mathbf{\beta }}_A\right)$ is ${\mathbf{\beta }}_A^q={\mathbf{\mu }}_A$, while the mode estimator ${\mathbf{\beta }}_I^q$ of ${\mathbf{\beta }}_I$ for the variational density $q_{{\beta }_I}^{\ast }\left({\mathbf{\beta }}_I\right)$ is ${\mathbf{\beta }}_I^q=0$.

The optimal density $q_{b_i}^{\ast }\left(b_i\right)$ is the multivariate normal distribution

$$q_{b_i}^{\ast }\left(b_i\right)\sim {\mathcal{N}}_q({\mathbf{\mu }}_b,{\mathsf{\Sigma }}_b),$$

(17)

where ${\mathsf{\Sigma }}_b^{-1}=E_Q^{\ast }\left(Q\right)+{\sum }_{j=1}^m{z}_{ij}{z}_{ij}^{\top }{E}_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)$ and ${\mathbf{\mu }}_b={\mathsf{\Sigma }}_b\left[{\sum }_{j=1}^m{z}_{ij}\{y_{ij}-x_{ijA}^{\top }{\mathrm{E}}_A^{\ast }\left({\mathbf{\beta }}_A\right)\}{\mathrm{E}}_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)\right]$. Then, the estimated posterior mean and variance matrix of $b_i$ for variational densities $q_{b_i}^{\ast }\left(b_i\right)$ are ${\mathrm{E}}_{b_i}^{\ast }\left(b_i\right)={\mathbf{\mu }}_b$ and ${\mathrm{var}}_{b_i}^{\ast }\left(b_i\right)={\mathsf{\Sigma }}_b$, respectively. Moreover, the mode estimator $b_i^q$ of $b_i$ for variational density $q_{b_i}^{\ast }\left(b_i\right)$ is $b_i^q={\mathbf{\mu }}_b$. The optimal densities $q_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)$ and $q_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)$ are given by

$$q_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)\sim \mathrm{IvG}(a_{0\xi k}^{\ast },b_{0\xi k}^{\ast })\mathrm{for}k\in I,q_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)\sim \mathrm{IvG}(a_{1\xi k}^{\ast },b_{1\xi k}^{\ast })\mathrm{for}k\in A,$$

(18)

respectively, where $a_{0\xi k}^{\ast }=\sqrt{{\mathrm{E}}_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)/{\mathrm{var}}_{{\beta }_k}^{\ast }\left({\beta }_k\right)}$, $a_{1\xi k}^{\ast }=\sqrt{{\mathrm{E}}_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)/[{\left\{{\mathrm{E}}_{{\beta }_k}^{\ast }\left({\beta }_k\right)\right\}}^2+{\mathrm{var}}_{{\beta }_k}^{\ast }\left({\beta }_k\right)]}$, $b_{0\xi k}^{\ast }={\mathrm{E}}_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$, $b_{1\xi k}^{\ast }={\mathrm{E}}_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$ and ${\mathrm{E}}_{{\lambda }_0}^{\ast }(·)$ and ${\mathrm{E}}_{{\lambda }_1}^{\ast }(·)$ are the expectations taken with respect to $q_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$ and $q_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$, respectively. In this case, we have ${\mathrm{E}}_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)=a_{0\xi k}^{\ast }$, ${\mathrm{E}}_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)=a_{1\xi k}^{\ast }$, ${\mathrm{var}}_{{\xi }_{0k}}^{\ast }\left({\xi }_{1k}^{-2}\right)={\left(a_{0\xi k}^{\ast }\right)}^3/b_{0\xi k}^{\ast }$ and ${\mathrm{var}}_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)={\left(a_{1\xi k}^{\ast }\right)}^3/b_{1\xi k}^{\ast }$. Moreover, the mode estimators ${\xi }_{0k}^{-2q}$ and ${\xi }_{1k}^{-2q}$ of ${\xi }_{0k}^{-2}$ and ${\xi }_{1k}^{-2}$ for variational densities $q_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)$ and $q_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)$ are ${\xi }_{0k}^{-2q}=a_{0\xi k}^{\ast }\sqrt{1+{(1.5a_{0\xi k}^{\ast }/b_{0\xi k}^{\ast })}^2}-1.5{\left(a_{0\xi k}^{\ast }\right)}^2/b_{0\xi k}^{\ast }$ for $k\in I$ and ${\xi }_{1k}^{-2q}=a_{1\xi k}^{\ast }\sqrt{1+{(1.5a_{1\xi k}^{\ast }/b_{1\xi k}^{\ast })}^2}-1.5{\left(a_{1\xi k}^{\ast }\right)}^2/b_{1\xi k}^{\ast }$ for $k\in A$, respectively.

To derive the optimal density $q_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$, we denote

$$\begin{array}{ccc}log\left({\varrho }_k\right)\hfill & =\hfill & {\mathrm{E}}_{\rho }^{\ast }(log\rho )-{\mathrm{E}}_{\rho }^{\ast }\{log(1-\rho )\}+\frac{1}{2}\left[{\mathrm{E}}_{{\xi }_{1k}}^{\ast }\{log\left({\xi }_{1k}^{-2}\right)\}-{\mathrm{E}}_{{\xi }_{0k}}^{\ast }\{log\left({\xi }_{0k}^{-2}\right)\}\right]\hfill \\ \hfill & \hfill & +{\mathrm{E}}_{{\beta }_k}^{\ast }\left({\beta }_k\right)\sum _{i=1}^n\sum _{j=1}^m\{y_{ij}-x_{ij,C_k}^{\top }{\mathrm{E}}_{\beta }^{\ast }\left({\mathbf{\beta }}_{C_k}\right)-z_{ij}^{\top }{\mathrm{E}}_{b_i}^{\ast }\left(b_i\right)\}{x}_{ijk}{\mathrm{E}}_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)\hfill \\ \hfill & \hfill & -{\displaystyle \frac{1}{2}}\left[{\mathrm{var}}_{{\beta }_k}^{\ast }\left({\beta }_k\right)+{\left\{{\mathrm{E}}_{{\beta }_k}^{\ast }\left({\beta }_k\right)\right\}}^2\right]\left\{{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^m}{x}_{ijk}^2{\mathrm{E}}_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)-{\mathrm{E}}_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)+{\mathrm{E}}_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)\right\},\hfill \end{array}$$

(19)

where $C_k=\{\ell :{\gamma }_{\ell }=1,\ell \ne k\in A\}=A\setminus \left\{k\right\}$, which is an index set with the kth index deleted from the set A. Thus, latent variable ${\gamma }_k$ is sampled from the Bernoulli distribution with the probability ${\varsigma }_k={\varrho }_k/({\varrho }_k+1)$, i.e., ${\gamma }_k|\mathcal{D},b,\mathbf{\sigma }\sim \mathrm{Bernoulli}\left({\varsigma }_k\right)$ for $k=1,\dots ,p$. In this case, the estimated posterior mean and variance of ${\gamma }_k$ for variational density $q_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$ are ${\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)={\varsigma }_k$ and ${\mathrm{var}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)={\varsigma }_k(1-{\varsigma }_k)$, respectively. Thus, the mode estimator ${\gamma }_k^q$ of ${\gamma }_k$ for variational density $q_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$ is ${\gamma }_k^q={\varsigma }_k$ for $k=1,\dots ,p$.

The optimal density $q_Q^{\ast }\left(Q\right)$ has the form

$$q_Q^{\ast }\left(Q\right)\sim {\mathrm{IW}}_q(S_0^{\ast },{\nu }_0^{\ast }),$$

(20)

where $S_0^{\ast }=S_0+n{\mathbf{\mu }}_b{\mathbf{\mu }}_b^{\top }+n{\mathsf{\Sigma }}_b$ with ${\mathbf{\mu }}_b$ and ${\mathsf{\Sigma }}_b$ defined in Equation (17) and ${\nu }_0^{\ast }={\nu }_0+n$. Then, we have ${\mathrm{E}}_Q^{\ast }\left(Q\right)=S_0^{\ast }/({\nu }_0^{\ast }-q-1)$. Moreover, the mode estimator $Q^q$ of Q is given by $Q^q=S_0^{\ast }/({\nu }_0^{\ast }+q+1)$.

The optimal density $q_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)$ ($j=1,\dots ,m$) has the form

$$q_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)\sim \mathsf{\Gamma }\left({\displaystyle \frac{n}{2}},b_{\sigma }^{\ast }\right),$$

(21)

where $b_{\sigma }^{\ast }=0.5{\sum }_{i=1}^n{h}_{ij}$, $h_{ij}={(y_{ij}-{\mu }_{ij}^{\ast })}^2+x_{ijA}^{\top }{\mathsf{\Sigma }}_A{x}_{ijA}+x_{ijI}^{\top }{\mathsf{\Sigma }}_I{x}_{ijI}+z_{ij}^{\top }{\mathsf{\Sigma }}_b{z}_{ij}$ and ${\mu }_{ij}^{\ast }=x_{ijA}^{\top }{\mathbf{\mu }}_A+z_{ij}^{\top }{\mathbf{\mu }}_b$. Thus, we have ${\mathrm{E}}_{{\sigma }_j}^{\ast }({\sigma }_j^{-2})=n/{\sum }_{i=1}^n{h}_{ij}$ and ${\mathrm{var}}_{{\sigma }_j}^{\ast }\left({\sigma }_j^{-2}\right)=2n/{\left({\sum }_{i=1}^n{h}_{ij}\right)}^2$. In this case, the mode estimator ${\sigma }_j^{-2q}$ of ${\sigma }_j^{-2}$ for variational density $q_{{\sigma }_j^2}^{\ast }({\sigma }_j^{-2})$ is ${\sigma }_j^{-2q}=(n-2)/{\sum }_{i=1}^n{h}_{ij}$ for $j=1,\dots ,m$.

The optimal density $q_{\rho }^{\ast }\left(\rho \right)$ can be expressed as

$$q_{\rho }^{\ast }\left(\rho \right)\sim \mathrm{Beta}(c_{\rho },d_{\rho }),$$

(22)

where $c_{\rho }=a_{\gamma }+{\sum }_{k=1}^p{\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$ and $d_{\rho }=b_{\gamma }+p-{\sum }_{k=1}^p{\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$. Thus, we have ${\mathrm{E}}_{\rho }^{\ast }\left(\rho \right)=c_{\rho }/(c_{\rho }+d_{\rho })$ and ${\mathrm{var}}_{\rho }^{\ast }\left(\rho \right)=c_{\rho }{d}_{\rho }/\left\{{(c_{\rho }+d_{\rho })}^2(c_{\rho }+d_{\rho }-1)\right\}$. In this case, the mode estimator of $\rho$ is given as ${\rho }^q=c_{\rho }/(c_{\rho }+d_{\rho })$.

The optimal densities $q_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$ and $q_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$ are

$$q_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)\sim \mathsf{\Gamma }\left(a_{0\lambda }^{\ast },b_{0\lambda }^{\ast }\right),q_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)\sim \mathsf{\Gamma }\left(a_{1\lambda }^{\ast },b_{1\lambda }^{\ast }\right),$$

(23)

respectively, where $a_{0\lambda }^{\ast }=c_0+p-{\sum }_{k=1}^p{\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$, $b_{0\lambda }^{\ast }=d_0+{\sum }_{k=1}^p\{1-{\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)\}{\mathrm{E}}_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^2\right)/2$, $a_{1\lambda }^{\ast }=c_1+{\sum }_{k=1}^p{\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$ and $b_{1\lambda }^{\ast }=d_1+{\sum }_{k=1}^p{\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right){\mathrm{E}}_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^2\right)/2$. In this case, we obtain ${\mathrm{E}}_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)=a_{0\lambda }^{\ast }/b_{0\lambda }^{\ast }$, ${\mathrm{var}}_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)=a_{0\lambda }^{\ast }/{\left(b_{0\lambda }^{\ast }\right)}^2$, ${\mathrm{E}}_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)=a_{1\lambda }^{\ast }/b_{1\lambda }^{\ast }$ and ${\mathrm{var}}_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)=a_{1\lambda }^{\ast }/{\left(b_{1\lambda }^{\ast }\right)}^2$. The mode estimators ${\lambda }_0^{2q}$ and ${\lambda }_1^{2q}$ of ${\lambda }_0^2$ and ${\lambda }_1^2$ for variational densities $q_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$ and $q_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$ are ${\lambda }_0^{2q}=(a_{0\lambda }^{\ast }-1)/b_{0\lambda }^{\ast }$ and ${\lambda }_1^{2q}=(a_{1\lambda }^{\ast }-1)/b_{1\lambda }^{\ast }$, respectively.

4.2. Optimizing $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ via Coordinate Ascent Algorithm

The elaborated steps for optimizing $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ via the coordinate ascent algorithm are given below:

• Step (a) Given the initial values of variational densities $q_{\beta }^{\ast }\left(\mathbf{\beta }\right)$, $q_{b_i}^{\ast }\left(b_i\right)$, $q_{\xi 0k}^{\ast }\left({\xi }_{0k}^{-2}\right)$, $q_{\xi 1k}^{\ast }\left({\xi }_{1k}^{-2}\right)$, $q_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$, $q_Q^{\ast }\left(Q\right)$, $q_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)$, $q_{\rho }^{\ast }\left(\rho \right)$, $q_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$ and $q_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$, compute the lower bound $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ (denoted as ${\mathbb{L}}^{\left(0\right)}\left\{q(\Xi )\right\}$) and set $\kappa =1$.
• Step (b) Compute variational density $q_{\beta }^{\ast }\left(\mathbf{\beta }\right)$ and update ${\mathrm{E}}_{\beta }^{\ast }\left(\mathbf{\beta }\right)$.
• Step (c) Compute variational density $q_{b_i}^{\ast }\left(b_i\right)$ and update ${\mathrm{E}}_{b_i}^{\ast }\left(b_i\right)$.
• Step (d) Compute variational density $q_{\xi 0k}^{\ast }\left({\xi }_{0k}^{-2}\right)$ and update ${\mathrm{E}}_{{\xi }_{0k}}^{\ast }\left({\xi }_{0k}^{-2}\right)$.
• Step (e) Compute variational density $q_{\xi 1k}^{\ast }\left({\xi }_{1k}^{-2}\right)$ and update ${\mathrm{E}}_{{\xi }_{1k}}^{\ast }\left({\xi }_{1k}^{-2}\right)$.
• Step (f) For $k=1,\dots ,p$, compute variational densities $q_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$ and update ${\mathrm{E}}_{{\gamma }_k}^{\ast }\left({\gamma }_k\right)$.
• Step (g) Compute variational density $q_Q^{\ast }\left(Q\right)$ and update ${\mathrm{E}}_Q^{\ast }\left(Q\right)$.
• Step (h) Compute variational densities $q_{{\sigma }_j^2}^{\ast }\left({\sigma }_j^{-2}\right)$ and update ${\mathrm{E}}_{{\sigma }_j}^{\ast }\left({\sigma }_j^{-2}\right)$.
• Step (i) Compute variational density $q_{\rho }^{\ast }\left(\rho \right)$ and update $E_{\rho }^{\ast }\left(\rho \right)$.
• Step (j) Compute variational density $q_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$ and update $E_{{\lambda }_0}^{\ast }\left({\lambda }_0^2\right)$.
• Step (k) Compute variational density $q_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$ and update $E_{{\lambda }_1}^{\ast }\left({\lambda }_1^2\right)$.
• Step (l) Based on variational densities from Steps (b)–(k), compute the ELB $\mathbb{L}\left\{q\right(\Xi \left)\right\}$ (denoted as ${\mathbb{L}}^{\left(\kappa \right)}\left\{q(\Xi )\right\}$) and the relative change

  $$\mathrm{RC}={\displaystyle \frac{|{\mathbb{L}}^{\left(\kappa \right)}\left\{q(\Xi )\right\}-{\mathbb{L}}^{(\ell -1)}\left\{q(\Xi )\right\}|}{{\mathbb{L}}^{(\kappa -1)}\left\{q(\Xi )\right\}}}.$$
• Step (m) Given sufficiently small $ϵ$, if $\mathrm{RC}<ϵ$, the algorithm is stopped. Otherwise, repeat Steps (b)–(l).

The preceding presented coordinate ascent algorithm for computing variational Bayesian estimates of parameters is summarized as Algorithm 1 and converges to the solution of the optimization problem (13) because it satisfies the well-known KKT condition for the considered model.   

Algorithm 1: Variational Bayesian estimation

4.3. Model Comparison

The Bayes factor is a vital statistic for model comparison within the Bayesian framework and is widely employed to choose a better model among the considered competing models due to its merits for model selection: (i) it is a consistent selector; (ii) it plays the part of an Occam’s razor, preferring the simpler model for the similar fits; (iii) it does not need the models to be nested. For instance, see [36] for structural equation models and [37] for non-ignorable missing data. Denote $f\left(Y\right|X,Z,{\Xi }_h,{\mathcal{H}}_h)$ as the probability density of the data $\{Y,X,Z\}$ associated with the model ${\mathcal{H}}_h$, where ${\Xi }_h$ is the parameter vector in the model ${\mathcal{H}}_h$. Define $f\left({\Xi }_h\right|{\mathcal{H}}_h)$ as the prior of ${\Xi }_h$ for $h=0,1$. The Bayes factor for comparing two competing models ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$ can be written as

$$B_{10}=\frac{\int f\left(Y\right|X,Z,{\Xi }_1,{\mathcal{H}}_1)f\left({\Xi }_1\right|{\mathcal{H}}_1)d{\Xi }_1}{\int f\left(Y\right|X,Z,{\Xi }_0,{\mathcal{H}}_0)f\left({\Xi }_0\right|{\mathcal{H}}_0)d{\Xi }_0}=\frac{f\left(Y\right|X,Z,{\mathcal{H}}_1)}{f\left(Y\right|X,Z,{\mathcal{H}}_0)},$$

(24)

where $f\left(Y\right|X,Z,{\mathcal{H}}_k)$ is the marginal likelihood for the model ${\mathcal{H}}_h$ for $h=0$ and 1. However, computing the Bayes factor $B_{10}$ is a non-trivial task for our considered high-dimensional linear mixed model because of the intractable integral involved. Considerable methods have been developed to compute the marginal likelihood $f\left(Y\right|X,Z,{\mathcal{H}}_k)$ or the Bayes factor, for example, Laplace’s method [38], annealed importance sampling [39], bridge sampling [40], path sampling (also called thermodynamic integration) [41], nested sampling [42], power posteriors [43], hybrid method combining simulation and asymptotic approximations [44]. For a comprehensive review, refer to [45]. Here, a path sampling or thermodynamic integration method is adopted to compute $B_{10}$ via a link model: ${\mathcal{H}}_{\zeta 01}=(1-\zeta ){\mathcal{H}}_0+\zeta {\mathcal{H}}_1$, where $\zeta$ is a continuous parameter taking value in the interval $[0,1]$. Thus, we have ${\mathcal{H}}_{\zeta 01}={\mathcal{H}}_0$ when $\zeta =0$ and ${\mathcal{H}}_{\zeta 01}={\mathcal{H}}_1$ when $\zeta =1$. Similarly to [41], we define the following class of probability densities:

$$\mathcal{Q}(\zeta )=f(Y|X,Z,\zeta )=\int f(Y,\zeta |X,Z,\Xi )f(\Xi )d\Xi ,$$

(25)

where $f(Y,\zeta |X,Z,\Xi )$ is the density of Y given X and Z under ${\mathcal{H}}_{\zeta }$ and $f(\Xi )$ is the prior of $\Xi$. Under the above definition, it is easily known that $\mathcal{Q}\left(0\right)=f\left(Y\right|X,Z,{\mathcal{H}}_0)$ and $\mathcal{Q}\left(1\right)=f\left(Y\right|X,Z,{\mathcal{H}}_1)$. Following the argument of [41], we obtain

$$log{B}_{10}=log\frac{\mathcal{Q}\left(1\right)}{\mathcal{Q}\left(0\right)}={\int }_0^1\mathrm{E}\left\{U(Y,\zeta ,\Xi |X,Z)\right\}d\zeta ,$$

(26)

where $\mathrm{E}(·)$ represents the expectation taken with respect to the conditional density $f(\Xi ,\zeta |Y,X,Z)$ and $U(Y,\zeta ,\Xi |X,Z)=dlogf(Y,\zeta ,\Xi |X,Z)/d\zeta$. Thus, applying the thermodynamic integration [41] or powered posteriors method [43] to Equation (26), $log{B}_{10}$ can be estimated by

$$\widehat{log{B}_{10}}=\frac{1}{2}\sum _{\ell =0}^{\mathbb{L}}\left({\zeta }_{(\ell +1)}-{\zeta }_{(\ell )}\right)\left({\overline{U}}_{(\ell +1)}+{\overline{U}}_{(\ell )}\right),$$

(27)

where $0={\zeta }_{\left(0\right)}<{\zeta }_{\left(1\right)}<\cdots <{\zeta }_{(\mathbb{L}+1)}=1$ and ${\overline{U}}_{(\ell )}={\mathcal{J}}^{-1}{\sum }_{\tau =1}^{\mathcal{J}}U(Y,{\zeta }_{(\ell )},{\Xi }_{\ell }^{(\tau )}|X,Z)$, in which $\{{\Xi }_{\ell }^{(\tau )}:\tau =1,\dots ,\mathcal{J}\}$ are observations sampled from the variational density $q^{\ast }(\Xi |{\zeta }_{(\ell )})$ for $\ell =1,\dots ,\mathbb{L}$. Following [46], ${\mathcal{H}}_1$ is selected when $\widehat{log{B}_{10}}>1$; otherwise, ${\mathcal{H}}_0$ is selected.

5. Simulation Studies

Several simulation studies are implemented to assess the performance of the introduced variational Bayesian methodologies. For comparison, we also take the Bayesian lasso method into consideration. In this simulation study, response variables $y_{ij}$s are independently sampled from the normal distribution: $y_{ij}\sim \mathcal{N}(x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i,{\sigma }_j^2)$, where $x_{ij}$, $z_{ij}$ and $b_i$ are independently drawn from the multivariate normal distributions ${\mathcal{N}}_p(0,{\mathsf{\Sigma }}_x)$, ${\mathcal{N}}_q(0,I)$ and ${\mathcal{N}}_q(0,Q)$, respectively, for $i=1,\dots ,n,j=1,\dots ,m$. The true value of $\mathbf{\beta }$ is taken to be ${(-0.5,0.8,2,0.8,0.5,0.0,\dots ,0.0)}^{\top }$, which implies that there are five active variables and $p-5$ inactive variables. As an illustration, we set $m=6$, $q=4$, $n=100$, 200 and 300, and $p=500,1000$ and 2000, which indicate that $n\ll p$. The true values of ${\sigma }_j^2$’s are set to be ${\sigma }_1^2={\sigma }_2^2=0.8$, ${\sigma }_3^2={\sigma }_4^2=0.9$ and ${\sigma }_5^2={\sigma }_6^2=1.0$. The true value of Q is taken with diagonal elements being 1.0 and remaining components being 0.1.

We consider the following two types of covariance structures for ${\mathsf{\Sigma }}_x={\left({\sigma }_{x_{jk}}\right)}_{p\times p}$.

• Type I. Components of covariate vector $x_{ij}$ are independent of each other, i.e., ${\sigma }_{x_{jk}}=0.0$ when $j\ne k$ and ${\sigma }_{x_{jj}}=1.0$ when $1\le j,k\le p$.
• Type II. $x_{ij}$ is an autoregressive correlation, i.e., ${\sigma }_{x_{jk}}={0.5}^{\mid j-k\mid }$ when $\forall j\ne k$ and ${\sigma }_{x_{jj}}=1.0$ when $1\le j,k\le p$.

In implementing the preceding presented variational Bayesian approach together with the spike and slab priors, we take the hyperparameters ${\nu }_0=1$ and $S_0=0.02I_{q\times q}$ leading to the flat prior for Q and set $a_{\gamma }=b_{\gamma }=0.5$. For the spike and slab priors of ${\beta }_k$s, to achieve appropriate shrinkage and model selection consistency, we take $c_0=500$ and $c_1=0.3$, indicating $c_1\ll c_0$, $d_0=5$ and $d_1=30$, implying $d_0\ll d_1$, guaranteeing the sparsity of the model. In this simulation, 100 replications are conducted to select active variables and estimate model parameters. To assess the accuracy of parameter estimation via the proposed variational Bayesian method, we calculate the average value of RMSes for unknown parameters, where “RMS“ indicates the root mean square between the Bayesian estimates based on 100 replications and true values of unknown parameters. To assess the performance of the variable selection procedure, we compute TP and FP, where TP represents the average number of active covariates correctly identified as active and FP denotes the average number of inactive covariates incorrectly detected as active. Generally, the closer to the true number of active covariates TP is or the smaller FP is, the better the variable selection method behaves. Results are reported in

Table 1. Performance of variable selection and parameter estimation in the first simulation study.

(${\mathit{a}}_{\mathit{\gamma }}$, ${\mathit{b}}_{\mathit{\gamma }}$)	${\mathsf{\Sigma }}_{\mathit{x}}$	n	Method	p = 500			p = 1000			p = 2000
				TP	FP	RMS	TP	FP	RMS	TP	FP	RMS
(0.5, 0.5)	I	100	$\mathrm{VB}$	3.91	0.00	0.11	3.79	0.00	0.08	3.84	0.00	0.06
			$\mathrm{LASSO}$	4.44	0.87	1.90	3.54	1.03	1.66	1.39	0.00	1.91
		200	$\mathrm{VB}$	4.71	0.00	0.11	4.68	0.00	0.08	4.65	0.00	0.06
			$\mathrm{LASSO}$	4.95	0.24	2.24	2.78	1.91	1.36	3.34	0.00	1.64
		300	$\mathrm{VB}$	4.89	0.00	0.11	4.81	0.00	0.08	4.91	0.00	0.06
			$\mathrm{LASSO}$	4.99	0.01	2.12	4.91	0.00	1.41	4.23	0.00	1.45
	II	100	$\mathrm{VB}$	3.79	0.00	0.11	3.84	0.00	0.08	3.76	0.00	0.06
			$\mathrm{LASSO}$	3.48	0.10	2.19	3.01	0.00	1.87	3.00	0.00	2.01
		200	$\mathrm{VB}$	3.97	0.00	0.11	3.96	0.00	0.08	3.98	0.00	0.06
			$\mathrm{LASSO}$	3.59	0.02	2.44	3.12	0.00	1.78	3.00	0.00	1.84
		300	$\mathrm{VB}$	3.98	0.00	0.11	3.96	0.00	0.08	3.98	0.00	0.06
			$\mathrm{LASSO}$	3.63	0.03	2.31	3.20	0.00	1.79	3.01	0.00	1.75
(0.1, 0.9)	I	100	$\mathrm{VB}$	3.88	0.00	0.11	3.79	0.00	0.08	3.84	0.00	0.06
			$\mathrm{LASSO}$	4.44	0.87	1.90	3.54	1.03	1.66	1.39	0.00	1.91
		200	$\mathrm{VB}$	4.71	0.00	0.11	4.66	0.00	0.08	4.64	0.00	0.06
			$\mathrm{LASSO}$	4.95	0.24	2.24	2.78	1.91	1.36	3.34	0.00	1.64
		300	$\mathrm{VB}$	4.89	0.00	0.11	4.81	0.00	0.08	4.91	0.00	0.06
			$\mathrm{LASSO}$	4.99	0.01	2.12	4.91	0.00	1.41	4.23	0.00	1.45

Note: VB represents variational Bayesian method and LASSO denotes Bayesian lasso method.

. Examination of Table 1 shows that the proposed variational Bayesian method behaves better than Bayesian lasso method, regardless of the values of p and n and covariance structures, in that TP values for the former are closer to the true number of active covariates and FP values for the former are closer to zero than those for the latter. For parameter estimation, the proposed variational Bayesian method behaves better than the Bayesian lasso method in that the average values of the RMSes for the former are smaller than those for the latter, regardless of the values of p and n and covariance structures. To investigate the sensitivity of the selection of the hyperparameters $a_{\gamma }$ and $b_{\gamma }$, we take $a_{\gamma }=0.1$ and $b_{\gamma }=0.9$ and calculate the corresponding results for the Type I structure of ${\mathsf{\Sigma }}_x$, which results are given in Table 1. These empirical results indicate that the proposed variational Bayesian method is not sensitive to the hyperparameters in that the same pattern is observed regardless of the hyperparameters $a_{\gamma }$ and $b_{\gamma }$.

As an illustration for model comparison via the proposed Bayes factor, we consider the second simulation study. In the simulation study, the data $\{(x_{ij},z_{ij},y_{ij}):i=1,\dots ,n,j=1,\dots ,m\}$ are generated as those in the first simulation study with covariance structure of ${\mathsf{\Sigma }}_x$ taken to be Type I. To this end, we consider the following competing models:

$$\begin{array}{cc}{\mathcal{H}}_0:\hfill & y_{ij}=x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i+{\epsilon }_{ij},{\epsilon }_{ij}\sim \mathcal{N}(0,{\sigma }_j^2),\hfill \\ {\mathcal{H}}_1:\hfill & y_{ij}=z_{ij}^{\top }{b}_i+{\epsilon }_{ij},{\epsilon }_{ij}\sim \mathcal{N}(0,{\sigma }_j^2),\hfill \\ {\mathcal{H}}_2:\hfill & y_{ij}=x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i+{\epsilon }_{ij},{\epsilon }_{ij}\sim \mathcal{N}(0,{\sigma }_0^2),\hfill \end{array}$$

where ${\mathcal{H}}_0$ represents the true linear mixed model and while ${\mathcal{H}}_1$ and ${\mathcal{H}}_2$ are two competing linear mixed models, ${\mathcal{H}}_1$ only containing random effects without fixed effects, and ${\mathcal{H}}_2$ misspecifying the distribution of measurement error. We define a path $t\in [0,1]$ to link any two of the above presented three models. For example, ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$ can be linked by ${\mathcal{H}}_{t01}:y_{ij}=(1-t)x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i+{\epsilon }_{ij}$, which indicates that ${\mathcal{H}}_{t01}$ is just ${\mathcal{H}}_0$ for $t=0$ and becomes ${\mathcal{H}}_1$ for $t=1$, and ${\mathcal{H}}_0$ and ${\mathcal{H}}_2$ are linked by ${\mathcal{H}}_{t02}:y_{ij}=x_{ij}^{\top }\mathbf{\beta }+z_{ij}^{\top }{b}_i+{\epsilon }_{ij}$ with ${\epsilon }_{ij}\stackrel{\mathrm{ind}}{\sim }\mathcal{N}(0,t^2{\sigma }_0^2+{(1-t)}^2{\sigma }_j^2)$, which implies that ${\mathcal{H}}_{t02}$ reduces to ${\mathcal{H}}_0$ with $t=0$ and ${\mathcal{H}}_{t02}$ becomes ${\mathcal{H}}_2$ with $t=1$.

To calculate the estimated log Bayes factors (i.e., $\widehat{log{B}_{10}}$ and $\widehat{log{B}_{20}}$) via the preceding proposed path sampling procedure, we take ${\zeta }_{(\ell )}=\ell /\mathbb{L}$ for $\ell =0,1,\dots ,\mathbb{L}$, $\mathbb{L}=10$, $\mathcal{J}=1000$ and ${\sigma }_0^2=0.5$ and the same priors as those given in the first simulation studies. Results are given in

Table 2. Estimated log Bayes factor in the second simulation study.

$\widehat{log{B}_{10}}$	n	p
		500	1000	2000
	100	−194	−102	−86
	200	−372	−272	−294
	300	−506	−544	−588
$\widehat{log{B}_{20}}$ ($\times {10}^7$)	100	−0.95	−4.03	−1.41
	200	−1.54	−3.68	−2.54
	300	−3.13	−3.58	−2.26

, which indicates that ${\mathcal{H}}_0$ is strongly selected as expected regardless of n and p.

6. An Empirical Example

As an illustration of the variational Bayesian method developed above, we consider the ADNI-2 data [47] published in 2003 and followed by ADNI-1, ADNI-GO and ADNI-2 groups. This study aims to predict the mini-mental state examination (MMSE) score, which is an important index for detecting Alzheimer’s disease (AD) stages in that different MMSE scores indicate different progression of a AD patient. AD is the most common type of dementia for elderly people and the sixth leading cause of death in the United States, and it results in the loss of memory and the impairment of cognitive and language skills. More importantly, there is no effective treatment to slow the progression of the disease [48]. The number of AD patients has grown exponentially with the speed of the aging population, bringing a socioeconomic burden to both families and society [49]. The details on the ADNI database can refer to the website http://adni.loni.usc.edu (accessed on 20 May 2021).

The ADNI-2 data were analyzed by [48] using the factor analysis model to impute missing values. As an illustration, we utilize 340 complete magnetic resonance imaging (MRI) features with 62 samples and 3 medical visits (6-month, 12-month and 24-month), take five features among 340 features as covariates associated with random effects and set the MMSE score as the response variable. That is, $n=62$, $p=340$, $q=5$ and $m=3$. In this case, covariates are high-dimensional compared with the sample size. Here, we assume that only a small fraction of covariates contribute to the response variable.

The preceding introduced variational Bayesian method together with the linear mixed model and the same priors as those in the first simulation study are utilized to fit the above-mentioned MRI data. Here, the hyperparameters are taken as ${\nu }_0=1$, $S_0=0.02I_{q\times q}$, $a_{\gamma }=b_{\gamma }=0.5$, $c_0=10$, $d_0=1$, $c_1=1$ and $d_1=10$ for ensuring the sparsity of the model. Thus, the proposed variational Bayesian method selects three features as active variables: thickness average of the right fusiform (denoted as “$x_1$”), thickness standard deviation of the right posterior cingulate (denoted as “$x_2$”) and thickness standard deviation of the left postcentral (denoted as “$x_3$"). Their corresponding parameter estimates are 1.9, 0.25 and 0.4, respectively, which show that the three active variables have positive effects on MMSE that are consistent with those given in [48]. Bayesian estimates of random effects $b_i$ are −0.003, −0.0021, −0.0013, −0.0058 and −0.0054, respectively, which imply that the selected five covariates associated with random effects have negative effects on MMSE.

Table 3. Performance of variational Bayesian method for the complete and selected models in the ADNI-2 data.

Model	n	p	RMSE	MAP
Complete	62	340	49.17	49.15
Selected	62	3	1.05	0.82

also presents the RMSE and MAP values for the models with 340 covariates (denoted as the “Complete“ model) and the selected three active covariates (denoted as the “Selected“ model), where RMSE and MAP are evaluated by RMSE=$\sqrt{n^{-1}{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2}$ and MAP=$n^{-1}{\sum }_{i=1}^n|{\widehat{y}}_i-y_i|$ and ${\widehat{y}}_i$ is the fitted value of response $y_i$. Examination of Table 3 shows that the selected model has smaller RMS and MAP values than the complete model, i.e., the selected model fits the ADNI-2 data better than the complete model. For the selected model, we also compute the Bayes factors for three competing models ${\mathcal{H}}_0,{\mathcal{H}}_1$ and ${\mathcal{H}}_2$ given in the second simulation study, which are $\widehat{\log B_{10}}=-558$ and $\widehat{\log B_{20}}=-46.93$, leading to the conclusion that ${\mathcal{H}}_0$ is strongly selected.

7. Discussions

This paper investigates simultaneously estimating model parameters and selecting variables in linear mixed models with high-dimensional fixed effects and low-dimensional random effects in the Bayesian framework. A novel variational Bayesian approach is developed to address the time-consuming problem of the traditional Bayesian lasso method due to the ill-posited problem and large matrix computation involved in the presence of high-dimensional data. The Gaussian spike and slab priors of population-specific fixed-effects regression coefficients are specified to identify important fixed effects by allowing the tuning parameters to tend to zero. For the sake of sampling observations, the Gaussian spike and slab priors are reformulated as a mixture of a normal distribution and an exponential distribution. In the variational Bayesian framework, the problem of best approximating the posterior density is transformed as an optimization problem, i.e., minimizing the evidence lower bound. For ease of computation, the coordinate ascent algorithm, implemented efficiently, is employed to optimize the evidence lower bound. For model comparison, the Bayes factor is computed by the path sampling method. Simulation studies are conducted to investigate the performance of the proposed variational Bayesian method, and a real example is illustrated by the proposed methodologies. Empirical results show that the proposed variational Bayesian method behaves better than the traditional Bayesian lasso method regardless of the accuracy of parameter estimation, the consistency of variable selection or computational flexibility and complexity.

The proposed variational Bayesian method has the following advantages:

• Overcoming the problem of selecting a high-dimensional vector of shrinkage parameters required for the Bayesian lasso method;
• Simultaneously estimating model parameters and variance–covariance matrices and selecting fixed-effects and random-effects components with a relatively low computational cost;
• Avoiding large matrix computations and the curse of dimensionality problem;
• Providing a flexible and efficient approach to compute the Bayes factor for model comparison.

The proposed variational Bayesian method can be extended to more complicated models, such as generalized linear mixed models with mixed discrete and missing data. However, their extensions have huge challenges, including the closed-form derivation of the optimal variational density, the specification of the priors, the learning of the data-driven hyperparameters and the computational complexity. In addition, this paper does not consider the selection of high-dimensional random effects, which is a rather challenging topic. In addition, to speed up the convergence of the chain, we might consider some important and relevant Gibbs sampling schemes, for example, the herded Gibbs sampling, which is a deterministic variant of the Gibbs sampling scheme and generates observations by matching the full-conditionals rather than by taking the full-conditionals at random [50], the recycling Gibbs sampler, which generates auxiliary observations whose information is eventually discarded and which can be recycled within the Gibbs algorithm for improving efficiency with no extra cost [51], and the blocking and parameterization method [52].

In addition, we did not consider BIC criterion for model comparison in that BIC is only an approximation to the Bayes factor of marginal likelihood of the data given each hypothesis. Moreover, due to the random effects involved in the considered models, BIC behaves unsteadily.