Feature Selection Using Nearest Neighbor Gaussian Processes

Abstract

We introduce a novel Bayesian approach for feature (variable) selection using Gaussian process regression, which is crucial for enhancing interpretability and model regularization. Our method employs nearest neighbor Gaussian processes as scalable approximations to classical Gaussian processes. Feature selection is performed by conditioning the process mean and covariance function on a random set representing the indices of relevant variables. A priori beliefs regarding this set control the feature selection, while reference priors are assigned to the remaining model parameters, ensuring numerical robustness in the process covariance matrix. For model inference, we propose a Metropolis-within-Gibbs algorithm. The effectiveness of the proposed feature selection approach is demonstrated through evaluation on simulated data, a computer experiment approximation, and two real-world data sets.

1. Introduction

Due to the ongoing and pervasive process of digitalization, increasing amounts of data are collected by companies, governments, and organizations of all kinds. Consequently, the demand for analyzing dependencies among observed features (variables) is also growing. Regression techniques are often the methods of choice for this task. The application of regression models is straightforward when a large number n of observations is available for a comparatively small number d of observed features. However, this is often not the case, and the problem becomes more challenging. Models tend to overfit when the model complexity cannot be supported by the available sample size [1]. Moreover, identifying the most influential features (variables) becomes increasingly difficult. For this reason, approaches that automatically detect the most important features have gained considerable attention in many fields, including genetics [2], astronomy [3], and economics [4]. In semiconductor manufacturing, a vast number of parameters are measured during the production process. Feature selection models can help pinpoint specific production parameters that have a significant impact on the quality of the final product.

This study introduces a novel Bayesian approach to addressing the feature selection problem. The novelty of this work lies in the combination of nearest neighbor Gaussian processes (NNGPs), reference priors, and a random set $\mathcal{A}$ that specifies the features contributing to the regression model. For model inference, we propose a Metropolis-within-Gibbs algorithm to sample from the posterior distribution. In particular, the proposed Markov chain Monte Carlo (MCMC) algorithm makes use of Hamiltonian Monte Carlo (HMC) [5]. HMC is a family of MCMC methods based on concepts from dynamical systems in physics. Virtual dynamical systems are constructed by augmenting the model parameters with momentum variables, and samples are then drawn from the joint distribution of the augmented system. In a Bayesian context, HMC exploits derivatives of the posterior density to generate efficient proposals. In many applications, HMC converges significantly faster than conventional approaches [5,6,7].

The contributions of this work can be summarized as follows. First, we adopt a nearest neighbor Gaussian process approximation that restricts dependence to local neighborhoods of data points, with an appropriate choice of the number m of neighbors. This construction preserves the essential dependence structure of the underlying Gaussian process while enabling stable inference and accurate prediction in high-dimensional settings. Second, we enhance the numerical stability of nearest neighbor Gaussian processes through the use of reference priors for the process parameters. Third, we develop a novel Bayesian approach for identifying the most influential features by placing suitable prior distributions on random subsets of feature indices.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 then presents the proposed feature selection approach. In Section 4, we assess the predictive performance of the method using carefully selected data sets. Finally, Section 5 summarizes and concludes the paper.

2. Related Work

If the dependencies of interest are purely linear, simple linear regression models are sufficient for the analysis. There exist many well-established approaches for feature selection in linear regression models [8,9,10,11,12,13,14,15].

Bhattacharya et al. [16] introduce and analyze the properties of Dirichlet–Laplace priors, emphasizing their utility for achieving optimal shrinkage in statistical modeling, particularly in high-dimensional settings. Bhadra et al. [17] address the challenges associated with estimating ultra-sparse signals, where the relevant signals are extremely sparse relative to a large amount of noise or irrelevant information. They introduce the Horseshoe+ estimator, which extends the well-known Horseshoe prior by incorporating additional features to improve performance in identifying and estimating ultra-sparse signals. Chen and Walker [18] introduce a Bayesian approach to variable selection in high-dimensional linear models using marginal solo spike-and-slab priors, which provide a computationally efficient framework for Bayesian variable selection.

However, linearity does not hold in many practical applications, and more flexible modeling approaches are required. Gaussian process regression models (GPRMs), originally proposed by O’Hagan [19], have gained substantial attention. GPRMs can be viewed as an alternative to neural networks, since a broad class of neural network–based regression models converges to a Gaussian process (GP) in the limit of infinitely wide networks [20]. Alternatively, GPs may be interpreted from the perspective of nonparametric Bayesian regression [7], where Gaussian priors are directly assigned to spaces of regression functions. Numerous empirical studies demonstrate that GPRMs perform competitively compared to other nonlinear models [21,22]. Moreover, GP regression has become a standard tool for the approximation of computationally expensive computer models and for Bayesian optimization; see, e.g., Santner et al. [23], Gramacy [24]. A comprehensive survey of high-dimensional GP modeling with applications to Bayesian optimization is provided by Binois and Wycoff [25]. Extensions of standard GP models allow, for instance, nonstationary behavior [26]. Alternatively, divide-and-conquer approaches such as treed GPs and local approximate GPs address computational bottlenecks associated with fitting classical GPs to large data sets [27,28]. Deep Gaussian processes further combine nonstationary and global modeling capabilities [29], albeit at the cost of increased computational complexity.

A large class of GPRMs automatically provides information regarding the relative importance of individual variables through a mechanism known as automatic relevance determination [7]. However, in settings with limited training data, such approaches may perform poorly, often resulting in high generalization error. One strategy to mitigate this issue is the introduction of penalty terms that encourage sparse representations, thereby favoring models in which only a subset of variables is deemed important. Such approaches have been proposed by Yi et al. [30] and Wu et al. [31]. Linkletter et al. [32] present a related Bayesian approach that employs spike-and-slab priors for variable selection. Chen and Wang [33] propose a Bayesian method based on a random indicator vector to specify which variables contribute to the model, while Zhang et al. [34] apply an indicator-based Bayesian variable selection approach to Gaussian process models in the context of computer experiments.

A further limitation of GPRMs is the potential numerical instability of the estimation procedure. To address this issue, Gu et al. [35] propose the concept of robust estimation. From a Bayesian perspective, robustness can be achieved by assigning reference priors to the GP parameters, as recommended in [36,37,38,39]. Following these recommendations, the reference priors employed in this work ensure well-conditioned correlation matrices, thereby avoiding issues related to near-singularity or near-diagonality. This also provides protection against effects associated with Jeffreys’ paradox, which is known to arise in linear regression models. Another well-known drawback of GPRMs is their computational cost. To alleviate this issue, Datta et al. [40] introduced nearest neighbor Gaussian processes (NNGPs), which constitute highly scalable approximations derived from classical Gaussian processes.

3. Methodology

In this section, we define the proposed feature selection approach, which combines nearest neighbor Gaussian processes (NNGPs) with reference priors. Feature selection is achieved through a random set $\mathcal{A}$ that specifies the features contributing to the regression model.

First, Section 3.1 briefly reconsiders nearest neighbor Gaussian processes. Next, the proposed reference prior is introduced in Section 3.2. In Section 3.3, we present the feature selection model, followed by the model inference procedure in Section 3.4. Finally, Section 3.5 describes how predictions are obtained from the model. We provide a summary of the key notation in Appendix A.

3.1. Nearest Neighbor Gaussian Process

A stochastic process $\left\{y\left(\mathbf{x}\right);\mathbf{x}\in X\subseteq {\mathbb{R}}^d\right\}$ is called a Gaussian process (GP) iff for every finite set of indices $\left\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_k\right\}\subseteq X$ the random vector $y_{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_k}={(y\left({\mathbf{x}}_1\right),\dots ,y\left({\mathbf{x}}_k\right))}^T$ is multivariate normally distributed. A GP is fully determined by its mean function $\mathit{\mu }:X\to \mathbb{R},\mathbf{x}\mapsto \mathbb{E}\left(y\right(\mathbf{x}\left)\right)$ and its covariance function $C:X\times X\to \mathbb{R},({\mathbf{x}}_1,{\mathbf{x}}_2)\mapsto Cov(y\left({\mathbf{x}}_1\right),y\left({\mathbf{x}}_2\right))$. In nonparametric Bayesian regression, GPs are often used as prior distribution for the unknown regression function. Hence, performing inference in such a model is therefore reduced to the estimation of some unknown parameters used to define the mean function $\mathit{\mu }(·)$ and the covariance function $C(·,·)$.

In the context of the regressions, the variables (features) ${\mathbf{x}}_i$ are also called covariates (or predictors). Let $D=\{({\mathbf{x}}_i,y\left({\mathbf{x}}_i\right))|i=1,\dots ,n\}$ be a finite realization of a GP, i.e., some training data. Further, let $S=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$ denote the set of covariate constellations for which observations are available. Then, ${\mathbf{C}}_S={\left[C({\mathbf{x}}_i,{\mathbf{x}}_j)\right]}_{{\mathbf{x}}_i,{\mathbf{x}}_j\in S}$ denotes the corresponding covariance matrix. Due to the form of the Gaussian density, the model inference is based on the inverse matrix ${\mathbf{C}}_S^{-1}$ and the determinant $|{\mathbf{C}}_S|$.

When dealing with large values of n, employing GP regression models poses challenges due to the computational expense involved in computing inverse matrices and matrix determinants. To overcome these issues, Datta et al. [40] introduce nearest neighbor Gaussian processes (NNGPs) as highly scalable approximations derived from classical GP models. The starting point of an NNGP is a zero mean GP, i.e., $\mathit{\mu }(·)\equiv 0$. Then, the main idea behind NNGPs is to write the joint density of ${\mathbf{y}}_S={(y\left({\mathbf{x}}_1\right),\dots ,y\left({\mathbf{x}}_n\right))}^T$ as a product of conditional densities, using the multiplication theorem, and subsequently replacing the conditioning sets with smaller ones of size at most $m<n$. Formally, this is expressed as

$$\begin{array}{cc}\hfill p\left({\mathbf{y}}_S\right)& =p\left(y\left({\mathbf{x}}_1\right)\right){\displaystyle \prod _{i=2}^n}p(y\left({\mathbf{x}}_i\right)|y\left({\mathbf{x}}_{i-1}\right),\dots ,y\left({\mathbf{x}}_1\right)),\hfill \\ \hfill \tilde{p}\left({\mathbf{y}}_S\right)& ={\displaystyle \prod _{i=1}^n}p(y\left({\mathbf{x}}_i\right)|y_{N\left({\mathbf{x}}_i\right)}),\hfill \end{array}$$

(1)

where $y_{N\left({\mathbf{x}}_i\right)}$ denotes the vector formed by stacking all the $y\left(\mathbf{x}\right)$ for which $\mathbf{x}$ lies in the so-called neighbor set of ${\mathbf{x}}_i$, i.e., $N\left({\mathbf{x}}_i\right)\subset S\setminus \left\{{\mathbf{x}}_i\right\}$, $|N({\mathbf{x}}_i)|\le m$.

By defining the neighbor set $N\left({\mathbf{x}}_i\right)$ to be any subset of $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{i-1}\}$, one can ensure that (1) is a valid probability density; see Datta et al. [40]. In this work, we follow the approach of Vecchia [41], who specified $N\left({\mathbf{x}}_i\right)$ to be the m nearest neighbors of ${\mathbf{x}}_i$ among $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{i-1}\}$ with respect to the Euclidean distance. In that case, the density $\tilde{p}\left({\mathbf{y}}_S\right)$ is given by

$$\begin{array}{c}\hfill \tilde{p}\left({\mathbf{y}}_S\right)={\displaystyle \prod _{i=1}^n}\mathcal{N}(y\left({\mathbf{x}}_i\right)|{\mathbf{B}}_{{\mathbf{x}}_i}{y}_{N\left({\mathbf{x}}_i\right)},F_{{\mathbf{x}}_i}),\end{array}$$

(2)

where ${\mathbf{B}}_{{\mathbf{x}}_i}={\mathbf{C}}_{{\mathbf{x}}_i,N\left({\mathbf{x}}_i\right)}{\mathbf{C}}_{N\left({\mathbf{x}}_i\right)}^{-1}$ and $F_{{\mathbf{x}}_i}=\mathbf{C}({\mathbf{x}}_i,{\mathbf{x}}_i)-{\mathbf{C}}_{{\mathbf{x}}_i,N\left({\mathbf{x}}_i\right)}{\mathbf{C}}_{N\left({\mathbf{x}}_i\right)}^{-1}{\mathbf{C}}_{N\left({\mathbf{x}}_i\right),{\mathbf{x}}_i}$. Equation (2) holds true due to (1) and the fact that for

$$\mathbf{x}=\left(\begin{array}{c}{\mathbf{x}}_1\\ {\mathbf{x}}_2\end{array}\right)\sim \mathcal{N}\left(\left(\begin{array}{c}{\mathit{\mu }}_1\\ {\mathit{\mu }}_2\end{array}\right),\left(\begin{array}{cc}{\mathbf{\Sigma }}_{11}& {\mathbf{\Sigma }}_{12}\\ {\mathbf{\Sigma }}_{21}& {\mathbf{\Sigma }}_{22}\end{array}\right)\right),$$

$({\mathbf{x}}_1|{\mathbf{x}}_2=\mathbf{a})\sim \mathcal{N}(\overline{\mathit{\mu }},\overline{\mathbf{\Sigma }})$ with $\overline{\mathit{\mu }}={\mathit{\mu }}_1+{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}(\mathbf{a}-{\mathit{\mu }}_2)$ and $\overline{\mathbf{\Sigma }}={\mathbf{\Sigma }}_{11}-{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}{\mathbf{\Sigma }}_{21}$.

Following Datta et al. [40], the density $\tilde{p}\left({\mathbf{y}}_S\right)$ can be extended to a stochastic process that satisfies Kolmogorov’s consistency criteria. Let $\mathbf{u}\in X\setminus S$.

Further, let $N\left(\mathbf{u}\right)$ be the set of m-nearest neighbors of $\mathbf{u}$ in S. For any finite set $U=\{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_r\}$ with $S\cap U=Ø$ we define the nearest neighbor density of ${\mathbf{y}}_U$ conditional on ${\mathbf{y}}_S$ as

$$\begin{array}{c}\hfill \tilde{p}({\mathbf{y}}_U|{\mathbf{y}}_S)={\displaystyle \prod _{i=1}^r}p(y\left({\mathbf{u}}_i\right)|{\mathbf{y}}_{N\left({\mathbf{u}}_i\right)}).\end{array}$$

(3)

This conditional density is similar to (1), except that all the neighbor sets are subsets of S. Indeed, (1) and (3) are sufficient to describe the joint nearest neighbor density of any finite set $V=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_n\}\subseteq X$

$$\begin{array}{c}\hfill \tilde{p}\left({\mathbf{y}}_V\right)=\int \tilde{p}({\mathbf{y}}_U|{\mathbf{y}}_S)\tilde{p}\left({\mathbf{y}}_S\right){\displaystyle \prod _{\{{\mathbf{s}}_i\in S\setminus V\}}}d\mathbf{y}\left({\mathbf{s}}_i\right),\end{array}$$

(4)

where $U=V\setminus S$. If $U=Ø$, then (3) implies that $\tilde{p}({\mathbf{y}}_U|{\mathbf{y}}_S)=1$ in (4). If $S\setminus V=Ø$ the integration in (4) is not needed. Moreover, Datta et al. [40] show that $\tilde{p}\left({\mathbf{y}}_V\right)$ is the density of a zero mean GP with covariance function,

$$\begin{array}{cc}\hfill \tilde{C}({\mathbf{v}}_1,{\mathbf{v}}_2)& =\left\{\begin{array}{cc}{\tilde{C}}_{{\mathbf{x}}_i,{\mathbf{x}}_j},\hfill & \mathrm{if}{\mathbf{v}}_1={\mathbf{x}}_i,{\mathbf{v}}_2={\mathbf{x}}_j\mathrm{are}\mathrm{both}\mathrm{in}S,\hfill \\ {\mathbf{B}}_{{\mathbf{v}}_1}{\tilde{\mathbf{C}}}_{N\left({\mathbf{v}}_1\right),{\mathbf{x}}_j},\hfill & \mathrm{if}{\mathbf{v}}_1\notin S,{\mathbf{v}}_2={\mathbf{x}}_j\in S,\hfill \\ {\mathbf{B}}_{{\mathbf{v}}_1}{\tilde{\mathbf{C}}}_{N\left({\mathbf{v}}_1\right),N\left({\mathbf{v}}_2\right)}{\mathbf{B}}_{{\mathbf{v}}_2}^T+{\delta }_{({\mathbf{v}}_1={\mathbf{v}}_2)}{F}_{{\mathbf{v}}_1},\hfill & \mathrm{if}{\mathbf{v}}_1,{\mathbf{v}}_2\notin S,\hfill \end{array}\right.\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill {\tilde{\mathbf{C}}}_S& ={\mathbf{B}}_S^{-1}{\mathbf{F}}_S{\left({\mathbf{B}}_S^T\right)}^{-1},\hfill \end{array}$$

(5)

with

$$\begin{array}{cc}\hfill {\mathbf{B}}_S& =\left(\begin{array}{c}1\\ & 1& & \mathcal{O}\\ & & \ddots \\ & B_{{\mathbf{x}}_i,j}& & 1\\ & & & & & 1\end{array}\right),B_{{\mathbf{x}}_i,j}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}i=j,\hfill \\ -{\mathbf{B}}_{{\mathbf{x}}_i}\left[l\right],\hfill & \mathrm{if}{\mathbf{x}}_j={\mathbf{x}}_i^l\mathrm{for}\mathrm{some}l,\hfill \\ 0,\hfill & \mathrm{else},\hfill \end{array}\right.\hfill \\ \hfill {\mathbf{B}}_{{\mathbf{v}}_i}& ={\mathbf{C}}_{{\mathbf{v}}_i,N\left({\mathbf{v}}_i\right)}{\mathbf{C}}_{N\left({\mathbf{v}}_i\right)}^{-1},\hfill \\ \hfill {\mathbf{F}}_S& =diag(F_{{\mathbf{x}}_1},\dots ,F_{{\mathbf{x}}_n}),\hfill \\ \hfill F_{{\mathbf{v}}_i}& =C_{{\mathbf{v}}_i}-{\mathbf{B}}_{{\mathbf{v}}_i}{\mathbf{C}}_{N\left({\mathbf{v}}_i\right),{\mathbf{v}}_i},\hfill \end{array}$$

where ${\mathbf{x}}_j={\mathbf{x}}_i^l$ means that ${\mathbf{x}}_j$ is the l-th neighbor of ${\mathbf{x}}_i$ ($N\left({\mathbf{x}}_i\right)=\left\{{\mathbf{x}}_i^1,{\mathbf{x}}_i^2,\dots ,{\mathbf{x}}_i^l,\dots ,{\mathbf{x}}_i^{|N({\mathbf{x}}_i)|}\right\}$), ${\mathbf{v}}_1,{\mathbf{v}}_2\in X$, ${\tilde{\mathbf{C}}}_{A,B}$ denotes submatrices of $\tilde{\mathbf{C}}$ indexed by the elements of the sets A and B. The Kronecker delta is denoted by ${\delta }_{({\mathbf{v}}_1={\mathbf{v}}_2)}$.

3.2. Reference Prior

Paulo [37] computes the general form of the reference prior for a broad class of Gaussian processes. Assume that $y(·)$ is a GP on $X\subseteq {\mathbb{R}}^d$ with mean and covariance given by

$$\begin{array}{c}\hfill \mathit{\mu }\left(\mathbf{x}\right)={\mathbf{x}}^T\mathit{\beta },C({\mathbf{x}}_1,{\mathbf{x}}_2)={\sigma }^{2}K({\mathbf{x}}_1,{\mathbf{x}}_2|\mathit{\xi }),\end{array}$$

where $\mathit{\beta }\in {\mathbb{R}}^d$, $\mathit{\xi }\in {\mathbb{R}}_{+}^r$ stands for a vector of additional correlation parameters, and ${\sigma }^2>0$ is the process variance. Some additional notation is required to define the likelihood function corresponding to the observed data D:

$$\begin{array}{cc}\hfill \mathbf{X}& ={({\mathbf{x}}_1\cdots {\mathbf{x}}_n)}^T,\hfill \\ \hfill \mathit{\eta }& ={({\sigma }^2,{\mathit{\beta }}^T,{\mathit{\xi }}^T)}^T,\hfill \\ \hfill {\mathbf{K}}_S& ={\left[K({\mathbf{x}}_i,{\mathbf{x}}_j|\mathit{\xi })\right]}_{{\mathbf{x}}_i,{\mathbf{x}}_j\in S}.\hfill \end{array}$$

Then, ${\mathbf{y}}_S|\mathit{\eta }\sim \mathcal{N}(\mathbf{X}\mathit{\beta },{\sigma }^2{\mathbf{K}}_S)$, which implies that the likelihood is given by

$$\begin{array}{c}\hfill L(\mathit{\eta }|{\mathbf{y}}_S)\propto {\left({\sigma }^2\right)}^{-{\displaystyle \frac{n}{2}}}{|{\mathbf{K}}_S|}^{-{\displaystyle \frac{1}{2}}}exp\left\{-{\displaystyle \frac{1}{2{\sigma }^2}}{({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })}^T{\mathbf{K}}_S^{-1}({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })\right\}.\end{array}$$

When defining the reference prior, ${({\sigma }^2,{\mathit{\xi }}^T)}^T$ is considered as a parameter of interest, and $\mathit{\beta }$ is considered as a nuisance parameter. Moreover, the distribution of the reference prior ${\pi }^R\left(\mathit{\eta }\right)$ is factored as ${\pi }^R\left(\mathit{\eta }\right)={\pi }^R(\mathit{\beta }|{\sigma }^2,\mathit{\xi }){\pi }^R({\sigma }^2,\mathit{\xi })$. The conditional prior of $\mathit{\beta }$ is specified proportional to 1, i.e., ${\pi }^R(\mathit{\beta }|{\sigma }^2,\mathit{\xi })\propto 1$, because this is the Jeffreys-rule prior (see Robert et al. [42]), which is a noninformative prior, for the model at hand when ${\sigma }^2$ and $\mathit{\xi }$ are considered to be known.

Next, ${\pi }^R({\sigma }^2,\mathit{\xi })$ is calculated as the Jeffreys-rule prior for the marginal experiment defined via the integrated likelihood function with respect to ${\pi }^R(\mathit{\beta }|{\sigma }^2,\mathit{\xi })$. The integrated likelihood function is given by

$$\begin{array}{cc}\hfill L^I({\sigma }^2,\mathit{\xi }|{\mathbf{y}}_S)& =\underset{{\mathbb{R}}^d}{\int }L(\mathit{\eta }|{\mathbf{y}}_S){\pi }^R(\mathit{\beta }|{\sigma }^2,\mathit{\xi })d\mathit{\beta },\hfill \\ \hfill & =\underset{{\mathbb{R}}^d}{\int }L(\mathit{\eta }|{\mathbf{y}}_S)d\mathit{\beta },\hfill \\ \hfill & \propto {\left({\sigma }^2\right)}^{-{\displaystyle \frac{n-d}{2}}}|{\mathbf{K}}_S{|}^{-{\displaystyle \frac{1}{2}}}||{\mathbf{X}}^T{\mathbf{K}}_S^{-1}{\mathbf{X}|}^{-{\displaystyle \frac{1}{2}}}exp\left\{-{\displaystyle \frac{S_{\mathit{\xi }}^2}{2{\sigma }^2}}\right\},\hfill \end{array}$$

where $S_{\mathit{\xi }}^2={\mathbf{y}}_S^T\mathbf{Q}{\mathbf{y}}_S$, $\mathbf{Q}={\mathbf{K}}_S^{-1}\mathbf{P}$, and $\mathbf{P}=\mathbf{I}-\mathbf{X}{\left({\mathbf{X}}^T{\mathbf{K}}_S^{-1}\mathbf{X}\right)}^{-1}{\mathbf{X}}^T{\mathbf{K}}_S^{-1}$.

Finally, the reference prior ${\pi }^R\left(\mathit{\eta }\right)$, see also [37], is of the form ${\pi }^R\left(\mathit{\eta }\right)\propto {\displaystyle \frac{{\pi }^R\left(\mathit{\xi }\right)}{{\sigma }^2}}$ with ${\pi }^R\left(\mathit{\xi }\right)\propto {|{\tilde{I}}_R\left(\mathit{\xi }\right)|}^{{\displaystyle \frac{1}{2}}}$ where

$$\begin{array}{cc}\hfill {\mathbf{W}}_k& ={\displaystyle \frac{\partial {\mathbf{K}}_S}{\partial {\mathit{\xi }}_k}}\mathbf{Q},k=1,\dots ,r,\hfill \\ \hfill {\tilde{I}}_R\left(\mathit{\xi }\right)& =\left(\begin{array}{ccccc}n-d& tr\left({\mathbf{W}}_1\right)& tr\left({\mathbf{W}}_2\right)& \dots & tr\left({\mathbf{W}}_r\right)\\ tr\left({\mathbf{W}}_1\right)& tr\left({\mathbf{W}}_1^2\right)& tr\left({\mathbf{W}}_1{\mathbf{W}}_2\right)& \dots & tr\left({\mathbf{W}}_1{\mathbf{W}}_r\right)\\ ⋮& ⋮& ⋮& & ⋮\\ tr\left({\mathbf{W}}_r\right)& tr\left({\mathbf{W}}_1{\mathbf{W}}_r\right)& tr\left({\mathbf{W}}_2{\mathbf{W}}_r\right)& \dots & tr\left({\mathbf{W}}_r^2\right)\end{array}\right).\hfill \end{array}$$

3.3. Feature Selection Model

In this section, we formally introduce the feature selection model. First, we define the likelihood function. Subsequently, prior distributions are assigned to the model parameters within a Bayesian framework.

3.3.1. Likelihood Function

Let $y\in \mathbb{R}$ denote a target variable of interest and let $\mathbf{x}\in {\mathbb{R}}^d$ denote covariates. It is assumed that the relationship

$$\begin{array}{c}\hfill y={\mathbf{x}}^T\mathit{\beta }+f\left(\mathbf{x}\right)+\epsilon ,\end{array}$$

(6)

holds, where f denotes an unknown function and $\epsilon$ additive noise. Moreover, let $\mathcal{A}$ denote a (random) subset of the index set $\left\{1,\dots ,d\right\}$, with $|\mathcal{A}|=k>0$. This set is used to specify the covariates that contribute to the regression model (6). For given $\mathcal{A}$, we assign a GP prior with zero mean and covariance function $\tilde{C}$ to the sum $z:=f+\epsilon$:

$$\begin{array}{c}\hfill z\sim GP(0,\tilde{C}(·,·|{\sigma }^2,\gamma ,\rho ,\mathcal{A})),\end{array}$$

(7)

with ${\sigma }^2>0,\rho >0$ and $\gamma \in (0,1)$. In particular, the GP prior $GP(0,\tilde{C}(·,·|{\sigma }^2,\gamma ,\rho ,\mathcal{A}))$ (7) is defined as the NNGP derived from the parent $GP(0,C(·,·|{\sigma }^2,\gamma ,\rho ,\mathcal{A}))$, with

$$\begin{array}{cc}\hfill C(\mathbf{x},{\mathbf{x}}^{\prime }|{\sigma }^2,\gamma ,\rho ,\mathcal{A})& ={\sigma }^2\underset{=\mathcal{K}(\mathbf{x},{\mathbf{x}}^{\prime }|\gamma ,\rho ,\mathcal{A})}{\underbrace{\left[{\delta }_{(\mathbf{x}={\mathbf{x}}^{\prime })}(1-\gamma )+\gamma K(\mathbf{x},{\mathbf{x}}^{\prime }|\rho ,\mathcal{A})\right]}},\hfill \end{array}$$

(8)

where

$$\begin{array}{cc}\hfill K(\mathbf{x},{\mathbf{x}}^{\prime }|\rho ,\mathcal{A})& =\left(1+{\displaystyle \frac{\sqrt{5}{d}_{\mathcal{A}}}{\rho }}+{\displaystyle \frac{5d_{\mathcal{A}}^2}{3{\rho }^2}}\right)exp\left(-{\displaystyle \frac{\sqrt{5}{d}_{\mathcal{A}}}{\rho }}\right),\hfill \end{array}$$

with

$$\begin{array}{cc}\hfill 1-1d_{\mathcal{A}}& =\sqrt{{\displaystyle \sum _{i\in \mathcal{A}}}{(x_i-x_i^{\prime })}^2}.\hfill \end{array}$$

(9)

Therefore, the parent GP has a Matérn-${\displaystyle \frac{5}{2}}$ covariance function [43]. The Matérn covariance function can be seen as a generalization of the Gaussian radial basis function.

The Euclidean distance function $d_{\mathcal{A}}$ (9) considers only predictors with indices in $\mathcal{A}$. Function $d_{\mathcal{A}}$ is also used to determine the neighbor sets required for the computation of the NNGP.

It is noteworthy that the parent GP results from assigning a zero mean GP with a covariance function ${\sigma }_f^{2}K(\mathbf{x},{\mathbf{x}}^{\prime }|\rho ,\mathcal{A})$ to f while assuming the noise $\epsilon$ to be $iid$$\mathcal{N}(0,{\sigma }_{\epsilon }^2)$. The reparameterization ${\sigma }^2={\sigma }_f^2+{\sigma }_{\epsilon }^2,\gamma =1-{\displaystyle \frac{{\sigma }_{\epsilon }^2}{{\sigma }^2}}$ in (8) gives the following:

$$\begin{array}{cc}\hfill C(\mathbf{x},{\mathbf{x}}^{\prime }|{\sigma }_f^2,{\sigma }_{\epsilon }^2,\rho ,\mathcal{A})& ={\sigma }_{\epsilon }^2{\delta }_{(\mathbf{x}={\mathbf{x}}^{\prime })}+{\sigma }_f^{2}K(\mathbf{x},{\mathbf{x}}^{\prime }|\rho ,\mathcal{A}),\hfill \\ \hfill & ={\sigma }^2\left[{\displaystyle \frac{{\sigma }_{\epsilon }^2}{{\sigma }^2}}{\delta }_{(\mathbf{x}={\mathbf{x}}^{\prime })}+{\displaystyle \frac{{\sigma }^2-{\sigma }_{\epsilon }^2}{{\sigma }^2}}K(\mathbf{x},{\mathbf{x}}^{\prime }|\rho ,\mathcal{A})\right],\hfill \\ \hfill & ={\sigma }^2\left[(1-\gamma ){\delta }_{(\mathbf{x}={\mathbf{x}}^{\prime })}+\gamma K(\mathbf{x},{\mathbf{x}}^{\prime }|\rho ,\mathcal{A})\right].\hfill \end{array}$$

The above specifications directly imply that y follows a GP with mean function $\mu \left(\mathbf{x}\right)={\mathbf{x}}^T\mathit{\beta }$ and covariance function $\tilde{C}(\mathbf{x},{\mathbf{x}}^{\prime }|{\sigma }^2,\gamma ,\rho ,\mathcal{A})$. Let $D=\{({\mathbf{x}}_i,y_i)|i=1\dots ,n\}$ denote some training data and, further, let $S=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$ denote the corresponding set of covariate constellations (experimental design). Consider the following additional notations:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{C}}}_S& ={\left[\tilde{C}({\mathbf{x}}_i,{\mathbf{x}}_j|{\sigma }^2,\gamma ,\rho ,\mathcal{A})\right]}_{{\mathbf{x}}_i,{\mathbf{x}}_j\in S}.\hfill \end{array}$$

We recall from (5) that ${\tilde{\mathbf{C}}}_S$ is defined as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{C}}}_S& ={\mathbf{B}}_S^{-1}{\mathbf{F}}_S{\left({\mathbf{B}}_S^T\right)}^{-1},\hfill \end{array}$$

with

$$\begin{array}{cc}\hfill {\mathbf{B}}_S& =\left(\begin{array}{c}1\\ & 1& & \mathcal{O}\\ & & \ddots \\ & B_{{\mathbf{x}}_i,j}& & 1\\ & & & & & 1\end{array}\right),B_{{\mathbf{x}}_i,j}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}i=j,\hfill \\ -{\mathbf{B}}_{{\mathbf{x}}_i}\left[l\right],\hfill & \mathrm{if}{\mathbf{x}}_j={\mathbf{x}}_i^l\mathrm{for}\mathrm{some}l,\hfill \\ 0,\hfill & \mathrm{else},\hfill \end{array}\right.\hfill \\ \hfill {\mathbf{B}}_{{\mathbf{x}}_i}& ={\mathbf{C}}_{{\mathbf{x}}_i,N\left({\mathbf{x}}_i\right)}{\mathbf{C}}_{N\left({\mathbf{x}}_i\right)}^{-1}={\sigma }^2{\mathcal{K}}_{{\mathbf{x}}_i,N\left({\mathbf{x}}_i\right)}{\left({\sigma }^2\right)}^{-1}{\mathcal{K}}_{N\left({\mathbf{x}}_i\right)}^{-1},\hfill \\ \hfill & ={\mathcal{K}}_{{\mathbf{x}}_i,N\left({\mathbf{x}}_i\right)}{\mathcal{K}}_{N\left({\mathbf{x}}_i\right)}^{-1}(\mathrm{independent}\mathrm{of}{\sigma }^2),\hfill \\ \hfill {\mathbf{F}}_S& =diag(F_{{\mathbf{x}}_1},\dots ,F_{{\mathbf{x}}_n}),\hfill \\ \hfill F_{{\mathbf{x}}_i}& =C_{{\mathbf{x}}_i}-{\mathbf{B}}_{{\mathbf{x}}_i}{\mathbf{C}}_{N\left({\mathbf{x}}_i\right),{\mathbf{x}}_i},\hfill \\ \hfill & ={\sigma }^2{\mathcal{K}}_{{\mathbf{x}}_i}-{\mathbf{B}}_{{\mathbf{x}}_i}{\sigma }^2{\mathcal{K}}_{N\left({\mathbf{x}}_i\right),{\mathbf{x}}_i},\hfill \\ \hfill & ={\sigma }^2{\displaystyle \underset{={\mathcal{F}}_{{\mathbf{x}}_i}}{\overset{(\mathrm{independent}\mathrm{of}{\sigma }^2)}{\underbrace{\left[{\mathcal{K}}_{{\mathbf{x}}_i}-{\mathbf{B}}_{{\mathbf{x}}_i}{\mathcal{K}}_{N\left({\mathbf{x}}_i\right),{\mathbf{x}}_i}\right]}}}}.\hfill \end{array}$$

Finally, we find that

$$\begin{array}{cc}\hfill {\tilde{\mathbf{C}}}_S& ={\sigma }^2\underset{=:{\tilde{\mathcal{K}}}_S}{\underbrace{{\mathbf{B}}_S^{-1}diag({\mathcal{F}}_{{\mathbf{x}}_1},\dots ,{\mathcal{F}}_{{\mathbf{x}}_n}){\left({\mathbf{B}}_S^T\right)}^{-1}}},\hfill \\ \hfill & ={\sigma }^2{\tilde{\mathcal{K}}}_S.\hfill \end{array}$$

Note that ${\tilde{\mathcal{K}}}_S$ is independent of ${\sigma }^2$, while it is dependent on $\rho$, $\gamma$, and $\mathcal{A}$. Thus, ${\mathbf{y}}_S|{\sigma }^2,\gamma ,\rho ,\mathcal{A}\sim \mathcal{N}(\mathbf{X}\mathit{\beta },{\sigma }^2{\tilde{\mathcal{K}}}_S)$, which implies that the likelihood function is given by

$$\begin{array}{c}\hfill L(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S)\propto {\left({\sigma }^2\right)}^{-{\displaystyle \frac{n}{2}}}{|{\tilde{\mathcal{K}}}_S|}^{-{\displaystyle \frac{1}{2}}}exp\left\{-{\displaystyle \frac{1}{2{\sigma }^2}}{({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })}^T{\tilde{\mathcal{K}}}_S^{-1}({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })\right\}.\end{array}$$

3.3.2. Reference Prior

For given $\mathcal{A}$ we denote the components of $\mathit{\beta }$ with indices in $\mathcal{A}$ as ${\mathit{\beta }}_{\mathcal{A}}$ and the complementary ones as ${\mathit{\beta }}_{{\mathcal{A}}^C}$. We set ${\mathit{\beta }}_{{\mathcal{A}}^C}$ equal to zero and assign a uniform prior to ${\mathit{\beta }}_{\mathcal{A}}$. More generally, we could also specify appropriate spike and slab priors for the parameters on the complementary set ${\mathcal{A}}^c$. For the use of such types of priors, we refer to Malsiner-Walli and Wagner [44] and Louzada et al. [45].

The joint prior density with respect to the measure ${(\lambda +{\delta }_0)}^d$ is given by

$$\begin{array}{c}\hfill p(\mathit{\beta }|\mathcal{A})=p({\mathit{\beta }}_{\mathcal{A}}|\mathcal{A})p({\mathit{\beta }}_{{\mathcal{A}}^C}|\mathcal{A})\propto I_{\{{\mathit{\beta }}_{{\mathcal{A}}^C}=\mathbf{0}\}}\left({\mathit{\beta }}_{{\mathcal{A}}^C}\right)I_{\{{\mathit{\beta }}_i\ne 0|\forall i\in \mathcal{A}\}}\left({\mathit{\beta }}_{\mathcal{A}}\right),\end{array}$$

where $\lambda$ denotes the Lebesgue measure and ${\delta }_0$ denotes the Dirac measure concentrated at zero. Thus, the integrated likelihood (compare to Section 3.2) is given by

$$\begin{array}{cc}\hfill L^I({\sigma }^2,\rho ,\gamma |{\mathbf{y}}_S,\mathcal{A})& =\underset{{\mathbb{R}}^d}{\int }L({\sigma }^2,\rho ,\gamma ,\mathit{\beta }|{\mathbf{y}}_S,\mathcal{A})p(\mathit{\beta }|\mathcal{A})d\lambda \left({\mathit{\beta }}_{\mathcal{A}}\right)d\left({\delta }_0\left({\mathit{\beta }}_{\mathcal{A}}^C\right)\right),\hfill \\ \hfill & \propto {\left({\sigma }^2\right)}^{-{\displaystyle \frac{n-|\mathcal{A}|}{2}}}|{\tilde{\mathcal{K}}}_S{|}^{-{\displaystyle \frac{1}{2}}}||{\mathbf{X}}_{\mathcal{A}}^T{\tilde{\mathcal{K}}}_S^{-1}{\mathbf{X}}_{\mathcal{A}}{|}^{-{\displaystyle \frac{1}{2}}}exp\left\{-{\displaystyle \frac{S_{\lambda }^2}{2{\sigma }^2}}\right\},\hfill \end{array}$$

where $\lambda ={(\rho ,\gamma ,\mathcal{A})}^T$, $S_{\lambda }^2={\mathbf{y}}_S^T\mathbf{Q}{\mathbf{y}}_S$, $\mathbf{Q}={\tilde{\mathcal{K}}}_S^{-1}\mathbf{P}$, $\mathbf{P}=\mathbf{I}-{\mathbf{X}}_{\mathcal{A}}{\left({\mathbf{X}}_{\mathcal{A}}^T{\tilde{\mathcal{K}}}_S^{-1}{\mathbf{X}}_{\mathcal{A}}\right)}^{-1}{\mathbf{X}}_{\mathcal{A}}^T{\tilde{\mathcal{K}}}_S^{-1}$, and ${\mathbf{X}}_{\mathcal{A}}$ denotes the corresponding to the set $\mathcal{A}$ reduced design matrix.

In accordance with Section 3.2 we use the prior

$$\begin{array}{cc}\hfill p({\sigma }^2,\rho ,\gamma )& \propto {\displaystyle \frac{|{\tilde{I}}_R{(\rho ,\gamma )|}^{\frac{1}{2}}}{{\sigma }^2}},\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill {\mathbf{W}}_{\rho }& ={\displaystyle \frac{\partial {\tilde{\mathcal{K}}}_S}{\partial \rho }}\mathbf{Q},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathbf{W}}_{\gamma }& ={\displaystyle \frac{\partial {\tilde{\mathcal{K}}}_S}{\partial \gamma }}\mathbf{Q},\hfill \\ \hfill {\tilde{I}}_R(\rho ,\gamma )& =\left(\begin{array}{ccc}n-|\mathcal{A}|& tr\left({\mathbf{W}}_{\rho }\right)& tr\left({\mathbf{W}}_{\gamma }\right))\\ tr\left({\mathbf{W}}_{\rho }\right)& tr\left({\mathbf{W}}_{\rho }^2\right)& tr\left({\mathbf{W}}_{\rho }{\mathbf{W}}_{\gamma }\right)\\ tr\left({\mathbf{W}}_{\gamma }\right)& tr\left({\mathbf{W}}_{\rho }{\mathbf{W}}_{\gamma }\right)& tr\left({\mathbf{W}}_{\gamma }^2\right)\end{array}\right).\hfill \end{array}$$

(11)

Details concerning the computation of the derivatives presented in Equations (10) and (11) are provided in Appendix B.

3.3.3. Prior for the Random Set

As in Posch et al. [15], we propose the following prior for $\mathcal{A}$:

$$\begin{array}{cc}\hfill & p(\mathcal{A}=\{{\alpha }_1,\dots ,{\alpha }_k\})\propto (p_{{\alpha }_1}+\dots +p_{{\alpha }_k}){\displaystyle \frac{1}{k}}\tilde{p}\left(k\right),\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill & \{p_{{\alpha }_1},\dots ,p_{{\alpha }_k}\}\subseteq \{p_1,\dots ,p_d\}\mathrm{with}{\displaystyle \sum _{i=1}^d}{p}_i=1\mathrm{and}{p}_i\ge 0\mathrm{for}i=1,\dots ,d\mathrm{and},\hfill \\ \hfill & \tilde{p}:\{1,\dots ,d\}\to {\mathbb{R}}_0^{+}.\hfill \end{array}$$

The mapping $\tilde{p}$ can be chosen arbitrarily and is intended to represent the a priori belief regarding the number of predictors contributing to the model. Moreover, the parameters $p_1,\dots ,p_d$ represent the a priori beliefs about the relative importance of the predictors $x_1,\dots ,x_d$.

3.4. Model Inference

Clearly, the joint posterior density $p(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S,\mathbf{X})$ is analytically intractable. To overcome this limitation, we propose a Metropolis-within-Gibbs algorithm to sample from this distribution. The algorithm produces a Markov Chain with the true posterior as the limiting distribution. In particular, the algorithm includes three main steps:

• Step 1: Sample from the conditional posterior $p({\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1}|{\mathbf{y}}_S,\mathbf{X},{\sigma }_t^2,{\gamma }_t,{\rho }_t)$ using a classical Metropolis–Hastings (MH) algorithm;
• Step 2: Sample from the conditional posterior $p({\sigma }_{t+1}^2|{\mathbf{y}}_S,\mathbf{X},{\gamma }_t,{\rho }_t,{\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1})$;
• Step 3: Sample from the conditional posterior $p({\gamma }_{t+1},{\rho }_{t+1}|{\mathbf{y}}_S,\mathbf{X},{\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1},{\sigma }_{t+1}^2)$ using Hamiltonian MCMC (HMC).

In order to run the procedure one has to define an initial sample ${\mathit{\beta }}_0,{\sigma }_0^2,{\gamma }_0,{\rho }_0,{\mathcal{A}}_0$ that lies in the support of $p(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S,\mathbf{X})$. Additionally, some tuning parameters for the proposal distribution of $\mathcal{A}$ are required, along with the step size S and the number of steps L for the leapfrog method, which is used within Step 3 (HMC). The Metropolis-within-Gibbs sampler and the HMC update for $(\gamma ,\rho )$ follow standard implementations as described in Neal [46], Roberts and Rosenthal [47], with only minor adaptations specific to the NNGP structure. Details concerning the three steps are provided below.

3.4.1. Step 1

To specify the proposal for the random set $\mathcal{A}$, at first, a Bernoulli-distributed random variable $c_h$ is introduced (as suggested by Posch et al. [15])

$$c_h\sim \mathrm{Bernoulli}\left(p_h\right),$$

(12)

where $p_h\in [0,1]$ can be considered as a tuning parameter. The event $c_h=1$ means that the model size (number of predictors used) changes, i.e., $k_{t+1}\ne k_t$. Moreover, a random variable $\alpha$ with support on the index set $\{1,\dots ,d\}$ is introduced in order to describe the model transition probabilities in the event of changing model size. A realization of the conditional random variable $\alpha |{\mathcal{A}}_t$ corresponds to the index of a predictor that is going to be added to or removed from the model. Transitions between models, which differ by two or more parameters, are not allowed. The probability mass function of $\alpha |{\mathcal{A}}_t$ is defined as

$$q(\alpha |{\mathcal{A}}_t)={\mathrm{I}}_{\{1,\dots ,d\}}\left(\alpha \right)\left\{\begin{array}{cc}{\tilde{p}}_{\alpha },\hfill & \mathrm{if}{k}_t>1,\alpha \notin {\mathcal{A}}_t,\hfill \\ {\displaystyle \frac{{\displaystyle \sum _{i\in {\mathcal{A}}_t}}{\tilde{p}}_i}{{\tilde{p}}_{\alpha }{\displaystyle \sum _{i\in {\mathcal{A}}_t}}1/{\tilde{p}}_i}},\hfill & \mathrm{if}{k}_t>1,\alpha \in {\mathcal{A}}_t,\hfill \\ 0,\hfill & \mathrm{if}{k}_t=1,\alpha \in {\mathcal{A}}_t,\hfill \\ {\displaystyle \frac{{\tilde{p}}_{\alpha }}{{\displaystyle \sum _{i\in \{1,\dots ,p\}\setminus {\mathcal{A}}_t}}{\tilde{p}}_i}},\hfill & \mathrm{else},\hfill \end{array}\right.$$

(13)

where the parameters ${\tilde{p}}_1,\dots ,{\tilde{p}}_d$ are greater or equal to zero, sum up to one, and can but must not be identical to the parameters $p_1,\dots ,p_d$ already used in the prior $p\left(\mathcal{A}\right)$. Using (12) and (13), the proposal distribution $q({\mathcal{A}}_{t+1}|{\mathcal{A}}_t)$ is finally defined by

$$\begin{array}{cc}\hfill & q({\mathcal{A}}_{t+1}|{\mathcal{A}}_t,c_h=1)=q([{\mathcal{A}}_{t+1}\setminus {\mathcal{A}}_t]\cup [{\mathcal{A}}_t\setminus {\mathcal{A}}_{t+1}]|{\mathcal{A}}_t){\mathrm{I}}_{\left\{1\right\}}(|k_{t+1}-k_t|),\hfill \\ \hfill & q({\mathcal{A}}_{t+1}|{\mathcal{A}}_t,c_h=0)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathcal{A}}_{t+1}={\mathcal{A}}_t,\hfill \\ 0,\hfill & \mathrm{else},\hfill \end{array}\right.\hfill \\ \hfill & q({\mathcal{A}}_{t+1}|{\mathcal{A}}_t)={\displaystyle \sum _{c_h\in \{0,1\}}}q({\mathcal{A}}_{t+1},c_h|{\mathcal{A}}_t),\hfill \\ \hfill & ={\displaystyle \sum _{c_h\in \{0,1\}}}q({\mathcal{A}}_{t+1}|c_h,{\mathcal{A}}_t)q\left(c_h\right).\hfill \end{array}$$

By straightforward algebraic manipulation, it is evident that

$$\begin{array}{cc}\hfill {\mathit{\beta }}_{t+1,{\mathcal{A}}_{t+1}}|{\mathbf{y}}_S,\mathbf{X},{\sigma }_t^2,{\gamma }_t,{\rho }_t,{\mathcal{A}}_{t+1}& \sim \mathcal{N}(\mathit{\mu }=\mathbf{\Sigma }{\mathbf{X}}_{{\mathcal{A}}_{t+1}}^T{\tilde{\mathbf{C}}}_S^{-1}{\mathbf{y}}_S,\mathbf{\Sigma }={\left({\mathbf{X}}_{{\mathcal{A}}_{t+1}}^T{\tilde{\mathbf{C}}}_S^{-1}{\mathbf{X}}_{{\mathcal{A}}_{t+1}}\right)}^{-1}),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathit{\beta }}_{t+1,{\mathcal{A}}_{t+1}^C}|{\mathbf{y}}_S,\mathbf{X},{\sigma }_t^2,{\gamma }_t,{\rho }_t,{\mathcal{A}}_{t+1}& \sim {\delta }_0.\hfill \end{array}$$

(15)

We use the conditional posterior (14) and (15) as a proposal for ${\mathit{\beta }}_{t+1}$. The Hastings ratio r is then given by (16).

$$\begin{array}{c}\hfill r={\displaystyle \frac{L({\mathit{\beta }}_{t+1},{\sigma }_t^2,{\gamma }_t,{\rho }_t,{\mathcal{A}}_{t+1}|{\mathbf{y}}_S)p({\mathit{\beta }}_{t+1}|{\mathcal{A}}_{t+1})p\left({\mathcal{A}}_{t+1}\right)q({\mathit{\beta }}_t|{\sigma }_t^2,{\gamma }_t,{\rho }_t,{\mathcal{A}}_t)q({\mathcal{A}}_t|{\mathcal{A}}_{t+1})}{L({\mathit{\beta }}_t,{\sigma }_t^2,{\gamma }_t,{\rho }_t,{\mathcal{A}}_t|{\mathbf{y}}_S)p({\mathit{\beta }}_t|{\mathcal{A}}_t)p\left({\mathcal{A}}_t\right)q({\mathit{\beta }}_{t+1}|{\sigma }_t^2,{\gamma }_t,{\rho }_t,{\mathcal{A}}_{t+1})q({\mathcal{A}}_{t+1}|{\mathcal{A}}_t)}}.\end{array}$$

(16)

3.4.2. Step 2

The conditional posterior $p({\sigma }_{t+1}^2|{\mathbf{y}}_S,\mathbf{X},{\gamma }_t,{\rho }_t,{\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1})$ can be derived in a closed analytical form. Upon straightforward algebraic manipulation, it comes out as an inverse gamma (IG) distribution:

$$\begin{array}{c}\hfill {\sigma }_{t+1}^2|{\mathbf{y}}_S,X,{\gamma }_t,{\rho }_t,{\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1}\sim IG(\alpha ={\displaystyle \frac{n}{2}},\beta ={\displaystyle \frac{1}{2}}{({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })}^T{\tilde{\mathcal{K}}}_S^{-1}({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })).\end{array}$$

(17)

3.4.3. Step 3

For the HMC algorithm we reparameterize the posterior $p({\gamma }_{t+1},{\rho }_{t+1}|{\mathbf{y}}_S,\mathbf{X},{\mathit{\beta }}_{t+1},$${\mathcal{A}}_{t+1},{\sigma }_{t+1}^2)$, such that the support of the reparameterized distribution is unbounded. This can easily be done using Jacobi transformation:

$$\begin{array}{cc}\hfill {\gamma }_{t+1}& ={\displaystyle \frac{1}{1+exp(-{\tilde{\gamma }}_{t+1})}}\mathrm{with}{\tilde{\gamma }}_{t+1}\in \mathbb{R},\hfill \\ \hfill {\rho }_{t+1}& =log(1+exp\left({\tilde{\rho }}_{t+1}\right))\mathrm{with}{\tilde{\rho }}_{t+1}\in \mathbb{R},\hfill \\ \hfill p({\tilde{\gamma }}_{t+1},{\tilde{\rho }}_{t+1}|{\mathbf{y}}_S,\mathbf{X},{\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1},{\sigma }_{t+1}^2)& \propto L({\mathit{\beta }}_{t+1},{\sigma }_{t+1}^2,{\gamma }_{t+1},{\rho }_{t+1},{\mathcal{A}}_{t+1}|{\mathbf{y}}_S)p({\rho }_{t+1},{\gamma }_{t+1})\hfill \\ \hfill & {\displaystyle \frac{exp({\tilde{\rho }}_{t+1}-{\tilde{\gamma }}_{t+1})}{(1+exp{(-{\tilde{\gamma }}_{t+1})}^2)(1+exp\left({\tilde{\rho }}_{t+1}\right))}}.\hfill \end{array}$$

(18)

Next we specify the so-called potential energy function:

$$\begin{array}{cc}\hfill E({\tilde{\rho }}_{t+1},{\tilde{\gamma }}_{t+1})& =-log\left(p({\tilde{\gamma }}_{t+1},{\tilde{\rho }}_{t+1}|{\mathbf{y}}_S,\mathbf{X},{\mathit{\beta }}_{t+1},{\mathcal{A}}_{t+1},{\sigma }_{t+1}^2)\right)\hfill \end{array}$$

Due to Equation (18), $E(·|·)$ is known up to an additive constant that vanishes if derivatives are computed. Moreover, the kinetic energy is defined as the negative logarithm of the density of a zero mean normal distribution with a diagonal covariance matrix:

$$\begin{array}{c}\hfill K(\mathit{v}={(v_{\tilde{\rho }},v_{\tilde{\gamma }})}^T)=c+{\displaystyle \frac{1}{2}}\left({\displaystyle \frac{v_{\tilde{\rho }}^2}{m_{\tilde{\rho }}}}+{\displaystyle \frac{v_{\tilde{\gamma }}^2}{m_{\tilde{\gamma }}}}\right).\end{array}$$

The parameters $m_{\tilde{\rho }}$ and $m_{\tilde{\gamma }}$ are tuning parameters of the algorithm and, further, c denotes an additive constant. Consequently, the partial derivatives of the kinetic energy function are given by

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial K\left(\mathit{v}\right)}{\partial v_{\tilde{\rho }}}}={\displaystyle \frac{v_{\tilde{\rho }}}{m_{\tilde{\rho }}}},{\displaystyle \frac{\partial K\left(\mathit{v}\right)}{\partial v_{\tilde{\gamma }}}}={\displaystyle \frac{v_{\tilde{\gamma }}}{m_{\tilde{\gamma }}}}.\end{array}$$

With the above considerations, the HMC algorithm proposes a sample from the conditional posterior by applying the following procedure:

1.

Draw a sample $\mathit{v}={(v_{\tilde{\rho }},v_{\tilde{\gamma }})}^T$ from $\mathcal{N}(\mathbf{0},diag(m_{\tilde{\rho }},m_{\tilde{\gamma }}))$.

2.

Use the leapfrog method to simulate ${\tilde{\rho }}^{\ast },{\tilde{\gamma }}^{\ast },v_{\tilde{\rho }}^{\ast },v_{\tilde{\gamma }}^{\ast }$.

(a)

Set ${\mathit{v}}^{\ast }=\mathit{v},{\tilde{\rho }}^{\ast }={\tilde{\rho }}_t,{\tilde{\gamma }}^{\ast }={\tilde{\gamma }}_t$.

(b)

For $j=1,\dots ,L:$

i.

$v_{\tilde{\rho }}^{\ast }=v_{\tilde{\rho }}^{\ast }-{\displaystyle \frac{\epsilon }{2}}{\displaystyle \frac{\partial E}{\partial {\tilde{\rho }}^{\ast }}}({\tilde{\rho }}^{\ast },{\tilde{\gamma }}^{\ast }),v_{\tilde{\gamma }}^{\ast }=v_{\tilde{\gamma }}^{\ast }-{\displaystyle \frac{\epsilon }{2}}{\displaystyle \frac{\partial E}{\partial {\tilde{\gamma }}^{\ast }}}({\tilde{\rho }}^{\ast },{\tilde{\gamma }}^{\ast })$;

ii.

${\tilde{\rho }}^{\ast }={\tilde{\rho }}^{\ast }+\epsilon {\displaystyle \frac{v_{\tilde{\rho }}^{\ast }}{m_{\tilde{\rho }}}},{\tilde{\gamma }}^{\ast }={\tilde{\gamma }}^{\ast }+\epsilon {\displaystyle \frac{v_{\tilde{\gamma }}^{\ast }}{m_{\tilde{\gamma }}}}$;

iii.

$v_{\tilde{\rho }}^{\ast }=v_{\tilde{\rho }}^{\ast }-{\displaystyle \frac{\epsilon }{2}}{\displaystyle \frac{\partial E}{\partial {\tilde{\rho }}^{\ast }}}({\tilde{\rho }}^{\ast },{\tilde{\gamma }}^{\ast }),v_{\tilde{\gamma }}^{\ast }=v_{\tilde{\gamma }}^{\ast }-{\displaystyle \frac{\epsilon }{2}}{\displaystyle \frac{\partial E}{\partial {\tilde{\gamma }}^{\ast }}}({\tilde{\rho }}^{\ast },{\tilde{\gamma }}^{\ast })$.

3.

Calculate the Hastings ratio $r=exp[K\left(\mathit{v}\right)+E({\tilde{\rho }}_t,{\tilde{\gamma }}_t)-K\left({\mathit{v}}^{\ast }\right)-E({\tilde{\rho }}^{\ast },{\tilde{\gamma }}^{\ast })]$.

4.

With probability $min(r,1)$ set ${\tilde{\rho }}_{t+1}={\tilde{\rho }}^{\ast },{\tilde{\gamma }}_{t+1}={\tilde{\gamma }}^{\ast }$. Otherwise, set ${\tilde{\rho }}_{t+1}={\tilde{\rho }}_t,{\tilde{\gamma }}_{t+1}={\tilde{\gamma }}_t$.

3.5. Prediction

Let ${\mathbf{X}}^{\ast }={({\mathbf{x}}_1^{\ast }\dots {\mathbf{x}}_v^{\ast })}^T$ be the design matrix corresponding to some predictor specifications for which predictions ${\widehat{\mathbf{y}}}^{\ast }$ are required. Note that ${\mathbf{y}}_S$ and ${\mathbf{y}}^{\ast }$ are joint normally distributed, i.e.,

$$\begin{array}{c}\hfill {({\mathbf{y}}^{\ast },{\mathbf{y}}_S)}^T|{\mathbf{X}}^{\ast },\mathbf{X},\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}\sim \mathcal{N}\left(\begin{array}{c}\left(\begin{array}{c}{\mathbf{X}}^{\ast }\mathit{\beta }\\ \mathbf{X}\mathit{\beta }\end{array}\right),\left(\begin{array}{cc}{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast }}& {\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}}\\ {\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}}& {\tilde{\mathbf{C}}}_{\mathbf{S}}\end{array}\right)\end{array}\right).\end{array}$$

(19)

This implies directly that

$$\begin{array}{c}\hfill {\mathbf{y}}^{\ast }|{\mathbf{y}}_S,{\mathbf{X}}^{\ast },\mathbf{X},\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}\sim \mathcal{N}({\mathbf{X}}^{\ast }\mathit{\beta }+{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}}{\tilde{\mathbf{C}}}_{\mathbf{S}}^{-1}({\mathbf{y}}_S-\mathbf{X}\mathit{\beta }),{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast }}-{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}}{\tilde{\mathbf{C}}}_{\mathbf{S}}^{-1}{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}})\end{array}$$

(20)

We use the expected value of the posterior predictive distribution as a prediction. Using Monte Carlo Integration, this quantity can easily be approximated as follows.

$$\begin{array}{cc}\hfill {\widehat{\mathbf{y}}}^{\ast }& =\int {\mathbf{y}}^{\ast }p({\mathbf{y}}^{\ast }|{\mathbf{y}}_S,{\mathbf{X}}^{\ast },\mathbf{X})d{\mathbf{y}}^{\ast },\hfill \\ \hfill & =\int {\mathbf{y}}^{\ast }\left\{\int \int \int \int \int p({\mathbf{y}}^{\ast }|{\mathbf{y}}_S,{\mathbf{X}}^{\ast },\mathbf{X},\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A})p(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S,\mathbf{X})\right.\hfill \\ \hfill & \left.d{\sigma }^{2}d\gamma d\rho \#\left(d\mathcal{A}\right){(\lambda +{\delta }_0)}^d\left(d\mathit{\beta }\right)\right\}d{\mathbf{y}}^{\ast },\hfill \\ \hfill & =\int \int \int \int \int \left\{\int {\mathbf{y}}^{\ast }p({\mathbf{y}}^{\ast }|{\mathbf{y}}_S,{\mathbf{X}}^{\ast },\mathbf{X},\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A})d{\mathbf{y}}^{\ast }\right\}p(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S,\mathbf{X})\hfill \\ \hfill & d{\sigma }^{2}d\gamma d\rho \#\left(d\mathcal{A}\right){(\lambda +{\delta }_0)}^d\left(d\mathit{\beta }\right),\hfill \\ \hfill & =\int \int \int \int \int \left\{{\mathbf{X}}^{\ast }\mathit{\beta }+{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}}{\tilde{\mathbf{C}}}_{\mathbf{S}}^{-1}({\mathbf{y}}_S-\mathbf{X}\mathit{\beta })\right\}p(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S,\mathbf{X})\hfill \\ \hfill & d{\sigma }^{2}d\gamma d\rho \#\left(d\mathcal{A}\right){(\lambda +{\delta }_0)}^d\left(d\mathit{\beta }\right),\hfill \\ \hfill & \approx {\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\mathbf{X}}^{\ast }{\mathit{\beta }}^{\left(i\right)}+{\tilde{\mathbf{C}}}_{{\mathbf{X}}^{\ast },\mathbf{S}}^{\left(i\right)}{{\tilde{\mathbf{C}}}_{\mathbf{S}}^{\left(i\right)}}^{-1}({\mathbf{y}}_S-\mathbf{X}{\mathit{\beta }}^{\left(i\right)}).\hfill \end{array}$$

(21)

where # denotes the counting measure. In (21), notation $\left(i\right)$ shall point out that each of the individual summands is computed for an $iid$ sample from the posterior $p(\mathit{\beta },{\sigma }^2,\gamma ,\rho ,\mathcal{A}|{\mathbf{y}}_S,\mathbf{X})$.

4. Evaluation

In this section, we evaluate and compare the predictive performance of our approach (VRNNGP) against several established feature selection methods. The performance comparison is based on two accuracy measures: the mean squared error (MSE) and the mean absolute deviation (MAD). These metrics are computed using test data sets that are distinct from the training data, ensuring that no observations are shared between the two sets. The methods considered in the comparison are as follows:

• Lasso. The least absolute shrinkage and selection operator (Lasso) [8] penalizes the absolute values of the regression coefficients in a classical linear model to obtain a sparse solution. The method is implemented using the glmnet R package (v4.1-10) [48]. Cross-validation is employed to determine the penalization strength.
• Adaptive Lasso (ALasso). The adaptive Lasso [11] extends the classical Lasso by allowing coefficient-specific penalization strengths. In this study, the individual penalization factors are defined by multiplying a global tuning parameter by the inverse of the absolute values of the regression coefficients obtained from a ridge-penalized model. The glmnet package is again used for implementation.
• Random Forest (RF). Random forests are ensembles of tree-based predictors introduced by Breiman [49]. We use the implementation provided by the textttrandomForest R package (v4.7-1.2) [50].
• Robust Gaussian Process (RGP). The RobustGaSP R package (v0.6.8) [51] implements robust Gaussian process regression models using reference priors and separable (anisotropic) covariance functions. We employ the default settings (e.g., a Matérn $5/2$ kernel), except that we force the method to estimate a nugget term. Note that GP models with anisotropic kernels allow for automatic relevance determination [7]; therefore, predictors are scaled prior to model inference.
• Bayesian Adaptive Sampling (BAS). The BAS R package (v2.0.2) implements Bayesian variable selection for linear models using Zellner’s g-prior or mixtures of g-priors, including the Zellner–Siow Cauchy prior and the mixture of g-priors proposed by Liang et al. [52].
• Variational Inference for Bayesian Variable Selection (VARBVS). The varbvs R package (v2.6-10) applies variational approximations originally developed for genetic association studies. Carbonetto and Stephens [53] demonstrate that this approach can yield posterior inferences that closely match exact values in certain settings.
• Bayesian Additive Regression Trees (BART). The BART R package (v2.9.10) [54] implements a Bayesian nonparametric, machine learning, ensemble predictive modeling method. It is a tree-based method that fits the outcome to an arbitrary random function of the covariates.
• Bayesian Additive Regression Trees Approach using Gaussian Processes (GP-BART). Standard BART models may perform poorly in settings where smoothness assumptions or explicit covariance structures are required. To address this limitation, Maia et al. [55] propose an extension of BART that assumes Gaussian process priors for the predictions at the terminal nodes of each tree.

We consider four benchmark data sets. The first three follow common practice in benchmarking feature selection methods: the Pepelyshev function (Section 4.1), the Sine function (Section 4.2), and the body fat data set (Section 4.3). In addition, we analyze a data set obtained from a semiconductor manufacturing process (Section 4.4).

For all data sets, we assume no a priori preference among predictors, setting the prior importance weights to $p_1=\dots =p_d=1/d$. For the data sets considered in Section 4.1, Section 4.2 and Section 4.3, the mapping $\tilde{p}(·)$, which encodes a priori beliefs about the number of predictors contributing to the model, is defined as $\tilde{p}\left(k\right)=1/k,\forall k\in \{1,\dots ,d\}$. This specification intuitively favors smaller models and reflects the absence of prior knowledge regarding predictor relevance. We observe that the MCMC chains converge without requiring informative specifications of the parameters ${\tilde{p}}_i$ after a reasonable number of iterations (on the order of a few thousand).

In contrast, for the semiconductor manufacturing data set (Section 4.4), $\tilde{p}(·)$ is chosen to be proportional to the third power of the probability mass function of a zero-truncated binomial distribution with size parameter d and success probability $1/d$, following the recommendation of Posch et al. [15]. This choice is motivated by the substantially larger number of potential predictors in this application.

Regarding model inference, the tuning parameter $p_h$ is set to $0.6$, implying that proposals for the random set $\mathcal{A}$ slightly favor changes over remaining unchanged. For simplicity, most tuning parameters in the HMC step are fixed at $m_{\tilde{\rho }}=m_{\tilde{\gamma }}=1$ and $L=2$. The only exception is the step size S of the leapfrog method, which is the most influential tuning parameter. Values between $0.15$ and $0.5$ are considered and selected via trial and error to ensure good mixing of the MCMC chains.

For all baseline methods, we used the default hyperparameter settings provided by the respective software packages, relying on their built-in tuning mechanisms in order to avoid method-specific bias and ensure a consistent comparison across approaches.

While quantitative feature selection accuracy can be assessed in synthetic experiments where the true active and inactive variables are known, such ground-truth information is generally unavailable for the real-world data sets considered in this study. To ensure a consistent evaluation across all experiments, we therefore focus on predictive performance. Prediction performance is assessed using MSE and MAD, reported as mean ± standard deviation over five independent runs ($k=5$).

4.1. Pepelyshev Function

We first evaluate the considered methods using the Pepelyshev function [56], which is commonly employed as a benchmark in computer experiments. In this context, Gaussian processes are frequently used to model the output of complex systems [39]. The highly curved Pepelyshev function is defined as

$$\begin{array}{c}\hfill f\left(\mathbf{x}\right)=4{(x_1-2+8x_2-8x_2^2)}^2+{(3-4x_2)}^2+16\sqrt{x_3+1}{(2x_3-1)}^2,\end{array}$$

(22)

for $x_1,x_2,x_3\in [0,1]$. By extending (22) with 17 inert variables $x_4,\dots ,x_{20}\in [0,1]$, we obtain a feature selection problem.

The evaluation is based on a training data set of size 31 and a testing data set of size 100. Both data sets are generated using maximin Latin hypercube designs, which are standard experimental designs in computer experiments [23]. These designs are constructed using the lhs R package (v1.2.0) [57]. Prior to analysis, both the training and testing data sets are scaled. The results are reported in

Table 1. Accuracy measures for all considered approaches applied to the Pepelyshev function with inert variables.

Method	MSE	MAD
VRNNGP ($m=5$)	0.3152 ± 0.2015	0.4163 ± 0.1498
VRNNGP ($m=10$)	0.1943 ± 0.0022	0.3152 ± 0.0006
VRNNGP ($m=20$)	0.1449 ± 0.0023	0.2775 ± 0.0019
RGP	1.4835 ± 0.0000	1.0195 ± 0.0000
RF	0.7656 ± 0.0217	0.7339 ± 0.0113
Lasso	1.1412 ± 0.1441	0.9066 ± 0.0494
ALasso	1.4273 ± 0.2009	0.9984 ± 0.0692
BART	1.4276 ± 0.0031	0.9812 ± 0.0011
GPBART	1.6171 ± 0.0031	1.0229 ± 0.0011
BAS	0.9985 ± 0.0000	0.8677 ± 0.0000
VARBVS	1.0196 ± 0.0000	0.8741 ± 0.0000

.

VRNNGP clearly outperforms all other considered methods. This superior performance arises because the competing methods fail to identify the active variables $x_1,x_2$, and $x_3$, instead incorporating inert variables into the model. However, it should be mentioned that VRNNGP cannot properly identify $x_1,x_2,x_3$. Within the VRNNGP model, variables $x_2$ and $x_3$ have posterior inclusion probabilities equal to 1, whereas all remaining predictors have probabilities close to zero.

In addition, we demonstrate that the proposed Metropolis-within-Gibbs algorithm (Section 3.4) exhibits fast convergence. Figure 1, Figure 2, Figure 3 and Figure 4 display the Markov chains of several representative model parameters. The displayed chains indicate that convergence is achieved after approximately 1000 iterations.

4.2. Sine Function

In a simulation study adapted from Savitsky et al. [58], the observed continuous response y is generated from a nonlinear relationship involving two variables ($x_1$ and $x_2$). The model includes 20 normally distributed variables; consequently, $x_3,x_4,\dots ,x_{20}$ are inert. The full model from which the data is generated reads as follows:

$$\begin{array}{cc}\hfill y& =sin\left(x_1\right)+sin\left(5x_2\right)+\epsilon ,\hfill \\ \hfill \epsilon & \sim \mathcal{N}(0,0.0025),\hfill \\ \hfill {(x_1,\dots ,x_{20})}^T& \sim \mathcal{N}(\mathbf{0},\mathbf{\Sigma }),\hfill \\ \hfill diag(\mathbf{\Sigma })& =\mathbf{1},\hfill \\ \hfill {\Sigma }_{ij}& =\left\{\begin{array}{cc}0.4,\hfill & \mathrm{i}\mathrm{f}(i,j)\in \left\{\right(1,13),(13,1\left)\right\},\hfill \\ 0.3,\hfill & \mathrm{i}\mathrm{f}(i,j)\in \left\{\right(1,12),(12,1\left)\right\},\hfill \\ 0.4,\hfill & \mathrm{i}\mathrm{f}(i,j)\in \left\{\right(2,14),(14,2\left)\right\},\hfill \\ 0.3,\hfill & \mathrm{i}\mathrm{f}(i,j)\in \left\{\right(2,15),(15,2\left)\right\},\hfill \\ 0,\hfill & \mathrm{else}.\hfill \end{array}\right.\hfill \end{array}$$

More specifically, 100 samples are simulated from the model to generate a data set, which is subsequently centered at zero and scaled. Performance evaluation is conducted using 5-fold cross-validation. The results are reported in

Table 2. Accuracy measures for all considered approaches applied to the Sine function simulation model.

Method	MSE	MAD
VRNNGP ($m=5$)	0.2577 ± 0.0796	0.3371 ± 0.0554
VRNNGP ($m=10$)	0.2195 ± 0.0509	0.3371 ± 0.0396
RGP	0.6405 ± 0.1343	0.6690 ± 0.0913
RF	0.6891 ± 0.1976	0.6559 ± 0.1094
Lasso	0.7911 ± 0.1992	0.7262 ± 0.1226
ALasso	0.7867 ± 0.1898	0.7258 ± 0.1154
BART	1.5037 ± 0.0146	0.9699 ± 0.0037
GP-BART	1.4946 ± 0.0598	0.9782 ± 0.0213
BAS	0.8010 ± 0.0232	0.7399 ± 0.0131
VARBVS	0.7981 ± 0.0210	0.7386 ± 0.0122

. We observe that VRNNGP performs significantly better than the competing methods. Specifically, this approach correctly identifies the active variables $x_1$ and $x_2$ in four out of the five train/test splits, whereas the other methods fail to do so.

4.3. Body Fat Data Set

The body fat data set [59] consists of 128 observations on 14 variables and is available in the mplot R package (v1.0.6) [60]. The target variable is the body fat percentage of male subjects, and the predictors include age, weight, height, and ten body circumference measurements. A 5-fold cross-validation is used to evaluate and compare the predictive performance of VRNNGP with that of the competing methods. Prior to analysis, the entire data set is centered at zero and scaled. The results are reported in

Table 3. Accuracy measures for all considered approaches applied to the body fat data set.

Method	MSE	MAD
VRNNGP ($m=10$)	0.2849 ± 0.0359	0.4291 ± 0.0364
RGP	0.3408 ± 0.0180	0.4706 ± 0.0125
RF	0.3235 ± 0.0056	0.4671 ± 0.0024
Lasso	0.2933 ± 0.0143	0.4349 ± 0.0108
ALasso	0.2952 ± 0.0083	0.4406 ± 0.0069
BART	1.7193 ± 0.0290	1.0621 ± 0.0110
GP-BART	0.6139 ± 0.0092	0.6282 ± 0.0069
BAS	0.2889 ± 0.0056	0.4291 ± 0.0077
VARBVS	0.3081 ± 0.0053	0.4449 ± 0.0072

.

We observe that VRNNGP achieves the best predictive performance, while the BAS approach yields nearly identical results. Figure 5 and Figure 6 visualize the importance of individual predictors (compare to Section 4.4) obtained from VRNNGP and RF, respectively. Although all three methods agree that the sixth predictor (abdomen circumference) is the most influential, they differ in their rankings of the remaining predictors.

4.4. Semiconductor Frontend Manufacturing Data

Semiconductor production involves a multitude of intricate steps, beginning with the frontend phase. Semiconductor manufacturing is renowned for its high-tech setups, characterized by complex processes, extensive automation, and a high degree of digitalization [61,62]. Following the frontend phase, wafers are transferred to an interim storage facility before entering the second and final production phase, known as the backend. In this stage, wafers are divided into individual chips, mounted onto lead frames, and ultimately packaged and sealed.

The frontend manufacturing process transforms semiconductor wafers through several hundred production steps. During each operation, equipment-integrated sensors record a variety of physical parameters, including gas flow, voltage, and temperature. From these sensor traces, a set of summary metrics—typically statistical descriptors such as means, standard deviations, and extrema—is computed. These derived metrics constitute the features used to assess and monitor process stability. Once frontend production is completed, each wafer undergoes extensive testing, resulting in wafer test data. These tests, which measure quantities such as current, voltage, or resistance, ensure that each device meets its specifications. If certain parameters from the wafer test data indicate potential quality issues, possibly corresponding to high failure rates, the underlying causes must be traced back to the frontend production stage. Feature selection models are therefore essential for identifying production parameters that significantly influence final product quality.

We apply our approach to a data set collected during semiconductor frontend production [63], comprising 228 observations derived from 77 potential predictors (key numbers). These predictors are extracted from data recorded during two selected process steps, which have been identified as the most critical for the final outcome. The target variable represents an analog measurement from the wafer test data and reflects wafer quality, with higher values indicating poorer quality. Prior to analysis, all predictors were standardized and the target variable was centered at zero. The data set was then randomly divided into two equally sized subsets: a training set and a testing set, ensuring an unbiased evaluation of model performance.

Table 4. Accuracy measures for all considered approaches applied to the semiconductor frontend manufacturing data.

Method	MSE	MAD
VRNNGP ($m=10$)	0.2389	0.4198
RGP	0.3108	0.4684
RF	0.2672	0.4423
Lasso	0.2845	0.4575
ALasso	0.2898	0.4528
BART	0.4493	0.5712
GP-BART	0.4374	0.5451
BAS	0.2988	0.4622
VARBVS	0.3029	0.4602

reports the predictive accuracy of the methods considered in this study. VRNNGP achieves the best overall performance, followed by the RF approach. The RGP model performs slightly worse than the linear methods in this application. Figure 7 and Figure 8 visualize the importance of individual predictors as determined by VRNNGP and RF. For VRNNGP, predictor importance is quantified by the relative frequency with which a predictor index appears in the posterior samples of the random set $\mathcal{A}$. This quantity can be interpreted as the posterior inclusion probability of a predictor. For RF, variable importance is measured by the mean decrease in accuracy resulting from the removal of a given predictor, as computed using the importance() function from the randomForest R package. The importance rankings differ across methods; however, the two most influential predictors identified by VRNNGP (variables 34 and 41) are also deemed important by RF. We hypothesize that the influence of variable 34 is nonlinear, which may explain why linear models fail to detect its importance. Since variables 34 and 41 consistently appear influential across methods, we refit the RGP and RF models using a reduced data set containing only these two predictors. The resulting predictive accuracies are reported in

Table 5. Accuracy measures for RGP and RF applied to the reduced semiconductor frontend manufacturing data using only variables 34 and 41.

Method	MSE	MAD
RGP	0.2053222	0.3692166
RF	0.2022327	0.3539577

. Both models perform similarly on the reduced data set and show substantial improvements compared to their performance on the full data set. Notably, restricting the analysis to variables 34 and 41 yields the best overall results. Figure 9 and Figure 10 illustrating the fitted response surfaces show the functions learned by RGP and RF based on the reduced data set. Both models exhibit similar behavior in regions associated with poor production quality. Finally, Figure 11 displays the true production quality of the test data ($y_{\mathrm{Test}}$) versus the corresponding predictions obtained from the RGP model fitted to the reduced data set. All samples exhibiting particularly poor quality ($y_{\mathrm{Test}}>0.5$) are correctly identified, demonstrating that the features selected by VRNNGP are highly informative for this application.

5. Conclusions

In this study, we introduced a novel Bayesian approach to the feature selection problem. Our method combines nearest neighbor Gaussian processes, reference priors, and a model selection strategy based on a random set. To the best of our knowledge, this specific combination has not been investigated in prior research.

The use of nearest neighbor Gaussian processes provides a flexible approximation to classical Gaussian processes, while the adoption of reference priors contributes to stable and robust model inference. The random set formulation enables the selection of the most predictive variables while incorporating relevant a priori knowledge. For model inference, we proposed a Metropolis-within-Gibbs algorithm that allows transitions between different sets of active predictors, that is, predictors that contribute to the regression model, and employs Hamiltonian dynamics for efficient inference of the Gaussian process correlation parameters.

The predictive performance of the proposed method was evaluated in a comprehensive computational study. Our results show that the approach successfully identifies truly relevant features in a variety of settings, including simulated data and an approximation problem arising from computer experiments. In contrast, linear models, anisotropic Gaussian processes (automatic relevance determination), and random forest models performed less effectively in these feature selection tasks. Finally, results obtained for a feature selection problem in semiconductor manufacturing highlight the potential of the proposed method for applications in high-tech manufacturing. A systematic comparison of training and inference times with RGP and other baselines is left for future work, as such an analysis requires carefully controlled implementations and computing environments.

An important direction for future work could be a systematic comparison between the proposed nearest neighbor Gaussian processes-based approach and exact Gaussian process models, particularly on smaller data sets where both methods are computationally feasible. Such a study would help quantify potential trade-offs between computational efficiency and predictive accuracy.