Sparse Robust Weighted Expectile Screening for Ultra-High-Dimensional Data

Abstract

This paper investigates robust feature screening for ultra-high dimensional data in the presence of outliers and heterogeneity. Considering the susceptibility of likelihood methods to outliers, we propose a Sparse Robust Weighted Expectile Regression (SRoWER) method that combines the $L_{2}E$ criterion with expectile regression. By utilizing the IHT algorithm, our method effectively incorporates correlations of covariates and enables joint feature screening. The proposed approach demonstrates robustness against heavy-tailed errors and outliers in data. Simulation studies and a real data analysis are provided to demonstrate the superior performance of the SRoWER method when dealing with outlier-contaminated explanatory variables and/or heavy-tailed error distributions.

1. Introduction

With the exponential growth of data sets in various fields over the past two decades, numerous exhaustive methods have been proposed to address the issue of coefficient sparsity in high-dimensional statistical models, such as bridge regression [1], LASSO [2], SCAD and other folded-concave penalties [3], and the Dantzig selector [4]. While these methods have demonstrated their effectiveness both theoretically and practically, real-world scenarios present new challenges such as identifying disease-causing genes among millions of other genes or analyzing key factors contributing to stock price fluctuations from vast amounts of business data. To tackle ultra-high dimensional data, a range of techniques has emerged. One notable technique is Sure Independent Screening (SIS), initially developed by Fan and Lv [5] for screening out irrelevant factors before conducting variable selection in ultra-high dimensional linear models. There are numerous further developments based on SIS [6,7,8,9]. However, these methods overlook the correlations of covariates, despite their computational efficiency. Consequently, additional procedures have been proposed to address this limitation, including ISIS [6], FR [10], SMLE [11].

The aforementioned approaches, which are all based on the maximum likelihood function or Pearson’s correlation, become invalid in the presence of outliers. Therefore, robust methods have been extensively studied in the literature. Although quantile regression [12] is effective in handling heterogeneous data, the significantly higher computational cost compared to least squares error necessitates an investigation of the asymmetric least squares (ALS) regression (i.e., expectile regression [13,14,15,16]). ALS regression provides a more comprehensive interpretation of the conditional distribution than the ordinary least squares (OLS) method by allocating different squared error losses to positive and negative residuals, respectively. Moreover, its smooth differentiability greatly reduces computational costs and facilitates theoretical research. Building upon ALS and quantile regression, numerous methods have been proposed to address heterogeneous data with high-dimensionality, such as [17,18] for variable selection and [19,20,21,22,23,24] for feature screening. The study of [25] proposed an expectile partial correlation screening (EPCS) procedure to sequentially identify important variables for expectile regressions in ultra-high dimensions, and proved that this procedure can lead to a sure screening set. Another robust parametric technique called DPD-SIS [26,27] has been developed for ultra-high-dimensional linear regression models and generalized linear models. This approach is based on the robust minimum density power divergence estimator [28], but it is still limited to addressing marginal aspects without accounting for the correlations between features. In addition, the DPD-SIS cannot handle heterogeneity, which is often a feature of ultra-high-dimensional data.

In the context of heterogeneity and outliers in the data, we propose a new method called Robust Weighted Expectile Regression (RoWER), which combines the $L_2$ error criterion with expectile regression to achieve the robustness and address the heterogeneity. Furthermore, we developed a sparse restricted RoWER (SRoWER) approach to achieve feature screening. Under general assumptions, we show that the SRoWER can be used for sure screening property. Numerical studies validate the robustness and efficacy of SRoWER. There are three advantages of our SRoWER method, including the following: (1) The SRoWER can provide more reliable screening results, particularly in the presence of outliers in both covariates and the response; (2) In the case of heteroscedasticity, the SRoWER yields superior performance in estimation and feature screening as demonstrated in simulation studies; (3) The SRoWER can be efficiently solved by an iterative hard-thresholding-based algorithm.

The remaining sections of this article are organized as follows. Section 2 introduces the model and the RoWER method. In Section 3, we present the SRoWER method for feature screening and establish sure independent screening property. Section 4 describes simulation studies and a real data analysis that evaluates the finite sample performance of the SRoWER method. Concluding remarks are provided in Section 5. The proofs of the main results can be found in the Appendix A.

2. Model and Method

2.1. $L_{2}E$ Criterion for Asymmetric Normal Distribution

To address the problem that likelihood methods are sensitive to outliers, Scott [29] proposed a $L_{2}E$ method, for which the objective function is

$${\int }_{-\infty }^{+\infty }f{(v|\theta )}^{2}dv-{\displaystyle \frac{2}{n}}\sum _{i=1}^{n}f(v_i|\theta ),$$

where $f(v|\theta )$ is a parametric probability density function (pdf) of a random variable V and $v_i,i=1,\dots ,n$ is a given random sample.

Here, we assume that V follows the asymmetric normal distribution, i.e., $V\sim AN(v;{\mu }_{\tau },{\sigma }^2,\tau )$. The corresponding pdf is

$$f(v;{\mu }_{\tau },{\sigma }^2,\tau )={\displaystyle \frac{2}{\sqrt{\pi {\sigma }^2}}}{\displaystyle \frac{\sqrt{\tau (1-\tau )}}{\sqrt{\tau }+\sqrt{1-\tau }}}\exp \left\{-{\rho }_{\tau }\left({\displaystyle \frac{v-{\mu }_{\tau }}{\sigma }}\right)\right\},$$

where ${\rho }_{\tau }\left(t\right)={\omega }_{\tau }\left(t\right)t^2$ is asymmetric squared error loss [13], ${\omega }_{\tau }\left(t\right)=|\tau -I(t\le 0)|$ with $I(·)$ being the indicator function. Moreover, ${\mu }_{\tau }$, $\sigma$ and $\tau$ are the location, scale and asymmetric parameters, respectively. The following proposition gives the $L_{2}E$ criterion of asymmetric normal distribution.

Proposition 1. 

Suppose $V\sim AN(v;{\mu }_{\tau },{\sigma }^2,\tau )$, then the $L_{2}E$ criterion is

$${\displaystyle \frac{1}{\sqrt{\pi {\sigma }^2}}}{\displaystyle \frac{\sqrt{2\tau (1-\tau )}}{\sqrt{\tau }+\sqrt{1-\tau }}}\left\{1-2\sqrt{2}{\displaystyle \frac{1}{n}}\sum _{i=1}^n\exp \left[-{\rho }_{\tau }\left({\displaystyle \frac{v_i-{\mu }_{\tau }}{\sigma }}\right)\right]\right\}.$$

(1)

2.2. RoWER

We consider $\tau$-mean [13] of the random variable $Z\in \mathbb{R}$,

$${\mathcal{E}}^{\tau }\left(Z\right)=arg\underset{a\in \mathbb{R}}{min}\mathbb{E}\left\{{\rho }_{\tau }(Z-a)\right\},\tau \in (0,1).$$

In fact, the $\tau$-mean corresponds to Efron’s w-mean [30], where $w=\tau /(1-\tau )$. In economics, the $\tau$-mean is also called the $\tau$-expectile. Let $\mathit{y}={(y_1,\dots ,y_n)}^T$ be the n-dimensional response vector, $\mathit{X}={({\mathit{x}}_1,\dots ,{\mathit{x}}_n)}^T$ be the $n\times p$ design matrix with ${\mathit{x}}_i={(x_{i1},\dots ,x_{ip})}^T,i=1,\dots ,n$. The ALS regression is carried out using the following

$$arg\underset{\mathsf{\beta }\in {\mathbb{R}}^p}{min}\sum _{i=1}^n{\rho }_{\tau }(y_i-{\mathit{x}}_i^T\mathsf{\beta }),$$

which degenerates to the OLS regression when $\tau =0.5$.

Consider the following linear model

$$\mathit{y}=\mathit{X}{\mathsf{\beta }}^{\tau }+{\mathit{\epsilon }}^{\tau },$$

(2)

where ${\mathsf{\beta }}^{\tau }={({\beta }_1\left(\tau \right),\dots ,{\beta }_p\left(\tau \right))}^T$ is a p-dimensional parameter vector and ${\mathit{\epsilon }}^{\tau }={({\epsilon }_1^{\tau },\dots ,{\epsilon }_n^{\tau })}^T$ is the vector of n independent errors satisfying ${\mathcal{E}}^{\tau }({\epsilon }_i^{\tau }|{\mathit{x}}_i)=0,i=1,\dots ,n$ for some $\tau \in (0,1)$. We adopt the sparsity assumption on ${\mathsf{\beta }}^{\tau }$, that is, the regression coefficient vector ${\mathsf{\beta }}^{\tau }$ has many zero components. In model (2), it is crucial to understand that varying $\tau$ allows for variations in the coefficient vector ${\mathsf{\beta }}^{\tau }$, so we can model different locations of the conditional distribution. For convenience, the superscript $\tau$ of ${\mathsf{\beta }}^{\tau }$ and ${\mathsf{\epsilon }}^{\tau }$ is omitted in the following so that no confusion arises.

By substituting ${\mu }_{\tau }={\mathit{x}}_i^T\mathsf{\beta }$ and $\sigma =1$ into (1), we can obtain the following loss function by disregarding the terms that are independent of $\mathsf{\beta }$,

$$L_n\left(\mathsf{\beta }\right)=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n\exp \{-{\rho }_{\tau }(y_i-{\mathit{x}}_i^T\mathsf{\beta })\}.$$

(3)

However, (3) may not be strictly convex, so we propose a new loss function in the following Proposition 2 by Taylor’s expansion and logarithmic transformation.

Proposition 2. 

Given a consistent estimator $\tilde{\mathsf{\beta }}$ of β, minimizing (3) is transformed into minimizing the following loss

$$D_n\left(\mathsf{\beta }\right)=\sum _{i=1}^n{\pi }_i\left(\tilde{\mathsf{\beta }}\right){\rho }_{\tau }(y_i-{\mathit{x}}_i^T\mathsf{\beta }),$$

(4)

where

$${\pi }_i\left(\tilde{\mathsf{\beta }}\right)={\displaystyle \frac{\exp \{-{\rho }_{\tau }(y_i-{\mathit{x}}_i^T\tilde{\mathsf{\beta }})\}}{{\sum }_{l=1}^n\exp \{-{\rho }_{\tau }(y_l-{\mathit{x}}_l^T\tilde{\mathsf{\beta }})\}}},i=1,\dots ,n,$$

which is abbreviated as ${\pi }_i$.

Here ${\pi }_i$’s can be treated as the weights of the asymmetric least squares loss, and the loss (4) is referred to as the RoWER. When $\tau =0.5$, the RoWER degenerates to the weighted least squares regression. This paper chooses the consistent estimator $\tilde{\mathsf{\beta }}$ as ${\mathsf{\beta }}_L$ based on Lemma A5. We assume that ${\pi }_i$’s are lower bounded.

3. The SRoWER and Sure Screening Property

Let $\mathcal{M}$ be any subset of $\{1,\dots ,p\}$, which corresponds to a submodel with the relevant regression coefficient vector ${\mathsf{\beta }}_{\mathcal{M}}={({\beta }_j,j\in \mathcal{M})}^T$ and the design matrix ${\mathit{x}}_{\mathcal{M}}={({\mathit{x}}_{1\mathcal{M}},\dots ,{\mathit{x}}_{n\mathcal{M}})}^T$, ${\mathit{x}}_{i\mathcal{M}}={(x_{ij},j\in \mathcal{M})}^T$. In addition, let ${\parallel ·\parallel }_2$ be the $L_2$-norm, and ${\parallel ·\parallel }_0$ be the $L_0$-norm, which denotes the number of non-zero components of a vector. The size of model $\mathcal{M}$ is denoted as $s\left(\mathcal{M}\right)$. The true model is represented by ${\mathcal{M}}^{*}=\{j:{\beta }_j^{*}\ne 0\}$, with ${\mathsf{\beta }}^{*}$ being the true regression coefficient vector, and $s\left({\mathcal{M}}^{*}\right)={\parallel {\mathsf{\beta }}^{*}\parallel }_0=k_0$.

3.1. The IHT Algorithm

For the objective function $D_n\left(\mathsf{\beta }\right)$, assuming that $\mathsf{\beta }$ is sparse with $s\left({\mathcal{M}}^{*}\right)=k_0⩽k$ for some known k, the RoWER method with sparsity restriction (SRoWER) yields an estimator of $\mathsf{\beta }$ defined as

$${\widehat{\mathsf{\beta }}}_{\left[k\right]}=arg\underset{\mathsf{\beta }}{min}{D}_n\left(\mathsf{\beta }\right),{\parallel \mathsf{\beta }\parallel }_0⩽k,$$

(5)

and $\widehat{\mathcal{M}}=\{j:{\widehat{\beta }}_{\left[k\right]j}\ne 0\}$ stands for the set of subscripts of the non-zero components of ${\widehat{\beta }}_{\left[k\right]}$.

For feature screening, the goal is to retain a relatively small number of features from p features. Currently, many studies have proposed methods to solve such problems. For example, Mallat and Zhang [31] proposed the matching pursuit algorithm. Moreover, the hard thresholding method proposed by Blumensath and Davies [32] is particularly effective for linear models. We now follow the idea of an iterative hard thresholding (IHT) algorithm to compute the SRoWER estimate. For $\gamma$ within the neighborhood of a given $\mathsf{\beta }$, the IHT uses the approximation of $D_n(·)$,

$$Q_n(\gamma ;\mathsf{\beta })=D_n\left(\mathsf{\beta }\right)+{(\gamma -\mathsf{\beta })}^T{\mathit{T}}_n\left(\mathsf{\beta }\right)+(u/2){\parallel \gamma -\mathsf{\beta }\parallel }_2^2,$$

(6)

where

$${\mathit{T}}_n\left(\mathsf{\beta }\right)={\displaystyle \frac{\partial D_n\left(\mathsf{\beta }\right)}{\partial \mathsf{\beta }}}=-\sum _{i=1}^{n}2{\pi }_i{\omega }_{\tau ,i}\left(\mathsf{\beta }\right)(y_i-{\mathit{x}}_i^T\mathsf{\beta }){\mathit{x}}_i,$$

${\omega }_{\tau ,i}\left(\mathsf{\beta }\right)=|\tau -I(y_i\le {\mathit{x}}_i^T\mathsf{\beta })|$, and $u>0$ is a scale parameter. Denote ${\mathit{T}}_n\left(\mathsf{\beta }\right)={(T_{nj}\left(\mathsf{\beta }\right),j=1,\dots ,p)}^T$.

By (6), the approximate solution of (5) can be obtained by the following iterative procedure

$${\mathsf{\beta }}^{(t+1)}=arg\underset{\gamma }{min}{Q}_n(\gamma ;{\mathsf{\beta }}^{\left(t\right)}),{\parallel \gamma \parallel }_0\le k.$$

(7)

The optimization of (7) is equivalent to

$${\mathsf{\beta }}^{(t+1)}=arg\underset{\gamma }{min}\parallel \gamma -u^{-1}\{u{\mathsf{\beta }}^{\left(t\right)}-{\mathit{T}}_n\left({\mathsf{\beta }}^{\left(t\right)}\right)\}{\parallel }_2^2,{\parallel \gamma \parallel }_0\le k.$$

(8)

If there is no constraint ${\parallel \gamma \parallel }_0\le k$, the analytic solution of (8) is $\tilde{\gamma }={\mathsf{\beta }}^{\left(t\right)}-u^{-1}{\mathit{T}}_n\left({\mathsf{\beta }}^{\left(t\right)}\right)$. However, due to the sparsity restriction, ${\mathsf{\beta }}^{(t+1)}$ can be obtained by selecting the component of $\tilde{\gamma }$ with the largest absolute value before k, i.e.,

$$\mathit{V}(\tilde{\gamma };k)={(V({\tilde{\gamma }}_1;r),\dots ,V({\tilde{\gamma }}_p;r))}^T,$$

where r is the k-th largest component of $(|\tilde{\gamma }{|}_1,\dots ,|\tilde{\gamma }{{|}_p)}^T$, and $V(\gamma ;r)=\gamma I\left(|\gamma |\ge r\right)$ is a hard thresholding function. Given the sparse solution ${\mathsf{\beta }}^{\left(t\right)}$ obtained at the t-th iteration, iterating (8) is equivalent to iterating the following expression

$${\mathsf{\beta }}^{(t+1)}=\mathit{V}({\mathsf{\beta }}^{\left(t\right)}-u^{-1}{\mathit{T}}_n\left({\mathsf{\beta }}^{\left(t\right)}\right);k).$$

The ultra-high dimensional case is often faced with a huge amount of computational tasks including matrix operations. However, the use of thresholding functions can eliminate this issue. Moreover, it naturally incorporates information on the correlations between predictors. Theorem 1 shows that the value of $D_n(·)$ decreases as the number of iterations increases.

Theorem 1. 

Let $\left\{{\mathsf{\beta }}^{\left(t\right)}\right\}$ be the sequence obtained by(7), ${\lambda }_{\max }$ be the maximum eigenvalue of ${\mathit{X}}^T\mathit{X}$. If $u/2-\overline{c}{\lambda }_{\max }\ge 0$ with $\overline{c}=\tau \vee (1-\tau )$, the value of $D_n(·)$ decreases as the number of iterations increases, i.e., $D_n\left({\mathsf{\beta }}^{(t+1)}\right)\le D_n\left({\mathsf{\beta }}^{\left(t\right)}\right)$.

3.2. Sure Screening Property

This subsection will prove the sure screening property of feature screening based on the SRoWER method. Define

$${\mathcal{M}}_{+}^k=\{\mathcal{M}:{\mathcal{M}}^{*}\subset {\mathcal{M};\parallel \mathcal{M}\parallel }_0\le k\},$$

$${\mathcal{M}}_{-}^k=\{\mathcal{M}:{\mathcal{M}}^{*}\not\subset {\mathcal{M};\parallel \mathcal{M}\parallel }_0\le k\}$$

as the collections of the over-fitted models and the under-fitted models, respectively. When p, $k_0$, k and ${\mathsf{\beta }}^{*}$ vary along with the sample size n, we provide the asymptotic property of ${\widehat{\mathsf{\beta }}}_{\left[k\right]}$. Additionally, we make the following assumptions, some of which are completely technical and only help us comprehend the SRoWER method theoretically.

(A1)

$\log p=O\left(n^{\alpha }\right)$ for some $0\le \alpha <1$.

(A2)

There exist $w_1,w_2>0$ and some non-negative constants ${\eta }_1,{\eta }_2$, such that

$${\min }_{j\in {\mathcal{M}}^{*}}|{\beta }_j^{*}|\ge w_1{n}^{-{\eta }_1}$$

and

$$k_0\le k\le w_2{n}^{{\eta }_2}.$$

(A3)

There exists a constant ${\eta }_3>0$, such that ${max}_{1\le i\le n}{max}_{1\le j\le p}|x_{ij}|\le {\eta }_3$.

(A4)

Suppose that the random errors ${\epsilon }_i$ are i.i.d. sub-Gaussian random variables satisfying ${\mathcal{E}}^{\tau }({\epsilon }_i|{\mathit{x}}_i)=0,i=1,\dots ,n$.

(A5)

Let $\mathsf{\delta }=\mathsf{\beta }-{\mathsf{\beta }}^{*}$ and

$$1/2R_n=D_n\left(\mathsf{\beta }\right)-D_n\left({\mathsf{\beta }}^{*}\right)-{\mathsf{\delta }}^T{\mathit{T}}_n\left({\mathsf{\beta }}^{*}\right).$$

There exists a constant $v>0$, such that, for sufficiently large n,

$$R_n\ge vn{\parallel {\mathsf{\delta }}_{{\mathcal{M}}^{*}}\parallel }_2^2$$

for $\parallel {\mathsf{\delta }}_{{\mathcal{M}}_c^{*}}{\parallel }_1\le 3{\parallel {\mathsf{\delta }}_{{\mathcal{M}}^{*}}\parallel }_1$ with ${\mathcal{M}}_c^{*}$ being the complement of ${\mathcal{M}}^{*}$.

Condition (A1) shows that p diverges exponentially with n, which is a common setting in the ultra-high dimension. The two requirements in Condition (A2) are crucial for establishing the sure screening property. The former one implies that the signals of the true model are stronger than the random errors, so they are detectable. The latter one implies that the sparsity of $\mathsf{\beta }$ makes sure screening possible with $s\left(\widehat{\mathcal{M}}\right)=k\ge k_0$. Condition (A3) is a regular condition for the theoretical derivation. Condition (A4) is the same as the assumption of [17]. Condition (A5) is similar to [11].

Theorem 2. 

Suppose that Conditions (A1)–(A5) are satisfied with $2{\eta }_1+{\eta }_2<1-\alpha$. Let $\widehat{\mathcal{M}}$ be the estimated model obtained by the SRoWER with size k; then, we have

$$\mathbb{P}\{{\mathcal{M}}^{*}\subset \widehat{\mathcal{M}}\}\to 1,\mathrm{as}n\to \infty .$$

By using feature screening, important features that are highly correlated with the response variable can be kept in $\widehat{\mathcal{M}}$. However, it is necessary to note that there is no explicit choice for k, because it depends on the different dimensions. Note that the IHT algorithm needs a initial estimate ${\mathsf{\beta }}^{\left(0\right)}$. To further enhance computational efficiency, the LASSO estimate is chosen as the initial value of the iterations. The following theorem shows that with the initial value ${\mathsf{\beta }}^{\left(0\right)}$ obtained using LASSO, the IHT-implemented SRoWER can satisfy the property of sure independent screening within a finite iteration.

Theorem 3. 

Let ${\mathsf{\beta }}^{\left(t\right)}$ be the t-th update of the IHT procedure. The scale parameter $u\ge \xi kn$ for some $\xi >0$, and let ${\mathcal{M}}^{\left(t\right)}=\{j:{\mathsf{\beta }}_j^{\left(t\right)}\ne 0\}$ be the screening features. The initial value of iteration is

$${\mathsf{\beta }}^{\left(0\right)}=arg\underset{\mathsf{\beta }}{min}\{D_n\left(\mathsf{\beta }\right)+{n\lambda \parallel \mathsf{\beta }\parallel }_1\},$$

where λ satisfies $\lambda n^{(1-\alpha )/2}\to \infty$ and $\lambda =o\left(n^{-({\eta }_1+{\eta }_2)}\right)$. Then, under Conditions (A1)–(A5), for any finite $t\ge 1$, we have

$$\mathbb{P}\{{\mathcal{M}}^{*}\subset {\mathcal{M}}^{\left(t\right)}\}\to 1,\mathrm{as}n\to \infty .$$

3.3. The Choice of k

For the SRoWER method, we need prespecified k, such as $log\left(n\right)n^{1/3}$ [4,6,11]. Here, we treat k as a tuning parameter to control model complexity, and determine k by minimizing the following EBIC score:

$$EBIC\left(k\right)=log\left\{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\pi }_i\left(\tilde{\mathsf{\beta }}\right){\rho }_{\tau }(y_i-{\mathit{x}}_{i\mathcal{M}}^T{\mathsf{\beta }}_{\mathcal{M}})\right\}+k{\displaystyle \frac{log\left(n\right)log\left(p\right)}{2n}},$$

where $k=s\left(\mathcal{M}\right)$. The study of [33] proposed the EBIC for model selection for large model spaces. Here, we use it to determine k for comparing the SRoWER with the EPCS proposed by [25], which also used the EBIC for model selection.

Note that the EBIC selector for determining k requires searching over $k=1,2,\dots ,p$. To balance the computation and model selection accuracy in practice, we minimize $EBIC\left(k\right)$ for $k=1,\dots ,log\left(n\right)n^{1/3}$.

4. Numerical Studies

4.1. Simulation Studies

In this subsection, the finite sample performance of SRoWER is evaluated using simulation studies and compared with EPCS [25] and SMLE [11] based on expectile regression, i.e., ${\pi }_i=1,i=1,\dots ,n$ in the SRoWER. The IHT algorithm is used to carry out feature screening based on SRoWER, and the iteration is stopped when $\parallel {\mathsf{\beta }}^{\left(t\right)}-{\mathsf{\beta }}^{(t-1)}{\parallel }_2\le {10}^{-3}$.

We take $p=500$, $n=100,200$, and expectile level $\tau =0.5,0.05$, which correspond to the mean regression and an extreme expectile regression, respectively. All simulation results are based on 200 replications (with standard deviations in parentheses). To evaluate the performance of the screening approach, we use three criteria: the number of true positive variables (TP), the percentage of correctly fitted models (CF), and the root mean-squared error (RMSE) $\parallel {\widehat{\mathsf{\beta }}}_{\left[k\right]}-{\mathsf{\beta }}^{*}{\parallel }_2.$

Example 1. 

Consider the linear model

$$y_i=3x_{i6}+1.5x_{i12}+x_{i15}+x_{i20}+[{\epsilon }_i-{\mathcal{E}}^{\tau }\left({\epsilon }_i\right)],i=1,\dots ,n,$$

where the candidate features ${\mathit{x}}_i$ are i.i.d. generated from multivariate normal distribution $N_p(\mathbf{0},\Sigma )$ with $\Sigma =\left({\sigma }_{jk}\right)$, and ${\sigma }_{jk}={\rho }^{|j-k|}$. We set $\rho =0$ and $0.5$. The true model is ${\mathcal{M}}^{*}=\{6,12,15,20\}$. ${\epsilon }_i$ is generated from the standard gumbel distribution (Gumbel), the standard normal distribution (Normal), and the t distribution with three degrees of freedom (T), respectively.

Example 1 considers a relatively simple case. The simulation results are given in Table 1 and Table 2. For $\tau =0.5$, we can see that all three considered methods can almost screen all important features for three different error distributions. No one method can control the other two methods in all cases, but the SRoWER performs better than the SMLE and EPCS in most instances. However, at extreme expectiles $(\tau =0.05)$, the SRoWER method performs much better than the SMLE and EPCS in terms of RMSE except for the case of Gumbel with $n=200$. In addition, all the results become better when the sample size increases from 100 to 200.

Table 1. Simulation results of Example 1 for $\tau =0.5$.

$\mathit{\rho }$	$\mathit{\epsilon }$	n	Method	TP	CF	RMSE
0	Normal	100	SRoWER	4 (0 )	0.880 (0.326)	0.985 (0.427)
			SMLE	4 (0 )	0.910 (0.287)	1.092 (0.418)
			EPCS	4 (0 )	0.935 (0.247)	0.951 (0.424)
		200	SRoWER	4 (0 )	0.95 (0.218)	0.514 (0.077)
			SMLE	4 (0 )	0.95 (0.218)	0.693 (0.162)
			EPCS	4 (0 )	0.95 (0.218)	0.696 (0.162)
	Gumbel	100	SRoWER	3.995 (0.071)	0.880 (0.326)	1.153 (0.349)
			SMLE	4.000 (0.000)	0.905 (0.294)	1.518 (0.514)
			EPCS	3.990 (0.100)	0.920 (0.272)	1.390 (0.416)
		200	SRoWER	4 (0 )	0.900 (0.301)	0.756 (0.156)
			SMLE	4 (0 )	0.965 (0.184)	0.877 (0.216)
			EPCS	4 (0 )	0.965 (0.184)	0.922 (0.244)
	T	100	SRoWER	3.900 (0.375)	0.765 (0.425)	1.665 (0.587)
			SMLE	3.845 (0.471)	0.795 (0.405)	2.351 (0.914)
			EPCS	3.795 (0.578)	0.765 (0.425)	2.450 (1.143)
		200	SRoWER	3.980 (0.199)	0.91 (0.287)	0.854 (0.241)
			SMLE	3.960 (0.262)	0.92 (0.272)	1.260 (0.451)
			EPCS	3.945 (0.320)	0.92 (0.272)	1.286 (0.457)
0.5	Normal	100	SRoWER	4 (0 )	0.935 (0.247)	0.714 (0.245)
			SMLE	4 (0 )	0.945 (0.229)	0.834 (0.330)
			EPCS	4 (0 )	0.955 (0.208)	0.741 (0.327)
		200	SRoWER	4 (0 )	0.905 (0.294)	0.594 (0.094)
			SMLE	4 (0 )	0.975 (0.157)	0.322 (0.214)
			EPCS	4 (0 )	0.985 (0.122)	0.290 (0.178)
	Gumbel	100	SRoWER	3.995 (0.071)	0.90 (0.301)	0.943 (0.379)
			SMLE	3.995 (0.071)	0.91 (0.287)	1.175 (0.501)
			EPCS	3.975 (0.186)	0.91 (0.287)	1.187 (0.519)
		200	SRoWER	4 (0 )	0.905 (0.294)	0.695 (0.169)
			SMLE	4 (0 )	0.975 (0.157)	0.570 (0.289)
			EPCS	4 (0 )	0.990 (0.100)	0.397 (0.167)
	T	100	SRoWER	3.870 (0.429)	0.775 (0.419)	1.924 (0.580)
			SMLE	3.780 (0.551)	0.770 (0.422)	2.724 (1.067)
			EPCS	3.525 (0.795)	0.550 (0.499)	2.711 (0.907)
		200	SRoWER	3.970 (0.222)	0.845 (0.363)	0.986 (0.277)
			SMLE	3.955 (0.252)	0.950 (0.218)	0.665 (0.234)
			EPCS	3.940 (0.327)	0.935 (0.247)	0.729 (0.223)

Table 2. Simulation results of Example 1 for $\tau =0.05$.

$\mathit{\rho }$	$\mathit{\epsilon }$	n	Method	TP	CF	RMSE
0	Normal	100	SRoWER	3.995 (0.071)	0.725 (0.448)	1.565 (0.422)
			SMLE	4.000 (0.000)	0.760 (0.428)	2.187 (0.661)
			EPCS	3.995 (0.071)	0.510 (0.501)	3.067 (0.951)
		200	SRoWER	4 (0 )	0.725 (0.448)	0.891 (0.229)
			SMLE	4 (0 )	0.690 (0.464)	1.635 (0.589)
			EPCS	4 (0 )	0.540 (0.500)	2.031 (0.697)
	Gumbel	100	SRoWER	3.980 (0.140)	0.875 (0.332)	1.050 (0.192)
			SMLE	3.995 (0.071)	0.950 (0.218)	0.932 (0.291)
			EPCS	3.995 (0.071)	0.830 (0.377)	1.391 (0.393)
		200	SRoWER	4 (0 )	0.885 (0.320)	0.601 (0.111)
			SMLE	4 (0 )	0.985 (0.122)	0.422 (0.131)
			EPCS	4 (0 )	0.960 (0.196)	0.619 (0.184)
	T	100	SRoWER	3.120 (1.049)	0.295 (0.457)	4.441 (1.483)
			SMLE	3.325 (0.907)	0.235 (0.425)	7.941 (3.112)
			EPCS	3.120 (1.025)	0.065 (0.247)	9.146 (3.500)
		200	SRoWER	3.765 (0.558)	0.400 (0.491)	2.918 (1.069)
			SMLE	3.800 (0.540)	0.180 (0.385)	7.217 (2.362)
			EPCS	3.800 (0.530)	0.125 (0.332)	7.767 (2.407)
0.5	Normal	100	SRoWER	3.985 (0.122)	0.79 (0.408)	1.355 (0.364)
			SMLE	3.990 (0.141)	0.83 (0.377)	1.746 (0.645)
			EPCS	3.880 (0.487)	0.50 (0.501)	2.710 (0.887)
		200	SRoWER	4 (0 )	0.665 (0.473)	0.962 (0.251)
			SMLE	4 (0 )	0.725 (0.448)	1.752 (0.515)
			EPCS	4 (0 )	0.705 (0.457)	1.733 (0.499)
	Gumbel	100	SRoWER	4.00 (0.000)	0.890 (0.314)	0.979 (0.246)
			SMLE	4.00 (0.000)	0.935 (0.247)	1.035 (0.253)
			EPCS	3.92 (0.405)	0.635 (0.483)	1.563 (0.618)
		200	SRoWER	4 (0 )	0.765 (0.425)	0.774 (0.120)
			SMLE	4 (0 )	0.935 (0.247)	0.670 (0.301)
			EPCS	4 (0 )	0.935 (0.247)	0.541 (0.372)
	T	100	SRoWER	3.260 (0.926)	0.270 (0.445)	4.738 (1.573)
			SMLE	3.415 (0.852)	0.265 (0.442)	7.569 (2.619)
			EPCS	2.615 (1.159)	0.065 (0.247)	8.153 (2.855)
		200	SRoWER	3.725 (0.593)	0.425 (0.496)	2.296 (0.635)
			SMLE	3.770 (0.518)	0.220 (0.415)	7.199 (2.344)
			EPCS	3.460 (0.801)	0.145 (0.353)	6.908 (2.204)

Example 2. 

For the linear model in Example 1, to examine the robustness of the SRoWER, we consider the case where there are outliers in the covariates. We first generate data as those in Example 1. Next, we artificially add outliers from $N(0,5^2)$ to random 50 covariates of 10% of the observations. The other settings are the same as those in Example 1.

Example 2 considers the case where both covariates and response variables have outliers. The simulation results are shown in Table 3 and Table 4. We can see that the SRoWER has the smallest RMSE compared with SMLE and EPCS. Three considered methods have similar performance in variable selection, except for the case of T. Both SRoWER and SMLE perform better than EPCS in terms of CF for the case of T.

Table 3. Simulation results of Example 2 for $\tau =0.5$.

$\mathit{\rho }$	$\mathit{\epsilon }$	n	Method	TP	CF	RMSE
0	Normal	100	SRoWER	3.685 (0.713)	0.645 (0.480)	1.695 (0.569)
			SMLE	3.770 (0.537)	0.685 (0.466)	2.771 (1.039)
			EPCS	3.700 (0.716)	0.690 (0.464)	2.880 (0.987)
		200	SRoWER	3.940 (0.277)	0.755 (0.431)	1.492 (0.503)
			SMLE	3.935 (0.285)	0.820 (0.385)	2.216 (0.806)
			EPCS	3.915 (0.372)	0.820 (0.385)	2.150 (0.711)
	Gumbel	100	SRoWER	3.655 (0.720)	0.635 (0.483)	1.553 (0.521)
			SMLE	3.680 (0.663)	0.695 (0.462)	2.986 (1.355)
			EPCS	3.605 (0.782)	0.675 (0.470)	2.806 (0.871)
		200	SRoWER	3.825 (0.464)	0.725 (0.448)	1.296 (0.551)
			SMLE	3.820 (0.478)	0.780 (0.415)	2.050 (0.688)
			EPCS	3.805 (0.528)	0.805 (0.397)	1.456 (0.585)
	T	100	SRoWER	3.495 (0.821)	0.560 (0.498)	2.030 (0.712)
			SMLE	3.445 (0.819)	0.585 (0.494)	3.532 (1.271)
			EPCS	3.380 (0.927)	0.540 (0.500)	3.123 (1.117)
		200	SRoWER	3.895 (0.380)	0.675 (0.470)	1.611 (0.518)
			SMLE	3.805 (0.508)	0.785 (0.412)	1.831 (0.558)
			EPCS	3.830 (0.482)	0.810 (0.393)	2.013 (0.693)
0.5	Normal	100	SRoWER	3.735 (0.646)	0.660 (0.475)	1.740 (0.586)
			SMLE	3.760 (0.587)	0.715 (0.453)	3.033 (0.881)
			EPCS	3.660 (0.760)	0.690 (0.464)	3.292 (1.296)
		200	SRoWER	3.950 (0.240)	0.645 (0.480)	1.588 (0.421)
			SMLE	3.860 (0.437)	0.690 (0.464)	2.469 (0.457)
			EPCS	3.855 (0.525)	0.745 (0.437)	2.582 (0.500)
	Gumbel	100	SRoWER	3.735 (0.622)	0.655 (0.477)	1.839 (0.567)
			SMLE	3.725 (0.601)	0.690 (0.464)	2.801 (1.184)
			EPCS	3.685 (0.615)	0.670 (0.471)	2.463 (1.101)
		200	SRoWER	3.885 (0.377)	0.640 (0.481)	1.617 (0.487)
			SMLE	3.840 (0.394)	0.675 (0.470)	2.523 (0.661)
			EPCS	3.820 (0.468)	0.715 (0.453)	2.624 (0.710)
	T	100	SRoWER	3.455 (0.801)	0.445 (0.498)	2.114 (0.549)
			SMLE	3.425 (0.792)	0.475 (0.501)	3.765 (1.072)
			EPCS	3.100 (1.070)	0.405 (0.492)	3.186 (1.153)
		200	SRoWER	3.910 (0.304)	0.520 (0.501)	1.949 (0.523)
			SMLE	3.775 (0.525)	0.685 (0.466)	2.756 (0.671)
			EPCS	3.670 (0.703)	0.660 (0.475)	2.877 (0.735)

Table 4. Simulation results of Example 2 for $\tau =0.05$.

$\mathit{\rho }$	$\mathit{\epsilon }$	n	Method	TP	CF	RMSE
0	Normal	100	SRoWER	3.530 (0.918)	0.385 (0.488)	2.090 (0.576)
			SMLE	3.755 (0.638)	0.535 (0.500)	4.484 (1.384)
			EPCS	3.575 (0.865)	0.325 (0.470)	4.070 (1.368)
		200	SRoWER	3.75 (0.671)	0.550 (0.499)	1.406 (0.587)
			SMLE	3.98 (0.140)	0.490 (0.501)	4.210 (1.145)
			EPCS	3.87 (0.463)	0.495 (0.501)	3.615 (1.053)
	Gumbel	100	SRoWER	3.490 (0.930)	0.395 (0.490)	1.649 (0.537)
			SMLE	3.745 (0.626)	0.630 (0.484)	4.217 (1.552)
			EPCS	3.535 (0.856)	0.475 (0.501)	3.409 (1.670)
		200	SRoWER	3.755 (0.606)	0.600 (0.491)	1.370 (0.578)
			SMLE	3.885 (0.391)	0.655 (0.477)	3.338 (1.196)
			EPCS	3.785 (0.592)	0.675 (0.470)	3.088 (1.085)
	T	100	SRoWER	2.795 (1.127)	0.165 (0.372)	3.357 (0.890)
			SMLE	2.985 (1.082)	0.185 (0.389)	8.103 (2.779)
			EPCS	2.735 (1.184)	0.070 (0.256)	8.865 (2.696)
		200	SRoWER	3.44 (0.900)	0.34 (0.475)	2.686 (0.759)
			SMLE	3.67 (0.635)	0.17 (0.377)	7.480 (2.210)
			EPCS	3.56 (0.692)	0.09 (0.287)	7.741 (2.436)
0.5	Normal	100	SRoWER	3.610 (0.966)	0.40 (0.491)	1.966 (0.582)
			SMLE	3.795 (0.524)	0.54 (0.500)	4.450 (1.279)
			EPCS	3.365 (1.170)	0.30 (0.459)	4.641 (1.729)
		200	SRoWER	3.835 (0.547)	0.480 (0.501)	1.556 (0.400)
			SMLE	3.925 (0.346)	0.515 (0.501)	3.623 (0.840)
			EPCS	3.760 (0.628)	0.545 (0.499)	3.360 (0.939)
	Gumbel	100	SRoWER	3.700 (0.702)	0.435 (0.497)	1.739 (0.621)
			SMLE	3.785 (0.529)	0.685 (0.466)	3.233 (1.333)
			EPCS	3.560 (0.895)	0.505 (0.501)	3.989 (1.709)
		200	SRoWER	3.835 (0.509)	0.530 (0.500)	1.620 (0.572)
			SMLE	3.910 (0.335)	0.620 (0.487)	3.390 (1.064)
			EPCS	3.770 (0.607)	0.665 (0.473)	3.121 (1.015)
	T	100	SRoWER	2.815 (1.099)	0.115 (0.320)	3.167 (0.756)
			SMLE	2.960 (0.992)	0.155 (0.363)	8.137 (2.316)
			EPCS	2.265 (1.068)	0.025 (0.157)	8.689 (2.627)
		200	SRoWER	3.445 (0.843)	0.305 (0.462)	2.703 (0.689)
			SMLE	3.620 (0.662)	0.155 (0.363)	7.437 (2.066)
			EPCS	3.115 (0.998)	0.145 (0.353)	8.051 (1.960)

Example 3. 

Here we consider a heterogeneous model. We first generate ${(z_1,\dots ,z_p)}^T$ from multivariate normal distribution $N_p(\mathbf{0},\Sigma )$ with $\Sigma =\left({\sigma }_{jk}\right)$, and ${\sigma }_{jk}={\rho }^{|j-k|}$. We set $\rho =0$ and $0.5$. Let $x_1=\Phi \left(z_1\right)$ and $x_j=z_j,j=2,3,\dots ,p$, where $\Phi (·)$ is the cumulative distribution function of standard normal distribution. The response is then simulated from the following normal linear heteroscedastic model

$$y_i=3x_{i6}+1.5x_{i12}+x_{i15}+x_{i20}+\left(0.7x_{i1}\right)[{\epsilon }_i-{\mathcal{E}}^{\tau }\left({\epsilon }_i\right)],$$

where ${\epsilon }_i\sim N(0,1)$. Meanwhile, the other settings are the same as those in Example 2.

From Table 5 and Table 6, we can see that the conclusions are similar to those of Examples 1 and 2. Hence, the SRoWER performs well even in the case that there are outliers in the heterogeneous model.

Table 5. Simulation results of Example 3 for $\tau =0.5$.

$\mathit{\rho }$	n	Method	TP	CF	RMSE
0	100	SRoWER	3.81 (0.579)	0.690 (0.464)	1.600 (0.642)
		SMLE	3.86 (0.471)	0.665 (0.473)	3.671 (1.112)
		EPCS	3.83 (0.532)	0.670 (0.471)	3.186 (1.061)
	200	SRoWER	3.945 (0.25)	0.785 (0.412)	0.968 (0.513)
		SMLE	3.965 (0.21)	0.765 (0.425)	1.956 (0.784)
		EPCS	3.965 (0.21)	0.815 (0.389)	1.762 (0.859)
0.5	100	SRoWER	3.895 (0.495)	0.65 (0.478)	1.432 (0.566)
		SMLE	3.915 (0.372)	0.64 (0.481)	2.811 (0.826)
		EPCS	3.845 (0.559)	0.70 (0.459)	2.841 (0.937)
	200	SRoWER	3.970 (0.198)	0.675 (0.470)	1.325 (0.557)
		SMLE	3.980 (0.140)	0.665 (0.473)	1.968 (0.585)
		EPCS	3.965 (0.184)	0.675 (0.470)	2.221 (0.641)

Table 6. Simulation results of Example 3 for $\tau =0.05$.

$\mathit{\rho }$	n	Method	TP	CF	RMSE
0	100	SRoWER	3.610 (0.907)	0.545 (0.499)	1.189 (0.659)
		SMLE	3.845 (0.531)	0.530 (0.500)	4.331 (1.956)
		EPCS	3.725 (0.783)	0.515 (0.501)	4.003 (1.500)
	200	SRoWER	3.750 (0.735)	0.600 (0.491)	0.947 (0.715)
		SMLE	3.945 (0.250)	0.535 (0.500)	4.572 (1.206)
		EPCS	3.900 (0.401)	0.560 (0.498)	3.445 (1.388)
0.5	100	SRoWER	3.73 (0.735)	0.435 (0.497)	1.602 (0.523)
		SMLE	3.92 (0.323)	0.550 (0.499)	3.820 (1.434)
		EPCS	3.73 (0.692)	0.520 (0.501)	3.843 (1.253)
	200	SRoWER	3.950 (0.240)	0.49 (0.501)	1.427 (0.527)
		SMLE	3.985 (0.122)	0.53 (0.500)	3.478 (1.163)
		EPCS	3.905 (0.383)	0.54 (0.500)	3.212 (0.929)

4.2. Real Data Example

This subsection applies the SRoWER method for feature screening to the Mid-Atlantic wage data with 3000 observations and 8 predictors from [34] that are available in the `ISLR’ package in R. A total of eight predictors (two continuous and six categorical) are considered. The continuous variables are year that wage information was recorded (year) and age of worker (age). The categorical factors include a factor with the following levels: 1. Never Married, 2. Married, 3. Widowed, 4. Divorced, and 5. Separated, which indicate marital status (marital). Another factor (race) contains the following levels: 1. White, 2. Black, 3. Asian and 4. Other. Another factor (education) contains the following levels 1. <HS Grad, 2. HS Grad, 3. Some College, 4. College Grad and 5. Advanced Degree. Another factor (jobclass) contains the following levels: 1. Industrial and 2. Information indicating type of job. Another factor (health) contains the following levels, 1. ≤Good and 2. ≥Very Good, indicating health level of worker. Another factor contains the following levels, 1. Yes and 2. No, indicating whether worker has health insurance (health_ins). We use the dummy variables to represent six categorical variables. Therefore, there are 16 covariates, and the response is the logarithm of wage. Following the set up of [35], to demonstrate the application in high dimension, we extend the data by introducing the following artificial covariates:

$$X_j={\displaystyle \frac{Z_j+2W}{3}},j=17,\dots ,500,$$

where $Z_j$ is the standard normal random variables and W follows the standard uniform distribution.

To test the prediction performance of SRoWER, EPCS, and SMLE, we randomly generatde 100 partitions of the full data, and divided the data into two parts, where $n_{tr}=1500$ samples are treated as training data and the remaining $n_{te}=1500$ samples are treated as testing data. The average of model size (Size), the number of selected noise variables (SNV) and expectile prediction error (EPE) at $\tau =0.5$ and $0.05$, where EPE is computed by the test data

$$EP{E}_{\tau }={\displaystyle \frac{1}{n_{te}}}\sum _{i=1}^{n_{te}}{\pi }_i\left(\tilde{\mathsf{\beta }}\right){\rho }_{\tau }(y_i-{\widehat{y}}_i),$$

where ${\widehat{y}}_i={\mathit{x}}_i^T{\widehat{\mathsf{\beta }}}_{\tau }$ is the $\tau$ expectile estimate of the i-th observation with ${\widehat{\mathsf{\beta }}}_{\tau }$ being calculated based on the training data set, and $y_i$ is the i-th observation in the test set.

The results are reported in Table 7. For $\tau =0.5$, both SRoWER and SMLE perform similarly in terms of EPE and SNV, while the SRoWER includes more than one variable compared with the SMLE. Although the model size of the SRoWER and EPCS are similar, the EPE of the EPCS is the largest among three methods. For $\tau =0.05$, the SRoWER performs best, while the EPCS performs worst. The selected model sizes vary for different $\tau$, it indicates the heteroscedasticity of the model. This conclusion agrees with the results of [36].

Table 7. Expectile prediction error (EPE), model size (Size), and selected noise variables (SNV) over 100 repetitions and their standard errors (in parentheses) for wage data.

$\mathit{\tau }$	Method	EPE	Size	SNV
0.5	SRoWER	0.041 (0.002)	6.63 (1.107)	0.06 (0.239)
	SMLE	0.042 (0.002)	5.44 (0.978)	0.04 (0.243)
	EPCS	0.048 (0.005)	6.45 (0.687)	0.00 (0.000)
0.05	SRoWER	0.017 (0.001)	4.60 (1.511)	0.33 (0.620)
	SMLE	0.030 (0.014)	6.03 (2.422)	1.12 (1.565)
	EPCS	0.275 (0.183)	10.11 (3.127)	3.62 (2.107)

5. Conclusions

To deal with the heterogeneity and the outliers in the covariates and/or the response, this paper proposes the RoWER method, which is further applied to screen the features in ultra-high dimensional data. We have also proposed an iterative hard-thresholding algorithm for implementing the feature screening procedure, and establish the sure screening property for the SRoWER method. Simulation studies and a real data analysis verify that the SRoWER method not only reduces the huge computational effort faced by ultra-high dimensional data, but also shows excellent robustness in heterogeneous data. Compared with ISIS [6], the SRoWER naturally accounts for the joint effects between features, and benefits from the advantage of the SMLE in terms of computational efficiency. Based on the proposed method, the problem of robust feature screening for classification data also presents a promising direction for future research.