Prediction and Variable Selection in High-Dimensional Misspecified Binary Classification

Abstract

In this paper, we consider prediction and variable selection in the misspecified binary classification models under the high-dimensional scenario. We focus on two approaches to classification, which are computationally efficient, but lead to model misspecification. The first one is to apply penalized logistic regression to the classification data, which possibly do not follow the logistic model. The second method is even more radical: we just treat class labels of objects as they were numbers and apply penalized linear regression. In this paper, we investigate thoroughly these two approaches and provide conditions, which guarantee that they are successful in prediction and variable selection. Our results hold even if the number of predictors is much larger than the sample size. The paper is completed by the experimental results.

1. Introduction

Large-scale data sets, where the number of predictors significantly exceeds the number of observations, become common in many practical problems from, among others, biology or genetics. Currently, the analysis of such data sets is a fundamental challenge in statistics and machine learning. High-dimensional prediction and variable selection are arguably the most popular and intensively studied topics in this field. There are many methods trying to solve these problems such as those based on penalized estimation [1,2]. The main representative of them is Lasso [3], that relates to $l_1$-norm penalization. Its properties in model selection, estimation and prediction are deeply investigated, among others, in [2,4,5,6,7,8,9,10]. The results obtained in the above papers can be applied only if some specific assumptions are satisfied. For instance, these conditions concern the relation between the response variable and predictors. However, it is quite common that a complex data set does not satisfy these model assumptions or they are difficult to verify, which leads to the fact that the considered model is specified incorrectly. The model misspecification problem is the core of the current paper. We investigate this topic in the context of high-dimensional binary classification (binary regression).

In the classification problem we are to predict or to guess the class label of the object on the basis of its observed predictors. The object is described by the random vector $(X,Y),$ where $X\in {\mathbb{R}}^p$ is a vector of predictors and $Y\in \{-1,1\}$ is the class label of the object. A classifier is defined as a measurable function $f:{\mathbb{R}}^p\to \mathbb{R},$ which determines the label of an object in the following way:

$$\mathrm{if}f\left(x\right)\ge 0,\mathrm{then}\mathrm{we}\mathrm{predict}\mathrm{that}y=1.$$

Otherwise, we guess that $y=-1.$

The most natural approach is to look for a classifier $f,$ which minimizes the misclassification risk (probability of incorrect classification)

$$R\left(f\right)=P(Y=1,f\left(X\right)<0)+P(Y=-1,f\left(X\right)\ge 0).$$

(1)

Let $\eta \left(x\right)=P(Y=1|X=x).$ It is clear that $f_B\left(x\right)=\mathrm{sign}(2\eta \left(x\right)-1)$ minimizes the risk (1) in the family of all classifiers. It is called the Bayes classifier and we denote its risk as $R_B=R\left(f_B\right).$ Obviously, in practice we do not know the function $\eta$, so we cannot find the Bayes classifier. However, if we possess a training sample $(X_1,Y_1),\dots ,(X_n,Y_n)$ containing independent copies of $(X,Y),$ then we can consider a sample analog of (1), namely the empirical misclassification risk

$$\frac{1}{n}\sum _{i=1}^n\left[\mathbb{I}(Y_i=1,f\left(X_i\right)<0)+\mathbb{I}(Y_i=-1,f\left(X_i\right)\ge 0)\right],$$

(2)

where $\mathbb{I}$ is the indicator function. Then a minimizer of (2) could be used as our estimator.

The main difficulty in this approach lies in discontinuity of the function (2). It entails that finding its minimizer is computationally difficult and not effective. To overcome this problem, one usually replaces the discontinuous loss function by its convex analog $\varphi :\mathbb{R}\to [0,\infty ]$, for instance the logistic loss, the hinge loss or the exponential loss. Then we obtain the convex empirical risk

$$\overline{Q}\left(f\right)=\frac{1}{n}\sum _{i=1}^n\varphi \left(Y_{i}f\left(X_i\right)\right).$$

(3)

In the high-dimensional case one usually obtains an estimator by minimizing the penalized version of (3). Those tricks have been successfully used in the classification theory and have allowed to invent boosting algorithms [11], support vector machines [12] or Lasso estimators [3]. In this paper we are mainly interested in Lasso estimators, because they are able to solve both variable selection and prediction problems simultaneously, while the first two algorithms are developed mainly for prediction.

Thus, we consider linear classifiers

$$f_b\left(x\right)=b_0+\sum _{j=1}^p{b}_j{x}_j,$$

(4)

where $b=(b_0,b_1,\dots ,b_p)\in {\mathbb{R}}^{p+1}.$ For a fixed loss function $\varphi$ we define the Lasso estimator as

$$\widehat{b}=arg\underset{b\in {\mathbb{R}}^{p+1}}{min}\overline{Q}\left(f_b\right)+\lambda \sum _{j=1}^p\left|b_j\right|,$$

(5)

where $\lambda$ is a positive tuning parameter, which provides a balance between minimizing the empirical risk and the penalty. The form of the penalty is crucial, because its singularity at the origin implies that some coordinates of the minimizer $\widehat{b}$ are exactly equal to zero, if $\lambda$ is sufficiently large. Thus, calculating (5) we simultaneously select significant predictors in the model and we estimate their coefficients, so we are also able to predict the class of new objects. The function $\overline{Q}\left(f_b\right)$ and the penalty are convex, so (5) is a convex minimization problem, which is an important fact from both practical and theoretical points of view. Notice that the intercept $b_0$ is not penalized in (5).

The random vector (5) is an estimator of

$$b_{*}=arg\underset{b\in {\mathbb{R}}^{p+1}}{min}Q\left(f_b\right),$$

(6)

where $Q\left(f_b\right)=\mathbb{E}\varphi \left(Y{f}_b\left(X\right)\right).$ In this paper we are mainly interested in minimizers (6) corresponding to quadratic and logistic loss functions. The latter has a nice information-theoretic interpretation. Namely, it can be viewed as the Kullback–Leibler projection of unknown $\eta$ on logistic models [13]. The Kullback–Leibler divergence [14] plays an important role in the information theory and statistics, for instance it is involved in information criteria in model selection [15] or in detecting influenctial observations [16].

In general, the classifier corresponding to (6) need not coincide with the Bayes classifier. Obviously, we want to have a “good” estimator, which means that its misclassification risk should be as close to the risk of the Bayes classifier as possible. In other words, its excess risk

$$\mathcal{E}(\widehat{b},f_B)=E_{D}R\left(\widehat{b}\right)-R_B$$

(7)

should be small, where $E_D$ is the expectation with respect to the data $D=\{(X_1,Y_1),\dots ,(X_n,Y_n)\}$ and we write simply $R\left(b\right)$ instead of $R\left(f_b\right).$ Our goal is to study the excess risk (7) for the estimator (5) with different loss functions $\varphi .$ We do it by looking for the upper bounds of (7).

In the excess risk (7) we compare two misclassification risks defined in (1). In the literature one can also find a different approach, which replaces the misclassification risks $R(·)$ in (7) by the convex risks $Q(·).$ In that case the excess risk depends on the loss function $\varphi .$ To deal with this fact one uses the results from [17,18], which state the relation between the excess risk (7) and its analog based on the convex risk $Q(·).$ In this paper we do not follow this way and work, right from the beginning, with the excess risk independent of $\varphi .$ Only the estimator (5) depends on the loss $\varphi .$

In this paper we are also interested in variable selection. We investigate this problem in the following semiparametric model

$$\eta \left(x\right)=g({\beta }_0+\sum _{j=1}^p{\beta }_j{x}_j),$$

(8)

where $\eta \left(x\right)=P(Y=1|X=x)$, $\beta \in {\mathbb{R}}^{p+1}$ is the true parameter and g is unknown function. Thus, we suppose that predictors influence class probability through the function g of the linear combination ${\beta }_0+{\displaystyle \sum _{j=1}^p}{\beta }_j{x}_j$. The goal of variable selection is the identification of the set of significant predictors

$$\mathbb{T}=\{1\le j\le p:{\beta }_j\ne 0\}.$$

(9)

Obviously, in the model (8) we cannot estimate an intercept ${\beta }_0$ and we can identify the vector $({\beta }_1,\dots ,{\beta }_p)$ only up to a multiplicative constant, because any shift or scale change in ${\beta }_0+{\displaystyle \sum _{j=1}^p}{\beta }_j{X}_j$ can be absorbed by g. However, we show in Section 5 that in many situations the Lasso estimator (5) can properly identify the set (9).

The literature on the classification problem is comprehensive. We just mention a few references: [12,19,20,21]. The predictive quality of classifiers is often investigated by obtaining upper bounds for their excess risks. It is an important problem and was studied thoroughly, among others in [17,18,22,23,24]. The variable selection and predictive properties of estimators in the high-dimensional scenario were studied, for instance, in [2,10,13,25,26]. In the current paper we investigate the behaviour of classifiers in possibly misspecified high-dimensional classification, which appears frequently in practice. For instance, while working with binary regression one often assumes incorrectly that the data follow the logistic regression model. Then the problem is solved using the Lasso penalized maximum likelihood method. Another approach to binary regression, which is widely used due to its computational simplicity, is just treating labels $Y_i$ as they were numbers and applying standard Lasso. For instance, such method is used in ([1], [Subsections 4.2 and 4.3]) or ([2], Subsection 2.4.1). These two approaches to classification sometimes give unexpectedly good results in variable selection and prediction, but the reason of this phenomenon has not been deeply studied in the literature. Among the above mentioned papers only [2,13,25] take up this issue. However, [25] focuses mainly on the predictive properties of Lasso classifiers with the hinge loss. Bühlmann and van de Geer [2] and Kubkowski and Mielniczuk [13] study general Lipschitz loss functions. The latter paper considers only the variable selection problem. In [2] one also investigates prediction, but they do not study classification with the quadratic loss.

In this paper we are interested in both variable selection and predictive properties of classifiers with convex (but not necessarily Lipschitz) loss functions. The prominent example is classification with the quadratic loss function, which has not been investigated so far in the context of the high-dimensional misspecified model. In this case the estimator (5) can be calculated efficiently using the existing algorithms, for instance [27] or [28], even if the number of predictors is much larger than the sample size. It makes this estimator very attractive, while working with large data sets. In [28] one provides also the efficient algorithm for Lasso estimators with the logistic loss in the high-dimensional scenario. Therefore, misspecified classification with the logistic loss plays an important role in this paper as well. Our goal is to study thoroughly such estimators and provide conditions, which guarantee that they are successful in prediction and variable selection.

The paper is organized as follows: in the next section we provide basic notations and assumptions, which are used in this paper. In Section 3 we study predictive properties of Lasso estimators with different loss functions. We will see that these properties depend strongly on the estimation quality of estimators, which is studied in Section 4. In Section 5 we consider variable selection. In Section 6 we show numerical experiments, which describe the quality of estimators in practice. The proofs and auxiliary results are relegated to Appendix A.

2. Assumptions and Notation

In this paper we work in the high-dimensional scenario $p>>n.$ As usual we assume that the number of predictors p can vary with the sample size $n,$ which could be denoted as $p\left(n\right)=p_n.$ However, to make notation simpler we omit the lower index and write p istead of $p_n.$ The same refers to the other objects appearing in this paper.

In the further sections we will need the following notation:

-

$X_i={(X_{i1},X_{i2},\dots ,X_{ip})}^{\top }$;

-

$\mathbb{X}={(X_1,X_2,\dots ,X_n)}^{\top }$ is the $(n\times p)$-matrix of predictors;

-

Let $A\subset \{1,\dots ,p\}.$ Then $A^c=\{1,\dots ,p\}\backslash A$ is a complement of A;

-

${\mathbb{X}}_A$ is a submatrix of $\mathbb{X},$ with columns whose indices belong to A;

-

$b_A$ is a restriction of a vector $b\in {\mathbb{R}}^p$ to the indices from $A;$

-

$\left|A\right|$ is the number of elements in $A;$

-

$\tilde{A}=A\cup \left\{0\right\},$ so the set $\tilde{A}$ contains indices from A and the intercept;

-

The $l_q$-norm of a vector is defined as ${\left|b\right|}_q={\left({\sum }_{j=1}^p{\left|b_j\right|}^q\right)}^{1/q}$ for $q\in [1,\infty ];$

-

For $x\in {\mathbb{R}}^p$ we denote $\tilde{x}={(1,x)}^{\top };$

-

$\tilde{\mathbb{X}}$ is the matrix $\mathbb{X}$ with the column of ones binded from the left side;

-

${\widehat{b}}^{quad},b_{*}^{quad}$ are minimizers in (5), (6), respectively, with the quadratic loss function;

-

${\widehat{b}}^{log},b_{*}^{log}$ are minimizers in (5), (6), respectively, with the logistic loss function;

-

The Kullback–Leibler (KL) distance [14] between two binary distributions with success probabilities ${\pi }_1$ and ${\pi }_2$ is defined as

$$KL({\pi }_1,{\pi }_2)={\pi }_{1}log\left(\frac{{\pi }_1}{{\pi }_2}\right)+(1-{\pi }_1)log\left(\frac{1-{\pi }_1}{1-{\pi }_2}\right).$$

(10)

Obviously, we have $KL({\pi }_1,{\pi }_2)\ge 0$ and $KL({\pi }_1,{\pi }_2)=0$ if only if ${\pi }_1={\pi }_2$. Moreover, the $KL$ distance need not be symmetric;

-

the set of nonzero coefficients of $b_{*}^{quad}$ is denoted as

$$T=\{1\le j\le p:{\left(b_{*}^{quad}\right)}_j\ne 0\}.$$

(11)

Notice that the intercept is not contained in (11) even if it is nonzero.

We also specify assumptions, which are used in this paper.

Assumption 1.

We assume that $(X,Y),(X_1,Y_1),\dots ,(X_n,Y_n)$ are i.i.d. random vectors. Moreover, predictors are univariate subgaussian, i.e., for each $a\in \mathbb{R}$ and $j\in \{1,\dots ,p\}$ we have $Eexp\left(a{X}_j\right)\le exp({\sigma }_j^2{a}^2/2)$ for positive numbers ${\sigma }_j.$ We also denote $\sigma =\underset{1\le j\le p}{max}{\sigma }_j.$ Finally, we suppose that the matrix $H=E\left[X{X}^{\top }\right]$ is positive definite and $H_{jj}=1$ for $j=1,\dots ,p.$

In Section 4 and Section 5 we need stronger version of Assumption 1.

Assumption 2.

We suppose that the subvector of predictors $X_T$ is subgaussian with the coefficient ${\sigma }_0>0$, i.e., for each $u\in {\mathbb{R}}^{\left|T\right|}$ we have $Eexp\left(u^{\top }{X}_T\right)\le exp({\sigma }_0^2{u}^{\top }{H}_{T}u/2),$ where $H_T={\left(E\left[X_{1j}{X}_{1k}\right]\right)}_{j,k\in T}.$ The remaining conditions are as in Assumption 1. We also denote $\sigma =max({\sigma }_0,{\sigma }_j,j\notin T).$

Subgaussianity of predictors is a standard assumption while working with random predictors in high-dimensional models, cf. [13]. In particular, Assumption 1 implies that $E\left[X\right]=0$ and $\sigma \ge 1$ [29].

3. Predictive Properties of Classifiers

In this part of the paper we study prediction properties of classifiers with convex loss functions. To do it we look for upper bounds of the excess risk (7) of estimators.

As usual the excess risk in (7) can be decomposed as

$$E_{D}R\left(\widehat{b}\right)-R\left(b_{*}\right)+R\left(b_{*}\right)-R_B.$$

(12)

The second term in (12) is the approximation risk and compares the predictive abilitity of the “best” linear classifier (6) to the Bayes classifier. The first term in (12) is called the estimation risk and describes how the estimation process influences the predictive property of classifiers.

In the next theorem we bound from above the estimation risk of classifiers. To make the result more transparent we use notations $P_D$ and $P_X$ in (13), which indicate explicitly which probability we consider, i.e., $P_D$ is probability with respect to the data D and $P_X$ is with respect to the new object X. In further results we omit these lower indexes and believe that it does not lead to confusion.

Theorem 1.

For $c>0$ we consider an event $\Omega =\left\{\right|\widehat{b}-b_{*}{|}_1\le c\}.$ We have

$$E_{D}R\left(\widehat{b}\right)-R\left(b_{*}\right)\le 2P_D\left({\Omega }^c\right)+P_X\left(\right|b_{*}^{\top }\tilde{X}|\le c|\tilde{X}{|}_{\infty }).$$

(13)

In Theorem 1 we obtain the upper bound for the estimation risk. This risk becomes small, if we establish that probability of the event ${\Omega }^c$ is small and the sequence $c,$ which is involved in $\Omega$ and in the second term on the right-hand side of (13), decreases sufficiently fast to zero. Therefore, Theorem 1 shows that to have a small estimation risk it is enough to prove that for each $\epsilon \in (0,1)$ there exists c such that

$$P\left(\right|\widehat{b}-b_{*}{|}_1\le c)\ge 1-\epsilon .$$

(14)

Moreover, numbers $\epsilon$ and c should be sufficiently small. This property will be studied thoroughly in the next section. Notice that the first term on the right-hand side of (13) relates to the fact, how well (5) estimates (6). Moreover, the second expression on the right-hand side of (13) can be bounded from above, if predictors are sufficiently regular, for instance subgaussian.

So far, we have been interested in the estimation risk of estimators. In the next result we establish the upper bound for the approximation risk as well. This bound combined with (13) enables us to bound from above the excess risk of estimators. We prove this fact for the quadratic loss $\varphi \left(t\right)={(1-t)}^2$ and the logistic loss $\varphi \left(t\right)=log(1+e^{-v})$, which play prominent roles in this paper.

Theorem 2.

Suppose that Assumption 1 is fulfilled. Moreover, a random variable $b_{*}^{\top }\tilde{X}$ has a density $h,$ which is continuous on the interval $U=[-2\sigma c\sqrt{logp},2\sigma c\sqrt{logp}]$ and $\tilde{h}=\underset{u\in U}{sup}h\left(u\right).$

(a) We have

$$\begin{array}{ccc}\hfill \mathcal{E}({\widehat{b}}^{quad},f_B)& \le & 2P\left({\Omega }^c\right)+4\sigma {\tilde{h}}^{quad}c\sqrt{logp}+2/p\hfill \end{array}$$

(15)

$$\begin{array}{cc}+& \sqrt{E{\left[2\eta \left(X\right)-1-{\left(b_{*}^{quad}\right)}^{\top }\tilde{X}\right]}^2},\hfill \end{array}$$

(16)

where ${\tilde{h}}^{quad}$ refers to the density h of ${\left(b_{*}^{quad}\right)}^{\top }\tilde{X}.$

(b) Let ${\eta }_{log}\left(u\right)=1/(1+exp(-u)).$ Then we obtain

$$\begin{array}{ccc}\hfill \mathcal{E}({\widehat{b}}^{log},f_B)& \le & 2P\left({\Omega }^c\right)+4\sigma {\tilde{h}}^{log}c\sqrt{logp}+2/p\hfill \end{array}$$

(17)

$$\begin{array}{cc}+& \sqrt{2E\left[KL\left(\eta \left(X\right),{\eta }_{log}\left({\left(b_{*}^{log}\right)}^{\top }\tilde{X}\right)\right)\right]}\hfill \end{array}$$

(18)

where $KL(·,·)$ is the Kullback–Leibler distance defined in (10) and ${\tilde{h}}^{log}$ refers to the density h of ${\left(b_{*}^{log}\right)}^{\top }\tilde{X}.$ Additionally, assuming that there exists $\delta \in (0,1)$ such that $\delta \le \eta \left(X\right)\le 1-\delta$ and $\delta \le {\eta }_{log}\left({\left(b_{*}^{log}\right)}^{\top }\tilde{X}\right)\le 1-\delta ,$ we have

$$E\left[KL\left(\eta \left(X\right),{\eta }_{log}\left({\left(b_{*}^{log}\right)}^{\top }\tilde{X}\right)\right)\right]\le {\left(2\delta (1-\delta )\right)}^{-1}E{\left[\eta \left(X\right)-{\eta }_{log}\left({\left(b_{*}^{log}\right)}^{\top }\tilde{X}\right)\right]}^2.$$

(19)

In Theorem 2 we establish upper bounds on the excess risks for Lasso estimators (5). They describe predictive properties of these classifiers. In this paper we consider linear classifiers, so the misclassification risk of an estimator is close to the Bayes risk, if the “truth” can be approximated linearly in a satisfactory way. For the classifier with the logistic loss this fact is described by (18) and (19), which measure the distance between true success probability and the one in logistic regression. In particular, when the true model is logistic, then (18) and (19) vanish. The expression (16) relates to the approximation error in the case of the quadratic loss. It measures how well the conditional expectation $E\left[Y\right|X]$ can be described by the “best” (with respect to the loss $\varphi$) linear function ${\left(b_{*}^{quad}\right)}^{\top }\tilde{X}.$

The right-hand sides of (15) and (17) relate to estimation risk. They have been already discussed after Theorem 1. Using subgaussianity of predictors we have made them more explicit. The main ingredient of bounds in Theorem 2, namely $P\left({\Omega }^c\right),$ is studied in the next section.

Results in Theorem 2 refer to Lasso estimators with quadratic and logistic loss functions. Similar results are given in ([2], Theorem 6.4). They refer to the case that the convex excess risk is considered, i.e., the misclassification risks $R(·)$ are replaced by the convex risks $Q(·)$ in (7). Moreover, these results do not consider Lasso estimators with the quadratic loss applied to classification, which is an approach playing a key role in the current paper. Furthermore, in ([2], Theorem 6.4) the estimation error $\widehat{b}-b_{*}$ is measured in the $l_1$-norm, which is enough for prediction. However, for variable selection the $l_{\infty }$-norm gives better results. Such results will be established in Section 4 and Section 5. Finally, results of [2] need more restrictive assumptions than ours. For instance, predictors should be bounded and a function $f_{b_{*}}$ should be sufficiently close to $f_B$ in the supremum norm.

Analogous bounds to those in Theorem 2 can be obtained for other loss functions, if we combine Theorem 1 with results of [17]. Finally, we should stress that the estimator $\widehat{b}$ need not rely on the Lasso method. All we require is that the bound (14) can be estiblished for this estimator.

4. On the Event $\Omega$

In this section we show that probability of the event $\Omega$ can be close to one. Such results for classification models with Lipschitz loss functions were established in [2,13]. Therefore, we focus on the quadratic loss function, which is obviously non-Lipschitz. This loss function is important from the practical point of view, but was not considered in these papers. Moreover, in our results the estimation error in $\Omega$ can be measured in the $l_q$-norms, $q\ge 1,$ not only in the $l_1$-norm as in [2,13]. Bounds in the $l_{\infty }$-norm lead to better results in variable selection, which are given in Section 5.

We start with introducing the cone invertibility factor (CIF), which plays a significant role in investigating properties of estimators based on the Lasso penalty [9]. In the case $n>p$ one usually uses the minimal eigenvalue of the matrix ${\mathbb{X}}^{\top }\mathbb{X}/n$ to express the strength of correlations between predictors. Obviously, in the high-dimensional scenario this value is equal to zero and the minimal eigenvalue needs to be replaced by some other measure of predictors interdependency, which would describe the potential of consistent estimation of model parameters.

For $\xi >1$ we define a cone

$$\mathcal{C}\left(\xi \right)=\{b\in {\mathbb{R}}^{p+1}:|b_{T^c}{|}_1\le \xi \left|b_{\tilde{T}}{|}_1\right\},$$

where we recall that $\tilde{T}=T\cup \left\{0\right\}.$ In the case when $p>>n$ three different characteristics measuring the potential for consistent estimation of the model parameters have been introduced:

-

The restricted eigenvalue [8]:

$$RE\left(\xi \right)=\underset{0\ne b\in \mathcal{C}\left(\xi \right)}{inf}\frac{b^{\top }{\tilde{\mathbb{X}}}^{\top }\tilde{\mathbb{X}}b/n}{{\left|b\right|}_2^2},$$

-

The compatibility factor [7]:

$$K\left(\xi \right)=\underset{0\ne b\in \mathcal{C}\left(\xi \right)}{inf}\frac{\left|T\right|b^{\top }{\tilde{\mathbb{X}}}^{\top }\tilde{\mathbb{X}}b/n}{|b_T{|}_1^2},$$

-

The cone invertibility factor (CIF, [9]): for $q\ge 1$

$${\overline{F}}_q\left(\xi \right)=\underset{0\ne b\in \mathcal{C}\left(\xi \right)}{inf}\frac{{\left|T\right|}^{1/q}{|{\tilde{\mathbb{X}}}^{\top }\tilde{\mathbb{X}}b/n|}_{\infty }}{{\left|b\right|}_q}.$$

In this article we will use CIF, because this factor allows for a sharp formulation of convergency results for all $l_q$ norms with $q\ge 1,$ see ([9], Section 3.2). The population (non-random) version of CIF is given by

$$F_q\left(\xi \right)=\underset{0\ne b\in \mathcal{C}\left(\xi \right)}{inf}\frac{{\left|T\right|}^{1/q}{\left|\tilde{H}b\right|}_{\infty }}{{\left|b\right|}_q},$$

where $\tilde{H}=E\left[\tilde{X}{\tilde{X}}^{\top }\right].$ The key property of the random and the population versions of CIF, ${\overline{F}}_q\left(\xi \right)$ and $F_q\left(\xi \right)$, is that, in contrast to the smallest eigenvalues of matrices ${\tilde{\mathbb{X}}}^{\top }\tilde{\mathbb{X}}/n$ and $\tilde{H},$ they can be close to each other in the high-dimensional setting, see ([30], Lemma 4.1) or ([31], Corollary 10.1). This fact is used in the proof of Theorem 3 (given below).

Next, we state the main results of this section.

Theorem 3.

Let $a\in (0,1),q\ge 1$ and $\xi >1$ be arbitrary. Suppose that Assumption 2 is satisfied and

$$n\ge \frac{K_1{\left|T\right|}^2{\sigma }^4{(1+\xi )}^{2}log(p/a)}{F_q^2\left(\xi \right)}$$

(20)

and

$$\lambda \ge K_2\frac{\xi +1}{\xi -1}{\sigma }^2\sqrt{\frac{log(p/a)}{n}},$$

(21)

where $K_1,K_2$ are universal constants. Then there exists a universal constant $K_3>0$ such that with probability at least $1-K_{3}a$ we have

$$|{\widehat{b}}^{quad}-b_{*}^{quad}{|}_q\le \frac{{2\xi \left|T\right|}^{1/q}\lambda }{(\xi +1)F_q\left(\xi \right)}.$$

(22)

In Theorem 3 we provide the upper bound for the estimation error of the Lasso estimator with the quadratic loss function. This result gives the conditions for estimation consistency of ${\widehat{b}}^{quad}$ in the high-dimensional scenario, i.e., the number of predictors can be significantly greater than the sample size. Indeed, consistency in the $l_{\infty }$-norm holds e.g., when $p=exp\left(n^{a_1}\right),\left|T\right|=n^{a_2},a=exp(-n^{a_1}),$ where $a_1+2a_2<1.$ Moreover, $\lambda$ is taken as the right-hand side of the inequality (21) and finally $F_{\infty }\left(\xi \right)$ is bounded from below (or slowly converging to 0) and $\sigma$ is bounded from above (or slowly diverging to ∞).

The choice of the $\lambda$ parameter is difficult in practice, which is a common drawback of Lasso estimators. However, Theorem 3 gives us a hint how to choose $\lambda$. The “safe” choice of $\lambda$ is the right-hand side of the inequality (21), so, roughly speaking, $\lambda$ should be proportional to $\sqrt{log\left(p\right)/n}.$ In the experimental part of the paper the parameter $\lambda$ is chosen using the cross-validation method. As we will observe it gives satisfatory results for the Lasso estimators in both prediction and variable selection.

Theorem 3 is a crucial fact, which gives the upper bound for (15) in Theorem 2. Namely, taking $q=1,a=1/p$ and $\lambda$ equal to the right-hand side of the inequality (21), we obtain the following consequence of Theorem 3.

Corollary 1.

Suppose that Assumptions 2 is satisfied. Moreover, assume that there exist ${\xi }_0>1$ and constants $C_1>0$ and $C_2<\infty$ such that $F_1\left({\xi }_0\right)\ge C_1$ and $\sigma \le C_2$. If $n\ge K_1{\left|T\right|}^{2}logp,$ then

$$P\left(|{\widehat{b}}^{quad}-b_{*}^{quad}{|}_1\le K_2\left|T\right|\sqrt{\frac{logp}{n}}\right)\ge 1-K_3/p,$$

(23)

where the constants $K_1$ and $K_2$ depend only on ${\xi }_0,C_1,C_2$ and $K_3$ is a universal constant provided in Theorem 3.

The above result works for Lasso estimators with the quadratic loss. In the case of the logistic loss analogous result is obtained in ([13], Theorem 1). In fact, their results relate to the case of quite general Lipschitz loss functions, which can be useful in extending Theorem 2 to such cases.

5. Variable Selection Properties of Estimators

In Section 3 we are interested in predictive properties of estimators. In this part of the paper we focus on variable selection, which is another important problem in high-dimensional statistics. As we have already noticed upper bounds for probability of the event $\Omega$ are crucial in proving results concerning prediction. It also plays a key role in establishing results relating to variable selection. In this section we again focus on the Lasso estimators with the quadratic loss functions. The analogous results for Lipschitz loss functions were considered in ([13], Corollary 1).

In the variable selection problem we want to find significant predictors, which, roughly speaking, give us some information on the observed phenomenon. We consider this problem in the semiparametric model, which is defined in (8). In this case the set of significant predictors is given by (9). As we have already mentioned vectors $\beta$ and $b_{*}^{quad}$ need not be the same. However, in [32] one proved that for a real number $\gamma$ the following relation

$${\left(b_{*}^{quad}\right)}_j=\gamma {\beta }_j,j=1,\dots ,p$$

(24)

holds under Assumption 3, which is now stated.

Assumption 3.

Let $\mathring{\beta }=({\beta }_1,\dots ,{\beta }_p).$ We assume that for each $\theta \in {\mathbb{R}}^p$ the conditional expectation $E\left[{\theta }^{\top }X|{\mathring{\beta }}^{\top }X\right]$ exists and

$$E\left[{\theta }^{\top }X|{\mathring{\beta }}^{\top }X\right]=d_{\theta }{\mathring{\beta }}^{\top }X$$

for a real number $d_{\theta }\in \mathbb{R}.$

The coefficient $\gamma$ in (24) can be easily calculated. Namely, we have

$$\gamma =\frac{E\left[Y{\mathring{\beta }}^{\top }X\right]}{{\mathring{\beta }}^{\top }H\mathring{\beta }}=\frac{2E\left[g\left({\beta }^{\top }\tilde{X}\right){\mathring{\beta }}^{\top }X\right]}{{\mathring{\beta }}^{\top }H\mathring{\beta }}.$$

Standard arguments [33] show that $\gamma$ is nonzero, if g is monotonic. In this case we have that the set $\mathbb{T}$ defined in (9) equals to T defined in (11).

Assumption 3 is a well-known condition in the literature, see e.g., [13,32,34,35,36]. It is always satisfied in the simple regression model (i.e., when $X_1\in \mathbb{R}$), which is often used for initial screening of explanatory variables, see, e.g., [37]. It is also satisfied when X comes from the elliptical distribution, like the multivariate normal distribution or multivariate t-distribution. In the interesting paper [38] one advocates that Assumption 3 is a nonrestrictive condition, when the number of predictors is large, which is the case that we focus on in this paper.

Now, we state the results of this part of the paper. We will use the notation $b_{min}^{quad}={min}_{j\in T}\left|{\left(b_{*}^{quad}\right)}_j\right|.$

Corollary 2.

Suppose that conditions of Theorem 3 are satisfied for $q=\infty .$ If $b_{min}^{quad}\ge \frac{4\xi \lambda }{(\xi +1)F_{\infty }\left(\xi \right)},$ then

$$P\left({\forall }_{j\in T,k\notin T}|{\widehat{b}}_j^{quad}|>|{\widehat{b}}_k^{quad}|\right)\ge 1-K_{3}a,$$

where $K_3$ is the universal constant from Theorem 3.

In Corollary 2 we show that the Lasso estimator with the quadratic loss is able to separate predictors, if the nonzero coefficients of ${\mathring{b}}_{*}^{quad}$ are large enough in absolute values. In the case that T equals (9) (i.e., T is the set of significant predictors) we can prove that the thresholded Lasso estimator is able to find the true model with high-probability. This fact is stated in the next result. The thresholded Lasso estimator is denoted by ${\widehat{b}}_{th}^{quad}$ and defined as

$${\left({\widehat{b}}_{th}^{quad}\right)}_j={\widehat{b}}_j^{quad}\mathbb{I}\left(\right|{\widehat{b}}_j^{quad}|\ge \delta ),j=1,\dots ,p,$$

(25)

where $\delta >0$ is a threshold. We set ${\left({\widehat{b}}_{th}^{quad}\right)}_0={\widehat{b}}_0^{quad}$ and denote ${\widehat{T}}_{th}=\{1\le j\le p:{\left({\widehat{b}}_{th}^{quad}\right)}_j\ne 0\}.$

Corollary 3.

Let g in (8) be monotonic. We suppose that Assumption 3 and conditions of Theorem 3 are satisfied for $q=\infty .$ If

$$\frac{2\xi \lambda }{(\xi +1)F_{\infty }\left(\xi \right)}<\delta \le b_{min}^{quad}/2,$$

then

$$P\left({\widehat{T}}_{th}=\mathbb{T}\right)\ge 1-K_{3}a,$$

where $K_3$ is the universal constant from Theorem 3.

Corollary 3 states that the Lasso estimator after thresholding is able to find the true model with high probability, if the threshold is appropriately chosen. However, Corollary 3 does not give a constructive way of choosing the threshold, because both endpoints of the interval $[\frac{2\xi \lambda }{(\xi +1)F_{\infty }\left(\xi \right)},b_{min}^{quad}/2]$ are unknown. It is not a surprising fact and has been already observed, for instance, in linear models ([9], Theorem 8). In the literature we can find methods, which help to choose a threshold in practice, for instance the approach relying on information criteria developed in [39,40].

Finally, we discuss the condition of Corollary 3 that $b_{min}^{quad}$ cannot be too small, i.e., $b_{min}^{quad}\ge \frac{4\xi \lambda }{(\xi +1)F_{\infty }\left(\xi \right)}$. We know that ${\left(b_{*}^{quad}\right)}_j=\gamma {\beta }_j$ for $j=1,\dots ,p,$ so the considered condition requires that

$$\underset{j\in \mathbb{T}}{min}\left|{\beta }_j\right|\ge \frac{4\xi \lambda }{\left|\gamma \right|(\xi +1)F_{\infty }\left(\xi \right)}.$$

(26)

Compared to the similar condition for the Lasso estimators in the well-specified models, we observe that the denominator in (26) contains an additional factor $\left|\gamma \right|$. This number is usually smaller than one, which means that in the misspecified models the Lasso estimator needs larger sample size to work well. This phenomenon is typical for misspecified models and the similar restrictions hold for competitors [13].

6. Numerical Experiments

In this section we present simulation study, where we compare the accuracy of considered estimators in prediction and variable selection.

We consider the model (8) with predictors generated from the p-dimensional normal distribution $N(0,H)$, where $H_{jj}=1$ and $H_{jk}=0.5$ for $j\ne k$. The true parameter is

$$\beta =(1,\underset{10}{\underbrace{\pm 1,\pm 1,\dots ,\pm 1}},0,0,\dots ,0),$$

(27)

where signs are chosen at random. The first coordinate in (27) corresponds to the intercept and the next ten coefficients relate to significant predictors in the model. We study two cases:

-

Scenario 1: $g\left(x\right)=exp\left(x\right)/(1+exp(x\left)\right);$

-

Scenario 2: $g\left(x\right)=arctan\left(x\right)/\pi +0.5.$

In each scenario we generate the data $(X_1,Y_1),\dots ,(X_n,Y_n)$ for $n\in \{100,350,600\}.$ The corresponding numbers of predictors are $p\in \{100,1225,3600\},$ so the number of predictors exceeds significantly the sample size in the experiments. For every model we consider two Lasso estimators with unpenalized intercepts (5): the first one with the logistic loss and the second one with the quadratic loss. They are denoted by “logistic” and “quadratic”, respectively. To calculate them we use the “glmnet” package [28] in the “R” software [41]. The tuning parameters $\lambda$ are chosen on the basis of 10-fold cross-validation.

Observe that applying the Lasso estimator with the logistic loss function to Scenario 1 leads to a well-specified model, while using the quadratic loss implies misspecification. In Scenario 2 both estimators work in misspecified models.

Simulations for each scenario are repeated 300 times.

To describe the quality of estimators in variable selection we calculate two values:

-

TD—the number of correctly selected relevant predictors;

-

sep—the number of relevant predictors, whose Lasso coefficients are greater in absolute value than the largest in absolute value Lasso coefficient corresponding to irrelevant predictors.

So, we want to confirm that the considered estimators are able to separate predictors, which we establish in Section 5. Using TD we also study “screening” properties of estimators, which are easier than separability.

The classification accuracy of estimators is measured in the following way: we generate a test sample containg 1000 objects. On this set we calculate

-

pred—the fraction of correctly predicted classes of objects for each estimator.

The results of experiments are collected in Table 1 and Table 2. By the “oracle” we mean the classifier, which works only with significant predictors and uses the function g from the true model (8) in the estimation process.

Table 1. Results for Scenario 1.

$\mathit{n}=100$	Quadratic	Logistic	Oracle
TD	6.3	6.1
sep	2.2	2.3
pred	0.734	0.736	0.810
$n=350$
TD	9.3	9.5
sep	6.0	6.3
pred	0.774	0.779	0.831
$n=600$
TD	9.8	9.9
sep	8.6	8.9
pred	0.791	0.795	0.832

Table 2. Results for Scenario 2.

$\mathit{n}=100$	Quadratic	Logistic	Oracle
TD	4.8	4.6
sep	1.4	1.4
pred	0.697	0.698	0.768
$n=350$
TD	8.1	8.2
sep	3.9	3.9
pred	0.730	0.731	0.805
$n=600$
TD	9.4	9.4
sep	6.8	6.9
pred	0.750	0.752	0.809

Finally, we also compare execution time of both algorithms. In Table 3 we show the averaged relative time difference

$$\frac{t^{log}-t^{quad}}{t^{quad}},$$

(28)

where $t^{quad}$ and $t^{log}$ is time of calculating Lasso with quadratic and logistic loss functions, respectively.

Table 3. Relative time difference (28) of algorithms.

	Scenario 1	Scenario 2
$n=350$	0.02	0.06
$n=600$	0.11	0.13

Looking at the results of experiments we observe that both estimators perform in a satisfactory way. Their predictive accuracy is relatively close to the oracle, especially when the sample size is larger. In variable selection we see that both estimators are able to find significant predictors and separate predictors in both scenarios. Again we can notice that properties of estimators become better, when n increases.

In Scenario 2 the quality of both estimators in prediction and variable selection is comparable. In Scenario 1, which is well-specified for Lasso with the logistic loss, we observe its dominance over Lasso with the quadratic loss. However, this dominance is not large. Therefore, using Lasso with the quadratic loss we obtain slightly worse accuracy of the procedure, but this algorithm is computationally faster. The computational efficiency is especially important, when we study large data sets. As we can see in Table 3 execution time of estimators is almost the same for $n=350$, but for $n=600$ the relative time difference becomes greater than $10\%.$