Sparse Multicategory Generalized Distance Weighted Discrimination in Ultra-High Dimensions

Abstract

Distance weighted discrimination (DWD) is an appealing classification method that is capable of overcoming data piling problems in high-dimensional settings. Especially when various sparsity structures are assumed in these settings, variable selection in multicategory classification poses great challenges. In this paper, we propose a multicategory generalized DWD (MgDWD) method that maintains intrinsic variable group structures during selection using a sparse group lasso penalty. Theoretically, we derive minimizer uniqueness for the penalized MgDWD loss function and consistency properties for the proposed classifier. We further develop an efficient algorithm based on the proximal operator to solve the optimization problem. The performance of MgDWD is evaluated using finite sample simulations and miRNA data from an HIV study.

1. Introduction

Classification problems appear in diverse practical applications, such as spam e-mail classification, disease diagnosis and drug discovery, among many others (e.g., [1,2,3]). In these classification problems, the goal is to predict class labels based on a given set of variables. Recent research has focused extensively on linear classification: see [4,5] for comprehensive introductions. Among many linear classification methods, support vector machines (SVMs) (see [6,7]) and distance-weighted discrimination (DWD) (see [8,9,10]) are two commonly used large-margin based classification methods.

Owing to the recent advent of new technologies for data acquisition and storage, classification with high dimensional features, i.e., a large number of variables, has become a ubiquitous problem in both theoretical and applied scientific studies. Typically, only a small number of instances are available, a setting we refer to as high-dimensional, low-sample size (HDLSS), as in [11]. In the HDLSS setting, a so-called “data-piling” phenomenon is observed in [8] for SVMs, occurring when projections of many training instances onto a vector normal to the separating hyperplane are nearly identical, suggesting severe overfitting. DWD was originally proposed to overcome data-piling in the HDLSS setting. In binary classification problems, linear SVMs seek a hyperplane maximizing the smallest margin for all data points, while DWD seeks a hyperplane minimizing the sum of inverse margins over all data points. Reference [8] suggests replacing the inverse margins by the q-th power of the inverse margins in a generalized DWD method; see [12] for a detailed description. Formally, for a training data set ${\left\{(y_i,{\mathit{X}}_i)\right\}}_{i=1}^N$ of N observations, where ${\mathit{X}}_i\in {\mathbb{R}}^p$ and $y_i\in \{-1,1\}$, binary generalized linear DWD seeks a proper separating hyperplane $\{\mathit{X}:a+{\mathit{X}}^{\top }\mathit{b}=0\}$ through the optimization problem

$$\begin{array}{cc}\hfill \underset{a,\mathit{b}}{argmax}& \sum _{i=1}^N\frac{1}{d_i^q}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.d_i& =y_i\left(a+{\mathbf{X}}_i^T\mathit{b}\right)+{\eta }_i\ge 0,\forall i,\hfill \\ \hfill {\eta }_i& \ge 0,\forall i,\sum _i{\eta }_i\le c,\hfill \\ \hfill {\parallel \mathit{b}\parallel }_2^2& =1,\hfill \end{array}$$

(1)

where a and $\mathit{b}$ are the intercept and slope parameters, respectively. The slack variable ${\eta }_i$ is introduced to ensure that the corresponding margin $d_i$ is non-negative and the constant $c>0$ is a tuning parameter to control the overlap between classes. Problem (1) can also be written in a loss-plus-penalty form (e.g., [12]) as

$$\begin{array}{c}\hfill (\widehat{a},\widehat{\mathit{b}})=\underset{a,\mathit{b}}{argmin}\left[\frac{1}{N}\sum _{i=1}^N{\varphi }_q\left\{y_i\left(a+{\mathit{X}}_i^{\top }\mathit{b}\right)\right\}+\lambda {\parallel \mathit{b}\parallel }_2^2\right],\end{array}$$

(2)

where

$$\begin{array}{c}\hfill {\varphi }_q\left(u\right)=\left\{\begin{array}{cc}1-u,\hfill & \mathrm{if}u\le Q\hfill \\ {\phi }_q\left(u\right),\hfill & \mathrm{if}u>Q,\hfill \end{array}\right.\end{array}$$

(3)

with $Q=\frac{q}{q+1}$, $q>0$ and ${\phi }_q\left(u\right)=(1-Q){\left(Q{u}^{-1}\right)}^q$. When $q=1$, (1) becomes the standard DWD problem in [8] while problem (2) appears in [9,13].

The binary classification problem (1) is well studied. However, in many applications such as image classification [1], cancer diagnosis [2] and speech recognition [3], to name a few, problems with more than two categories are commonplace. To solve these multicategory problems with the DWD classifier, approaches based on either formulation (1) or (2) are common. One common strategy is to extend problem (1) to multiple classes by solving a series of binary problems in a one-versus-one (OVO) or one-versus-rest (OVR) method (e.g., [14]). Instead of reducing the multicategory problem to a binary one, another strategy based on problem (1) considers all classes at once. As shown in [14], this approach generally works better than the OVO and OVR methods. Based on an extension of problem (2), [15] proposes multicategory DWD, written in a loss-plus-penalty form as

$$\begin{array}{c}\hfill \underset{a_k,b_k}{min}\frac{1}{N}\sum _{i=1}^N{\varphi }_q\left(a_{y_i}+{\mathit{X}}_i^{\top }{\mathit{b}}_{y_i}\right)+\lambda \sum _{k=1}^K{\parallel {\mathit{b}}_k\parallel }_2^2\\ \hfill \mathrm{s}.\mathrm{t}.\sum _{k=1}^K{a}_k=0;\sum _{k=1}^K{b}_{jk}=0,\forall j=1,\dots ,p,\end{array}$$

(4)

with $y_i,k\in \{1,\dots ,K\}$ and where $a_k$ and ${\mathit{b}}_k=(b_{1k},\cdots ,b_{pk})$ are the intercept and slope parameters for each category k, respectively. Although these methods can be applied to multicategory classification in the HDLSS setting, both problems (2) and (4) use the $L_2$ penalty and do not perform feature selection. As discussed in [16], for high dimensional classification, taking all features into consideration does not work well for two reasons. First, based on prior knowledge, only a small number of variables are relevant to the classification problem: a good classifier in high dimensions should have the ability to sparsely select important variables and discard redundant ones. Second, classifiers using all available variables in high-dimensional settings may have poor classification performance.

Much of the SVM literature has considered variable selection in high-dimensional classification problems to improve performance (e.g., [17,18,19]). Among the DWD literature, to the best of our knowledge, only [16] considered variables selection and classification simultaneously. Wang and Zou [16] considered an $L_1$ rather than an $L_2$ penalty in problem (2) to improve interpretability through sparsity in the binary classification. Moreover, [16] made selections based on the strengths of input variables within individual classes but ignored the strengths of input variable groupings, thereby selecting more factors than necessary for each class. To overcome this weakness in this paper, we developed a multicategory generalized DWD method that is capable of performing variable selection and classification simultaneously. Our approach incorporates sparsity and group structure information via the sparse group lasso penalty (see [20,21,22,23,24]).

Although DWD is well studied, it is less popular than the SVM for binary classification, arguably for computational and theoretical reasons. For an up-to-date list of works on DWD mostly focused on the $q=1$ case, see [14,15]. Theoretical asymptotic properties of large-margin classifiers in high dimensional settings were studied in [25], and [26] derived an expression for asymptotic generalization error. In terms of computation, [8] solved the standard DWD problem in (1) as a second-order cone programming (SOCP) problem using a primal-dual interior-point method that is computationally expensive when N or p is large. To overcome computational bottlenecks, [12] proposed an approach based on a novel formulation of the primal DWD model in (1): this method, proposed in [12], does not scale to large data sets and requires further work. Lam et al. [27] designed a new algorithm for large DWD problems with $q\ge 2$ and $K=2$ based on convergent multi-block ADMM-type methods (see [28]). Wang and Zou [16] solved the lasso-penalized binary DWD problem by combining majorization–minimization and coordinate descent methods: the lasso penalty does not directly permit a SOCP solution. In fact, solution identifiability in the generalized DWD problem with $q>1$ requires more constraints and remains an open research problem (see [8]). To the best of our knowledge, no work focusing on computational aspects of lasso penalized multicategory generalized DWD (MgDWD) exists. The same holds for sparse group lasso-penalized MgDWD.

The theoretical and computational contributions of this paper are as follows. First, we establish the uniqueness of the minimizer in the population form of the MgDWD problem. Second, we prove a non-asymptotic $L_2$ estimation error bound for the sparse group lasso-regularized MgDWD loss function in the ultra-high dimensional setting under mild regularity conditions. Third, we develop a fast, efficient algorithm able to solve the sparse group lasso-penalized MgDWD problem using proximal methods.

The rest of this paper is organized as follows. In Section 2.1, we introduce the MgDWD problem with sparse group lasso penalty. In Section 2.2 and Section 2.3, we establish theoretical properties of the population classifier and regularized empirical loss. We propose a computational algorithm in Section 2.4. Section 3 illustrates the finite sample performance of our method through simulation studies and a real data analysis. Proofs for major theorems are given in the Appendix A.

2. Methodology

2.1. Model Setup

We begin with some basic set-up and notation. Consider the multicategory classification problem for a random sample ${\left\{(y_i,{\mathit{X}}_i)\right\}}_{i=1}^N$ of N independent and identically distributed (i.i.d.) observations from some distribution $\mathbb{P}(y,\mathit{X})$. Here, y is the categorical response taking values in $\mathcal{Y}=\{1,\dots ,K\}$, and $\mathit{X}={(x_1,\dots ,x_p)}^{\top }\in \mathcal{X}\subset {\mathbb{R}}^p$ is the covariate vector. We wish to obtain a proper separating hyperplane $\{\mathit{X}\in \mathcal{X}|a_k+{\mathit{X}}^{\top }{\mathit{b}}_k=0\}$ for each category $k\in \mathcal{Y}$, where $a_k$ and ${\mathit{b}}_k={(b_{1k},\dots ,b_{pk})}^{\top }$ are intercept and slope parameters, respectively.

In this paper, we consider MgDWD with sparse group lasso regularization. That is, we estimate a classification boundary by solving the constrained optimization problem

$$\begin{array}{c}{\displaystyle \underset{a_k,{\mathit{b}}_k}{min}}{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\varphi }_q\left(a_{y_i}+{\mathit{X}}_i^{\top }{\mathit{b}}_{y_i}\right)+{\lambda }_1{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{j=1}^p}\left|b_{jk}\right|+{\lambda }_2{\displaystyle \sum _{j=1}^p}\sqrt{{\displaystyle \sum _{k=1}^K}{b}_{jk}^2}\\ s.t.{\displaystyle \sum _{k=1}^K}{a}_k=0;{\displaystyle \sum _{k=1}^K}{b}_{jk}=0,\forall j=1,\dots ,p,\end{array}$$

(5)

where ${\varphi }_q$ is as defined in (3).

To approach this problem, we apply the concept of a “margin vector” to extend the definition of a (binary) margin to the multicategory case. Denote the margin vector of an observation ${\mathit{X}}_i$ as ${\mathit{F}}_i={(f_{i1},\dots ,f_{iK})}^{\top }$, with $f_{ik}=a_k+{\mathit{X}}_i^{\top }{\mathit{b}}_k$ satisfying ${\sum }_{k=1}^K{f}_{ik}=0$. Let ${\mathit{E}}_i={(e_{i1},\dots ,e_{iK})}^{\top }$ be the class indicator vector with $e_{ik}=\mathbb{𝟙}\{y_i=k\}$. The multicategory margin of the data point $(y_i,{\mathit{X}}_{\mathit{i}})$ is then given as $f_{i{y}_i}=a_{y_i}+{\mathit{X}}_i^{\top }{\mathit{b}}_{y_i}={\mathit{E}}_i^{\top }{\mathit{F}}_i$. Therefore, the MgDWD loss can be rewritten as

$$\begin{array}{c}\hfill {\varphi }_q(a_{y_i}+{\mathit{X}}_i^{\top }{\mathit{b}}_{y_i})={\varphi }_q\left({\mathit{E}}_i^{\top }{\mathit{F}}_i\right)={\mathit{E}}_i^{\top }{\varphi }_q\left({\mathit{F}}_i\right)=\sum _{k=1}^K\mathbb{𝟙}\{y_i=k\}{\varphi }_q(a_k+{\mathit{X}}_i^{\top }{\mathit{b}}_k).\end{array}$$

(6)

Based on (6), Lemma 1 describes the Fisher consistency of the MgDWD loss.

Lemma 1.

Given $\mathit{X}=\mathit{u}$, the minimizer of the conditional expectation of (6) is $\tilde{\mathit{F}}(\mathit{u})={\left({\tilde{f}}_1\left(\mathit{u}\right),\dots ,{\tilde{f}}_K\left(\mathit{u}\right)\right)}^{\top }$, satisfying

$$\begin{array}{c}\hfill \underset{k\in \mathcal{Y}}{\mathrm{argmax}}{\tilde{f}}_k\left(\mathit{u}\right)=\underset{k\in \mathcal{Y}}{\mathrm{argmax}}Pr\{y=k|\mathit{X}=\mathit{u}\},\end{array}$$

where

$$\begin{array}{c}\hfill {\tilde{f}}_k\left(\mathit{u}\right)=\left\{\begin{array}{cc}{\displaystyle Q\sqrt[q]{{\displaystyle \frac{Pr\{y=k|\mathit{X}=\mathit{u}\}}{Pr\{y=k_{\ast }|\mathit{X}=\mathit{u}\}}}},}\hfill & k\ne k_{\ast }\hfill \\ {\displaystyle -Q\sum _{l\ne k_{\ast }}\sqrt[q]{{\displaystyle \frac{Pr\{y=l|\mathit{X}=\mathit{u}\}}{Pr\{y=k_{\ast }|\mathit{X}=\mathit{u}\}}}},}\hfill & k=k_{\ast }.\hfill \end{array}\right.\end{array}$$

and $k_{\ast }=\underset{k\in \mathcal{Y}}{\mathrm{argmin}}Pr\{y=k|\mathit{X}=\mathit{u}\}$.

Consequently, ${\tilde{f}}_k\left(\mathit{u}\right)$ can be treated as an effective proxy of $Pr\{y=k|\mathit{X}=\mathit{u}\}$ and, for any new observation ${\mathit{X}}_{\ast }$, a reasonable prediction of its label $y_{\ast }$ is

$$\begin{array}{c}\hfill {\widehat{y}}_{\ast }=\underset{k\in \mathcal{Y}}{\mathrm{argmax}}\{a_k+{\mathit{X}}_{\ast }^{\top }{\mathit{b}}_k\}.\end{array}$$

Speaking to the sparse group lasso (SGL) regularization in (5), the $L_1$ penalty encourages an element-wise sparse estimator that selects important variables for each category, indicated by ${\widehat{b}}_{jk}\ne 0$. Assuming that parameters in different categories share the same information, we use an $L_2$ penalty to encourage a group-wise sparsity structure that removes covariates that are irrelevant across all categories, that is, where ${\widehat{\mathit{\beta }}}_j={(b_{1j},\dots ,b_{Kj})}^{\top }=\mathbf{0}$. Specifically, let ${\mathit{x}}_j={(x_{1j},\cdots ,x_{Nj})}^T$ and $\mathbf{B}=\left(b_{jk}\right)\in {\mathbb{R}}_{jk}^{p\times K}$, where the k-th column ${\mathit{b}}_k$ is the slope vector for the category label k and the j-th row ${\mathit{\beta }}_j^{\top }$ is the group coefficient for the variable ${\mathit{x}}_j$. If ${\mathit{x}}_j$ is noise in the classification problem or is not relevant to category label k, then the entry $b_{jk}$ of $\mathbf{B}$ should be shrunk to exactly zero. The SGL penalty of (5) can be written as a convex combination of the lasso and group lasso penalties in terms of ${\mathit{\beta }}_j$ as

$$\begin{array}{c}\hfill {\lambda }_1\sum _{k=1}^K\sum _{j=1}^p\left|b_{jk}\right|+{\lambda }_2\sum _{j=1}^p\sqrt{\sum _{k=1}^K{b}_{jk}^2}=\lambda \sum _{j=1}^p\left\{\tau {\parallel {\mathit{\beta }}_j\parallel }_1+(1-\tau ){\parallel {\mathit{\beta }}_j\parallel }_2\right\},\end{array}$$

(7)

where $\lambda >0$ is the scale of the penalty and $\tau \in [0,1]$ tunes the propensity between the element-wise and group-wise sparsity structure.

2.2. Population MgDWD

In this subsection, some basic results pertaining to unpenalized population MgDWD are given. These results are necessary for further theoretical analysis.

Denote the marginal probability mass of y as $Pr(y=k)={\pi }_k$ with ${\pi }_k>0$ and ${\sum }_{k=1}^K{\pi }_k=1$, and the conditional probability density functions of $\mathit{X}$ given $y=k$ by $g(\mathit{X}\mid y=k)=g_k\left(\mathit{X}\right)$. Let $\mathsf{\Theta }=({\mathbf{\theta }}_1,\dots ,{\mathbf{\theta }}_K)$ be the collection of coefficient vectors ${\mathbf{\theta }}_k={(a_k,{\mathit{b}}_k^{\top })}^{\top }$ for all labels and $\mathit{Z}={(1,{\mathit{X}}^{\top })}^{\top }$. The population version of the MgDWD problem in (6) is

$$\begin{array}{c}\hfill \mathcal{L}\left(\mathbf{\vartheta }\right)=\mathbb{E}\left\{\mathbb{I}{\left(\mathcal{Y}\right)}^{\top }{\varphi }_q\left({\mathsf{\Theta }}^{\top }\mathit{Z}\right)\right\}=\sum _{k=1}^K{\pi }_k{\int }_{\mathcal{X}}{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)g_k\left(\mathit{x}\right)\mathrm{d}\mathit{x},\end{array}$$

(8)

where $\mathbf{\vartheta }=\mathrm{vec}\left\{\mathsf{\Theta }\right\}$ is the vectorization of the matrix $\mathsf{\Theta }$ and $\mathbb{I}\left(\mathcal{Y}\right)={(\mathbb{𝟙}\{y=1\},\dots ,\mathbb{𝟙}\{y=K\})}^{\top }$ is a random vector. Denote the true parameter value ${\mathbf{\vartheta }}^{\ast }$ as a minimizer of the population MgDWD problem, namely,

$$\begin{array}{c}\hfill {\mathbf{\vartheta }}^{\ast }\in \underset{\mathbf{\vartheta }\in \mathcal{C}}{\mathrm{argmin}}\mathcal{L}\left(\mathbf{\vartheta }\right),\end{array}$$

where $\mathcal{C}=\left\{\mathbf{\vartheta }\in {\mathbb{R}}^{K(p+1)}|\mathbf{C}\mathbf{\vartheta }={\mathbf{0}}_K\right\}$ is the set of sum-constrained $\mathbf{\vartheta }$ with $\mathbf{C}={\mathbf{1}}_K^{\top }\otimes {\mathbf{I}}_{p+1}$, where ⊗ denotes the Kronecker product.

To facilitate our theoretical analysis, we first define the gradient vector and Hessian matrix of the population MgDWD loss function. We then introduce some regularity conditions necessary to derive theoretical properties of this problem. Let $\mathrm{diag}\left\{\mathit{v}\right\}$ be a diagonal matrix constructed from the vector $\mathit{v}$, and let ∘ and ⊕ be the Hadamard product and the direct matrix sum, respectively. Denote the gradient vector of the population MgDWD loss function (8) as

$$\begin{array}{c}\hfill \mathcal{S}\left(\mathbf{\vartheta }\right)=\mathbb{E}\left(\{\mathbb{I}\left(\mathcal{Y}\right)\circ {\varphi }_q^{\prime }\left({\mathsf{\Theta }}^{\top }\mathit{Z}\right)\}\otimes \mathit{Z}\right)=\mathrm{vec}\left({\mathit{S}}_1,\cdots ,{\mathit{S}}_K\right),\end{array}$$

with

$$\begin{array}{c}\hfill {\mathit{S}}_k=\mathbb{E}\left\{\mathbb{𝟙}\{y=k\}{\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\mathit{Z}\right\}={\pi }_k{\int }_{\mathcal{X}}{\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\theta }_k\right)\mathit{Z}{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X},\end{array}$$

and its Hessian matrix as

$$\begin{array}{c}\hfill \mathcal{H}\left(\mathbf{\vartheta }\right)=\mathbb{E}\left\{\mathrm{diag}\left\{\mathbb{I}(\mathcal{Y},\mathcal{X})\circ {\phi }_q^{″}\left({\mathsf{\Theta }}^{\top }\mathit{Z}\right)\right\}\otimes \left({\mathit{ZZ}}^{\top }\right)\right\}=\underset{k=1}{\overset{K}{⨁}}{\mathit{H}}_k,\end{array}$$

where ${\phi }_q^{″}$ denotes the second derivative of the function ${\phi }_q$; $\mathbb{I}(\mathcal{Y},\mathcal{X})=\mathbb{I}\left(\mathcal{Y}\right)\circ \mathbb{I}\left(\mathcal{X}\right)$ is a random vector with $\mathbb{I}\left(\mathcal{X}\right)={(\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_1\},\dots ,\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k\})}^{\top }$ and ${\mathcal{X}}_k=\left\{\mathit{X}\in \mathcal{X}|{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\right\}$; and

$$\begin{array}{c}\hfill {\mathit{H}}_k=\mathbb{E}\left\{\mathbb{𝟙}\{y=k,\mathit{X}\in {\mathcal{X}}_k\}{\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right){\mathit{ZZ}}^{\top }\right\}={\pi }_k{\int }_{{\mathcal{X}}_k}{\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right){\mathit{ZZ}}^{\top }{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}.\end{array}$$

The block structure of $\mathcal{H}\left(\mathbf{\vartheta }\right)$ implies a parallel relationship between each category. The relationship between the ${\mathbf{\theta }}_k$ is reflected by the sum-to-zero constraint in the definition of $\mathcal{C}$.

We assume the following regularity conditions.

(C1) The densities of $\mathit{X}$ given $y=k\in \mathcal{Y}$, i.e., the $g_k\left(\mathit{X}\right)$, are continuous and have finite second moments.

(C2) $0<Pr\{\mathit{X}\in {\mathcal{X}}_k^{\ast }|y=k\}<1$ for all $k\in \mathcal{Y}$, where ${\mathcal{X}}_k^{\ast }=\left\{\mathit{X}\in \mathcal{X}|{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>Q\right\}$.

(C3) $\mathrm{Var}\left\{\mathit{X}|\mathit{X}\in {\mathcal{X}}_k^{\ast },y=k\right\}\succ {\mathbf{O}}_p$ for all $k\in \mathcal{Y}$.

Remark 1.

Condition (C1) ensures that $\mathcal{L}$, $\mathcal{S}$ and $\mathcal{H}$ are well defined and continuous in $\mathbf{\vartheta }$. For the theoretically optimal hyperplane $\{\mathit{X}\in \mathcal{X}|{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }=0\}$, the case with ${\mathbf{\theta }}_k^{\ast }={\mathbf{0}}_{p+1}$ leaves $\mathcal{X}$ useless for classification. On the other hand, when $a_k^{\ast }\ne 0$ and ${\mathit{b}}_k^{\ast }={\mathbf{0}}_p$, the hyperplane is the empty set and is similarly meaningless. Condition (C2) is proposed to avoid the case where ${\mathit{b}}_k^{\ast }={\mathbf{0}}_p$ so that ${\mathbf{\vartheta }}^{\ast }$ always contains information relevant to the classification problem. For bounded random variables, condition (C2) should be assumed with caution. Condition (C3) implies the positive definiteness of $\mathcal{H}\left({\mathbf{\vartheta }}^{\ast }\right)$.

By convexity and the second-order Lagrange condition, the following theorem shows that the local minimizer of the population MgDWD problem exists and is unique.

Theorem 1.

Under the regularity conditions (C1)–(C3), the true parameter ${\mathbf{\vartheta }}^{\ast }\in \mathcal{C}$ is the unique minimizer of $\mathcal{L}\left(\mathbf{\vartheta }\right)$ with ${\mathit{b}}_k^{\ast }\ne {\mathbf{0}}_p$, and

$$\begin{array}{c}\hfill \mathcal{L}\left({\mathbf{\vartheta }}^{\ast }\right)=\sum _{k=1}^{K}A(k,q){\pi }_k,\end{array}$$

with $0\le u(k,q)\le A(k,q)\le v(k,q)\le 1$, where

$$\begin{array}{c}A(k,q)=1-\mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q\right\}|y=k\right\},\\ u(k,q)=Pr\left\{\mathit{X}\notin {\mathcal{X}}_k^{\ast }|y=k\right\}+Q^{2q}Pr\left\{Q<{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\le Q^{-1}|y=k\right\},\\ v(k,q)=Pr\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\le 1|y=k\right\}+\underset{ϵ>0}{inf}{\left({\displaystyle \frac{Q}{1+ϵ}}\right)}^{q}Pr\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>1+ϵ|y=k\right\}.\end{array}$$

The bounds in Theorem 1 show how q affects the loss function $\mathcal{L}\left({\mathbf{\vartheta }}^{\ast }\right)$. The upper bound $v(k,q)$ is a decreasing function of q with

$$\begin{array}{c}\underset{q\to 0}{lim}v(k,q)=1\mathrm{and}\underset{q\to \infty }{lim}v(k,q)=Pr\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\le 1|y=k\right\}.\end{array}$$

In the lower bound $u(k,q)$, the first term $Pr\left\{\mathit{X}\notin {\mathcal{X}}_k^{\ast }|y=k\right\}$ is an increasing function of q and the last term $Q^{2q}Pr\left\{Q<{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\le Q^{-1}|y=k\right\}$ is a decreasing function of q, with

$$\begin{array}{c}\hfill \underset{q\to 0}{lim}u(k,q)=1\mathrm{and}\underset{q\to \infty }{lim}u(k,q)=Pr\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\le 1|y=k\right\}.\end{array}$$

Consequently, for the given population $\mathbb{P}(y,\mathit{X})$, a larger q encourages the population MgDWD estimator to focus more on the regions $\{\mathit{X}\notin {\mathcal{X}}_k,y=k\}$ that correspond to misclassifications. As a result, the estimator’s performance will be similar to the hinge loss as $q\to \infty$. Setting q too small will lead to an ineffective classifier due to the unreasonable penalty placed on the well classified region $\{\mathit{X}\in {\mathcal{X}}_k,y=k\}$. This variation in the lower bound with respect to q provides a necessary condition for the existence of an optimal q.

Remark 2.

The explicit relationship between q and ${\mathbf{\vartheta }}^{\ast }$ is complicated. While it may be more desirable to prove that a greater value of q results in a smaller value of the loss function $\mathcal{L}\left(\mathbf{\vartheta }\right)$, there is no explicit formula for the optimal value ${\mathbf{\vartheta }}^{\ast }$ in terms of q.

2.3. Estimator Consistency

Under the unpenalized framework presented in the previous subsection, all covariates will contribute to the classification task for each category: this scenario may lead to a classifier that overfits to the training data set. In this subsection, we study the consistency of the estimator for (5) in ultra-high dimensional settings.

To achieve structural sparsity in the estimator, the regularization parameter $\lambda$ in (7) must be large enough to dominate the gradient of the empirical MgDWD loss evaluated at the theoretical minimizer ${\mathbf{\vartheta }}^{\ast }=\mathrm{vec}\left\{{\mathsf{\Theta }}^{\ast }\right\}$ with high probability. On the other hand, $\lambda$ should also be as small as possible to reduce the bias incurred by the SGL regularization term

$$\begin{array}{c}\hfill P\left(\mathit{\beta }\right)=\sum _{j=1}^p\tau {\parallel {\mathit{\beta }}_j\parallel }_1+(1-\tau ){\parallel {\mathit{\beta }}_j\parallel }_2.\end{array}$$

Lemma 2 provides a suitable choice of $\lambda$ under the following assumption.

(A1) The predictors $\mathit{X}=(x_1,\dots ,x_p)\in {\mathbb{R}}^p$ are independent sub-Gaussian random vectors satisfying $\mathbb{E}\mathit{X}={\mathbf{0}}_p$, and where $\mathrm{Var}\left(\mathit{X}\right)=\mathsf{\Sigma }$, there exists a constant $\kappa >0$ such that for any $\mathbf{\gamma }\in {\mathbb{R}}^p$, $\mathbb{E}exp\left({\mathbf{\gamma }}^{\top }{\mathsf{\Sigma }}^{-1/2}\mathit{X}\right)\le exp({\parallel \mathbf{\gamma }\parallel }_2^2{\kappa }^2/2)$. From here on, we define ${\varsigma }_1^2$ as the largest eigenvalue of $\mathsf{\Sigma }$.

Lemma 2.

Denote $\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)=({\mathbf{I}}_K\otimes {\mathbf{Z}}^{\top })\mathrm{diag}\left(\mathrm{vec}\left\{\mathbf{E}\right\}\right)\mathrm{vec}\left\{{\varphi }_q^{\prime }\left(\mathbf{Z}{\mathsf{\Theta }}^{\ast }\right)\right\}$, where $\mathbf{E}={\left({\mathit{E}}_1,\dots ,{\mathit{E}}_N\right)}^{\top }$, $\mathbf{Z}={\left({\mathit{Z}}_1,\dots ,{\mathit{Z}}_N\right)}^{\top }$ with ${\mathit{Z}}_i={(1,{\mathit{X}}_i^{\top })}^{\top }$, and ${\mathbf{I}}_K$ is the identity matrix of size K. Under condition (A1),

$$\begin{array}{c}\hfill \tilde{P}\left\{\mathbf{P}\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)\right\}\le \tau {\mathsf{\Lambda }}_1+(1-\tau ){\mathsf{\Lambda }}_2\end{array}$$

with probability at least $1-2{\left(Kp\right)}^{1-c_1^2}-p^{1-c_2^2}$, where

$$\begin{array}{c}\mathbf{P}=({\mathbf{I}}_K-K^{-1}{\mathbf{1}}_K{\mathbf{1}}_K^{\top })\otimes {\mathbf{I}}_{p+1},\\ {\mathsf{\Lambda }}_1=max\{{\varsigma }_1\kappa ,1\}\left(1-{\displaystyle \frac{1}{K}}\right)\sqrt{{\displaystyle \frac{2log\left(pK\right)}{N}}},\\ {\mathsf{\Lambda }}_2=max\{2\sqrt{2}{\varsigma }_1\kappa ,1\}\left\{c_2\sqrt{\left(1-{\displaystyle \frac{1}{K}}\right){\displaystyle \frac{2log\left(p\right)}{N}}}+\sqrt{{\displaystyle \frac{K-1}{N}}}\right\},\end{array}$$

for constants $c_1,c_2>1$.

It is difficult to obtain a closed form for the conjugate of the SGL penalty, say, $\overline{P}\left(\mathit{v}\right)={sup}_{\mathit{u}\in \mathcal{C}\backslash \left\{0\right\}}\frac{⟨\mathit{u},\mathit{v}⟩}{P\left(\mathit{u}\right)}$. Instead, we use a regularized upper bound $\tilde{P}\left(\mathit{v}\right)\ge \overline{P}\left(\mathit{v}\right)$. Based on Lemma 2, we propose a theoretical tuning parameter value

$$\begin{array}{c}\hfill \lambda =c_0\sqrt{{\displaystyle \frac{log\left(pK\right)}{N}}},\end{array}$$

(9)

where $c_0$ is some given constant satisfying $\lambda >\tau {\mathsf{\Lambda }}_1+(1-\tau ){\mathsf{\Lambda }}_2$.

Before we can derive an error bound for the estimator in (5), we impose two additional assumptions.

(A2) For the true parameter value ${\mathbf{\vartheta }}^{\ast }$, there is a $(s_e,s_g)$-sparse structure in the coefficients ${\mathbf{B}}^{\ast }$ with element-wise and group-wise support sets

$$\begin{array}{c}\mathcal{E}=\left\{(j,k)\in \{1,\dots ,p\}\times \{1,\dots ,K\}|b_{jk}^{\ast }\ne 0\right\}\mathrm{and}\mathcal{G}=\left\{j\in \{1,\dots ,p\}|{\mathit{\beta }}_j^{\ast }\ne {\mathbf{0}}_K\right\}\end{array}$$

with cardinality $\left|\mathcal{E}\right|=s_e$ and $\left|\mathcal{G}\right|=s_g$, respectively.

(A3) There exist some positive constants ${\varsigma }_2$ and ${\varsigma }_3$ such that

$$\begin{array}{c}\hfill {\varsigma }_2^2=\underset{\mathbf{\gamma }\in \mathcal{V}}{max}{\displaystyle \frac{\parallel \mathrm{diag}\left\{\mathrm{vec}\left({\mathbf{E}}^{\top }\right)\right\}(\mathbf{Z}\otimes {\mathbf{I}}_K){\mathbf{\gamma }\parallel }_2^2}{{N\parallel \mathbf{\gamma }\parallel }_2^2}}\mathrm{and}{\varsigma }_3^2=\underset{\mathbf{\gamma }\in \mathcal{U}}{min}{\displaystyle \frac{{\mathbf{\gamma }}^{\top }\mathcal{H}\left({\mathbf{\vartheta }}^{\ast }\right)\mathbf{\gamma }}{{\mathbf{\gamma }}^{\top }\mathbf{\gamma }}}\end{array}$$

with $\mathcal{V}=\left\{\mathit{v}\in {\mathbb{R}}^{K(p+1)}{|0<\parallel \mathit{v}\parallel }_0\le s_e+K\right\}$ and

$$\begin{array}{c}\hfill \mathcal{U}=\left\{\mathbf{\delta }\in {\mathbb{R}}^{K(p+1)}|{\displaystyle \frac{\tau }{1-\tau }}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\ge C_0\left({\displaystyle \frac{\tau }{1-\tau }}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1+\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right)\right\},\end{array}$$

where $C_0=\frac{(c_0-1)}{(c_0+1)}$, ${\mathcal{E}}^{\mathrm{c}}$ is the complement of $\mathcal{E}$, ${\mathcal{E}}_{+}=\mathcal{E}\cup \{l=1+(k-1)(p+1)|k=1,\dots ,K\}$, and ${\mathcal{G}}_{+}=\mathcal{G}\cup \left\{0\right\}$.

Under the choice of $\lambda$ given in (9), we show the $L_2$-consistency of the estimator in (5).

Theorem 2.

Suppose that conditions (A1)–(A3) hold. Then with $\lambda =c_0\sqrt{\frac{log\left(pK\right)}{N}}$ in (5), we have that

$$\begin{array}{c}\hfill \parallel \widehat{\mathbf{\vartheta }}-{\mathbf{\vartheta }}^{\ast }{\parallel }_2\le \left\{C_1\sqrt{s_e+K}+C_2\sqrt{s_g+1}\right\}\sqrt{{\displaystyle \frac{log\left(pK\right)}{N}}}\end{array}$$

with probability at least $1-2{\left(Kp\right)}^{2(s_e+K)(1-c_3^2)}$, where $C_1=2{\varsigma }_3^{-2}\{c_0\tau +(\sqrt{2}+2c_3){\varsigma }_2\}$ and $C_2=2{\varsigma }_3^{-2}{c}_0(1-\tau )$.

Remark 3.

The sub-Gaussian distribution assumption (A1) is common in high-dimensional scenarios. This assumption characterizes the tail behavior of a collection of random variables including Gaussian, Bernoulli, and bounded variables as special cases. Assumption (A2) describes structural sparsity at two levels. The element-wise size $s_e<p$ is the size of the underlying generative model, and the group-wise size $s_g<pK$ is the size of the signal covariate set. Both $s_e$ and $s_g$ are allowed to depend on the sample size N. As a result, the dimension p is allowed to increase with the sample size N. Assumption (A3) guarantees that eigenvalues are positive in this sparse scenario.

Remark 4.

In practice, the tuning parameters λ and τ in (7) are commonly chosen by M-fold cross validation. That is, we choose the pair $(\tau ,\lambda )$ with the highest prediction accuracy among the sub-data sets ${\mathcal{D}}_m$, specifically,

$$\begin{array}{c}\hfill CV(\tau ,\lambda )=\sum _{m=1}^M\sum _{i\in {\mathcal{D}}_m},\mathbb{𝟙}\{y_i={\widehat{y}}_i(\tau ,\lambda )\}\end{array}$$

where ${\widehat{y}}_i(\tau ,\lambda )=\underset{k\in \mathcal{Y}}{\mathrm{argmax}}{\mathit{Z}}_i^{\top }{\widehat{\mathbf{\theta }}}_k(\tau ,\lambda )$.

2.4. Computational Algorithm

In this section, we propose an efficient algorithm to solve problem (5). Our approach uses the proximal algorithm (see [29]) for solving high dimensional regularization problems. In two main steps, this approach obtains a solution to the constrained optimization problem by applying the proximal operator to the solution to the unconstrained problem.

Since regularization is not needed for the intercept terms $\mathbf{\alpha }={(a_1,\dots ,a_K)}^{\top }$, it can be separated from the coefficients in $\mathbf{B}$. The empirical MgDWD loss of (8) is given by

$$\begin{array}{c}\hfill L\left(\mathbf{\vartheta }\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{E}}_i^{\top }{\varphi }_q\left({\mathit{F}}_i\right)={\displaystyle \frac{1}{N}}\mathrm{tr}\left\{\mathbf{E}{\varphi }_q\left({\mathbf{F}}^{\top }\right)\right\}={\displaystyle \frac{1}{N}}\mathrm{vec}{\left\{{\mathbf{E}}^{\top }\right\}}^{\top }\mathrm{vec}\left\{{\varphi }_q\left({\mathbf{F}}^{\top }\right)\right\}\end{array}$$

where $\mathbf{F}={\left(f_{ik}\right)}_{N\times K}=\mathbf{Z}\mathsf{\Theta }={\mathbf{1}}_N{\mathbf{\alpha }}^{\top }+\mathbf{XB}$. Various properties of the loss function $L\left(\mathbf{\vartheta }\right)$ follow below.

Lemma 3.

The loss function $L\left(\mathbf{\vartheta }\right)$ has Lipschitz continuous partial derivatives. In particular, for $\mathit{S}\left(\mathbf{\alpha }\right)={\displaystyle \frac{\partial L\left(\mathbf{\theta }\right)}{\partial \mathbf{\alpha }}}={\displaystyle \frac{1}{N}}{\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }{\mathbf{1}}_N$ and any $\mathit{u},\mathit{v}\in {\mathbb{R}}^K$, we have that

$$\begin{array}{c}\parallel \mathit{S}\left(\mathit{u}\right)-\mathit{S}\left(\mathit{v}\right){\parallel }_2\le \sqrt{\frac{n_{max}}{N}}{\displaystyle \frac{{(q+1)}^2}{q}}{\parallel \mathit{u}-\mathit{v}\parallel }_2,\end{array}$$

where $n_{max}$ is the largest group sample size. For $\mathit{S}\left(\mathbf{B}\right)={\displaystyle \frac{\partial L\left(\mathbf{\theta }\right)}{\partial \mathbf{B}}}={\displaystyle \frac{1}{N}}{\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }\mathbf{X}$ and any $\mathbf{U},\mathbf{V}\in {\mathbb{R}}^{p\times K}$, we have that

$$\begin{array}{c}\parallel \mathrm{vec}\{\mathit{S}\left(\mathbf{U}\right)-\mathit{S}\left(\mathbf{V}\right)\}{\parallel }_2\le {\displaystyle \frac{{max}_k{\parallel \mathrm{diag}\left({\mathit{e}}_k\right)\mathbf{X}\parallel }_2^2}{N}}{\displaystyle \frac{{(q+1)}^2}{q}}{\parallel \mathrm{vec}\{\mathbf{U}-\mathbf{V}\}\parallel }_2,\end{array}$$

where ${\mathit{e}}_k$ is the k-th column of $\mathbf{E}$ and indicates the observations belonging to the k-th group.

Hence, following the majorization–minimization scheme, we can majorize the empirical MgDWD loss $L\left(\mathbf{\vartheta }\right)$ by a quadratic function, that is,

$$\begin{array}{cc}\hfill L\left(\mathbf{\vartheta }\right)\le & L\left({\mathbf{\vartheta }}_{\ast }\right)+\mathit{S}{\left({\mathbf{\alpha }}_{\ast }\right)}^{\top }(\mathbf{\alpha }-{\mathbf{\alpha }}_{\ast })+{\displaystyle \frac{L_{\mathbf{\alpha }}}{2}}{\parallel \mathbf{\alpha }-{\mathbf{\alpha }}_{\ast }\parallel }_2^2\hfill \\ \hfill & +\mathrm{vec}{\left\{\mathit{S}\left(\mathbf{B}\right)\right\}}^{\top }\mathrm{vec}\{\mathbf{B}-{\mathbf{B}}_{\ast }\}+{\displaystyle \frac{L_{{\mathbf{B}}_{\ast }}}{2}}{\parallel \mathrm{vec}\{\mathbf{B}-{\mathbf{B}}_{\ast }\}\parallel }_2^2,\hfill \end{array}$$

for some ${\mathbf{\vartheta }}_{\ast }=\mathrm{vec}\left\{{({\mathbf{\alpha }}_{\ast },{\mathbf{B}}_{\ast }^{\top })}^{\top }\right\}$, where $L_{\mathbf{\alpha }}$ and $L_{\mathbf{B}}$ denote the Lipschitz constants in Lemma 3. Instead of minimizing $L\left(\mathbf{\vartheta }\right)$ directly, we apply gradient descent to minimize its surrogate upper bound function. The gradient descent updates are given by

$$\begin{array}{c}{\mathbf{\alpha }}_{\ast }=\mathbf{\alpha }-\frac{q{(q+1)}^{-2}}{\sqrt{n_{max}N}}{\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }{\mathbf{1}}_N,\end{array}$$

(10)

$$\begin{array}{c}{\mathbf{B}}_{\ast }=\mathbf{B}-{\displaystyle \frac{q{(q+1)}^{-2}}{{max}_k{\parallel \mathrm{diag}\left({\mathit{e}}_k\right)\mathbf{X}\parallel }_2^2}}{\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }\mathbf{X}.\end{array}$$

(11)

Next, we address the problem’s constraints and regularization simultaneously by applying the proximal operator. For ${\mathbf{\alpha }}_{\ast }$, it is clear that

$$\begin{array}{c}\hfill {\mathbf{\alpha }}_{\mathrm{new}}=\underset{{\mathbf{\alpha }}^{\top }{\mathbf{1}}_K=0}{\mathrm{argmin}}{\parallel \mathbf{\alpha }-{\mathbf{\alpha }}_{\ast }\parallel }_2^2={\mathbf{P}}_K{\mathbf{\alpha }}_{\ast },\end{array}$$

(12)

where ${\mathbf{P}}_K={\mathbf{I}}_K-k^{-1}{\mathbf{1}}_K{\mathbf{1}}_K$. For ${\mathbf{B}}_{\ast }={({\mathit{\beta }}_{1\ast },\dots ,{\mathit{\beta }}_{p\ast })}^{\top }$, the minimization problem can be expressed as

$$\begin{array}{cc}\hfill {\mathbf{B}}_{\mathrm{new}}=& \underset{\mathbf{B}{\mathbf{1}}_K={\mathbf{0}}_p}{\mathrm{argmin}}{\displaystyle \frac{1}{2}}\parallel \mathrm{vec}\{\mathbf{B}-{\mathbf{B}}_{\ast }\}{\parallel }_2^2+{\displaystyle \frac{{\lambda }_1}{L_{\mathbf{B}}}}\parallel \mathrm{vec}\left\{\mathbf{B}\right\}{\parallel }_1+{\displaystyle \frac{{\lambda }_2}{L_{\mathbf{B}}}}{\parallel \mathrm{vec}\left\{\mathbf{B}\right\}\parallel }_{1,2}\hfill \\ \hfill =& \underset{\mathbf{B}{\mathbf{1}}_K={\mathbf{0}}_p}{\mathrm{argmin}}\sum _{j=1}^p{\displaystyle \frac{1}{2}}\parallel {\mathit{\beta }}_j-{\mathit{\beta }}_{j\ast }{\parallel }_2^2+{\displaystyle \frac{{\lambda }_1}{L_{\mathbf{B}}}}\parallel {\mathit{\beta }}_j{\parallel }_1+{\displaystyle \frac{{\lambda }_2}{L_{\mathbf{B}}}}{\parallel {\mathit{\beta }}_j\parallel }_2,\hfill \end{array}$$

(13)

which implies that we can implement minimization for p groups in parallel. The following theorem provides the solution to (13).

Theorem 3.

Let ${\rho }_1,{\rho }_2\ge 0$ and ${\mathit{\beta }}_{\ast }\in {\mathbb{R}}^K$. Then the constrained regularization problem

$$\begin{array}{c}\underset{\mathit{\beta }\in {\mathbb{R}}^K}{min}{\displaystyle \frac{1}{2}}\parallel \mathit{\beta }-{\mathit{\beta }}_{\ast }{\parallel }_2^2+{\rho }_1{\parallel \mathit{\beta }\parallel }_1+{\rho }_2{\parallel \mathit{\beta }\parallel }_2\\ s.t.{\mathit{\beta }}^{\top }{\mathbf{1}}_K=0\end{array}$$

has a solution of the form

$$\begin{array}{c}\hfill {\mathit{\beta }}^{\ast }={\left\{1-{\displaystyle \frac{{\rho }_2}{\parallel {\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}){\parallel }_2}}\right\}}_{+}{\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u})\end{array}$$

(14)

for some $\mathit{u}\in {\partial \parallel \mathit{\beta }\parallel }_1$.

In the special case with ${\rho }_2=0$, the constrained regularization problem in Theorem 3 reduces to the constrained lasso problem with solution ${\tilde{\mathit{\beta }}}^{\ast }={\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u})$. Combined with (14), the proximal operator $\mathcal{U}$, given by

$$\begin{array}{c}\hfill {\mathit{\beta }}^{\ast }=\mathcal{U}({\tilde{\mathit{\beta }}}^{\ast },{\rho }_2)={\left\{1-{\displaystyle \frac{{\rho }_2}{\parallel {\tilde{\mathit{\beta }}}^{\ast }{\parallel }_2}}\right\}}_{+}{\tilde{\mathit{\beta }}}^{\ast },\end{array}$$

(15)

can be introduced to realize the group sparsity of ${\tilde{\mathit{\beta }}}^{\ast }$.

For the standard lasso problem, the subgradient $\mathit{u}$ has a closed form given by ${\tilde{\mathit{\beta }}}^{\ast }={\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}=\mathcal{S}({\mathit{\beta }}_{\ast },{\rho }_1)$, with $\mathcal{S}(u,v)=\mathrm{sign}\left(u\right){\left(\right|u|-v)}_{+}$. However, under the constraint on ${\tilde{\mathit{\beta }}}^{\ast }$, the naive solution ${\mathbf{P}}_K\mathcal{S}({\mathit{\beta }}_{\ast },{\rho }_1)$ is misleading in that it satisfies the constraint but does not achieve shrinkage, let alone loss function minimization. The term ${\mathbf{P}}_K\mathit{u}$ is suggestive of the intersection between the subdifferential set ${\partial \parallel \mathit{\beta }\parallel }_1$ and the constraint set $\{\mathit{\beta }\in {\mathbb{R}}^K|{\mathit{\beta }}^{\top }{\mathbf{1}}_K=0\}$; in this sense, ${\tilde{\mathit{\beta }}}^{\ast }$ might not have a closed form. Here we consider using coordinate descent to solve the constrained lasso problem. For some fixed coordinate m, since ${\mathit{\beta }}^{\top }{\mathbf{1}}_K=0$, we have that $b_m=-{\sum }_{l\ne m}{b}_l$. Rewriting the objective function of the lasso-constrained problem in a coordinate-wise form, we obtain

$$\begin{array}{cc}\hfill \sum _{l=1}^K{\displaystyle \frac{1}{2}}{(b_l-b_{l\ast })}^2+{\rho }_1\left|b_l\right|=& {\left(b_k-{\displaystyle \frac{(b_{k\ast }-b_{m\ast })}{2}}+{\displaystyle \frac{1}{2}}\sum _{l\ne k,m}^K{b}_l\right)}^2+{\rho }_1\left\{|b_k|+|b_k+\sum _{l\ne k,m}^K{b}_l|\right\}\hfill \\ \hfill & +{\displaystyle \frac{1}{4}}{\left(b_{k\ast }+b_{m\ast }+\sum _{l\ne k,m}^K{b}_l\right)}^2+\sum _{l\ne k,m}^K{\displaystyle \frac{1}{2}}{(b_l-b_{l\ast })}^2+{\rho }_1\left|b_l\right|.\hfill \end{array}$$

(16)

Next, Theorem 4 provides the solution to the optimization problem (16).

Theorem 4.

Suppose that $t,s\in \mathbb{R}$ and $\varrho \ge 0$. Then the regularization problem

$$\begin{array}{c}\hfill \underset{b\in \mathbb{R}}{min}{\displaystyle \frac{1}{2}}{(b-t)}^2+\varrho \left\{\right|b|+|b+s\left|\right\}\end{array}$$

has solution

$$\begin{array}{cc}\hfill b^{\ast }=& \left\{\begin{array}{cc}t,\hfill & \left|t\right|<C(s,t)\hfill \\ -C(s,t),\hfill & C(s,t)\le \left|t\right|\le C(s,t)+2\varrho \hfill \\ \mathrm{sign}\left(t\right)\left(\right|t|-2\varrho ),\hfill & \left|t\right|>C(s,t)+2\varrho \hfill \end{array}\right.\hfill \\ \hfill =& t-\mathcal{S}\left(t,C(s,t)\right)+\mathcal{S}\left\{\mathcal{S}\left(t,C(s,t)\right),2\varrho \right\},\hfill \end{array}$$

where $C(s,t)={\displaystyle \frac{1-\mathrm{sign}\left(s\right)\mathrm{sign}\left(t\right)}{2}}\left|s\right|$.

By Theorem 4, given some $m\in \{1,\dots ,K\}$, the coordinate-wise minimizer for any $k\ne m$ can be expressed as the proximal operator

$$\begin{array}{c}\hfill b_k=\mathcal{T}(t,s,{\rho }_1)=t-\mathcal{S}\left(t,C(s,t)\right)+\mathcal{S}\left\{\mathcal{S}\left(t,C(s,t)\right),{\rho }_1\right\},\end{array}$$

(17)

with $s={\sum }_{l\ne k,m}{b}_l$ and $t=(b_{k\ast }-b_{m\ast }-s)/2$. If we fix m during iteration, then the shrinkage of $b_m$ will be indirectly reflected in the other $b_k$. We propose that m change with k in the coordinate-wise minimization process to ensure that every coordinate can be equally shrunk. We summarize our proposed algorithm in Algorithm 1.

Algorithm 1: Proximal gradient descent algorithm for SGL-MgDWD.

Input:${\lambda }_1,{\lambda }_2.$

Initialization:${\mathbf{\alpha }}^{\left(0\right)}={\mathbf{0}}_K$, ${\mathbf{B}}^{\left(0\right)}={\mathbf{O}}_{p\times K}$, $l=0$.

1: repeat 2:     Update $\mathbf{\alpha }$ according to (10) and (12): $$\begin{array}{c}\hfill {\mathbf{\alpha }}^{(l+1)}={\mathbf{P}}_K\{{\mathbf{\alpha }}^{\left(l\right)}-L_{\mathbf{\alpha }}^{-1}\mathit{S}\left({\mathbf{\alpha }}^{\left(l\right)}\right)\}.\end{array}$$ 3:     Update $\tilde{\mathbf{B}}$ according to (11): $$\begin{array}{c}\hfill \tilde{\mathbf{B}}={\mathbf{B}}^{\left(l\right)}-L_{\mathbf{B}}^{-1}\mathit{S}\left({\mathbf{B}}^{\left(l\right)}\right).\end{array}$$ 4:     Set ${\mathbf{B}}^{(l+1)}\leftarrow \tilde{\mathbf{B}}$. 5:     repeat 6:         for $m=1$ to K do 7:            for k in $\{1,\dots ,K\}\backslash m$ do 8:                Update $(\mathit{t},\mathit{s})$: $$\begin{array}{c}\hfill \mathit{t}={\tilde{\mathit{b}}}_k-{\tilde{\mathit{b}}}_m,\mathit{s}=\sum _{r=1}^K{\mathit{b}}_r^{(l+1)}-{\mathit{b}}_k^{(l+1)}-{\mathit{b}}_m^{(l+1)}.\end{array}$$ 9:                Update ${\mathit{b}}_k^{(l+1)}$ according to (17) and ${\mathit{b}}_m^{(l+1)}$ by constraint: $$\begin{array}{c}\hfill {\mathit{b}}_k^{(l+1)}=\mathcal{T}(\mathit{t},\mathit{s},L_{\mathbf{B}}^{-1}{\lambda }_1),{\mathit{b}}_m^{(l+1)}=-\mathit{s}-{\mathit{b}}_k^{(l+1)}.\end{array}$$ 10:            end for 11:         end for 12:     until ${\mathbf{B}}^{(l+1)}$ convergence. 13:     Update ${\mathbf{B}}^{(l+1)}$ according to (15): $$\begin{array}{c}\hfill {\mathbf{B}}^{(l+1)}=\mathcal{U}({\mathbf{B}}^{(l+1)},L_{\mathbf{B}}^{-1}{\lambda }_2).\end{array}$$ 14:     Set $l\leftarrow l+1$. 15: until some condition is met.

Output:${\mathbf{\alpha }}^{\left(l\right)}$ and ${\mathbf{B}}^{\left(l\right)}$.

3. Numerical Analysis

In the following section, we use both simulated and real data sets to evaluate the finite sample properties of our proposed method. We compare the finite sample performance of SGL-MgDWD with $L_1$-regularized multinomial logistic regression ($L_1$-logistic).

3.1. Simulation Studies

The data is generated from the following model. Consider the K-category classification problem where ${\pi }_k=K^{-1}$ and $g_k\left(\mathit{X}\right)$ is the density function of a normal distribution with mean vector ${\mathbf{\mu }}_k={({\mu }_{1k},{\mu }_{2k},{\mathbf{0}}_{p-2}^{\top })}^{\top }$ and covariance matrix ${\mathbf{I}}_p$, where $({\mu }_{1k},{\mu }_{2k})=(2cos\left(\pi r_k\right),2sin\left(\pi r_k\right))$ with $r_k=\frac{2(k-1)}{K}$, for $k=1,\dots ,K$. In this model, only the first two variables contribute to the classification and their corresponding parameter vectors ${\mathit{\beta }}_1$ and ${\mathit{\beta }}_2$ form two groups of coefficients. The true model has the sparsity structure $(s_e,s_g)=(2K,2)$ for a total of $K(p+1)$ coefficients. We set the sample size for each category to $n_k=50,100,200$ and 400, and the number of classes to $K=5$ and 11. We consider dimensionality $p=100$ and 1000.

In what follows, we compare the proposed SGL-MgDWD method with the OVR method based on SGL-MgDWD with $K=2$ (OVR-SGL-gDWD). For SGL-MgDWD, logistic regression and OVR, the tuning parameter $\lambda$ is optimized over a discrete set by minimizing the prediction error using 5-fold cross validation. In each simulation, we conduct 100 runs and use a testing set of equal size to evaluate each method’s performance using the following criteria:

• Testing set accuracy, measuring the rate of correct classification;
• Signal, as the average number of correctly-selected element-wise and group-wise signals, that is, with ${\widehat{b}}_{jk}\ne 0$ and ${\widehat{\mathit{\beta }}}_j\ne \mathbf{0}$, respectively, denoted by the pair $(s_e^{+},s_g^{+})$;
• Noise, as the average number of incorrectly-selected element-wise and group-wise components, that is, with ${\widehat{b}}_{jk}=0$ and ${\widehat{\mathit{\beta }}}_j=0$, respectively, denoted by the pair $(n_e^{+},n_g^{+})$.

Simulation results are summarized in Table 1 and Table 2.

Table 1. Simulation results for the SGL-MgDWD, $L_1$-logistic, and OVR methods with $K=5$. Time is measured relative to a baseline logistic regression model with $K=5$, $p=100$, and $N=50$. Numbers in parentheses denote standard deviations.

${\mathit{n}}_{\mathit{k}}$	p	Method	Test Accuracy	Signal $({\mathit{s}}_{\mathit{e}}^{+},{\mathit{s}}_{\mathit{g}}^{+})$	Noise $({\mathit{n}}_{\mathit{e}}^{+},{\mathit{n}}_{\mathit{g}}^{+})$	Time (SD)
50	100	SGL-MgDWD	0.980	(9.99, 2)	(0, 0)	1.150 (0.173)
		$L_1$-logistic	0.979	(9.00, 2)	(116.98, 26.17)	1.000 (0.153)
		OVR-SGL-gDWD	0.912	-	-	-
	1000	SGL-MgDWD	0.979	(10, 2)	(6.96, 1.94)	5.290 (0.166)
		$L_1$-logistic	0.966	(10, 2)	(2793.65, 722.38)	5.130 (0.063)
		OVR-SGL-gDWD	0.740	-	-	-
100	100	SGL-MgDWD	0.981	(10, 2)	(0.07, 0.03)	1.453 (0.155)
		$L_1$-logistic	0.980	(8.82, 2)	(35.18, 3.98)	1.258 (0.127)
		OVR-SGL-gDWD	0.828	-	-	-
	1000	SGL-MgDWD	0.980	(10, 2)	(1.01, 0.25)	4.863 (0.150)
		$L_1$-logistic	0.978	(9.93, 2)	(1380.38, 192.37)	4.703 (0.061)
		OVR-SGL-gDWD	0.546	-	-	-
200	100	SGL-MgDWD	0.980	(10, 2)	(7.67, 2.08)	1.776 (0.164)
		$L_1$-logistic	0.980	(9.39, 2)	(13.1, 0.72)	1.709 (0.175)
		OVR-SGL-gDWD	0.934	-	-	-
	1000	SGL-MgDWD	0.982	(10, 2)	(1.09, 0.29)	8.641 (0.186)
		$L_1$-logistic	0.981	(9.79, 2)	(199.02, 2.51)	2.505 (0.121)
		OVR-SGL-gDWD	0.950	-	-	-
400	100	SGL-MgDWD	0.981	(10, 2)	(0.02, 0)	2.792 (0.159)
		$L_1$-logistic	0.981	(10, 2)	(4.72, 3.95)	2.828 (0.115)
		OVR-SGL-gDWD	0.979	-	-	-
	1000	SGL-MgDWD	0.981	(10, 2)	(4.72, 3.95)	15.800 (0.221)
		$L_1$-logistic	0.981	(9.6, 2)	(16.17, 0.02)	17.915 (1.585)
		OVR-SGL-gDWD	0.964	-	-	-

Table 2. Simulation results for the SGL-MgDWD, $L_1$-logistic, and OVR methods with $K=11$. Time is measured relative to a baseline logistic regression model with $K=5$, $p=100$, and $N=50$. Numbers in parentheses denote standard deviations.

${\mathit{n}}_{\mathit{k}}$	p	Method	Test Accuracy	Signal $({\mathit{s}}_{\mathit{e}}^{+},{\mathit{s}}_{\mathit{g}}^{+})$	Noise $({\mathit{n}}_{\mathit{e}}^{+},{\mathit{n}}_{\mathit{g}}^{+})$	Time (SD)
50	100	SGL-MgDWD	0.735	(21.41, 2)	(0.14, 0.02)	1.661 (0.143)
		$L_1$-logistic	0.735	(20.13, 2)	(337.77, 22.07)	1.610 (0.110)
		OVR-SGL-gDWD	0.647	-	-	-
	1000	SGL-MgDWD	0.733	(21.25, 2)	(0, 0)	7.105 (0.205)
		$L_1$-logistic	0.566	(20.67, 2)	(3805.97, 265.82)	6.933 (0.205)
		OVR-SGL-gDWD	0.382	-	-	-
100	100	SGL-MgDWD	0.737	(21.82, 2)	(0.06, 0.01)	2.518 (0.099)
		$L_1$-logistic	0.721	(20, 2)	(173.17, 5.81)	2.418 (0.103)
		OVR-SGL-gDWD	0.609	-	-	-
	1000	SGL-MgDWD	0.737	(21.88, 2)	(5.4, 0.77)	12.371 (0.109)
		$L_1$-logistic	0.697	(20.15, 2)	(1859.51, 9.04)	12.279 (0.114)
		OVR-SGL-gDWD	0.214	-	-	-
200	100	SGL-MgDWD	0.738	(22, 2)	(0, 0)	5.191 (0.079)
		$L_1$-logistic	0.730	(20, 2)	(50.7, 0.08)	4.246 (0.100)
		OVR-SGL-gDWD	0.609	-	-	-
	1000	SGL-MgDWD	0.738	(21.98, 2)	(0.23, 0.04)	21.950 (0.241)
		$L_1$-logistic	0.730	(20, 2)	(523.08, 1.07)	22.158 (0.163)
		OVR-SGL-gDWD	0.490	-	-	-
400	100	SGL-MgDWD	0.740	(22, 2)	(0, 0)	7.025 (0.172)
		$L_1$-logistic	0.738	(20, 2)	(3.71, 3.48)	7.997 (0.122)
		OVR-SGL-gDWD	0.709	-	-	-
	1000	SGL-MgDWD	0.738	(22, 2)	(0.68, 0.11)	38.301 (0.200)
		$L_1$-logistic	0.734	(20, 2)	(38.84, 35.37)	41.059 (2.064)
		OVR-SGL-gDWD	0.556	-	-	-

As shown in Table 1 and Table 2, the proposed SGL-MgDWD method performs better than the $L_1$-logistic and OVR methods. Specifically, in each scenario, predictions from the SGL-MgDWD method had higher accuracy relative to the other two methods. Similarly, the SGL-MgDWD method correctly selected the signal components of the model with fewer incorrectly-selected noise components, again relative to the $L_1$-logistic and OVR methods. These simulation results also demonstrate that test accuracy increases with increasing sample size $n_k$ and that test accuracy decreases with higher dimension p at fixed $n_k$. This is consistent with the derived theoretical properties. All computations were performed on a Tensorflow 2.3 CPU on Threadripper 2950X at 4.1 Ghz.

3.2. HIV Data Analysis

Symptomatic distal sensory polyneuropathy (sDSP) is a common debilitating condition among people with HIV. This condition leads to neuropathic pain and is associated with substantial comorbidities and increased health care costs. Plasma miRNA profiles show differences between HIV patients with and without sDSP, and several miRNA biomarkers are reported to be associated with the presence of sDSP in HIV patients (see [30]). The corresponding binary classification problem was analyzed in [30] using random forest classifiers. However, the HIV data set can be further classified into four classes. The HIV data set has 1715 miRNA measures for 40 patients and is partitioned into four groups ($K=4$) with $n_k=10$ patients each category: non-HIV, HIV with no brain damage (HIVNBD), HIV with brain damage but stable (HIVBDS) and HIV with brain damage and unstable (HIVBDU). In the following analysis, we apply our proposed method to this classification problem. The primary aim was to identify critical miRNA biomarkers for each of the four groups. Beyond achieving a finer classification, this analysis is helpful in assessing related pathogenic effects for each patient group.

Given the small sample size of $N=40$, we chose the tuning parameter $\lambda$ by maximizing leave-one-out cross validation accuracy. We fixed $(q,\tau )=(1,0.1)$. Table 3 shows the signal for coefficient estimates obtained from the SGL-MgDWD method using the selected $\lambda$. We conclude that there are 22 critical miRNA biomarkers important to the classification problem. In particular, the biomarkers miR-25-star, miR-3171, miR-3924 and miR-4307 are not relevant to the non-HIV group; miR-4641, miR-4655-3p and miR-660 are not relevant to the HIVNBD group; miR-217 and miR-4683 are not relevant to the HIVBDS group; and miR-217 and miR-4307 are not relevant to the HIVBDU group.

Table 3. Signal for the coefficient estimates obtained from the SGL-MgDWD method with $(q,\tau )=(1,0.1)$ for the HIV data set. The symbols “+” and “-” denote positive and negative coefficient estimates, respectively, while “0” denotes a zero coefficient (i.e., an irrelevant variable).

	Non-HIV	HIVNBD	HIVBDS	HIVBDU
interception	+	+	-	+
miR-255b	-	+	-	+
miR-217	+	-	0	0
miR-25-star	0	+	+	-
miR-3136-5p	-	-	+	-
miR-3152-3p	+	-	-	+
miR-3159	-	-	-	+
miR-3171	0	+	-	-
miR-33b	-	-	-	+
miR-34c-3p	-	-	+	+
miR-3545-5p	-	+	-	+
miR-3654	-	-	-	+
miR-3924	0	-	+	-
miR-4307	0	-	+	0
miR-4474-5p	-	+	+	+
miR-4526	+	-	-	-
miR-4641	+	0	-	-
miR-4655-3p	+	0	-	-
miR-4680-5p	-	-	+	-
miR-4683	-	-	0	+
miR-589	-	+	+	-
miR-619	+	-	-	+
miR-660	+	0	-	+

Appendix A.1. Proof of Lemma 1

Proof. 

For simplicity, we write $p_j=P(y=j|\mathit{X})$ and $f_k=f_k\left(\mathit{X}\right)$. Using the Lagrange multiplier method, we define

$$\begin{array}{c}\hfill L\left(\mathit{F}\right)=\mathbb{E}\left\{\sum _{k=1}^K\mathbb{𝟙}\{y=k\}{\varphi }_q\left\{\mathit{F}\left(\mathit{X}\right)\right\}|\mathit{X}=\mathit{u}\right\}+\mu {\mathbf{1}}_K^{\top }\mathit{F}\left(\mathit{X}\right)=\sum _{k=1}^K{p}_k{\varphi }_q\left(f_k\right)+\mu f_k.\end{array}$$

Then for each k,

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial L\left(\mathit{F}\right)}{\partial f_k}}={\varphi }_q^{\prime }\left(f_k\right)p_k+\mu =0\end{array}$$

(A1)

with

$$\begin{array}{c}\hfill {\varphi }_q^{\prime }\left(f_j\right)=\left\{\begin{array}{cc}-1,\hfill & f_k\le Q\hfill \\ -{\left(Q{f}_k^{-1}\right)}^q,\hfill & f_k>Q.\hfill \end{array}\right.\end{array}$$

Without loss of generality, assume that $p_1>p_2\ge p_3\ge \cdots \ge p_{K-1}>p_K$. Note that $-1\le {\varphi }_q^{\prime }<0$, and so $p_j\ge -{\varphi }_q^{\prime }\left(f_k\right)p_k=\mu >0$ and $\mu =p_k$ if and only if $f_k\le Q$.

If $\mu <p_K<p_k$, then $p_K\ne \mu$ when $f_K>Q$, which implies that $f_k>f_K>Q$ for all $1\le k\le K$. Hence, substituting ${\varphi }_q^{\prime }\left(f_k\right)=-{\left(Q{f}_k^{-1}\right)}^q$ into (A1) yields

$$\begin{array}{c}\hfill f_k=Q\sqrt[q]{p_k{\mu }^{-1}}>Q>0.\end{array}$$

However, ${\sum }_{k=1}^K{f}_k>0$, contradicting the sum-to-zero constraint. Therefore, $\mu =p_K<p_k$ for $k<K$ and the result follows. □

Appendix A.2. Proof of Theorem 1

Lemma A1.

Under (C1), $\mathcal{L}\left(\mathbf{\vartheta }\right)$ exists, and it is convex on $\mathbf{\vartheta }$.

Proof. 

The existence of $\mathcal{L}\left(\mathbf{\vartheta }\right)$ will be satisfied if

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathit{X}|y}\left\{|{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)||y=k\right\}={\int }_{\mathcal{X}}\left|{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\right|g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}<\infty .\end{array}$$

We divide $\mathcal{X}$ into two disjoint subsets. Defining ${\mathcal{X}}_k=\{\mathit{X}\in \mathcal{X}\mid {\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}$, it is clear that

$$\begin{array}{c}\hfill {\int }_{{\mathcal{X}}_k}\left|{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\right|g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\le {(q+1)}^{-1}{\int }_{{\mathcal{X}}_k}{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}<\infty .\end{array}$$

Note that $0<{\varphi }_q\left(u\right)<{(1+q)}^{-1}<1$ when $u>Q$. On the other hand, for ${\mathcal{X}}_k^{\mathrm{c}}=\{\mathit{X}\in \mathcal{X}\mid {\mathit{Z}}^{\top }{\mathbf{\theta }}_k\le Q\}$,

$$\begin{array}{c}\hfill {\int }_{{\mathcal{X}}_k^{\mathrm{c}}}|{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)|g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\le |1-a_k|+\sum _{j=1}^p{b}_{jk}{\int }_{\mathcal{X}}\left|x_j\right|g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}<\infty ,\end{array}$$

if ${\mathbb{E}}_{\mathit{X}|y}\left\{|x_j||y=k\right\}<\infty$ for all $k\in \mathcal{Y}$. This completes the proof of the existence of $\mathcal{L}\left(\mathbf{\vartheta }\right)$.

Recall that

$$\begin{array}{c}\hfill \mathcal{L}\left(\mathbf{\vartheta }\right)=\sum _{k=1}^K{\pi }_k{\int }_{\mathcal{X}}{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X},\end{array}$$

where ${\varphi }_q\left(u\right)$ is a convex function of u, so its composition with the affine mapping $u={\mathit{Z}}^{\top }{\mathbf{\theta }}_k$ is still convex in ${\mathbf{\theta }}_k$. Clearly, $g_k\left(\mathit{X}\right),{\pi }_k>0$, so the non-negatively-weighted integral and sum both preserve convexity. □

Lemma A2.

Existence of minimizers of $\mathcal{L}\left(\mathbf{\vartheta }\right)$ on $\mathcal{C}=\left\{\mathbf{\vartheta }\in {\mathbb{R}}^{K(p+1)}|\mathbf{C}\mathbf{\vartheta }={\mathbf{0}}_K\right\}$, where $\mathbf{C}={\mathbf{1}}_K^{\top }\otimes {\mathbf{I}}_{p+1}$.

Proof. 

By Jensen’s inequality, for any $\mathbf{\vartheta }\in \mathcal{C}$, we have that

$$\begin{array}{c}\hfill \mathcal{L}\left(\mathbf{\vartheta }\right)\ge {\varphi }_q\left(\sum _{k=1}^K{\pi }_k\mathbb{E}\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right|y=k\}\right).\end{array}$$

Let $\mathbf{\mu }=\mathrm{vec}\left\{{\left({\pi }_k\mathbb{E}\left\{z_j\right|y=k\}\right)}_{jk}\right\}$, where ${\parallel \mathbf{\mu }\parallel }_2\ge {\left({\sum }_{k=1}^K{\pi }_k^2\right)}^{\frac{1}{2}}\ge K^{-\frac{1}{2}}>0$. For some $C>0$, we have that

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathbf{\vartheta }\right)\ge & {\varphi }_q\left({\mathbf{\mu }}^{\top }\mathbf{\vartheta }\right)=\mathbb{𝟙}\{{\mathbf{\mu }}^{\top }\mathbf{\vartheta }<Q\}(1-{\mathbf{\mu }}^{\top }\mathbf{\vartheta })+\mathbb{𝟙}\{{\mathbf{\mu }}^{\top }\mathbf{\vartheta }\ge Q\}{\phi }_q\left({\mathbf{\mu }}^{\top }\mathbf{\vartheta }\right)\hfill \\ \hfill \ge & \mathbb{𝟙}\{{\mathbf{\mu }}^{\top }\mathbf{\vartheta }<Q\}|1-\left|{\mathbf{\mu }}^{\top }\mathbf{\vartheta }\right||\hfill \\ \hfill =& \mathbb{𝟙}\{{\mathbf{\mu }}^{\top }\mathbf{\vartheta }<-(C+1)\}\left(\right|{\mathbf{\mu }}^{\top }\mathbf{\vartheta }|-1)+\mathbb{𝟙}\{-(C+1)<{\mathbf{\mu }}^{\top }\mathbf{\vartheta }<-1\}\left(\right|{\mathbf{\mu }}^{\top }\mathbf{\vartheta }|-1)\hfill \\ \hfill & +\mathbb{𝟙}\{-1<{\mathbf{\mu }}^{\top }\mathbf{\vartheta }<Q\}(1-|{\mathbf{\mu }}^{\top }\mathbf{\vartheta }\left|\right)\hfill \\ \hfill >& \mathbb{𝟙}\{\parallel \mathbf{\mu }{\parallel }_2\parallel \mathbf{\vartheta }{\parallel }_2>C+1\}C\hfill \\ \hfill =& \mathbb{𝟙}\left\{{\parallel \mathbf{\vartheta }\parallel }_2>{\displaystyle \frac{C+1}{{\parallel \mathbf{\mu }\parallel }_2}}\right\}C.\hfill \end{array}$$

Note that $1-{\mathbf{\mu }}^{\top }\mathbf{\vartheta }>1-Q>0$ when ${\mathbf{\mu }}^{\top }\mathbf{\vartheta }<Q$. By the Cauchy–Schwarz inequality, $-{\mathbf{\mu }}^{\top }\mathbf{\vartheta }=|{\mathbf{\mu }}^{\top }{\mathbf{\vartheta }|\le \parallel \mathbf{\mu }\parallel }_2{\parallel \mathbf{\vartheta }\parallel }_2$.

Hence, if ${\parallel \mathbf{\vartheta }\parallel }_2>{\displaystyle \frac{C+1}{{\parallel \mathbf{\mu }\parallel }_2}}>0$, then $\mathcal{L}\left(\mathbf{\vartheta }\right)>C>0$. The contrapositive of this result implies the existence of a minimizer in the unconstrained problem. That is, the closed set $\left\{\mathbf{\vartheta }\in \mathcal{C}\left|\mathcal{L}\right(\mathbf{\vartheta })\le C\right\}$ is bounded for some large enough C. This guarantees the existence of a solution, as desired. □

Lemma A3.

Under (C1), $\mathcal{S}\left(\mathbf{\vartheta }\right)$ exists and

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \mathcal{L}\left(\mathbf{\vartheta }\right)}{\partial \mathbf{\vartheta }}}=\mathcal{S}\left(\mathbf{\vartheta }\right).\end{array}$$

Proof. 

The existence of $\mathcal{S}\left(\mathbf{\vartheta }\right)$ will follow if

$$\begin{array}{c}\hfill {\int }_{\mathcal{X}}|{\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\theta }_k\right)z_j|{\pi }_k{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\le {\pi }_k{\int }_{\mathcal{X}}\left|z_j\right|g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}<\infty \end{array}$$

for $j=1,\dots ,p+1$. Note that $|{\varphi }_q^{\prime }\left(u\right)|\le 1$ when $u>Q$.

For every ${\theta }_{kj}\in \mathbb{R}$, ${\varphi }_q\left({\mathit{Z}}^{\top }{\theta }_k\right)$ is a Lebesgue integrable function of $\mathit{X}$. For any $u\in \mathbb{R}$, ${\varphi }_q^{\prime }\left(u\right)$ exists and $|{\varphi }_q^{\prime }\left(u\right)|\le 1$. Hence, by the Leibniz integral rule, we have that

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial }{\partial {\theta }_{jk}}}{\int }_{\mathcal{X}}{\varphi }_q\left({\mathit{Z}}^{\top }{\theta }_k\right){\pi }_k{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}=& {\int }_{\mathcal{X}}{\displaystyle \frac{\partial {\varphi }_q\left({\mathit{Z}}^{\top }{\theta }_k\right)}{\partial {\theta }_{jk}}}{\pi }_k{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\hfill \\ \hfill =& {\int }_{\mathcal{X}}{\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\theta }_k\right)z_j{\pi }_k{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\hfill \end{array}$$

and for any $l\ne k$,

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial }{\partial {\theta }_{jl}}}{\int }_{\mathcal{X}}{\varphi }_q\left({\mathit{Z}}^{\top }{\theta }_k\right){\pi }_k{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}=0,\end{array}$$

which is sufficient to show that

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \mathcal{L}\left(\mathbf{\vartheta }\right)}{\partial \mathbf{\vartheta }}}=\mathcal{S}\left(\mathbf{\vartheta }\right).\end{array}$$

□

Lemma A4.

Suppose (C1) is satisfied. Then (C2) implies that ${\mathit{b}}_k^{\ast }\ne \mathbf{0}$.

Proof. 

We can rewrite ${\varphi }_q\left(u\right)$ as

$$\begin{array}{cc}\hfill {\varphi }_q\left(u\right)=& \mathbb{𝟙}\{u\le Q\}(1-u)+\mathbb{𝟙}\{u>Q\}(1-Q){\left({\displaystyle \frac{Q}{u}}\right)}^q\hfill \\ \hfill =& \left\{-\mathbb{𝟙}\{u\le Q\}-\mathbb{𝟙}\{u>Q\}{\left({\displaystyle \frac{Q}{u}}\right)}^{q+1}\right\}u+\mathbb{𝟙}\{u\le Q\}+\mathbb{𝟙}\{u>Q\}{\left({\displaystyle \frac{Q}{u}}\right)}^q\hfill \\ \hfill =& {\varphi }_q^{\prime }\left(u\right)u+\mathbb{𝟙}\{u\le Q\}+\mathbb{𝟙}\{u>Q\}{\left({\displaystyle \frac{Q}{u}}\right)}^q.\hfill \end{array}$$

Then for any $\mathbf{\gamma }\in {\mathbb{R}}^{p+1}$ and its corresponding ${\mathcal{X}}_k=\{\mathit{X}\in \mathcal{X}|{\mathit{Z}}^{\top }\mathbf{\gamma }>Q\}$, we have that

$$\begin{array}{cc}\hfill \mathbb{E}\{\mathbb{𝟙}& \{y=k\}{\varphi }_q\left({\mathit{Z}}^{\top }\mathbf{\gamma }\right)\}\hfill \\ \hfill =& \mathbb{E}\left\{\mathbb{𝟙}\{y=k\}{\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }\mathbf{\gamma }\right){\mathit{Z}}^{\top }\mathbf{\gamma }\right\}+\mathbb{E}\left\{\mathbb{𝟙}\{y=k,{\mathit{Z}}^{\top }\mathbf{\gamma }\le Q\}\right\}\hfill \\ \hfill & +\mathbb{E}\left\{\mathbb{𝟙}\{y=k,{\mathit{Z}}^{\top }\mathbf{\gamma }>Q\}{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }\mathbf{\gamma }}}\right)}^q\right\}\hfill \\ \hfill =& {\mathit{S}}_k^{\top }\left(\mathbf{\gamma }\right)\mathbf{\gamma }+Pr\{y=k,\mathit{X}\notin {\mathcal{X}}_k\}+\mathbb{E}\left\{\mathbb{𝟙}\{y=k,\mathit{X}\in {\mathcal{X}}_k\}{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }\mathbf{\gamma }}}\right)}^q\right\}\hfill \\ \hfill =& {\mathit{S}}_k^{\top }\left(\mathbf{\gamma }\right)\mathbf{\gamma }+{\pi }_k\left(1-\mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k\}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }\mathbf{\gamma }}}\right)}^q\right\}|y=k\right\}\right).\hfill \end{array}$$

Let ${\mathbf{\vartheta }}^{\ast }\in \mathcal{C}$ be a local minimizer. It follows that $\mathbf{P}\mathcal{S}\left({\mathbf{\vartheta }}^{\ast }\right)=\mathbf{0}$ and ${\sum }_{k=1}^K{\mathit{S}}_k^{\top }\left({\mathbf{\theta }}_k^{\ast }\right){\mathbf{\theta }}_k^{\ast }={\mathcal{S}}^{\top }\left({\mathbf{\vartheta }}^{\ast }\right){\mathbf{\vartheta }}^{\ast }=0$ since ${\mathbf{\vartheta }}^{\ast }=\mathbf{P}{\mathbf{\vartheta }}^{\ast }$ and $\mathbf{P}=\left({\mathbf{I}}_K-K^{-1}{\mathbf{1}}_K{\mathbf{1}}_K^{\top }\right)\otimes {\mathbf{I}}_{p+1}$. Therefore,

$$\begin{array}{cc}\hfill \mathcal{L}\left({\mathbf{\vartheta }}^{\ast }\right)=& \mathbb{E}\left\{\mathbb{𝟙}\{y=k\}{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\right)\right\}\hfill \\ \hfill =& \sum _{k=1}^K{\pi }_k\left(1-\mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q\right\}|y=k\right\}\right)\hfill \\ \hfill =& \sum _{k=1}^K{\pi }_k\left(1-Pr\{\mathit{X}\in {\mathcal{X}}_k^{\ast }|y=k\}\mathbb{E}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q|y=k,\mathit{X}\in {\mathcal{X}}_k^{\ast }\right\}\right).\hfill \end{array}$$

(A2)

For any $\mathbf{\gamma }\in {\mathbb{R}}^{p+1}$ and its corresponding ${\mathcal{X}}_k=\{\mathit{X}\in \mathcal{X}|{\mathit{Z}}^{\top }\mathbf{\gamma }>Q\}$, we always have that

$$\begin{array}{c}\hfill 0<\mathbb{E}\left\{{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }\mathbf{\gamma }}}\right)}^q|y=k,\mathit{X}\in {\mathcal{X}}_k\right\}<1.\end{array}$$

If $\mathbf{\gamma }={\mathbf{0}}_{p+1}$, then ${\mathcal{X}}_k=⌀$ so that $Pr\{y=k,\mathit{X}\notin {\mathcal{X}}_k\}={\pi }_k$ and $Pr\{y=k,\mathit{X}\in {\mathcal{X}}_k\}=0$. If ${\gamma }_1\le Q$ and ${\mathbf{\gamma }}_{/1}={\mathbf{0}}_p$, then ${\mathcal{X}}_k=⌀$, giving the same conclusions as the previous case. If ${\gamma }_1>Q$ and ${\mathbf{\gamma }}_{/1}={\mathbf{0}}_p$, then ${\mathcal{X}}_k=\mathcal{X}$ so that $Pr\{y=k,\mathit{X}\notin {\mathcal{X}}_k\}=0$ and $Pr\{y=k,\mathit{X}\in {\mathcal{X}}_k\}={\pi }_k$. Consequently, when $0<Pr\{\mathit{X}\in {\mathcal{X}}_k|y=k\}<1$, then neither ${\mathcal{X}}_k$ nor $\mathcal{X}$ equal ⌀, so ${\mathit{b}}_k\ne 0$ follows.

Note that $Pr\{\mathit{X}\notin {\mathcal{X}}_k|y=k\}>0$ implies that $Pr\{0<{\mathit{Z}}^{\top }\mathbf{\gamma }\le Q|y=k\}>0$ or $Pr\{{\mathit{Z}}^{\top }\mathbf{\gamma }\le 0|y=k\}>0$, and so special attention should be paid to bounded random variables. □

Lemma A5.

Under (C1), $\mathcal{H}\left(\mathbf{\vartheta }\right)$ exists and

$$\begin{array}{c}\hfill {\displaystyle \frac{{\partial }^2\mathcal{L}\left(\mathbf{\vartheta }\right)}{\partial \mathbf{\vartheta }\partial {\mathbf{\vartheta }}^{\top }}}=\mathcal{H}\left(\mathbf{\vartheta }\right).\end{array}$$

Furthermore, $\mathcal{H}\left({\mathbf{\vartheta }}^{\ast }\right)\succ {\mathbf{O}}_{K(p+1)}$ when (C2) and (C3) hold.

Proof. 

The existence of $\mathcal{H}\left(\mathbf{\vartheta }\right)$ follows if its all entries are absolutely integrable, that is, for any $j,k=1,\dots ,p+1$,

$$\begin{array}{c}{\int }_{\mathcal{X}}\left|\mathbb{𝟙}\right\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\left\}{\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)z_j{z}_l\right|{\pi }_k{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\hfill \\ \le (q+q^{-1}+2){\int }_{{\mathcal{X}}_k^{\mathrm{c}}}\left|z_j{z}_l\right|g_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\hfill \\ <\infty .\hfill \end{array}$$

Equivalently, the result follows if ${\mathbb{E}}_{\mathit{X}|y}\left\{|z_j{z}_l\left|\right|y=k\right\}<\infty$ for all $k\in \mathcal{Y}$. Note that $0<{\phi }_q^{″}\left(u\right)\le q+q^{-1}+2$ when $u>Q$.

Let $\eta$ be a test function belonging to the Schwartz space $\mathcal{D}$. Then ${\eta }^{\prime }\in \mathcal{D}$ with some support denoted by $\mathrm{supp}\left({\eta }^{\prime }\right)$.

Clearly, ${\varphi }_q^{\prime }\left(u\right)$ is not differentiable at Q but is Lipschitz continuous. Therefore, the measurable function ${\mathit{S}}_k\left({\mathbf{\theta }}_k\right)$ is a locally integrable function of ${\mathbf{\theta }}_k$. Then the (regular) generalized functions ${\mathit{S}}_k\left({\mathbf{\theta }}_k\right)$ belong to the dual space of $\mathcal{D}$.

For the distributional derivative of ${\mathit{S}}_k\left({\mathbf{\theta }}_k\right)$ with respect to ${\theta }_{jk}$, we have that

$$\begin{array}{cc}\hfill \left|⟨{\displaystyle \frac{\partial {\mathit{S}}_k\left({\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}},\eta \left({\theta }_{jk}\right)⟩\right|=& \left|-⟨{\mathit{S}}_k\left({\mathbf{\theta }}_k\right),{\displaystyle \frac{\mathrm{d}\eta \left({\theta }_{jk}\right)}{\mathrm{d}{\theta }_{jk}}}⟩\right|\hfill \\ \hfill \le & {\int }_{\mathbb{R}}\left|{\mathit{S}}_k\left({\mathbf{\theta }}_k\right){\eta }^{\prime }\left({\theta }_{jk}\right)\right|\mathrm{d}{\theta }_{jk}\hfill \\ \hfill \le & \underset{{\theta }_{jk}\in \mathrm{supp}\left({\eta }^{\prime }\right)}{max}|{\eta }^{\prime }\left({\theta }_{jk}\right)|{\int }_{\mathrm{supp}\left({\eta }^{\prime }\right)}\left|{\mathit{S}}_k\left({\mathbf{\theta }}_k\right)\right|\mathrm{d}{\theta }_{jk}\hfill \\ \hfill <& \infty \hfill \end{array}$$

implying that the function $f({\theta }_{jk},\mathit{X})={\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\mathit{Z}{\pi }_k{g}_k\left(\mathit{X}\right){\eta }^{\prime }\left({\theta }_{jk}\right)$ is integrable on $\mathbb{R}\times \mathcal{X}$. Therefore, by Fubini’s Theorem,

$$\begin{array}{cc}\hfill ⟨{\displaystyle \frac{\partial {\mathit{S}}_k\left({\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}},\eta \left({\theta }_{jk}\right)⟩=& -⟨{\mathit{S}}_k\left({\mathbf{\theta }}_k\right),{\displaystyle \frac{\mathrm{d}\eta \left({\theta }_{jk}\right)}{\mathrm{d}{\theta }_{jk}}}⟩\hfill \\ \hfill =& {\int }_{\mathcal{X}}-⟨{\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\mathit{Z}{\pi }_k{g}_k\left(\mathit{X}\right),{\displaystyle \frac{\mathrm{d}\eta \left(\theta jk\right)}{\mathrm{d}{\theta }_{jk}}}⟩\mathrm{d}\mathit{X}\hfill \\ \hfill =& {\int }_{\mathcal{X}}⟨{\displaystyle \frac{\partial {\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}}\mathit{Z}{\pi }_k{g}_k\left(\mathit{X}\right),\eta \left({\theta }_{jk}\right)⟩\mathrm{d}\mathit{X}\hfill \\ \hfill =& ⟨\mathbb{E}\left\{{\displaystyle \frac{\partial {\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}}\mathit{Z}\mathbb{𝟙}\{y=k\}\right\},\eta \left({\theta }_{jk}\right)⟩,\hfill \end{array}$$

which implies that

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathit{S}}_k\left({\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}}=\mathbb{E}\left\{{\displaystyle \frac{\partial {\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}}\mathit{Z}\mathbb{𝟙}\{y=k\}\right\}.\end{array}$$

Recall that ${\varphi }_q^{\prime }$ can be written as

$$\begin{array}{c}\hfill {\varphi }_q^{\prime }\left(u\right)={\phi }_q^{\prime }\left(u\right)\mathbb{𝟙}\{u>Q\}+(-1)\mathbb{𝟙}\{u\le Q\}=({\phi }_q^{\prime }\left(u\right)+1)\mathbb{𝟙}\{u>Q\}-1,\end{array}$$

which contains a Schwartz product between the differentiable function ${\phi }_q^{\prime }\left(u\right)$ and the generalized function $\mathbb{𝟙}\{u>Q\}$. Note that

$$\begin{array}{cc}\hfill \mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}=& \mathbb{𝟙}\{z_j>0,{\theta }_{jk}>c_{jk}\}+\mathbb{𝟙}\{z_j\le 0,{\theta }_{jk}\le c_{jk}\}\hfill \\ \hfill =& (2\mathbb{𝟙}\{z_j>0\}-1)\mathbb{𝟙}\{{\theta }_{jk}>c_{jk}\}+(1-\mathbb{𝟙}\{z_j>0\})\hfill \\ \hfill =& \mathrm{sign}\left(z_j\right)\mathbb{𝟙}\{{\theta }_{jk}>c_{jk}\}+\mathbb{𝟙}\{z_j\le 0\},\hfill \end{array}$$

where $c_{jk}=(Q-{\sum }_{l\ne j}{z}_l{\theta }_{lk})/z_j$ and

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}}{\partial {\theta }_{jk}}}+0& =\mathrm{sign}\left(z_j\right)\delta ({\theta }_{jk}-c_{jk})\hfill \\ & =\mathrm{sign}\left(z_j\right)\left|z_j\right|\delta ({\mathit{Z}}^{\top }{\mathbf{\theta }}_k-Q)\hfill \\ & =z_j\delta ({\mathit{Z}}^{\top }{\mathbf{\theta }}_k-Q),\hfill \end{array}$$

where $\delta \left(x\right)$ is the Dirac delta function and the distributional derivative of $\mathbb{𝟙}\{x>0\}$. Recall that $\delta \left(cx\right)=\delta \left(x\right)/\left|c\right|$ and $f\left(x\right)\delta (x-c)=f\left(c\right)\delta (x-c)$ for some constant c and function f.

Thus, by the product rule for the distributional derivative of the Schwartz product,

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial {\varphi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}}=& {\displaystyle \frac{\partial ({\phi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)+1)}{\partial {\theta }_{jk}}}\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}+({\phi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)+1){\displaystyle \frac{\partial \mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}}{\partial {\theta }_{jk}}}\hfill \\ \hfill =& {\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)z_j\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}+({\phi }_q^{\prime }\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)+1)z_j\delta ({\mathit{Z}}^{\top }{\mathbf{\theta }}_k-Q)\hfill \\ \hfill =& {\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)z_j\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}.\hfill \end{array}$$

Substituting the above expression, we obtain

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathit{S}}_k\left({\mathbf{\theta }}_k\right)}{\partial {\theta }_{jk}}}=\mathbb{E}\left\{{\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\mathit{Z}{z}_j\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k>Q\}\mathbb{𝟙}\{y=k\}\right\}.\end{array}$$

Similarly, for $l\ne k$, we have the distributional derivative

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\mathit{S}}_k\left({\mathbf{\theta }}_k\right)}{\partial {\theta }_{jl}}}=0.\end{array}$$

Recall that the distributional derivative does not depend on the order of differentiation and agrees with the classical derivative whenever the latter exists. To summarize, we have that

$$\begin{array}{cc}\hfill {\mathit{H}}_k\left({\mathbf{\theta }}_k\right)=& {\displaystyle \frac{{\partial }^2\mathcal{L}\left(\mathbf{\vartheta }\right)}{\partial {\mathbf{\theta }}_k\partial {\mathbf{\theta }}_k^{\top }}}={\displaystyle \frac{\partial {\mathit{S}}_k\left({\mathbf{\theta }}_k\right)}{\partial {\mathbf{\theta }}_k^{\top }}},\mathcal{H}\left(\mathbf{\vartheta }\right)=\underset{k=1}{\overset{K}{⨁}}{\mathit{H}}_k\left({\mathbf{\theta }}_k\right).\hfill \end{array}$$

The ${\mathit{H}}_k\left({\mathbf{\theta }}_k\right)$ are symmetric matrices, so $\mathcal{H}\left(\mathbf{\vartheta }\right)$ is also symmetric.

In the sense of generalized functions, differentiation is a continuous operation with respect to convergence in ${\mathcal{D}}^{\prime }$. Therefore, ${\varphi }_0^{\prime }=\underset{q\to 0}{lim}{\varphi }_q^{\prime }=-\mathbb{𝟙}\{u\le 0\}$ and ${\varphi }_0^{″}=\underset{q\to 0}{lim}{\varphi }_q^{″}=\delta \left(u\right)$; ${\varphi }_{\infty }^{\prime }=\underset{q\to \infty }{lim}{\varphi }_q^{\prime }=-\mathbb{𝟙}\{u\le 1\}$ and ${\varphi }_{\infty }^{″}=\underset{q\to \infty }{lim}{\varphi }_q^{″}=\delta (u-1)$, which coincides with results from the hinge loss.

Next, $\mathcal{H}\left(\mathbf{\vartheta }\right)\succ {\mathbf{O}}_{K(p+1)}$ if and only if both ${\mathit{H}}_1\left({\mathbf{\theta }}_1\right)$ and its Schur complement ${⨁}_{k=2}^K{\mathit{H}}_k\left({\mathbf{\theta }}_k\right)$ are both symmetric and positive definite. We can deduce that $\mathcal{H}\left(\mathbf{\vartheta }\right)\succ {\mathbf{O}}_{K(p+1)}$ if and only if ${\mathit{H}}_k\left({\mathbf{\theta }}_k\right)\succ {\mathbf{O}}_{p+1}$ for all k.

Note that there exists $c>0$ such that ${\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right)\ge c$ on ${\mathcal{X}}_k$. Then for any $\mathbf{\gamma }\in {\mathbb{R}}^{p+1}$,

$$\begin{array}{cc}\hfill {\mathbf{\gamma }}^{\top }{\mathit{H}}_k\left({\mathbf{\theta }}_k\right)\mathbf{\gamma }=& {\pi }_k{\int }_{{\mathcal{X}}_k}{\phi }_q^{″}\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k\right){\left({\mathit{Z}}^{\top }\mathbf{\gamma }\right)}^2{g}_k\left(\mathit{X}\right)\mathrm{d}\mathit{X}\hfill \\ \hfill \ge & cPr\{\mathit{X}\in {\mathcal{X}}_k,y=k\}\mathbb{E}\left\{{\left({\mathit{Z}}^{\top }\mathbf{\gamma }\right)}^2\right|\mathit{X}\in {\mathcal{X}}_k,y=k\}\hfill \\ \hfill \ge & cPr\{\mathit{X}\in {\mathcal{X}}_k,y=k\}\left({\gamma }_0^2+{\mathbf{\gamma }}_1^{\top }\mathrm{Var}\left\{\mathit{X}\right|\mathit{X}\in {\mathcal{X}}_k,y=k\}{\mathbf{\gamma }}_1\right),\hfill \end{array}$$

which implies that ${\mathbf{\gamma }}^{\top }{\mathit{H}}_k\left({\mathbf{\theta }}_k\right)\mathbf{\gamma }=0$ if and only if $\mathbf{\gamma }={\mathbf{0}}_{p+1}$ when $\mathrm{Var}\left\{\mathit{X}\right|\mathit{X}\in {\mathcal{X}}_k,y=k\}$ is assumed to be non-singular. Assuming that $\mathrm{Var}\left\{\mathit{X}\right|y=k\}\succ \mathbf{O}$ implies that $\mathrm{Var}\left\{\mathit{X}\right|\mathit{X}\in {\mathcal{X}}_k,y=k\}⪰\mathbf{O}$. □

Proof of Theorem 1

By Lemma A2, a minimizer ${\mathbf{\vartheta }}^{\ast }\in \mathcal{C}$ exists with ${\mathit{b}}_k^{\ast }\ne {\mathbf{0}}_p$ (by Lemma A4) and $\mathcal{H}\left({\mathbf{\vartheta }}^{\ast }\right)\succ {\mathbf{O}}_{K(p+1)}$ (by Lemma A5). By the second-order Lagrange condition and the convexity of $\mathcal{L}\left(\mathbf{\vartheta }\right)$ (by Lemma A1), a minimizer of the population MgDWD loss is unique.

Recall from (A2) that

$$\begin{array}{cc}\hfill \mathcal{L}\left({\mathbf{\vartheta }}^{\ast }\right)=& \mathbb{E}\left\{\mathbb{𝟙}\{y=k\}{\varphi }_q\left({\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }\right)\right\}\hfill \\ \hfill =& \sum _{k=1}^K{\pi }_k\left(1-\mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q\right\}|y=k\right\}\right)\hfill \\ \hfill =& \sum _{k=1}^{K}A(k,q){\pi }_k.\hfill \end{array}$$

It follows that

$$\begin{array}{cc}\hfill 0\le & \mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q\right\}|y=k\right\}\hfill \\ \hfill <& \mathbb{E}\left\{\mathbb{𝟙}\{{\mathit{Z}}^{\top }\mathbf{\gamma }>1+q^{-1}\}+\mathbb{𝟙}\{Q<{\mathit{Z}}^{\top }\mathbf{\gamma }\le 1+q^{-1}\}\left\{1-{\left({\displaystyle \frac{Q}{1+q^{-1}}}\right)}^q\right\}|y=m\right\}\hfill \\ \hfill =& Pr\left\{{\mathit{Z}}^{\top }\mathbf{\gamma }>Q|y=m\right\}-Pr\left\{Q<{\mathit{Z}}^{\top }\mathbf{\gamma }\le Q^{-1}|y=m\right\}{Q}^{2q}\hfill \\ \hfill \le & 1\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill 1\ge & \mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}\left\{1-{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q\right\}|y=k\right\}\hfill \\ \hfill >& \mathbb{E}\left\{\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>1+ϵ\}\left\{1-{\left({\displaystyle \frac{Q}{1+ϵ}}\right)}^q\right\}|y=k\right\}\hfill \\ \hfill \ge & \underset{ϵ>0}{sup}\left\{1-{\left({\displaystyle \frac{Q}{1+ϵ}}\right)}^q\right\}Pr\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>1+ϵ|y=m\right\}\hfill \\ \hfill \ge & 0.\hfill \end{array}$$

Consequently, $0\le u(k,q)\le A(k,q)\le v(k,q)\le 1$.

Note that $\underset{q\to \infty }{lim}{(1+ϵ)}^{-q}{Q}^q={\mathrm{e}}^{-1}$ when $ϵ=0$ and $\underset{q\to \infty }{lim}{(1+ϵ)}^{-q}{Q}^q=0$ when $ϵ>0$. The difference between these two results is attributed to pointwise convergence.

Let $f_m=1-A(k,m)\in {\mathcal{D}}^{\prime }$ with $m=1,2,\dots$ and $\eta \in \mathcal{D}$. By Fubini’s theorem and the dominated convergence theorem,

$$\begin{array}{cc}\hfill \underset{m\to \infty }{lim}⟨f_m,\eta ⟩=& \underset{m\to \infty }{lim}⟨\mathbb{E}\left\{\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q|y=k\right\},\eta \left(\mathbf{\gamma }\right)⟩\hfill \\ \hfill =& \underset{m\to \infty }{lim}\mathbb{E}\left\{⟨\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q,\eta \left({\mathbf{\theta }}_k^{\ast }\right)⟩|y=k\right\}\hfill \\ \hfill =& \mathbb{E}\left\{\underset{m\to \infty }{lim}⟨\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q,\eta \left({\mathbf{\theta }}_k^{\ast }\right)⟩|y=k\right\}\hfill \\ \hfill =& 0=⟨0,\eta \left({\mathbf{\theta }}_k^{\ast }\right)⟩.\hfill \end{array}$$

Similarly,

$$\begin{array}{cc}\hfill \underset{m\to 0}{lim}⟨f_m,\eta ⟩=& \mathbb{E}\left\{\underset{m\to 0}{lim}⟨\mathbb{𝟙}\{\mathit{X}\in {\mathcal{X}}_k^{\ast }\}{\left({\displaystyle \frac{Q}{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }}}\right)}^q,\eta \left(\mathbf{\gamma }\right)⟩|y=k\right\}\hfill \\ \hfill =& \mathbb{E}\left\{⟨\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>0\},\eta \left(\mathbf{\gamma }\right)⟩|y=k\right\}\hfill \\ \hfill =& ⟨\mathbb{E}\left\{\mathbb{𝟙}\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>0\}|y=k\right\},\eta \left({\mathbf{\theta }}_k^{\ast }\right)⟩\hfill \\ \hfill =& ⟨Pr\left\{{\mathit{Z}}^{\top }{\mathbf{\theta }}_k^{\ast }>0|y=k\right\},\eta \left({\mathbf{\theta }}_k^{\ast }\right)⟩,\hfill \end{array}$$

hence

$$\begin{array}{c}A(k,\infty )=\underset{q\to \infty }{lim}A(k,q)=Pr\left\{\mathit{X}\notin {\mathcal{X}}_k^{\ast }|y=k\right\},\mathrm{and}A(k,0)=\underset{q\to 0}{lim}A(k,q)=1.\end{array}$$

As a result, $A(k,\infty )$ coincides with the population hinge/SVM loss and $A(k,0)$ is independent of ${\mathbf{\theta }}_k^{\ast }$. □

Appendix A.3. Proof of Lemma 2

Proof. 

By the definition of $\tilde{P}$,

$$\begin{array}{c}\hfill \tilde{P}\left\{\mathbf{P}\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)\right\}=\tau {\parallel \mathbf{P}\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)\parallel }_{\infty }+(1-\tau )\underset{j}{max}\left\{{\parallel {\mathbf{P}}_K\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right)\parallel }_2,{\parallel {\mathbf{P}}_K\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right)\parallel }_2\right\},\end{array}$$

where

$$\begin{array}{c}{\mathbf{P}}_K\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right)={\mathbf{P}}_K{\left(\mathbf{E}\circ {\varphi }_q^{\prime }\left\{\mathbf{F}\left({\mathbf{\vartheta }}^{\ast }\right)\right\}\right)}^{\top }{\mathbf{1}}_K={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathbf{P}}_K\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right),\\ {\mathbf{P}}_K\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right)={\mathbf{P}}_K{\left(\mathbf{E}\circ {\varphi }_q^{\prime }\left\{\mathbf{F}\left({\mathbf{\vartheta }}^{\ast }\right)\right\}\right)}^{\top }{\mathit{x}}_j={\displaystyle \frac{1}{N}}\sum _{i=1}^N{x}_{ij}{\mathbf{P}}_K\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right),\end{array}$$

${\mathbf{P}}_K=({\mathit{p}}_1,\dots ,{\mathit{p}}_K)$ with ${\mathit{p}}_k=\left(p_{lk}\right)=\mathbb{𝟙}\{l=k\}-K^{-1}$, and

$$\begin{array}{c}\mathbb{E}\left\{{\mathbf{P}}_K\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right)\right\}={\mathbf{P}}_K\mathcal{S}\left({\mathbf{\alpha }}^{\ast }\right)={\mathbf{0}}_K,\mathbb{E}\left\{{\mathbf{P}}_K\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right)\right\}={\mathbf{P}}_K\mathcal{S}\left({\mathit{\beta }}_j^{\ast }\right)={\mathbf{0}}_K.\end{array}$$

Denoting

$$\begin{array}{c}\hfill d_{ik}=\left\{{\mathit{p}}_k^{\top }\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right)\right\}=\sum _{l=1}^K\left(\mathbb{𝟙}\{y_i=k\}-{\displaystyle \frac{1}{K}}\right)e_{il}{\varphi }_q^{\prime }\left(f_{il}^{\ast }\right),\end{array}$$

we have that $|d_{ik}|\le 1-K^{-1}$. Note that the $d_{ik}$ are N i.i.d. random variables with

$$\begin{array}{c}{\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{E}\left(d_{ik}\right)={\mathit{p}}_k^{\top }\mathcal{S}\left({\mathbf{\alpha }}^{\ast }\right)=0\mathrm{and}{\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{E}\left(d_{ik}{x}_{ij}\right)={\mathit{p}}_k^{\top }\mathcal{S}\left({\mathit{\beta }}_j^{\ast }\right)=0.\end{array}$$

By Hoeffding’s inequality, we have that

$$\begin{array}{c}\hfill Pr\left\{|{\mathit{p}}_k^{\top }\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right)|>c_1\left(1-{\displaystyle \frac{1}{K}}\right)\sqrt{{\displaystyle \frac{2log\left(pK\right)}{N}}}\right\}\le 2{\left(pK\right)}^{-c_1^2},\end{array}$$

(A3)

where $c_1>1$.

Regarding the $d_{ik}{x}_{ij}$, we have that

$$\begin{array}{c}\mathbb{E}exp\left\{d_{ik}{x}_{ij}\right\}\le \mathbb{E}exp\{(1-K^{-1})|x_{ij}|\}\le exp\left\{4{(1-K^{-1})}^2{\varsigma }_1^2{\kappa }^2\right\},\end{array}$$

which implies that the $d_{ik}{x}_{ij}$ are N independent sub-Gaussian random variables with variance proxy ${(1-K^{-1})}^2{\varsigma }_1^2{\kappa }^2$. Taking $c_1>1$, we have that

$$\begin{array}{c}\hfill Pr\left\{|{\mathit{p}}_k^{\top }\mathit{S}({\mathit{\beta }}_j^{\ast })|>c_1{\varsigma }_1\kappa \left(1-{\displaystyle \frac{1}{K}}\right)\sqrt{{\displaystyle \frac{2log\left(pK\right)}{N}}}\right\}\le 2{\left(pK\right)}^{-c_1^2}.\end{array}$$

(A4)

Then by (A3) and (A4),

$$\begin{array}{c}\hfill Pr\left\{\underset{j}{max}\left\{|{\mathit{p}}_k^{\top }\mathit{S}({\mathbf{\alpha }}^{\ast })|,|{\mathit{p}}_k^{\top }\mathit{S}({\mathit{\beta }}_j^{\ast })|\right\}>{\mathsf{\Lambda }}_1\right\}\le 2{\left(pK\right)}^{-c_1^2}\end{array}$$

(A5)

with

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_1=max\{{\varsigma }_1\kappa ,1\}{c}_1\left(1-{\displaystyle \frac{1}{K}}\right)\sqrt{{\displaystyle \frac{2log\left(pK\right)}{N}}}.\end{array}$$

Taking a union bound over the $Kp$ entries of $\mathbf{P}\mathit{S}\left({\mathit{\beta }}^{\ast }\right)$ yields that

$$\begin{array}{cc}\hfill Pr\left\{{\parallel \mathbf{P}\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)\parallel }_{\infty }\ge {\mathsf{\Lambda }}_1\right\}=& Pr\left\{\underset{j,k}{max}\left\{\left|{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{p}}_k^{\top }\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right)\right|,\left|{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{p}}_k^{\top }\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right)\right|\right\}\ge {\mathsf{\Lambda }}_1\right\}\hfill \\ \hfill \le & 2K(p+1){\left(Kp\right)}^{-c_1^2}.\hfill \end{array}$$

On one hand,

$$\begin{array}{c}\parallel \mathbf{P}\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right){\parallel }_2^2=\parallel ({\mathit{E}}_i-K^{-1})\circ {\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right){\parallel }_2^2\le \sum _{l=1}^K{(e_{il}-K^{-1})}^2·1=1-K^{-1},\end{array}$$

so for any $\mathbf{\gamma }\in {\mathbb{R}}^K$,

$$\begin{array}{c}\hfill |{\mathbf{\gamma }}^{\top }\mathbf{P}\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right){|\le \parallel \mathbf{\gamma }\parallel }_2\sqrt{1-{\displaystyle \frac{1}{K}}}\end{array}$$

and $\mathbb{E}\left\{{\mathbf{\gamma }}^{\top }\mathbf{P}\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right)\right\}=0$. Applying Hoeffding’s lemma,

$$\begin{array}{c}\hfill \mathbb{E}exp\left\{{\mathbf{\gamma }}^{\top }{\mathbf{P}}_K\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right)\right\}=\prod _{i=1}^N\mathbb{E}exp\left\{{\displaystyle \frac{1}{N}}{\mathbf{\gamma }}^{\top }{\mathbf{P}}_K\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right)\right\}\le exp\left\{{\displaystyle \frac{{\parallel \mathbf{\gamma }\parallel }_2^2}{2N}}\left(1-{\displaystyle \frac{1}{K}}\right)\right\}.\end{array}$$

Applying a square root to Theorem 2.1 of [31] with $c_2>1$, we have that

$$\begin{array}{c}Pr\left\{\parallel \mathbf{P}\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right){\parallel }_2\ge \sqrt{{\displaystyle \frac{K-1}{N}}}+c_2\sqrt{\left(1-{\displaystyle \frac{1}{K}}\right){\displaystyle \frac{2log\left(p\right)}{N}}}\right\}\le p^{-c_2^2}.\end{array}$$

(A6)

On the other hand, since the $x_{ij}$ are N independent sub-Gaussian random variables with variance proxy ${\varsigma }_1^2{\kappa }^2$,

$$\begin{array}{cc}\hfill \mathbb{E}exp\left\{{\mathbf{\gamma }}^{\top }\mathbf{P}\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right)\right\}=& \prod _{i=1}^N\mathbb{E}exp\left\{{\displaystyle \frac{x_{ij}}{N}}\left\{{\mathbf{\gamma }}^{\top }\mathbf{P}\mathrm{diag}\left\{{\mathit{E}}_i\right\}{\varphi }_q^{\prime }\left({\mathit{F}}_i^{\ast }\right)\right\}\right\}\hfill \\ \hfill \le & \prod _{i=1}^N\mathbb{E}exp\left\{\sqrt{1-{\displaystyle \frac{1}{K}}}{\displaystyle \frac{{\parallel \mathbf{\gamma }\parallel }_2}{N}}\left|x_{ij}\right|\right\}\hfill \\ \hfill \le & =exp\left\{{\displaystyle \frac{{\parallel \mathbf{\gamma }\parallel }_2^2}{2}}\left(1-{\displaystyle \frac{1}{K}}\right){\displaystyle \frac{8{\varsigma }_1^2{\kappa }^2}{N}}\right\}\hfill \end{array}$$

and $\mathbb{E}\{{\mathbf{P}}_K\mathit{S}({\mathit{\beta }}_j^{\ast })\}={\mathbf{0}}_K$. Similarly, we have that

$$\begin{array}{c}Pr\left\{\parallel \mathbf{P}\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right){\parallel }_2\ge 2\sqrt{2}{\varsigma }_1\kappa \left\{\sqrt{{\displaystyle \frac{K-1}{N}}}+c_2\sqrt{\left(1-{\displaystyle \frac{1}{K}}\right){\displaystyle \frac{2log\left(p\right)}{N}}}\right\}\right\}\le p^{-c_2^2}\end{array}$$

(A7)

for a constant $c_2>1$.

Therefore, by (A6) and (A7),

$$\begin{array}{c}\hfill Pr\left\{\underset{j}{max}\left\{\parallel \mathbf{P}\mathit{S}\left({\mathbf{\alpha }}^{\ast }\right){\parallel }_2,{\parallel \mathbf{P}\mathit{S}\left({\mathit{\beta }}_j^{\ast }\right)\parallel }_2\right\}\ge {\mathsf{\Lambda }}_2\right\}\le p^{-c_2^2}\end{array}$$

with

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_2=max\{2\sqrt{2}{\varsigma }_1\kappa ,1\}\left\{\sqrt{{\displaystyle \frac{K-1}{N}}}+c_2\sqrt{\left(1-{\displaystyle \frac{1}{K}}\right){\displaystyle \frac{2log\left(p\right)}{N}}}\right\}.\end{array}$$

Applying the union bound to (A5), it follows that

$$\begin{array}{c}\hfill Pr\left\{\tilde{P}\left\{\mathbf{P}\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)\right\}\ge \tau {\mathsf{\Lambda }}_1+(1-\tau ){\mathsf{\Lambda }}_2\right\}\le 2K(p+1){\left(pK\right)}^{1-c_1^2}+p^{1-c_2^2},\end{array}$$

and the desired result follows. □

Appendix A.4. Proof of Theorem 2

Lemma A6.

Suppose that $\lambda =c_0\sqrt{{\displaystyle \frac{log\left(pK\right)}{N}}}$. Then $\widehat{\mathbf{\vartheta }}-{\mathbf{\vartheta }}^{\ast }\in \mathcal{U}$, where

$$\begin{array}{c}\hfill \mathcal{U}=\left\{\mathbf{\delta }\in {\mathbb{R}}^{K(p+1)}|{\displaystyle \frac{\tau }{1-\tau }}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\ge C_0\left({\displaystyle \frac{\tau }{1-\tau }}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1+\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right)\right\},\end{array}$$

$C_0=\frac{(c_0-1)}{(c_0+1)}$, ${\mathcal{E}}^{\mathrm{c}}$ denotes the complement of $\mathcal{E}$, ${\mathcal{E}}_{+}=\mathcal{E}\cup \{l=1+(k-1)(p+1)|k=1,\dots ,K\}$, and ${\mathcal{G}}_{+}=\mathcal{G}\cup \left\{0\right\}$.

Proof. 

Since $\widehat{\mathbf{\vartheta }}={\mathbf{\vartheta }}^{\ast }+\mathbf{\delta }$ is the minimizer, we have that

$$\begin{array}{cc}\hfill L\left({\mathbf{\vartheta }}^{\ast }\right)+\lambda P\left({\mathit{\beta }}^{\ast }\right)\ge & L\left(\widehat{\mathbf{\vartheta }}\right)+\lambda P\left(\widehat{\mathit{\beta }}\right)\hfill \\ \hfill \lambda \left\{P\left({\mathit{\beta }}^{\ast }\right)-P({\mathit{\beta }}^{\ast }+\tilde{\mathbf{\delta }})\right\}\ge & L({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })-L\left({\mathbf{\vartheta }}^{\ast }\right),\hfill \end{array}$$

(A8)

where ${\mathit{\beta }}^{\ast }$ is the vector ${\mathbf{\vartheta }}^{\ast }$ without the $a_k$ components, replacing $\tilde{\mathbf{\delta }}$ for $\mathbf{\delta }$. Then

$$\begin{array}{cc}\hfill P\left({\mathit{\beta }}^{\ast }\right)-P({\mathit{\beta }}^{\ast }+\tilde{\mathbf{\delta }})=& \tau \left({\parallel {\mathit{\beta }}_{\mathcal{E}}^{\ast }\parallel }_1-{\parallel {\mathit{\beta }}_{\mathcal{E}}^{\ast }+{\tilde{\mathbf{\delta }}}_{\mathcal{E}}\parallel }_1-{\parallel {\tilde{\mathbf{\delta }}}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1\right)\hfill \\ \hfill & +(1-\tau )\left(\sum _{j\in \mathcal{G}}{\parallel {\mathit{\beta }}_j^{\ast }\parallel }_2-\sum _{j\in \mathcal{G}}{\parallel {\mathit{\beta }}_j^{\ast }+{\mathbf{\delta }}_j\parallel }_2-\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j^{\ast }\parallel }_2\right)\hfill \\ \hfill \le & \tau \left({\parallel {\tilde{\mathbf{\delta }}}_{\mathcal{E}}\parallel }_1-{\parallel {\tilde{\mathbf{\delta }}}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1\right)+(1-\tau )\left(\sum _{j\in \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2-\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j^{\ast }\parallel }_2\right)\hfill \\ \hfill \le & \tau \left({\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1-{\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1\right)+(1-\tau )\left(\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2-\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right).\hfill \end{array}$$

By the convexity of L,

$$\begin{array}{c}\hfill L({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })-L\left({\mathbf{\vartheta }}^{\ast }\right)\ge ⟨\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right),\mathbf{\delta }⟩\ge -\overline{P}\left\{\mathbf{P}\mathit{S}\left({\mathbf{\vartheta }}^{\ast }\right)\right\}P\left(\mathbf{\delta }\right)\ge -{\displaystyle \frac{\lambda }{c_0}}P\left(\mathbf{\delta }\right).\end{array}$$

Note that

$$\begin{array}{c}\hfill P\left(\mathbf{\delta }\right)=\tau \left({\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+{\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1\right)+(1-\tau )\left(\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2+\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right).\end{array}$$

Combining the above results, we have that

$$\begin{array}{c}\lambda \left\{P\left({\mathbf{\vartheta }}^{\ast }\right)-P({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })\right\}\ge \left\{L({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })-L\left({\mathbf{\vartheta }}^{\ast }\right)\right\}\\ (c+1)\tau {\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+(1-\tau )\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\ge (c-1)\tau {\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1+(1-\tau )\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\\ {\displaystyle \frac{\tau }{1-\tau }}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\ge C_0\left({\displaystyle \frac{\tau }{1-\tau }}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1+\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right).\end{array}$$

□

Lemma A7.

Assume that conditions (A1)–(A3) are satisfied. Then

$$\begin{array}{c}\hfill \underset{\mathit{v}\in \mathcal{V}}{sup}{\displaystyle \frac{\left|\mathsf{\Delta }L\right(\mathit{u},\mathit{v})-\mathbb{E}\{\mathsf{\Delta }L(\mathit{u},\mathit{v})\left\}\right|}{{\parallel \mathit{v}\parallel }_2}}>{\mathsf{\Lambda }}_3\end{array}$$

with probability at most $2{\left(Kp\right)}^{2(s_e+K)(1-c_3^2)}$, where

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_3=(1+\sqrt{2}{c}_3){\varsigma }_2\sqrt{{\displaystyle \frac{2(s_e+K)log\left(pK\right)}{N}}}\end{array}$$

and $\mathsf{\Delta }L(\mathit{u},\mathit{v})=L(\mathit{u}+\mathit{v})-L\left(\mathit{u}\right)$ for any $\mathit{u},\mathit{v}\in {\mathbb{R}}^{K(p+1)}$ and for some constant $c_3>1$.

Proof. 

Given any $\mathit{u}\in {\mathbb{R}}^{K(p+1)}$ and $\mathit{v}\in \mathcal{V}$ with $\mathcal{V}=\left\{\mathit{v}\in {\mathbb{R}}^{K(p+1)}{|0<\parallel \mathit{v}\parallel }_0\le s_e+K\right\}$,

$$\begin{array}{cc}\hfill \mathsf{\Delta }L(\mathit{u},\mathit{v})=& {\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathit{E}}_i^{\top }\left({\varphi }_q\left\{{(\mathbf{U}+\mathbf{V})}^{\top }{\mathit{Z}}_i\right\}-{\varphi }_q\left\{{\mathbf{U}}^{\top }{\mathit{Z}}_i\right\}\right)\hfill \\ \hfill =& {\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{k=1}^K{e}_{ik}\left({\varphi }_q\left\{{\mathit{Z}}_i^{\top }({\mathit{u}}_k+{\mathit{v}}_k)\right\}-{\varphi }_q\left\{{\mathit{Z}}_i^{\top }\left({\mathit{u}}_k\right)\right\}\right)\hfill \\ \hfill =& {\displaystyle \frac{1}{N}}\sum _{i=1}^N{d}_i(\mathit{u},\mathit{v}),\hfill \end{array}$$

where $\mathit{u}=\mathrm{vec}\left\{\mathbf{U}\right\}$, $\mathit{v}=\mathrm{vec}\left\{\mathbf{V}\right\}$.

The bounded gradient implies the Lipschitz continuity of ${\varphi }_q$ so that $|{\varphi }_q(u+v)-{\varphi }_q\left(u\right)|\le |v|$. Since $e_{ik}\in \{0,1\}$, we have that

$$\begin{array}{cc}\hfill |d_i(\mathit{u},\mathit{v})|\le & \sum _{k=1}^K\left|e_{ik}\left\{{\varphi }_q\left\{{\mathit{Z}}_i^{\top }({\mathit{u}}_k+{\mathit{v}}_k)\right\}-{\varphi }_q\left({\mathit{Z}}_i^{\top }{\mathit{u}}_k\right)\right\}\right|\hfill \\ \hfill \le & \sum _{k=1}^K\left|e_{ik}{\mathit{Z}}_i^{\top }{\mathit{v}}_k\right|\le {\mathit{E}}_i^{\top }\mathrm{vec}\left\{{\mathbf{V}}^{\top }{\mathit{Z}}_i\right\}\hfill \\ \hfill =& {\mathit{v}}^{\top }({\mathit{Z}}_i\otimes {\mathbf{I}}_K){\mathit{E}}_i.\hfill \end{array}$$

Note that

$$\begin{array}{c}\hfill \sum _{i=1}^N{\left({\mathit{v}}^{\top }({\mathit{Z}}_i\otimes {\mathbf{I}}_K){\mathit{E}}_i\right)}^2={\parallel \mathrm{diag}\left\{\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\right\}(\mathbf{Z}\otimes {\mathbf{I}}_K)\mathit{v}\parallel }_2^2.\end{array}$$

By Hoeffding’s inequality, we have that

$$\begin{array}{cc}\hfill & Pr\left\{|{\displaystyle \frac{1}{N}}\sum _{i=1}^N{d}_i(\mathit{u},\mathit{v})-\mathbb{E}\left({\displaystyle \frac{1}{N}}\sum _{i=1}^N{d}_i(\mathit{u},\mathit{v})\right)|>t\right\}\hfill \\ \hfill \le & 2exp\left\{-{\displaystyle \frac{2N^2{t}^2}{4\parallel \mathrm{diag}\left\{\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\right\}(\mathbf{Z}\otimes {\mathbf{I}}_K){\mathit{v}\parallel }_2^2}}\right\}\hfill \\ \hfill \le & 2exp\left\{-{\displaystyle \frac{N{t}^2}{2{\varsigma }_2^2{\parallel \mathit{v}\parallel }_2^2}}\right\}.\hfill \end{array}$$

Thus $Pr\{R\left(\mathit{v}\right)>{\mathsf{\Lambda }}_3\}\le 2{\left(Kp\right)}^{-(s_e+K)c_3^2}$ with

$$\begin{array}{c}\hfill R\left(\mathit{v}\right)={\displaystyle \frac{\left|\mathsf{\Delta }L\right(\mathit{u},\mathit{v})-\mathbb{E}\{\mathsf{\Delta }L(\mathit{u},\mathit{v})\left\}\right|}{{\parallel \mathit{v}\parallel }_2}}\mathrm{and}{\mathsf{\Lambda }}_3=c_3{\varsigma }_2\sqrt{{\displaystyle \frac{2(s_e+K)log\left(pK\right)}{N}}}.\end{array}$$

Next, we consider covering $\mathcal{V}$ with $ϵ$-balls such that for any ${\mathit{v}}_1$ and ${\mathit{v}}_2$ in the same ball, $|\frac{{\mathit{v}}_1}{\parallel {\mathit{v}}_1{\parallel }_2}-\frac{{\mathit{v}}_1}{\parallel {\mathit{v}}_1{\parallel }_2}|\le ϵ,$ where $ϵ$ is a small positive number. The number of $ϵ$-balls required to cover a m-dimensional unit ball is bounded by ${(\frac{2}{ϵ}+1)}^m$. Then for those $\frac{\mathit{v}}{{\parallel \mathit{v}\parallel }_2}$, we require a covering number of at most ${(3\left(Kp\right)/ϵ)}^{s_e+K}$. Let $\mathcal{N}$ denote such an $ϵ$-net. We have that

$$\begin{array}{c}Pr\left\{\underset{\mathit{v}\in \mathcal{N}}{sup}R\left(\mathit{v}\right)>{\mathsf{\Lambda }}_3\right\}\le {\left({\displaystyle \frac{3Kp}{ϵ}}\right)}^{s_e+K}2{\left(Kp\right)}^{-(s_e+K)c_3^2}=2{\left\{{\displaystyle \frac{3}{ϵ}}{\left(Kp\right)}^{1-c_3^2}\right\}}^{s_e+K}.\end{array}$$

Furthermore, for any ${\mathit{v}}_1,{\mathit{v}}_2\in \mathcal{V}$,

$$\begin{array}{cc}\hfill |R\left({\mathit{v}}_1\right)-R\left({\mathit{v}}_2\right)|\le & {\displaystyle \frac{2}{N}}\parallel \mathrm{diag}\left\{\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\right\}(\mathbf{Z}\otimes {\mathbf{I}}_K)\left({\displaystyle \frac{{\mathit{v}}_1}{\parallel {\mathit{v}}_1{\parallel }_2}}-{\displaystyle \frac{{\mathit{v}}_1}{\parallel {\mathit{v}}_1{\parallel }_2}}\right){\parallel }_1\hfill \\ \hfill \le & {\displaystyle \frac{2}{\sqrt{N}}}\parallel \mathrm{diag}\left\{\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\right\}(\mathbf{Z}\otimes {\mathbf{I}}_K)\left({\displaystyle \frac{{\mathit{v}}_1}{\parallel {\mathit{v}}_1{\parallel }_2}}-{\displaystyle \frac{{\mathit{v}}_1}{\parallel {\mathit{v}}_1{\parallel }_2}}\right){\parallel }_2\hfill \\ \hfill \le & 2{\varsigma }_2ϵ.\hfill \end{array}$$

Therefore ${sup}_{\mathit{v}\in \mathcal{V}}R\left(\mathit{v}\right)\le {sup}_{\mathit{v}\in \mathcal{N}}R\left(\mathit{v}\right)+2{\varsigma }_2ϵ$. Taking $ϵ=\sqrt{{\displaystyle \frac{(s_e+K)log\left(pK\right)}{2N}}}$, we have that

$$\begin{array}{cc}\hfill Pr\left\{\underset{\mathit{v}\in \mathcal{V}}{sup}R\left(\mathit{v}\right)>{\mathsf{\Lambda }}_3\right\}\le & Pr\left\{\underset{\mathit{v}\in \mathcal{N}}{sup}R\left(\mathit{v}\right)>(c_3-1){\varsigma }_1\sqrt{{\displaystyle \frac{2(s_e+K)log\left(pK\right)}{N}}}\right\}\hfill \\ \hfill \le & 2{\left\{\sqrt{{\displaystyle \frac{2N}{(s_e+K)log\left(pK\right)}}}3{\left(Kp\right)}^{1-{(c_3-1)}^2}\right\}}^{s_e+K}\hfill \\ \hfill \le & 2{\left\{{\left(Kp\right)}^{2-{(c_3-1)}^2}\right\}}^{s_e+K}.\hfill \end{array}$$

Setting $c_3=1+\sqrt{2}{c}_4$ and $c_4>1$, we obtain the desired result that

$$\begin{array}{c}\hfill Pr\left\{\underset{\mathit{v}\in \mathcal{V}}{sup}R\left(\mathit{v}\right)>(1+\sqrt{2}{c}_4){\varsigma }_2\sqrt{{\displaystyle \frac{2(s_e+K)log\left(pK\right)}{N}}}\right\}\le 2{\left(Kp\right)}^{2(s_e+K)(1-c_4^2)}.\end{array}$$

□

Proof of Theorem 2.

Consider a disjoint partition on the coordinate set $\mathbf{\delta }=\widehat{\mathbf{\vartheta }}-{\mathbf{\vartheta }}^{\ast }$, that is, $\mathbf{\delta }={\sum }_{m=1}^M{\mathit{v}}_m$ with ${\mathit{v}}_m\in \mathcal{V}$. Note that, each subvector ${\mathit{v}}_m$ has at most $s_e+K$ non-zero coordinates. Denote ${\mathit{v}}_0=\mathbf{0}$ and ${\mathit{u}}_m={\mathbf{\vartheta }}^{\ast }+{\sum }_{l=0}^{m-1}{\mathit{v}}_l$ so that ${\mathit{u}}_1={\mathbf{\vartheta }}^{\ast }$ and ${\mathit{u}}_M+{\mathit{v}}_M={\mathbf{\vartheta }}^{\ast }+\mathbf{\delta }$. We have the decomposition

$$\begin{array}{cc}\hfill \mathsf{\Delta }L({\mathbf{\vartheta }}^{\ast },\mathbf{\delta })=& L\left({\mathbf{\vartheta }}^{\ast }+\sum _{m=1}^M{\mathit{v}}_m\right)-L\left({\mathbf{\vartheta }}^{\ast }\right)=\sum _{m=1}^{M}L\left({\mathbf{\vartheta }}^{\ast }+\sum _{l=0}^m{\mathit{v}}_l\right)-L\left({\mathbf{\vartheta }}^{\ast }+\sum _{l=0}^{m-1}{\mathit{v}}_l\right)\hfill \\ \hfill =& \sum _{m=1}^{M}L({\mathit{u}}_m+{\mathit{v}}_m)-L\left({\mathit{u}}_m\right)=\sum _{m=1}^M\mathsf{\Delta }L({\mathit{u}}_m,{\mathit{v}}_m).\hfill \end{array}$$

By Lemma A7,

$$\begin{array}{c}\hfill \sum _{m=1}^M\mathsf{\Delta }L({\mathit{u}}_m,{\mathit{v}}_m)\ge \sum _{m=1}^M\mathbb{E}\left\{\mathsf{\Delta }L({\mathit{u}}_m,{\mathit{v}}_m)\right\}-{\mathsf{\Lambda }}_3\parallel {\mathit{v}}_m{\parallel }_2=\mathbb{E}\left\{\mathsf{\Delta }L({\mathbf{\vartheta }}^{\ast },\mathbf{\delta })\right\}-{\mathsf{\Lambda }}_3{\parallel \mathbf{\delta }\parallel }_2\end{array}$$

with high probability. By Lemma A5, $\mathcal{L}$ is twice differentiable so that

$$\begin{array}{cc}\hfill \mathbb{E}\left\{\mathsf{\Delta }L({\mathbf{\vartheta }}^{\ast },\mathbf{\delta })\right\}& ={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{E}\left({\mathit{E}}_i^{\top }{\varphi }_q\left\{{\mathit{F}}_i({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })\right\}\right)-\mathbb{E}\left({\mathit{E}}_i^{\top }{\varphi }_q\left\{{\mathit{F}}_i\left({\mathbf{\vartheta }}^{\ast }\right)\right\}\right)\hfill \\ & =\mathcal{L}({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })-\mathcal{L}\left({\mathbf{\vartheta }}^{\ast }\right)\hfill \\ & =\mathcal{S}{\left({\mathbf{\vartheta }}^{\ast }\right)}^{\top }\mathbf{\delta }+{\displaystyle \frac{1}{2}}{\mathbf{\delta }}^{\top }\mathcal{H}\left({\mathbf{\vartheta }}^{\ast }\right)\mathbf{\delta }+o\left({\parallel \mathbf{\delta }\parallel }_2^2\right)\hfill \\ & \ge 0+{\displaystyle \frac{{\varsigma }_3^2}{2}}{\parallel \mathbf{\delta }\parallel }_2^2+o\left({\parallel \mathbf{\delta }\parallel }_2^2\right).\hfill \end{array}$$

Consequently, $\mathsf{\Delta }L({\mathbf{\vartheta }}^{\ast },\mathbf{\delta })$ is bounded below by ${\displaystyle \frac{{\varsigma }_3^2}{2}}{\parallel \mathbf{\delta }\parallel }_2^2-{\mathsf{\Lambda }}_3{\parallel \mathbf{\delta }\parallel }_2$ with high probability.

Note that

$$\begin{array}{cc}\hfill P\left({\mathit{\beta }}^{\ast }\right)-P({\mathit{\beta }}^{\ast }+\tilde{\mathbf{\delta }})\le & \tau \left({\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1-{\parallel {\mathbf{\delta }}_{{\mathcal{E}}^{\mathrm{c}}}\parallel }_1\right)+(1-\tau )\left(\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2-\sum _{j\notin \mathcal{G}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right)\hfill \\ \hfill \le & \left(\tau {\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+(1-\tau )\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right).\hfill \end{array}$$

From (A8),

$$\begin{array}{cc}\hfill L\left({\mathbf{\vartheta }}^{\ast }\right)+\lambda P\left({\mathit{\beta }}^{\ast }\right)\ge & L\left(\widehat{\mathbf{\vartheta }}\right)+\lambda P\left(\widehat{\mathit{\beta }}\right)\hfill \\ \hfill \lambda \left\{P\left({\mathit{\beta }}^{\ast }\right)-P({\mathit{\beta }}^{\ast }+\tilde{\mathbf{\delta }})\right\}\ge & L({\mathbf{\vartheta }}^{\ast }+\mathbf{\delta })-L\left({\mathbf{\vartheta }}^{\ast }\right)\hfill \\ \hfill \lambda \left(\tau {\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+(1-\tau )\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right)\ge & {\displaystyle \frac{{\varsigma }_3^2}{2}}{\parallel \mathbf{\delta }\parallel }_2^2-{\mathsf{\Lambda }}_3{\parallel \mathbf{\delta }\parallel }_2.\hfill \end{array}$$

Clearly, ${\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1\le \sqrt{s_e+K}{\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_2\le \sqrt{s_e+K}{\parallel \mathbf{\delta }\parallel }_2$ and ${\sum }_{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\le \sqrt{s_g+1}{\parallel \mathbf{\delta }\parallel }_2$. We conclude that

$$\begin{array}{cc}\hfill {\displaystyle \frac{{\varsigma }_3^2}{2}}{\parallel \mathbf{\delta }\parallel }_2^2\le & \lambda \left(\tau {\parallel {\mathbf{\delta }}_{{\mathcal{E}}_{+}}\parallel }_1+(1-\tau )\sum _{j\in {\mathcal{G}}_{+}}{\parallel {\mathbf{\delta }}_j\parallel }_2\right)+{\mathsf{\Lambda }}_3{\parallel \mathbf{\delta }\parallel }_2\hfill \\ \hfill {\parallel \mathbf{\delta }\parallel }_2^2\le & 2{\varsigma }_3^{-2}\left\{\lambda \left(\tau \sqrt{s_e+K}+(1-\tau )\sqrt{s_g+1}\right)+{\mathsf{\Lambda }}_3\right\}{\parallel \mathbf{\delta }\parallel }_2,\hfill \end{array}$$

after which the desired result follows from straightforward algebraic manipulation. □

Appendix A.5. Proof of Lemma 3

Proof. 

Since

$$\begin{array}{c}\hfill \mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }=\mathrm{vec}{\left\{{({\mathbf{1}}_N{\mathbf{\alpha }}^{\top }+\mathbf{XB})}^{\top }\right\}}^{\top }={\mathbf{\alpha }}^{\top }({\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K)+\mathrm{vec}{\left({\mathbf{B}}^{\top }\right)}^{\top }({\mathbf{X}}^{\top }\otimes {\mathbf{I}}_K),\end{array}$$

we have that

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\displaystyle \frac{\partial \mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }}{\partial \mathbf{\alpha }}}={\displaystyle \frac{\partial {\mathbf{\alpha }}^{\top }({\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K)}{\partial \mathbf{\alpha }}}={\displaystyle \frac{\partial {\mathbf{\alpha }}^{\top }}{\partial \mathbf{\alpha }}}({\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K)={\mathbf{I}}_K({\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K)={\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K\hfill \\ \\ {\displaystyle \frac{\partial \mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }}{\partial \mathrm{vec}\left({\mathbf{B}}^{\top }\right)}}={\displaystyle \frac{\partial \mathrm{vec}{\left({\mathbf{B}}^{\top }\right)}^{\top }({\mathbf{X}}^{\top }\otimes {\mathbf{I}}_K)}{\partial \mathrm{vec}\left({\mathbf{B}}^{\top }\right)}}={\mathbf{I}}_{pK}({\mathbf{X}}^{\top }\otimes {\mathbf{I}}_K)={\mathbf{X}}^{\top }\otimes {\mathbf{I}}_K.\hfill \end{array}\right.\end{array}$$

The derivative with respect to $\mathbf{\alpha }$ is

$$\begin{array}{cc}\hfill N\mathit{S}\left(\mathbf{\alpha }\right)=& N{\displaystyle \frac{\partial L\left(\mathbf{\theta }\right)}{\partial \mathbf{\alpha }}}={\displaystyle \frac{\partial }{\partial \mathbf{\alpha }}}\mathrm{vec}{\left\{{\mathbf{E}}^{\top }\right\}}^{\top }\mathrm{vec}\left\{{\varphi }_q\left({\mathbf{F}}^{\top }\right)\right\}\hfill \\ \hfill =& {\displaystyle \frac{\partial \mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }}{\partial \mathbf{\alpha }}}{\displaystyle \frac{\partial {\varphi }_q\left\{\mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }\right\}}{\partial \mathrm{vec}\left({\mathbf{F}}^{\top }\right)}}\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\hfill \\ \hfill =& ({\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K)\mathrm{diag}\left(\mathrm{vec}\left\{{\varphi }_q^{\prime }\left({\mathbf{F}}^{\top }\right)\right\}\right)\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\hfill \\ \hfill =& \mathrm{vec}\left({\mathbf{I}}_K{\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }{\mathbf{1}}_N\right)\hfill \\ \hfill =& {\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }{\mathbf{1}}_N.\hfill \end{array}$$

Thus,

$$\begin{array}{cc}\hfill {\parallel \mathit{S}\left(\mathbf{\alpha }\right)|}_{\mathit{v}}^{\mathit{u}}{\parallel }_2^2=& \parallel \mathit{S}\left(\mathit{u}\right)-\mathit{S}\left(\mathit{v}\right){\parallel }_2^2=N^{-2}\parallel ({\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K)\mathrm{vec}\left\{{\left(\mathbf{E}\circ {\varphi }_q^{\prime }\{\mathbf{F}(\mathbf{\alpha })\}{|}_{\mathit{v}}^{\mathit{u}}\right)}^{\top }\right\}{\parallel }_2^2\hfill \\ \hfill \le & N^{-2}\parallel {\mathbf{1}}_N^{\top }\otimes {\mathbf{I}}_K{\parallel }_2^2\parallel \mathrm{vec}\left\{{\left(\mathbf{E}\circ {\varphi }_q^{\prime }\{\mathbf{F}(\mathbf{\alpha })\}{|}_{\mathit{v}}^{\mathit{u}}\right)}^{\top }\right\}{\parallel }_2^2\hfill \\ \hfill =& N^{-1}\sum _{k=1}^K\sum _{i=1}^N{e}_{ik}^2{\left({\varphi }_q^{\prime }\left\{f_{ik}\left(u_k\right)\right\}-{\varphi }_q^{\prime }\left\{f_{ik}\left(v_k\right)\right\}\right)}^2\hfill \\ \hfill \le & N^{-1}\sum _{k=1}^K\left(\sum _{i=1}^N{e}_{ik}\right)L_q^2{(u_k-v_k)}^2\hfill \\ \hfill \le & N^{-1}{n}_{max}{L}_q^2{\parallel \mathit{u}-\mathit{v}\parallel }_2^2,\hfill \end{array}$$

where $L_q=\frac{{(q+1)}^2}{q}$ is the Lipschitz constant of ${\varphi }_q^{\prime }$. We have that $L_{\mathbf{\alpha }}=\sqrt{\frac{n_{max}}{N}}{L}_q$.

The derivative with respect to $\mathrm{vec}\left({\mathbf{B}}^{\top }\right)$ is

$$\begin{array}{cc}\hfill N{\displaystyle \frac{\partial L\left(\mathbf{\theta }\right)}{\partial \mathrm{vec}\left({\mathbf{B}}^{\top }\right)}}=& {\displaystyle \frac{\partial }{\partial \mathrm{vec}\left({\mathbf{B}}^{\top }\right)}}\mathrm{vec}{\left\{{\mathbf{E}}^{\top }\right\}}^{\top }\mathrm{vec}\left\{{\varphi }_q\left({\mathbf{F}}^{\top }\right)\right\}\hfill \\ \hfill =& {\displaystyle \frac{\partial \mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }}{\partial \mathrm{vec}\left({\mathbf{B}}^{\top }\right)}}{\displaystyle \frac{\partial {\varphi }_q\left\{\mathrm{vec}{\left({\mathbf{F}}^{\top }\right)}^{\top }\right\}}{\partial \mathrm{vec}\left({\mathbf{F}}^{\top }\right)}}\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\hfill \\ \hfill =& ({\mathbf{X}}^{\top }\otimes {\mathbf{I}}_K)\mathrm{diag}\left(\mathrm{vec}\left\{{\varphi }_q^{\prime }\left({\mathbf{F}}^{\top }\right)\right\}\right)\mathrm{vec}\left\{{\mathbf{E}}^{\top }\right\}\hfill \\ \hfill =& \mathrm{vec}\left({\mathbf{I}}_K{\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }\mathbf{X}\right)\hfill \\ \hfill =& \mathrm{vec}\left({\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}}^{\top }\mathbf{X}\right).\hfill \end{array}$$

Therefore, the derivative with respect to $\mathbf{B}$ is $\mathit{S}\left(\mathbf{B}\right)=N^{-1}{\mathbf{X}}^{\top }\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}$. Note that

$$\begin{array}{cc}\hfill \mathrm{vec}\left({\mathbf{X}}^{\top }\left\{\mathbf{E}\circ {\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}\right)=& ({\mathbf{I}}_K\otimes {\mathbf{X}}^{\top })\mathrm{diag}\left\{\mathrm{vec}\left(\mathbf{E}\right)\right\}\mathrm{vec}\left\{{\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}\hfill \\ \hfill =& \left\{\underset{k=1}{\overset{K}{⨁}}{\mathbf{X}}^{\top }\mathrm{diag}\left({\mathbf{e}}_k\right)\right\}\mathrm{vec}\left\{{\varphi }_q^{\prime }\left(\mathbf{F}\right)\right\}\hfill \end{array}$$

and

$$\begin{array}{c}\hfill \sum _{i=1}^N{\left\{e_{ik}{\mathit{X}}_i^{\top }({\mathit{u}}_k-{\mathit{v}}_k)\right\}}^2={∥\mathrm{diag}\left({\mathit{e}}_k\right)\mathbf{X}({\mathit{u}}_k-{\mathit{v}}_k)∥}_2^2\le \parallel \mathrm{diag}\left({\mathit{e}}_k\right){\mathbf{X}\parallel }_2^2{\parallel {\mathit{u}}_k-{\mathit{v}}_k\parallel }_2^2;\end{array}$$

thus

$$\begin{array}{cc}\hfill N^2\parallel \mathrm{vec}\{\mathit{S}\left(\mathbf{U}\right)-\mathit{S}\left(\mathbf{V}\right)\}{\parallel }_2^2=& \sum _{k=1}^K\parallel {\mathbf{X}}^{\top }\mathrm{diag}\left({\mathit{e}}_k\right){\varphi }_q\left\{{\mathit{f}}_k\left({\mathit{b}}_k\right)\right\}{|}_{{\mathit{v}}_k}^{{\mathit{u}}_k}{\parallel }_2^2\hfill \\ \hfill \le & \sum _{k=1}^K\parallel {\mathbf{X}}^{\top }\mathrm{diag}\left({\mathit{e}}_k\right){\parallel }_2^2\parallel \mathrm{diag}\left({\mathit{e}}_k\right){\varphi }_q\left\{{\mathit{f}}_k\left({\mathit{b}}_k\right)\right\}{{|}_{{\mathit{v}}_k}^{{\mathit{u}}_k}\parallel }_2^2\hfill \\ \hfill \le & \sum _{k=1}^K{\parallel \mathrm{diag}\left({\mathit{e}}_k\right)\mathbf{X}\parallel }_2^2\sum _{i=1}^N{e}_{ik}{\left({\varphi }_q\left\{{\mathit{f}}_{ik}\left({\mathit{u}}_k\right)\right\}-{\varphi }_q\left\{{\mathit{f}}_{ik}\left({\mathit{v}}_k\right)\right\}\right)}^2\hfill \\ \hfill \le & L_q^2\sum _{k=1}^K{\parallel \mathrm{diag}\left({\mathit{e}}_k\right)\mathbf{X}\parallel }_2^2\sum _{i=1}^N{\left\{e_{ik}{\mathit{X}}_i^{\top }({\mathit{u}}_k-{\mathit{v}}_k)\right\}}^2\hfill \\ \hfill \le & L_q^2\sum _{k=1}^K\parallel \mathrm{diag}\left({\mathit{e}}_k\right){\mathbf{X}\parallel }_2^4{\parallel {\mathit{u}}_k-{\mathit{v}}_k\parallel }_2^2\hfill \\ \hfill \le & \underset{k}{max}{\left\{\parallel \mathrm{diag}\left({\mathit{e}}_k\right){\mathbf{X}\parallel }_2^2\right\}}^2{\parallel \mathrm{vec}(\mathbf{U}-\mathbf{V})\parallel }_2^2.\hfill \end{array}$$

We conclude that $L_{\mathbf{B}}=L_q{N}^{-1}{max}_k{\parallel \mathrm{diag}\left({\mathit{e}}_k\right)\mathbf{X}\parallel }_2^2$. □

Appendix A.6. Proof of Theorem 3

Lemma A8.

The indicator function

$$\begin{array}{c}\hfill {\delta }_{\mathcal{R}}\left(\mathit{x}\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\mathit{x}\in \mathcal{R}\hfill \\ \infty ,\hfill & \mathrm{if}\mathit{x}\notin \mathcal{R},\hfill \end{array}\right.\end{array}$$

where $\mathcal{R}=\{\mathit{x}\in {\mathbb{R}}^p\mid {\mathbf{1}}_p^{\top }\mathit{x}=0\}$, has subdifferential

$$\begin{array}{c}\partial {\delta }_{\mathcal{R}}\left(\mathit{x}\right)=\left\{\begin{array}{cc}\{\mathit{g}\in {\mathbb{R}}^p\mid \mathit{g}=s{\mathbf{1}}_p,s\in \mathbb{R}\},\hfill & \mathrm{if}\mathit{x}\in \mathcal{R}\hfill \\ ⌀,\hfill & \mathrm{if}\mathit{x}\notin \mathcal{R}.\hfill \end{array}\right.\end{array}$$

Proof. 

Suppose that $\mathit{x}\in \mathcal{R}$. Then $\mathit{g}\in \partial {\delta }_{\mathcal{R}}\left(\mathit{x}\right)$ if and only if both

$$\begin{array}{c}{\delta }_{\mathcal{R}}\left(\mathit{y}\right)\ge {\delta }_{\mathcal{R}}\left(\mathit{x}\right)+⟨\mathit{g},\mathit{y}-\mathit{x}⟩\mathrm{for}\mathrm{all}\mathit{y}\in \mathcal{R}\mathrm{and}\\ {\mathit{\omega }}^{\top }(\mathit{y}-\mathit{x})\le 0.\end{array}$$

Let $\mathit{z}=\mathit{y}-\mathit{x}$. Then $\mathit{z}\in \mathcal{R}$ since ${\mathbf{1}}_p^{\top }(\mathit{y}-\mathit{X})=0$. Thus, ${\mathit{g}}^{\top }\mathit{z}\le 0$. If ${\mathit{g}}^{\top }\mathit{z}=0$, then $\mathit{g}\in \{\mathit{g}\in {\mathbb{R}}^p\mid \mathit{g}=s{\mathbf{1}}_p,s\in \mathbb{R}\}$. If there exists $\mathit{g}\in \partial {\delta }_{\mathcal{R}}\left(\mathit{x}\right)$ satisfying ${\mathit{g}}^{\top }\mathit{z}<0$ for some $\mathit{z}\in \mathcal{R}$, then $-\mathit{z}\in \mathcal{R}$, so we must have that ${\mathit{g}}^{\top }\mathit{z}>0$. This is a contradiction.

Now, for any $\mathit{x}\notin \mathcal{R}$, we have that $\mathit{g}\in \partial {\delta }_{\mathcal{R}}\left(\mathit{x}\right)$ if and only if both

$$\begin{array}{c}{\delta }_{\mathcal{R}}\left(\mathit{y}\right)\ge {\delta }_{\mathcal{R}}\left(\mathit{x}\right)+⟨\mathit{g},\mathit{y}-\mathit{x}⟩\mathrm{for}\mathrm{all}\mathit{y}\in \mathcal{R}\mathrm{and}\\ {\mathit{\omega }}^{\top }(\mathit{x}-\mathit{y})\ge \infty .\end{array}$$

For $\mathit{x}\notin \mathcal{R}$ and $\mathit{y}\in \mathcal{R}$, since $\mathit{z}=\mathit{x}-\mathit{y}\in {\mathbb{R}}^p$ and ${\mathit{g}}^{\top }\mathit{z}\ge \infty$, it must be that $\mathit{g}\in ⌀$. □

Proof of Theorem 3.

It is sufficient to minimize the objective function

$$\begin{array}{c}\hfill G\left(\mathit{\beta }\right)=\frac{1}{2}\parallel \mathit{\beta }-{\mathit{\beta }}_{\ast }{\parallel }_2^2+{\rho }_1{\parallel \mathit{\beta }\parallel }_1+{\rho }_2{\parallel \mathit{\beta }\parallel }_2+{\delta }_{\mathcal{R}}\left(\mathit{\beta }\right),\end{array}$$

where $\mathcal{R}=\{\mathit{x}\in {\mathbb{R}}^K\mid {\mathbf{1}}_K^{\top }\mathit{\beta }=0\}$. Then the subdifferential of $G\left(\mathit{\beta }\right)$ is

$$\begin{array}{c}\partial G\left(\mathit{\beta }\right)=\mathit{\beta }-{\mathit{\beta }}_{\ast }+{\rho }_1{\partial \parallel \mathit{\beta }\parallel }_1+{\rho }_2\partial {\parallel \mathit{\beta }\parallel }_2+\partial {\delta }_{\mathcal{R}}\left(\mathit{\beta }\right).\end{array}$$

For an optimal solution ${\mathit{\beta }}^{\ast }\in \mathcal{R}$, we have that ${\mathbf{0}}_p\in \partial G\left({\mathit{\beta }}^{\ast }\right)$ if and only if there exist $\mathit{u}\in {\partial \parallel \mathit{\beta }\parallel }_1$, $\mathit{v}\in {\partial \parallel \mathit{\beta }\parallel }_2$ and $s\in \mathbb{R}$ such that ${\mathit{\beta }}^{\ast }={\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}-{\rho }_2\mathit{v}-s{\mathbf{1}}_p$. Since ${\mathbf{1}}^{\top }{\mathit{\beta }}^{\ast }=0$, we have that $s=p^{-1}{\mathbf{1}}_p^{\top }({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}-{\rho }_2\mathit{v})$, so

$$\begin{array}{cc}\hfill {\mathit{\beta }}^{\ast }& ={\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}-{\rho }_2\mathit{v}).\hfill \end{array}$$

If ${\mathit{\beta }}^{\ast }={\mathbf{0}}_p$, then $|u_j|<1$ for $j=1,\dots ,p$, ${\parallel \mathit{v}\parallel }_2\le 1$ and

$$\begin{array}{c}\hfill \parallel {\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}){\parallel }_2={\rho }_2\parallel {\mathbf{P}}_K{\mathit{v}\parallel }_2\le {\rho }_2\parallel {\mathbf{P}}_K{\parallel }_2{\parallel \mathit{v}\parallel }_2={\rho }_2{\parallel \mathit{v}\parallel }_2\le {\rho }_2;\end{array}$$

If ${\mathit{\beta }}^{\ast }\ne {\mathbf{0}}_K$, then $\mathit{u}\in {\partial \parallel \mathit{X}\parallel }_1$, $\mathit{v}=\frac{{\mathit{\beta }}^{\ast }}{\parallel {\mathit{\beta }}^{\ast }{\parallel }_2}$ and

$$\begin{array}{cc}\hfill {\mathit{\beta }}^{\ast }=& {\mathbf{P}}_K\left({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}-{\rho }_2\frac{{\mathit{\beta }}^{\ast }}{\parallel {\mathit{\beta }}^{\ast }{\parallel }_2}\right)\hfill \\ \hfill \left(1+{\displaystyle \frac{{\rho }_2}{\parallel {\mathit{\beta }}^{\ast }{\parallel }_2}}\right){\mathit{\beta }}^{\ast }=& {\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}).\hfill \end{array}$$

Note that ${\mathit{\beta }}^{\ast }={\mathbf{P}}_K{\mathit{\beta }}^{\ast }\in \mathcal{R}$. Taking the norm of both sides, we see that

$$\begin{array}{cc}\hfill \left(1+{\displaystyle \frac{{\rho }_2}{\parallel {\mathit{\beta }}^{\ast }{\parallel }_2}}\right){\parallel {\mathit{\beta }}^{\ast }\parallel }_2& =\parallel {\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}){\parallel }_2\hfill \\ \hfill \parallel {\mathit{\beta }}^{\ast }{\parallel }_2& =\parallel {\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}){\parallel }_2-{\rho }_2>0.\hfill \end{array}$$

Substituting this result back into the ${\mathit{\beta }}^{\ast }\ne {\mathbf{0}}_K$ case, we have that

$$\begin{array}{c}\hfill {\mathit{\beta }}^{\ast }=\left\{1-{\displaystyle \frac{{\rho }_2}{\parallel {\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}){\parallel }_2}}\right\}{\mathbf{P}}_K({\mathit{\beta }}_{\ast }-{\rho }_1\mathit{u}).\end{array}$$

Combining the above two cases gives the desired result. □

Appendix A.7. Proof of Theorem 4

Proof. 

Denote the objective function by

$$\begin{array}{c}\hfill G\left(b\right)={\displaystyle \frac{1}{2}}{(b-t)}^2+\varrho \left\{\right|b|+|b+s\left|\right\}.\end{array}$$

When $s=0$, we obtain a lasso problem with

$$\begin{array}{c}\hfill b^{\ast }=\underset{b\in \mathbb{R}}{\mathrm{argmin}}{\displaystyle \frac{1}{2}}{(b-t)}^2+2\varrho \left|x\right|=\mathcal{S}(t,2\varrho ).\end{array}$$

When $s\ne 0$, the subdifferential of $G\left(b\right)$ is

$$\begin{array}{c}\hfill \partial G\left(b\right)=b-t+\varrho \{\partial |x|+\partial |x+s\left|\right\}.\end{array}$$

We see that $0\in \partial G\left(b^{\ast }\right)$ if and only if there exist $u\in \partial \left|b\right|$ and $v\in \partial |b+s|$ with

$$\begin{array}{c}\hfill b^{\ast }=b-\varrho (u+v).\end{array}$$

If $b^{\ast }=0$, then $\left|u\right|<1$ and $v=\mathrm{sign}\left(s\right)$, hence

$$\begin{array}{c}\hfill b^{\ast }=0\mathrm{if}|t-\varrho \mathrm{sign}\left(s\right)|\le \varrho .\end{array}$$

If $s>0$, then $\mathrm{sign}\left(s\right)=1$ and $0\le t\le 2\varrho$. If $s<0$, then $\mathrm{sign}\left(s\right)=-1$, and $-2\varrho \le t\le 0$. Note that if $t\ne 0$, then $\mathrm{sign}\left(s\right)=\mathrm{sign}\left(t\right)$ or $\mathrm{sign}\left(s\right)\mathrm{sign}\left(t\right)=1$.

When $b^{\ast }=-s$, then $u=-\mathrm{sign}\left(s\right)$ and $\left|v\right|<1$, hence

$$\begin{array}{c}\hfill b^{\ast }=-s\mathrm{if}|t+s+\varrho \mathrm{sign}\left(s\right)|\le \varrho .\end{array}$$

If $s>0$, then $\mathrm{sign}\left(s\right)=1$ and $-(s+2\lambda )\le t\le -s<0$. If $s<0$, then $\mathrm{sign}\left(s\right)=-1$ and $0<-s\le t\le -(s-2\lambda )$. Note that $\mathrm{sign}\left(s\right)=-\mathrm{sign}\left(t\right)$ is equivalent to $\mathrm{sign}\left(s\right)\mathrm{sign}\left(t\right)=-1$.

Let $C(s,t)={\displaystyle \frac{1-\mathrm{sign}\left(s\right)\mathrm{sign}\left(t\right)}{2}}\left|s\right|\ge 0$. We can summarize the two cases above as

$$\begin{array}{c}\hfill b^{\ast }=-C(s,t)\mathrm{if}0\le C(s,t)\le \left|t\right|\le C(s,t)+2\varrho .\end{array}$$

(A9)

If $b^{\ast }\ne 0,-s$, then $u=\mathrm{sign}\left(b^{\ast }\right)$ and $v=\mathrm{sign}(b^{\ast }+s)$, thus

$$\begin{array}{c}{b}^{\ast }=t-\varrho \left\{\mathrm{sign}\left(b^{\ast }\right)+\mathrm{sign}(b^{\ast }+s)\right\}\\ b^{\ast }+s=t+s-\varrho \left\{\mathrm{sign}\left(b^{\ast }\right)+\mathrm{sign}(b^{\ast }+s)\right\}.\end{array}$$

If $\mathrm{sign}\left(b^{\ast }\right)=-\mathrm{sign}(b^{\ast }+s)=1$, then $b^{\ast }(b^{\ast }+s)<0$ or $0<t<-s$. Thus $b^{\ast }=t>0$ if $0<t<-s$. If $\mathrm{sign}\left(b^{\ast }\right)=-\mathrm{sign}(b^{\ast }+s)=-1$, then $b^{\ast }(b^{\ast }+s)<0$ or $-s<t<0$. Thus $b^{\ast }=t<0$ if $-s<t<0$. Rewriting the two cases above, we have that

$$\begin{array}{c}{b}^{\ast }=t\mathrm{if}0<\left|t\right|<C(s,t).\end{array}$$

(A10)

If $\mathrm{sign}\left(b^{\ast }\right)=\mathrm{sign}(b^{\ast }+s)=1$, then

$$\begin{array}{c}min\{b^{\ast },b^{\ast }+s\}>0\\ t-2\varrho +{\displaystyle \frac{s-\left|s\right|}{2}}>0\\ \mathrm{sign}\left(t\right)\left|t\right|>\mathrm{sign}\left(t\right)\left({\displaystyle \frac{\left|s\right|}{2}}+2\varrho \right)-{\displaystyle \frac{s}{2}}>0.\end{array}$$

Note that $t>0$ and $\mathrm{sign}\left(x\right)=\mathrm{sign}\left(t\right)$. If $\mathrm{sign}\left(b^{\ast }\right)=\mathrm{sign}(b^{\ast }+s)=-1$, then

$$\begin{array}{c}max\{b^{\ast },b^{\ast }+s\}<0\\ t+2\varrho +{\displaystyle \frac{s+\left|s\right|}{2}}>0\\ \mathrm{sign}\left(t\right)\left|t\right|<\mathrm{sign}\left(t\right)\left({\displaystyle \frac{\left|s\right|}{2}}+2\varrho \right)-{\displaystyle \frac{s}{2}}<0.\end{array}$$

Note that $t<0$ and $\mathrm{sign}\left(x\right)=\mathrm{sign}\left(t\right)$. Rewriting the two cases above, we have that

$$\begin{array}{c}{b}^{\ast }=t-2\varrho \mathrm{sign}\left(t\right)\mathrm{if}\left|t\right|>2\varrho +C(s,t).\end{array}$$

(A11)

Summarizing (A9)–(A11),

$$\begin{array}{c}\hfill b^{\ast }=\left\{\begin{array}{cc}t,\hfill & \left|t\right|<C(s,t),\hfill \\ -C(s,t),\hfill & C(s,t)\le \left|t\right|\le C(s,t)+2\varrho ,\hfill \\ \mathrm{sign}\left(t\right)\left(\right|t|-2\varrho ),\hfill & \left|t\right|>C(s,t)+2\varrho ,\hfill \end{array}\right.\end{array}$$

with $C(s,t)={\displaystyle \frac{1-\mathrm{sign}\left(s\right)\mathrm{sign}\left(t\right)}{2}}\left|s\right|\ge 0$. On one hand, when $s\ne 0$,

$$\begin{array}{c}\hfill b^{\ast }=t-\mathcal{S}\left(t,C(s,t)\right)+\mathcal{S}\left\{\mathcal{S}\left(t,C(s,t)\right),2\varrho \right\}.\end{array}$$

On the other hand, when $s=0$, it follows that $b^{\ast }=\mathcal{S}(t,2\varrho )$ since $\mathcal{S}(z,0)=z$. □