Sparse Density Estimation with Measurement Errors

Abstract

This paper aims to estimate an unknown density of the data with measurement errors as a linear combination of functions from a dictionary. The main novelty is the proposal and investigation of the corrected sparse density estimator (CSDE). Inspired by the penalization approach, we propose the weighted Elastic-net penalized minimal ${\ell }_2$-distance method for sparse coefficients estimation, where the adaptive weights come from sharp concentration inequalities. The first-order conditions holding a high probability obtain the optimal weighted tuning parameters. Under local coherence or minimal eigenvalue assumptions, non-asymptotic oracle inequalities are derived. These theoretical results are transposed to obtain the support recovery with a high probability. Some numerical experiments for discrete and continuous distributions confirm the significant improvement obtained by our procedure when compared with other conventional approaches. Finally, the application is performed in a meteorology dataset. It shows that our method has potency and superiority in detecting multi-mode density shapes compared with other conventional approaches.

1. Introduction

Over the years, the mixture models have been extensively applied to model unknown distributional shapes in astronomy, biology, economics, and genomics (see [1] and references therein). The distributions of real data involving potential complex variables often show multi-mode and heterogeneity. Due to the flexibility, it also appears in various distribution-based statistical techniques, such as cluster analysis, discriminant analysis, survival analysis, and empirical Bayesian inference. Flexible mixture models can naturally represent how the data are generated as mathematical artifacts. Theoretical results show that the mixture can approximate any density in the Euclidean space well, and the amount of the mixture can also be finite (for example, a mixture of several Gaussian distributions). Although the mixture model is inherently attractive to the statistical modeling, it is well-known that it is difficult to infer (see [2,3]). From the computational aspect, the optimization problems of mixture models are non-convex. Although existing computational methods, such as EM and various MCMC algorithms, can make the mixture model fit the data relatively easily. It should be emphasized that the mixture problems are essentially challenging, even unrecognizable, and the number of components (says, the order selection) is hard to determine (see [4]). There is a large amount of literature on its approximation theory, and various methods have been proposed to estimate the components (see [5] and references therein).

The nonparametric method and combinatorial method in density estimation were well studied in [6,7], as well as [8]. These can consistently estimate the number of the mixture’s components when the components have known functional forms. When the number of candidate components is large, the non-parametric method becomes computationally infeasible. Fortunately, the high-dimensional inference would compensate for this gap and guarantee the corrected identification of the mixture components with a probability tending to one. With the advancement of technology, high-dimensional problems have been applied at the forefront of statistical research. The high-dimensional inference method has been applied to the infinite mixture models with a sparse mixture of $p\to \infty$ components, which is an interesting and challenging problem (see [9,10]). We propose an improvement of the sparse estimation strategy proposed in [9], in which Bunea et al. propose a ${\ell }_1$-type penalty [11] to obtain a sparse density estimate (SPADES). At the same time, we add a ${\ell }_2$-type penalty and extend the oracle-inequality results to our new estimator.

In the real data, we often encounter the situation that the i.i.d. samples $X_i=Z_i+{\epsilon }_i$ are contained by some zero-mean measurement errors ${\left\{{\epsilon }_i\right\}}_{i=1}^n$; see [12,13,14,15,16]. For density estimation of ${\left\{Z_i\right\}}_{i=1}^n$, if there exists orthogonal basis functions, the estimation method is quite easy. In the measurement errors setting, however, finding an orthogonal-based density function is not easy (see [17]). Ref. [17] suggests the assumption that the conditional distribution function of $X_i$ given $Z_i$ is known. This condition is somewhat strong since most conditional distributions cannot obtain the explicit formula (except the Gaussian distribution). To address this predicament, particularly with nonorthogonal base functions, the SPADES model is attractive and makes the situation easier to deal with. Based on the SPADES method, our approach is an Elastic-net calibration approach, which is simpler and more interpreted than the conditional inference procedure proposed by [17]. In this paper, we proposed the corrected loss function to debiasing the measurement errors, and this is motivated by [18]. The main problem of considering measurement errors in various statistical models is that they are responsible for the bias of the classical statistical estimates; this is true, e.g., in linear regression, which has traditionally been the main focus of studies related to measurement errors. Debiasing represents an important task in various statistical models. In linear regression, it can be performed in the basic measurement errors (ME) model, which is also denoted as the Errors-in-variables (EIV or EV) model, if it is possible to estimate the variability of measurement errors (see [19,20]). We derive the real variable selection consistency based on weighted ${\ell }_1$+ ${\ell }_2$ penalty [21]. At the same time, some theoretical results of SPADES only contain the situation of the equal weights setting, which is not plausible in the sense of adaptive (data-dependent) penalized estimation. Moreover, we perform the Poisson mixture model to approximate the complex discrete distribution in the simulation part, while existing papers only emphasize the performance of continuous distribution models. Note that the multivariate kernel density estimator can only deal with a continuous distribution, and it requires a multivariate bandwidths section, while our method is dimensional-free (the number of the required tuning parameters is only two). There has been quite a lot of work in this area, starting with [22].

There are several differences between our article and [9]. The first point is that the upper bound of non-asymptotic oracle inequality in in our Theorems 1 and 2 is tighter than Theorems 1 and 2 in [9], and the optimal weighted tuning parameters are derived. The second point is that the ${\ell }_1$-penalized techniques are applied in [9] to estimate the sparse density. Still, this paper considers the estimation of density functions in the presence of a classical measurement error. We opt to use an Elastic-net criterion function to estimate the density, which is taken to be approximated by a series of basis functions. The third point is that the tuning parameters are chosen by the coordinate descent algorithm in [9], and the mixture weights are calculated by the generalized bisection method (GBM). However, this paper directly calculates the optimal weights, so our algorithm is more accessible to implement than [9].

This paper is presented as follows. Section 2 introduces the density estimator, which can deal with measurement errors. This section introduces data-dependent weights for the Lasso penalty, and the weights are derived by the event of KKT conditions holding a high probability. In Section 3, we give a condition that can accurately estimate the weights of the mixture, with a probability tending to 1. We show that, in an increasing dimensional mixture model under the local coherence assumption, if the tuning parameter is higher than the noise level, the recovery of the mixture components can hold with a high probability. In Section 4, we study the performance of our approach on artificial data generated from mixture Gaussian or Poisson distributions compared with other conventional methods, which indeed shows the improvement by employing our procedures. Moreover, the simulation also demonstrates that our method is better than the traditional EM algorithm, even under a low-dimensional model. Considering the multi-modal density aspect of the meteorology dataset, our proposed estimator has a stronger ability to detect multiple modes for the underlying distribution compared with other methods, such as SPADES or un-weighted Elastic-net estimator. Section 5 is the summary, and the proof of theoretical results is delivered in the Appendix A.

2. Density Estimation

2.1. Mixture Models

Suppose that ${\left\{Z_i\right\}}_{i=1}^n\in {\mathbb{R}}^d$ are independent random variables with a common unknown density h. However, the observations are contaminated with measurement errors ${\left\{{\epsilon }_i\right\}}_{i=1}^n$ as latent variables, the observed data are actually $X_i=Z_i+{\epsilon }_i$. Let ${\left\{h_j\right\}}_{j=1}^W$ be a series of density functions (such as Gaussian density or Poisson mass function), and ${\left\{h_j\right\}}_{j=1}^W$ are also called basis functions. Assume that the h belongs to the linear combination of ${\left\{h_j\right\}}_{j=1}^W$. The $W:=W_n$ is a function of n, which is particularly intriguing for us since there may be $W\gg n$ (the high-dimensional setting). Let ${\beta }^{*}:=({\beta }_1^{*},\cdots ,{\beta }_W^{*})\in {\mathbb{R}}^W$ be the unknown true parameter. Assume that

• (H.1): the $h:=h_{{\beta }^{*}}$ is defined as

  $$Z\sim h\left(z\right):=h_{{\beta }^{*}}\left(z\right)=\sum _{j=1}^W{\beta }_j^{*}{h}_j\left(z\right),\mathrm{with}\sum _{j=1}^W{\beta }_j^{*}=1.$$

  (1)

If the base is orthogonal and there are no measurement errors, a perfectly natural method is to estimate h by an orthogonal series of estimators in the form of $h_{\tilde{\beta }}$, where $\tilde{\beta }$ has the coordinates ${\tilde{\beta }}_j=\frac{1}{n}{\sum }_{i=1}^n{h}_j\left(X_i\right)$ (see [17]). However, this estimator depends on the choice of W, and a data-driven selection of W or the threshold needs to be adaptive. This estimator can only be applied to $W\le n$. Nevertheless, we want to solve more general problems for $W>n$, and the base functions ${\left\{h_j\right\}}_{j=1}^W$ may not orthogonal.

We aim to achieve the best convergence for the estimator when the W is not necessarily less than n. Theorem 33.2 in [5] states that any smooth density can be well-approximated by a finite mixture of some continuous functions. However, Theorem 33.2 in [5] does not confirm how many components W are required for the mixture. Thus, the hypothesis of the increasing-dimensional W is reasonable. For discrete distributions, there is also a similar mixture density approximation—see Remark of Theorem 33.2 in [5].

2.2. The Density Estimation with Measurement Errors

This subsection aims to construct a sparse estimator for the density $h\left(z\right):=h_{{\beta }^{*}}\left(z\right)$ as a linear combination of known densities.

Recall the definition of the $L_2\left({\mathbb{R}}^d\right)$ norm $\parallel f\parallel ={\left({\int }_{{\mathbb{R}}^d}{f}^2\left(x\right)dx\right)}^{\frac{1}{2}}$. For $f,g\in L_2\left({\mathbb{R}}^d\right)$, let $<f,g>={\int }_{{\mathbb{R}}^d}f\left(x\right)g\left(x\right)dx$ be the inner product. If two functions f and g satisfy $<f,g>=0$, then we call these two functions are orthogonal. Note that if the density $h\left(z\right)$ belongs to $L_2\left({\mathbb{R}}^d\right)$ and assume that ${\left\{X_i\right\}}_{i=1}^n$ has the same distribution X, for any $f\in L_2$, we have $<f,h>={\int }_{{\mathbb{R}}^d}f\left(x\right)h\left(x\right)dx=Ef\left(X\right)$. If $h\left(x\right)$ is the density function for a discrete distribution, the integral is replaced by summation, and we can define the inner product as $<f,h>:={\sum }_{k\in {\mathbb{Z}}^d}f\left(k\right)h\left(k\right)$.

For true observations ${\left\{Z_i\right\}}_{i=1}^n$, we minimize the ${\parallel h_{\beta }-h\parallel }^2$ on $\beta \in {\mathbb{R}}^W$ to obtain the estimate of $h\left(z\right):=h_{{\beta }^{*}}\left(z\right)$, i.e., minimizing

$$\begin{array}{c}\hfill {\parallel h_{\beta }-h\parallel }^2={\parallel h\parallel }^2+{\parallel h_{\beta }\parallel }^2-2<h_{\beta },h>={\parallel h\parallel }^2+{\parallel h_{\beta }\parallel }^2-2E{h}_{\beta }\left(Z\right)\propto -2E{h}_{\beta }\left(Z\right)+{\parallel h_{\beta }\parallel }^2,\end{array}$$

which implies that minimizing the ${\parallel h_{\beta }-h\parallel }^2$ is equivalent to minimizing

$$\begin{array}{c}\hfill -2E{h}_{\beta }\left(Z\right)+{\parallel h_{\beta }\parallel }^2\approx -\frac{2}{n}\sum _{i=1}^n{h}_{\beta }\left(Z_i\right)+{\parallel h_{\beta }\parallel }^2.\end{array}$$

(2)

It is plausible to assign more constrains for the candidate set of $\beta$ in the optimization, for example, the ${\ell }_1$ constrains ${\parallel \beta \parallel }_1\le a$, where a is the tuning parameter. More adaptively, we prefer to use the weighted ${\ell }_1$ restriction ${\sum }_{j=1}^W{\omega }_j\left|{\beta }_j\right|\le a$, where the weights ${\omega }_j$’s are data-dependent and will be specified later. From [23], we add Elastic-net penality $2{\sum }_{j=1}^W{\omega }_j\left|{\beta }_j\right|+c{\sum }_{j=1}^W{\beta }_j^2$ with tuning parameter c, which is in regards to the measurement errors (see [24,25]) for a similar purpose. We would have $c=0$ in the situation without measurement errors. The c indeed becomes larger if the measurement errors become more serious, i.e., we can say that the c is proportional to the increasing variability of the measurement errors. It is different from SPADES since adjusting for the measurement errors is important for accurately describing the relationship between the observed varables and the outcomes of interest.

From the discussion above, now we propose the following Corrected Sparse Density Estimator (CSDE):

$$\begin{array}{c}\hfill \widehat{\beta }:=\widehat{\beta }({\omega }_1,\cdots ,{\omega }_W)=\underset{\beta \in {\mathbb{R}}^W}{\arg \min }\left\{-\frac{2}{n}\sum _{i=1}^n{h}_{\beta }\left(X_i\right)+{\parallel h_{\beta }\parallel }^2+2\sum _{j=1}^W{\omega }_j\left|{\beta }_j\right|+c\sum _{j=1}^W{\beta }_j^2\right\}\end{array}$$

(3)

where the c is the tuning parameter for ${\ell }_2$-penalty, and the c also presents the correction for adjusting the measurement errors in our observations.

For CSDE, if ${\left\{h_j\right\}}_{j=1}^W$ is an orthogonal system, it can be clearly seen that the CSDE estimator is consistent with the soft thresholding estimator, and the explicit solution is ${\widehat{\beta }}_j=\frac{{(1-{\omega }_j/|{\tilde{\beta }}_j\left|\right)}_{+}{\tilde{\beta }}_j}{1+c}$, where ${\tilde{\beta }}_j=\frac{1}{n}{\sum }_{i=1}^n{h}_j\left(X_i\right)$ and $x_{+}=\max (0,x)$. In this case, we can see that ${\omega }_j$ is the threshold of the j-th component of the simplest mean estimator $\tilde{\beta }=({\tilde{\beta }}_1,\cdots ,{\tilde{\beta }}_W)$.

From the sub-differential of the convex optimization, the corresponding Karush–Kuhn–Tucker conditions (necessary and sufficient first-order condition) for the minimizer in Equation (3) is

Lemma 1

(KKT conditions, Lemma 4.2 of [26]). Let $k\in \{1,2,\cdots ,W\}$ and $c>0$. Then, a necessary and sufficient condition for CSDE to be a solution of Equation (3)

• ${\widehat{\beta }}_k:\ne 0$if$\frac{1}{n}{\sum }_{i=1}^n{h}_k\left(X_i\right)-{\sum }_{j=1}^W{\widehat{\beta }}_j<h_j,h_k>-c{\widehat{\beta }}_k=w_k\mathrm{sign}\left({\widehat{\beta }}_k\right).$
• ${\widehat{\beta }}_k=0$if$\left|\frac{1}{n}{\sum }_{i=1}^n{h}_k\left(X_i\right)-{\sum }_{j=1}^W{\widehat{\beta }}_j<h_j,h_k>-c{\widehat{\beta }}_k\right|\le w_k.$

Since all values of ${\beta }_j^{*}$ are non-negative, when conducting minimization in Equation (3), we have to put a non-negative restriction for optimizing Equation (3).

Due to the computational feasibility and optimal first-order conditions, we prefer an adaptively weighted Lasso penalty as a convex adaptive ${\ell }_1$ penalization. We require that the larger weights are assigned to the coefficients of unimportant covariates, while significant covariates accompany the smaller weights. So, the weights represent the importance of the covariates. The larger (smaller) weights shrink to zero more easily (difficultly) than the unweighted Lasso, with appropriate or even optimal weights, leading to less bias and more efficient variable selection. The derivation of the weights will be given in Section 3.1.

In the end of this part, we will illustrate that in the mixture models, even without measurement errors, Equation (1) cannot be partially transformed into the linear model $Y={\mathrm{X}}^T\beta +\epsilon$, where Y is the n-dimensional response variables, $\mathrm{X}$ is the $W\times n$-dimensional fixed design matrix, $\beta$ is a W-dimensional vector of model parameters, the $\epsilon$ is a $n\times 1$-dimensional vector for random error terms with zero mean and finite variance. Consider the least square objective function $U\left(\beta \right)$ for estimating $\beta$, $U\left(\beta \right)={(Y-{\mathrm{X}}^T\beta )}^T(Y-{\mathrm{X}}^T\beta )=-2Y^T{\mathrm{X}}^T\beta +{\beta }^T\mathrm{X}{\mathrm{X}}^T\beta +Y^{T}Y.$ Minimizing $U\left(\beta \right)$ is equivalent to minimizing $U^{*}\left(\beta \right)$ in Formula (4)

$$U^{*}\left(\beta \right)=-2Y^T{\mathrm{X}}^T\beta +{\beta }^T\mathrm{X}{\mathrm{X}}^T\beta .$$

(4)

Comparing the objective function in Formula (4) with Equation (2), it is easy to obtain $Y={(\frac{1}{n},\frac{1}{n},\cdots ,\frac{1}{n})}^T,\beta ={({\beta }_1,{\beta }_2,\cdots ,{\beta }_W)}^T,\mathrm{X}=\left(\begin{array}{ccc}{h}_1\left(X_1\right)& \cdots & h_1\left(X_n\right)\\ ⋮& \ddots & ⋮\\ h_W\left(X_1\right)& \cdots & h_W\left(X_n\right)\end{array}\right).$ Substituting Y, $\mathrm{X}$ and $\beta$ into a linear regression model, we obtain

$${\left(\begin{array}{c}\frac{1}{n}\\ ⋮\\ \frac{1}{n}\end{array}\right)}_{n\times 1}={\left(\begin{array}{ccc}{h}_1\left(X_1\right)& \cdots & h_W\left(X_1\right)\\ ⋮& \ddots & ⋮\\ h_1\left(X_n\right)& \cdots & h_W\left(X_n\right)\end{array}\right)}_{n\times W}{\left(\begin{array}{c}{\beta }_1\\ ⋮\\ {\beta }_W\end{array}\right)}_{W\times 1}+{\left(\begin{array}{c}{\epsilon }_1\\ ⋮\\ {\epsilon }_n\end{array}\right)}_{n\times 1}.$$

Then,

$${\epsilon }_i=\frac{1}{n}-\sum _{j=1}^W{\beta }_j{h}_j\left(X_i\right),i=1,2,\cdots ,n.$$

(5)

It can be seen from Equation (5) that the value of ${\epsilon }_i$ is no longer random if $\mathrm{X}$ was the fixed design matrix. Furthermore, even for a random design $\mathrm{X}$, take the expectation on both sides of Equation (5), and one can find that the left side is not equal to the right side, that is, $E\left({\epsilon }_i\right)=0=\frac{1}{n}-{\sum }_{j=1}^W{\beta }_{j}E{h}_j\left(X_i\right).$ It leads to an additional requirement ${\sum }_{j=1}^W{\beta }_{j}E{h}_j\left(X_i\right)=\frac{1}{n}\to 0$, which is meaningless as $n\to \infty$, since all ${\beta }_j$ and $h_j$ are positive. This is a contradiction to ${\sum }_{j=1}^W{\beta }_{j}E{h}_j\left(X_i\right)>0$ for all n.

Both of the two situations above contradict the definition of the assumed linear regression model. Hence, we cannot convert the estimation of Equation (1) into the estimation problem of linear models. Thus, the existing oracle inequalities are not applicable anymore, and we will propose new ones later. However, we can transform the mixture models to a corrected score Dantzig selector, such as in [27]. Although [10] studies the oracle inequalities for adaptive Dantzig density estimation, their study does not contain the error-in-variables framework and the support recovery content.

3. Sparse Mixture Density Estimation

In this section, we will present the oracle inequalities for estimators $\widehat{\beta }$ and $h_{\widehat{\beta }}$. The core of this section consists of five main results corresponding to the oracle inequalities for estimated density (Theorems 1 and 2), upper bounds on ${\ell }_1$-estimation error (Corollaries 1 and 2) and support consistency (Theorem 3) as the byproduct of Corollary 2.

3.1. Data-Dependent Weights

The weights ${\omega }_j$’s are chosen adequately such that the KKT conditions for stochastic optimization problems have a high probability of being satisfied.

As mentioned before, the weights in Equation (3) rely on the observed data since we calculate the weights, ensuring the KKT conditions hold with a high probability. The weighted Lasso estimates could have less ${\ell }_1$ estimation error than Lasso estimates (see the simulation part and [28]). Next, we need to consider what kind of data-dependent weight configuration can enable the KKT conditions to be satisfied with a high probability. A way to obtain data-dependent weights is to apply a concentration inequality for a weighted sum of independent random variables. Moreover, the weights should be a known data function without any unknown parameters. A criterion can help obtain the weights grounded on McDiarmid’s inequality (see [29] for more details).

Lemma 2.

Suppose $X_1,\cdots ,X_n$ are independent random variables, and all values belong to a set A. Let $f:A^n\to \mathbb{R}$ be a function and satisfy the bounded difference conditions

$${sup}_{x_1,\cdots ,x_n,x_s^{\prime }\in A}|f(x_1,\cdots ,x_n)-f(x_1,\cdots ,x_{s-1},x_s^{\prime },x_{s+1},\cdots ,x_n)|\le C_s,$$

then for all$t>0$, $P\left\{|f(X_1,\cdots ,X_n)-Ef(X_1,\cdots ,X_n)|\ge t\right\}\le 2\exp \left\{-\frac{2t^2}{{\sum }_{s=1}^n{C}_s^2}\right\}.$

We define the KKT conditions of optimization evaluated at ${\beta }^{*}$ (it is from the sub-gradient of the optimization function evaluated at ${\beta }^{*}$) by the events below:

$${\mathcal{F}}_k\left({\omega }_k\right):=\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-\sum _{j=1}^W{\beta }_j^{*}<h_j,h_k>-c{\beta }_k^{*}\right|\le {\omega }_k\right\},k=1,2,\cdots ,W.$$

Assume that

• (H.2): $\exists L_k>0$ s.t. ${∥h_k∥}_{\infty }=\underset{1\le i\le n}{\max }\left|h_k\left(X_i\right)\right|\le 2L_k$;
• (H.3): $0<\underset{1\le j\le W}{\max }\left|{\beta }_j^{*}\right|\le B$.

(H.2) is an assumption in sparse ${\ell }_1$ estimation, and the assumption (H.3) is a classical compact parameter space assumption in sparse high-dimensional regressions (see [9,25]).

Next, we check that the event ${\mathcal{F}}_k\left({\omega }_k\right)$ is hold with high probability. Note that $E{h}_k\left(X_i\right)={\sum }_{j=1}^W{\beta }_j^{*}<h_j,h_k>$ (which is free of $X_i$), we find

$$\begin{array}{c}\hfill \frac{1}{n}\left|\sum _{i=1}^n{h}_k\left(X_i\right)-(\sum _{i\ne s}^n{h}_k\left(X_i\right)+h_k\left(X_s^{\prime }\right))\right|=\frac{1}{n}\left|h_k\left(X_s\right)-h_k\left(X_s^{\prime }\right)\right|\le \frac{1}{n}\left(\right|h_k\left(X_s\right)|+|h_k\left(X_s^{\prime }\right)\left|\right)\le \frac{4L_k}{n},\end{array}$$

where the last inequality is due to $|h_k\left(X_i\right)|\le \underset{1\le i\le n}{\max }\left|h_k\left(X_i\right)\right|\le 2L_k$.

Next, we apply the McDiarmid’s inequality on the event ${\mathcal{F}}_k^c\left({\omega }_k\right)$ by (H.3). Then

$$\begin{array}{cc}\hfill P\left({\mathcal{F}}_k^c\left({\omega }_k\right)\right)& =P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-\sum _{j=1}^W{\beta }_j^{*}<h_j,h_k>-c{\beta }_k^{*}\right|\ge {\omega }_k\right\}\hfill \\ \hfill \left(\mathrm{by}(\mathrm{H}.3)\right)& \le P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E{h}_k\left(X_i\right)\right|\ge {\omega }_k-cB\right\}\hfill \\ \hfill (\mathrm{define}{\tilde{\omega }}_k:={\omega }_k-cB>0)& \le 2\exp \left\{-\frac{2{\tilde{\omega }}_k^2}{16L_k^2/n}\right\}=2\exp \left\{-\frac{n{\tilde{\omega }}_k^2}{8L_k^2}\right\}=:\frac{\delta }{W},0<\delta <1.\hfill \end{array}$$

Considering the previous line,

$${\omega }_k:=2\sqrt{2}{L}_k\sqrt{\frac{1}{n}\log \frac{2W}{\delta }}+cB=:2\sqrt{2}{L}_{k}v(\delta /2)+cB,\mathrm{where}v=v\left(\delta \right):=\sqrt{\frac{1}{n}\log \frac{W}{\delta }}.$$

(6)

The weight ${\omega }_k$ in our paper is different from [9], which gives the un-shift version ($\check{{\omega }_k}=4L_k\sqrt{\frac{1}{n}\log \frac{W}{\delta /2}}$), due to the Elastic-net penalty. Define the modified KKT conditions:

$${\mathcal{K}}_k\left({\omega }_k\right):=\left\{\right|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-\sum _{j=1}^W{\beta }_j^{*}<h_j,h_k>|\le {\tilde{\omega }}_k\},k=1,2,\cdots ,W$$

(7)

which hold with probability of at least $1-2\exp \left\{-\frac{n{\tilde{\omega }}_k^2}{8L_k^2}\right\}$.

3.2. Non-Asymptotic Oracle Inequalities

Introduced by [30], oracle inequality is a powerful non-asymptotic and analytical tool that seeks to provide the distance between the obtained estimator and a true estimator. The sharp oracle inequality connects the optimal convergence of an obtained estimator compared with the true parameter (see [31,32]).

For $\forall \beta \in {\mathbb{R}}^W$, let $I\left(\beta \right)=\{j\in \{1,\cdots ,W\}:{\beta }_j\ne 0\}$ be the indices corresponding to the non-zero components of the vector $\beta$, i.e., the support in mathematical jargon. If there is no ambiguity, we would like to write $I\left({\beta }^{*}\right)$ as $I_{*}$ for simplicity. Define $W\left(\beta \right)={\sum }_{j=1}^{W}I({\beta }_j\ne 0)$ as the number of its non-zero components, where $I(·)$ represents the indicative function. Let ${\sigma }_j^2=Var\left(h_j\left(X_1\right)\right),1\le j\le W$.

Below, we will state the non-asymptotic oracle inequalities for $h_{\widehat{\beta }}$ (with the probability at least $1-\delta (W,n)$ for any integer W and n), which measures the $L_2$ distance between $h_{\widehat{\beta }}$ and h. For $\beta \in {\mathbb{R}}^W$, define the correlation for the two base densities: $h_i$ and $h_j$, ${\rho }_W(i,j)=\frac{<h_i,h_j>}{\parallel h_i\parallel \parallel h_j\parallel }$, $i,j=1,\cdots ,W.$ Our results will be established under the local coherence condition, and we define the maximal local coherence as:

$$\rho \left(\beta \right)={\max }_{i\in I\left(\beta \right)}{\max }_{j\ne i}\left|{\rho }_W(i,j)\right|.$$

It is easy to see that $\rho \left(\beta \right)$ measures the separation of the variables in the set $I\left(\beta \right)$ from one another and the rest. The degree of separation is measured in terms of the size of the correlation coefficients. However, the regular condition introduced by this coherence may be too strong. It may exclude cases that the “correlation” can be relatively significant for a small number of pairs $(i,j)$ and almost zero otherwise. Thus, we consider the definition of the cumulative local coherence given by [9]: ${\rho }_{*}\left(\beta \right)={\sum }_{i\in I\left(\beta \right)}{\sum }_{j>i}\left|{\rho }_W(i,j)\right|$. Define $H\left(\beta \right)={\max }_{j\in I\left(\beta \right)}\frac{{\omega }_j}{v(\delta /2)\parallel h_j\parallel },F={\max }_{1\le j\le W}\frac{v(\delta /2)\parallel h_j\parallel }{{\tilde{\omega }}_j}={\max }_{1\le j\le W}\frac{\parallel h_j\parallel }{2\sqrt{2}{L}_j},$ where ${\tilde{\omega }}_j:=2\sqrt{2}{L}_{j}v(\delta /2)$.

By using the definition of ${\rho }_{*}\left(\beta \right)$ and the notations above, we present the main results of this paper, which lays the foundation for the oracle inequality of the estimated mixture coefficients.

Theorem 1.

Under (H.1)–(H.3), let $c=\frac{{\min }_{1\le j\le W}\left\{{\tilde{\omega }}_j\right\}}{B}$ and a given constant $0<\gamma \le 1$. If the true base functions ${\left\{h_j\right\}}_{j=1}^W$ conform to the cumulative local coherence assumption for all $\beta \in {\mathbb{R}}^W$,

$$12FH\left(\beta \right){\rho }_{*}\left(\beta \right)\sqrt{W\left(\beta \right)}\le \gamma ,$$

(8)

then the $\widehat{\beta }$ of the optimization problem in Equation (3) has the following oracle inequality with a probability at least $1-\delta$,

$$\begin{array}{cc}\hfill & {\parallel h_{\widehat{\beta }}-h\parallel }^2+\frac{{\alpha }_{opt1}(1-\gamma )}{({\alpha }_{opt1}-1)}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\frac{{\alpha }_{opt1}}{{\alpha }_{opt1}-1}\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill \le & \frac{{\alpha }_{opt1}+1}{{\alpha }_{opt1}-1}{\parallel h_{\beta }-h\parallel }^2+\frac{18{\alpha }_{opt1}^2}{{\alpha }_{opt1}-1}{H}^2\left(\beta \right)v^2(\delta /2)W\left(\beta \right),\hfill \end{array}$$

where ${\alpha }_{opt1}=1+\sqrt{1+\frac{{\parallel h_{\beta }-h\parallel }^2}{9H^2\left(\beta \right)v^2(\delta /2)W\left(\beta \right)}}$.

It is worthy to note that here we use $\sqrt{W\left(\beta \right)}$ instead of $W\left(\beta \right)$, and the latter is used in [9]. The upper bound of the oracle inequality by Theorem 1 is sharper than the upper bound of Theorem 1 in [9]. Further, we give the value of the optimal ${\alpha }_{opt1}$, but [9] did not give it. The reason for this phenomenon is quite clean actually: from the proof, it is due to ineuqality (A5). Now, let us address the sparse Gram matrix ${\mathsf{\Psi }}_W={(<h_i,h_j>)}_{1\le i,j\le W}$ with a small number of non-zero elements in off-diagonal positions, define ${\psi }_W(i,j)$ as the element $(i,j)$-th of position ${\psi }_W$. Condition (8) in Theorem 1 can be transformed to the condition

$$12SH\left(\beta \right)\sqrt{W\left(\beta \right)}\le \gamma ,$$

where the number S is called the sparse index of matrix ${\mathsf{\Psi }}_W$, which is defined as $S=\left|\right\{(i,j):i,j\in \{1,\cdots ,W\},i>j\mathrm{and}{\psi }_W(i,j)\ne 0\left\}\right|$, where $\left|A\right|$ is the number of elements of set A.

Sometimes the assumption in Condition (8) does not imply the positive definiteness of ${\mathsf{\Psi }}_W$. Next, we give a similar oracle inequality that is valid under the hypothesis that the Gram matrix ${\mathsf{\Psi }}_W$ is positive definite.

Theorem 2.

Under the assumption of (H.1)–(H.3) and that the Gram matrix ${\mathsf{\Psi }}_W$ is positive definite with a minimum eigenvalue greater than or equal to ${\lambda }_W>0$. For all $\beta \in {\mathbb{R}}^W$, the $\widehat{\beta }$ of the optimization problem in Equation (3) has the following oracle inequality with probability at least $1-\delta$,

$$\begin{array}{cc}& {||h_{\widehat{\beta }}-h\parallel }^2+\frac{{\alpha }_{opt2}}{{\alpha }_{opt2}-1}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\frac{{\alpha }_{opt2}}{{\alpha }_{opt2}-1}\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill \le & \frac{{\alpha }_{opt2}+1}{{\alpha }_{opt2}-1}{\parallel h_{\beta }-h\parallel }^2+\frac{576{\alpha }_{opt2}^2}{{\alpha }_{opt2}-1}\frac{G}{{\lambda }_W}{v}^2(\delta /2),\hfill \end{array}$$

where$G=G\left(\beta \right):={\sum }_{j\in I\left(\beta \right)}{L}_j^2$and${\alpha }_{opt2}=1+\sqrt{1+\frac{{\parallel h_{\beta }-h\parallel }^2}{288\frac{G}{{\lambda }_W}{v}^2(\delta /2)}}$.

Remark 1.

The argument and result of Theorem 1 in this paper is more refined than the conclusion of Theorem 1 in [9] for Lasso by letting $\gamma =1/2$ and $c=0$. In addition, Theorems 1 and 2 of this paper, respectively, give the optimal α value of the density estimation oracle inequalities, namely ${\alpha }_{opt1}$, ${\alpha }_{opt2}$. It provides a potentially sharper bound for the ${\ell }_1$-estimation error bound.

Next, we will present the ${\ell }_1$-estimation error for the estimator $\widehat{\beta }$ by Equation (3), and the weights are defined by Equation (6). For the technical point, we consider that $\parallel h_j\parallel =1$ for all j in Equation (3), i.e., the base functions are normalized. This normalization mimics the covariates’ standardization procedure when doing penalized estimations in generalized linear models. For simplicity, we put $L:={\max }_{1\le j\le W}{L}_j$.

For any other choice of $v(\delta /2)$ greater than or equal to $\sqrt{\frac{1}{n}\log \frac{2W}{\delta }}$, the conclusions of Section 3 are valid with a high probability. It imposes a restriction on the predictive performance of CSDE. As pointed out in [33], for the ${\ell }_1$-penalty in the regression, the adjusted sequence ${\omega }_j$ required for the corrected selection is usually larger than the adjusted sequence ${\omega }_j$ that produces a good prediction. The selection of the mixture density shown below is also true. Specifically, we will take the value $\beta ={\beta }^{*}$ and $v=v(\delta /2W)=\sqrt{\frac{\log (2W^2/\delta )}{n}}$, then ${\alpha }_{opt1},{\alpha }_{opt2}=2$. Below, we give the Corollaries of Theorems 1 and 2.

Corollary 1.

Given the same conditions as Theorem 1 with $\parallel h_j\parallel =1$ for all j, let ${\alpha }_{opt1}=2$, then we have the following ${\ell }_1$-estimation error oracle inequality:

$$\begin{array}{c}\hfill \sum _{j=1}^W|{\widehat{\beta }}_j-{\beta }_j^{*}|\le \frac{72\sqrt{2}v(\delta /2W)W\left({\beta }^{*}\right)}{1-\gamma }\frac{{(L+L_{\min })}^2}{L_{\min }}\end{array}$$

(9)

with probability at least $1-\delta /W$, where $L_{\min }={\min }_{1\le j\le W}{L}_j$.

Corollary 2.

Given the same conditions as Theorem 2 with $\parallel h_j\parallel =1$ for all j, let ${\alpha }_{opt2}=2$, then we have the following oracle inequality, with probability at least $1-\delta /W$,

$${\sum }_{j=1}^W|{\widehat{\beta }}_j-{\beta }_j^{*}|\le \frac{288\sqrt{2}v(\delta /2W)G^{*}}{L_{\min }{\lambda }_W}, where G^{*}={\sum }_{j\in I_{*}}{L}_j^2.$$

If the number $W\left({\beta }^{*}\right)$ of the mixture indicator elements is much smaller than $\sqrt{n}$, then inequality (9) guarantees that the estimated $\widehat{\beta }$ is close to the true ${\beta }^{*}$, and the ${\ell }_1$-estimation error will be presented in the numerical simulation in Section 4. Our results of Corollaries 1 and 2 are non-asymptotic for any W and n. The oracle inequalities are guiders for us to find an optimal tuning parameter with order $O\left(\sqrt{\frac{\log W}{n}}\right)$ for a sharper estimation error and better prediction performance. This is also an intermediate and crucial result, which leads to the main results of correctly identifying the mixture components in Section 3.3. In the following section, we turn to cope with the identification of $I_{*}$. Corrected components are selected by the proposed oracle inequality for the weighted ${\ell }_1$+ ${\ell }_2$ penalty.

3.3. Corrected Support Identification of Mixture Models

In this section, we will study the results of the support recovery of our CSDE estimator. There are few results on support recovery, while most of the results are the consistency of the ${\ell }_1$-error and prediction errors. Here, we borrow the framework of [25,33]. They give many proof techniques to deal with the corrected support identification in linear models by ${\ell }_1+{\ell }_2$ regularization. Let $\widehat{I}$ be the set of indicators consisting of non-zero elements of $\widehat{\beta }$ in the given Equation (3). In other words, $\widehat{I}$ is an estimate of the true support set $I\left({\beta }^{*}\right):=I_{*}$. We will study $P(\widehat{I}=I\left({\beta }^{*}\right))\ge 1-\epsilon$ for a given $0<\epsilon <1$ under some mild conditions.

To identify the $I_{*}$ consistently, we need more assumptions about some special correlation conditions than ${\ell }_1$-error consistency.

Condition (A): ${\rho }_{*}\left({\beta }^{*}\right)\le \frac{L{L}_{\min }{\lambda }_W}{288G^{*}}.$

Moreover, we need an additional condition that the minimal signal should be higher than a threshold level and quantified by order of tuning parameter. Therefore, we state it as follows:

Condition (B): ${\min }_{j\in I^{*}}\left|{\beta }_j^{*}\right|\ge 4\sqrt{2}v\left(\frac{\delta }{2W}\right)L$, where $v\left(\frac{\delta }{2W}\right):=\sqrt{\frac{1}{n}\log \frac{2W^2}{\delta }}$.

When performing simulation, Condition (B) is the theoretical guarantee that the minor magnitude of ${\beta }_j$ must be greater than a threshold value as a minimal signal condition. It is also called the Beta-min condition (see [26]).

Theorem 3.

Let $0<\delta <\frac{1}{2}$ and define ${ϵ}_k:=|E\left[h_k\left(X_1\right)\right]-E\left[h_k\left(Z_1\right)\right]|$. Assume that both conditions (A) and (B) are true and give the same conditions as Corollary 2, then

$$P(\widehat{I}=I_{*})\ge 1-\left(4W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+2\delta \right), where {ϵ}_k^{*}={ϵ}_k/\sqrt{2}v(\delta /2W)L.$$

Under the Beta-min condition, the support estimation is very close to the true support of ${\beta }_j^{*}$. The probability of the event $\{\widehat{I}=I_{*}\}$ is high when W is growing. The $\widehat{\beta }$ recovers the corrected support with probability at least $1-(4W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+2\delta )$. The result is non-asymptotic and it is true for any fixed W and n. There is a similar conclusion about support consistency (see Theorem 6 of [25]).

4. Simulation and Real Data Analysis

Ref. [9] proposes the SPADES estimation to deal with the samples for sparse mixture density, and they also derive an algorithm from complementing their theoretical result. Their findings successfully handle the high-dimensional adaptive density estimation to some degree. However, their algorithm is costly and unstable. In this section, we deal with the tuning parameter directly and compare our CSDE method with the SPADES method in [9] and other similar methods. In all cases here, we fix $n=100$ for $W=81,131,211,321$, which is known as the dimension of the unknown parameter ${\beta }^{*}$. The performance of each estimator is evaluated by the ${\ell }_1$-estimation error and the total variation (TV) distance between the estimator and the true value of ${\beta }^{*}$. The total variation (TV) error is defined as: $\mathrm{TV}(h_{{\beta }^{*}},h_{\widehat{\beta }})=\int |h_{{\beta }^{*}}\left(x\right)-h_{\widehat{\beta }}\left(x\right)|dx.$

4.1. Tuning Parameter Selection

In [9], the ${\lambda }_1$ is chosen by the coordinate descent method, while the mixture weights are detected by GBM. However, in our article, the optimal weights can be computed directly. Thus, it is much easier to carry out than [9]. The ${\ell }_1$-penalty term ${\sum }_{j=1}^W{\omega }_j\left|{\beta }_j\right|$ with optimal weights are defined by ${\omega }_k:=2\sqrt{2}{L}_{k}v(\delta /2)+cB$, where $L_j={\parallel h_j\parallel }_{\infty }$, which usually can be computed easily for a continuous $h_j$.

For a discrete base density ${\left\{h_j\right\}}_{j=1}^W$, it can be estimated as the following approximation by using concentration inequalities from Exercise 4.3.3 of [34]: $|\mathrm{med}\left(X\right)-E\left(X\right)|\le \sqrt{2Var\left(X\right)},\overline{x}\approx x_{med}\left(1+O\left(n^{-1}\right)\right)\approx h^{-1}\left(L_j\right)\left(1+O\left(n^{-1}\right)\right)$, where $\overline{x}$ and $x_{med}$ represent the sample mean and sample median, respectively, in each simulation, then we only need to select the ${\lambda }_1$ and $c={\lambda }_2$, and they can be detected by the nesting coordinate descent method. Moreover, the precision level is assigned as $\xi =0.001$ in our simulation.

4.2. Multi-Modal Distributions

First, we examine our method in a multi-modal Gaussian model that is similar to the first model in [9]. However, our mixture Gaussian has a different variance, which leads the meaningful weights to our estimation. The density function for the i.i.d. sample $Z_1,\dots ,Z_n$ is assigned as follows:

$$h_{\beta }^{*}\left(x\right)=\sum _{j=1}^W{\beta }_j^{*}\varphi \left(x|aj,{\sigma }_j\right),$$

(10)

where $\varphi \left(x|aj,{\sigma }_j\right)$ is the density of $N(aj,{\sigma }_j^2)$. However, to estimate ${\beta }^{*}$, we only observe i.i.d. data $X_1,\dots ,X_n$ with density $g_{{\beta }_j^{*}}\left(x\right)={\sum }_{j=1}^W{\beta }_j^{*}\varphi \left(x\right|aj,\sqrt{1.1}{\sigma }_j)$. Put $a=0.5,n=100$ and

$${\beta }^{*}={\left({\mathbf{0}}_8^T,0.2,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_5^T,0.1,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_5^T,0.15,{\mathbf{0}}_{10}^T,0.15,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_{W-76}^T\right)}^T,$$

(11)

with $\sigma ={\left({\mathbf{1}}_{20}^T,{\mathbf{0.8}}_6^T,{\mathbf{0.6}}_{11}^T,{\mathbf{0.4}}_{11}^T,{\mathbf{0.6}}_6^T,{\mathbf{0.8}}_{11}^T,{\mathbf{1.2}}_{W-76}^T\right)}^T$.

We replicate the simulation $N=100$ times. Simulation results are presented in Table 1, from which we can see our method has more and more excellent performances as the W increases, which matches the non-asymptotic results in the previous section. The best performance is far better than the other three methods when $W=321$. It is worthy to note that the better approximation follows the increase in W, matching Equation (7) and Theorem 3 in our previous section.

Table 1. The simulation results in Section 4.2. The mean and standard deviation of the errors in the four estimators of ${\beta }^{*}$ under $N=100$ simulations, with $n=100$. The quasi-optimal ${\lambda }_2$ is $c=0.002$ for Elastic-net, while $c=0.027$ is for the CSDE.

	W	${\lambda }_1$	$L_1$ Error	$\mathrm{TV}$ Error
Lasso	81	0.065	2.133 (2.467)	1.137 (1.115)
Elastic-net			2.061 (1.439)	1.114 (0.805)
SPADES		0.053	1.922 (2.211)	1.258 (1.296)
CSDE			2.191 (4.812)	1.405 (2.329)
Lasso	131	0.068	2.032 (0.985)	1.352 (0.712)
Elastic-net			2.236 (2.498)	1.409 (1.056)
SPADES		0.056	1.880 (2.644)	0.972 (1.204)
CSDE			1.635 (0.342)	0.863 (0.402)
Lasso	211	0.071	2.572 (4.187)	1.605 (2.702)
Elastic-net			2.061 (1.883)	1.353 (1.516)
SPADES		0.058	1.764 (1.041)	0.832 (0.610)
CSDE			1.648 (0.168)	0.791 (0.415)
Lasso	321	0.074	2.120 (2.842)	1.146 (1.115)
Elastic-net			10.173 (82.753)	7.839 (67.887)
SPADES		0.061	2.106 (4.816)	0.818 (1.565)
CSDE			1.623 (0.085)	0.634 (0.199)

We plot the solution path to compare the performance of the four estimators in ${\beta }_j\in I\left(\beta \right)$ for every W in Figure 1 (the result of Elastic-net in $W=321$ is not be shown due to its poor performance.). These figures also provide strong support for the above analysis. Meanwhile, we plot the probability densities of the several estimators and the true density to complement the visual sensory of the advantage in our method in Figure 2. The robust competency of detecting the multi-mode is shown (whereas other methods only find the most strongest signal, ignoring other meaningful but relatively weak signals).

Figure 1. The simulation result in Section 4.2. The estimated support of ${\beta }^{*}$ by the four types of estimators, and the W is varying. The circles represent the means of the estimators under the four specific approaches, while the half vertical lines mean the standard deviations.

Figure 2. The simulation results in Section 4.2. The density map of the four estimators. The result of Elastic-net in $W=321$ is not be shown due to its poor performance.

4.3. Mixture of Poisson Distributions

We study the mixture of discrete distribution: the mixture Poisson distribution

$$h_{{\beta }^{*}}\left(x\right)=\sum _{j=1}^W{\beta }_j^{*}p\left(x|{\lambda }_j=a·j\right),$$

(12)

where $p\left(x|{\lambda }_j=a·j\right)$ is the probability mass function (p.m.f.) of the Poisson distribution with mean ${\lambda }_j$. We set $a=0.1$ and

$${\beta }^{*}={\left({\mathbf{0}}_8^T,0.2,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_5^T,0.1,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_5^T,0.15,{\mathbf{0}}_{10}^T,0.15,{\mathbf{0}}_{10}^T,0.1,{\mathbf{0}}_{W-75}^T\right)}^T.$$

(13)

The adjusted weights are calculated by Equation (3), and in discrete distributions, we define $⟨f,g⟩={\sum }_{k=1}^{\infty }f\left(k\right)g\left(k\right)$. Meanwhile, the Poisson random variable with measurement errors can be treated as a negative binomial random variable. Let $n\left(x\right|{\lambda }_j,r)$ be the p.m.f. of the Poisson distribution with the mean ${\lambda }_j$ and dispersion parameter r. Suppose the observed data with sample size $n=100$ has the p.m.f.

$$g_{{\beta }^{*}}\left(x\right)=\sum _{j=1}^W{\beta }_j^{*}n\left(x|{\lambda }_j=a·j,r\right),$$

(14)

where $r=6$, which leads to an increment of variance from Poisson to the negative binomial distribution. Similarly, we replicate each simulation to estimate the parameter $N=100$ times with the sample from the mixture negative binomial distribution above. The result is shown in Table 2. The result is actually akin to that in the previous mixture Gaussian distribution, while the strong performance of our method is shown clearly when W is considerable.

Table 2. The simulation result in Section 4.3. The mean and standard deviation of the errors in the four estimators of ${\beta }^{*}$ under $N=100$ simulations. The ${\lambda }_2$ is chosen as $c=0.005$ for Elastic-net, while $c=0.203$ for the CSDE.

	W	${\lambda }_1$	$L_1$ Error	$\mathrm{TV}$ Error
Lasso	81	0.048	1.796 (0.006)	0.002 (0.001)
Elastic-net			1.796 (0.006)	0.002 (0.001)
SPADES		0.138	1.811 (0.013)	0.002 (0.005)
CSDE			1.806 (0.008)	0.003 (0.005)
Lasso	131	0.051	1.828 (0.006)	0.003 (0.001)
Elastic-net			1.830 (0.009)	0.004 (0.002)
SPADES		0.145	1.880 (0.006)	0.002 (0.005)
CSDE			1.854 (0.006)	0.002 (0.004)
Lasso	211	0.053	1.935 (0.010)	0.005 (0.003)
Elastic-net			2.061 (0.014)	0.007 (0.008)
SPADES		0.152	1.935 (0.008)	0.005 (0.003)
CSDE			1.861 (0.005)	0.003 (0.002)
Lasso	321	0.055	1.927 (0.031)	0.005 (0.002)
Elastic-net			2.123 (0.026)	0.009 (0.009)
SPADES		0.158	1.938 (0.008)	0.005 (0.003)
CSDE			1.852 (0.002)	0.002 (0.001)

4.4. Low-Dimensional Mixture Model

Surprisingly, our method has more competitive efficacy than some popular methods (such as EM algorithm), even the dimension W is relatively small. To see this, we introduce the following numerical experiments to estimate the weights of the low-dimensional Gaussian mixture model: the samples $X_1,\cdots ,X_n$ come from the model: $h_{{\beta }^{*}}\left(x\right)={\sum }_{j=1}^W{\beta }_j^{*}\varphi \left(x|{\mu }_j,{\sigma }_j\right).$ The updated equation for the EM algorithm in t-th step is:

$${\omega }_{ij}^{\left(t\right)}=\frac{p_j^{\left(t\right)}\varphi \left(x_i;{\mu }_t,{\sigma }_t\right)}{{\sum }_{s=1}^W{p}_s^{\left(t\right)}\varphi \left(x_i;{\mu }_s,{\sigma }_s\right)},{\beta }_j^{(t+1)}=\frac{{\sum }_{i=1}^W{\omega }_{ij}^{\left(t\right)}}{{\sum }_{i=1}^n{\sum }_{j=1}^W{\omega }_{ij}^{\left(t\right)}}.$$

Here, we consider two scenarios:

$$\begin{array}{cc}\hfill & \left(1\right)W=6,\beta ={(0.3,0,0,0.3,0,0.4)}^T,\mu ={(0,10,20,30,40,50)}^T,\sigma ={(1,2,3,4,5,6)}^T;\hfill \\ \hfill & \left(2\right)W=7,\beta ={(0.1,0,0,0.8,0,0,0.1)}^T,\mu ={(0,1,2,3,4,5,6)}^T,\hfill \\ & \sigma ={(0.3,0.2,0.2,0.1,0.2,0.2,0.3)}^T.\hfill \end{array}$$

For each scenario $n=50$, and the fitter levels (cessation level) in the EM approach and our method are both $\xi ={10}^{-4}$. A well-advised initial value in the EM approach is the equal weight.

We replicate the simulation $N=100$ times, and the optimal tuning parameters stem from the cross-validation (CV). Thus, under each simulation, they are not the same, albeit they are very close to each other. The result can be seen in Table 3.

Table 3. The low-dimensional simulation result in Section 4.4.

		$L_1$ Error	$\mathrm{TV}$ Error
Scenario 1	EM	0.255 (0.122)	0.205 (0.098)
	CSDE	0.206 (0.145)	0.185 (0.104)
Scenario 2	EM	0.111 (0.055)	0.111 (0.055)
	CSDE	0.109 (0.037)	0.108 (0.037)

4.5. Real Data Examples

Practically, we consider using our method to estimate some densities in the environmental science field. Wind, which is mercurial, has been an advisable object to study for a long time in meteorology. Please note that the wind’s speed at one specific location may not be diverse so we will use the wind’s azimuth angle with a more sparse density at two sites in China. Many types of research about the estimated density for wind exist, so there is a possibility of using our approach to cope with some difficulties in meteorology science.

There have been some very credible meteorological dataset. We used the ERA5 hourly data in [35] to continue our analysis. We want chose a continental area and a coastal area in China, so we chose Beijing Nongzhanguan and Qingdao Coast. The locations of these two areas are: (116.3125${}^{°}$ E, 116.4375${}^{°}$ E) × (39.8125${}^{°}$ N, 39.8125${}^{°}$ N). Take notice that the wind in one day may be highly correlated. Therefore, using the data at a specific time point of each day in a consecutive period as i.i.d. samples is reasonable. The sample histograms at 6 am in Beijing Nongzhanguan and at midnight on the Qingdao Coast are shown in Figure 3. Here, we used the data from 1 January 2013 to 12 December 2015.

Figure 3. The sample histogram of the azimuth in Beijing Nongzhanguan at 6am and Qingdao Coast at 12 am.

As we can see, their density does multi-peak (we used 1095 samples). Now, we can use our approach to estimate the multi-mode densities based on a relatively small size of samples, which is only a tiny part of the whole data from 1 January 2013 to 12 December 2015. Because one year has about 360 days, we may assume that every day is a latent factor that forms the base density. Thus, the model is designed as $h_{{\beta }^{*}}\left(x\right)={\sum }_{j=1}^{360}{\beta }_j^{*}\varphi \left(x|{\mu }_j,{\sigma }_j\right)$ with the mean and variance parameters $\mu ={(1,2,\dots ,360)}^T$, $\sigma =t·{\mathbf{1}}_{360}^T$, where t is the bandwidth (or tuning parameter). With the different sub-samples, the computed values are different.

Another critical issue is how to choose the tuning parameters ${\lambda }_1$ and ${\lambda }_2$. Then, we apply the cross-validation criterion, namely choosing ${\lambda }_i$, to minimize the difference between the two estimators derived from the separated samples in a random dichotomy.

Now, start to construct the samples for the estimating procedure. Assume that an observatory wants to figure out some information about the two areas’ wind. However, it does not have intact data due to the limited budget at its inception. The only samples it has are several days’ information each month for the two areas, and these days scatter randomly. Furthermore, sample size $n=168$ exactly. These imperfect data increase the challenge of estimating a trustworthy density. We compared our method with other previous methods, in which appraising the difference between the complete data sample histogram and the estimated density under each method is for the evaluation. Notice that the samples are only a tiny part of the data, so the $n=168<M(=360)$ is relatively small. The small sample and large dimension setting coincide with the non-asymptotic theory provided in the previous section. The estimated density has been shown in Figure 4.

Figure 4. The density map of the four estimators’ approaches for the three random sub-samples from the real-world data in Section 4.5.

In this practical application, our method vindicates its more efficient estimating performance and stability from its propinquity of the complete sample histogram, namely the productive capacity of detecting the shape of the multi-mode density and the stronger inclination to bear a resemblance to each other sub-sample (although some subtle nuances do exist because of the different sub-sample). An alternative approach can be to consider principles and tools of circular statistics, which has been reviewed in [36].

5. Summary and Discussions

The paper deals with the deconvolution problem using Lasso-type methods: the observations $X_1,\cdots ,X_n$ are independent and generated from $X_i=Z_i+{\epsilon }_i$, and the goal is to estimate the unknown density h of the $Z_i$. We assume that the function h can be written as $h(·)=h_{{\beta }^{*}}(·)={\sum }_{j=1}^W{\beta }_j^{*}{h}_j(·)$ based on some functions ${\left\{h_j\right\}}_{j=1}^W$ from a specific dictionary and propose estimating the coefficients of this decomposition with the Elastic-net method. For this estimator, we show that under some classical assumptions of the model, such as coherence of the Gram matrix, finite sample bounds for the estimation and the prediction errors valid with a relatively high probability can be obtained. Moreover, we prove a variable selection consistency result under a beta-min condition and conduct an extensive numerical study. The following estimation problem is also similar to the CSDE.

For future study, it is also interesting and meaningful to do hypothesis testing about the coefficients ${\beta }^{*}\in {\mathbb{R}}^W$ in sparse mixture models. For a general function $h:{\mathbb{R}}^W\mapsto {\mathbb{R}}^m$ and a nonempty closed set $\mathsf{\Omega }\in {\mathbb{R}}^m$, we can consider

$$H_0:h\left({\beta }^{*}\right)\in \mathsf{\Omega }\mathrm{vs}.H_1:h\left({\beta }^{*}\right)\notin \mathsf{\Omega }.$$

It is possible to use [37] as a general approach to hypothesis testing within models with measurement errors.

Appendix A.1. Proof of Lemma A1

According to the definition of $\widehat{\beta }$, for any $\beta \in {\mathbb{R}}^W$, we find $-\frac{2}{n}{\sum }_{i=1}^n{h}_{\widehat{\beta }}\left(X_i\right)+{\parallel h_{\widehat{\beta }}\parallel }^2+2{\sum }_{j=1}^W{\omega }_j|{\widehat{\beta }}_j|+c{\sum }_{j=1}^W{\widehat{\beta }}_j^2\le -\frac{2}{n}{\sum }_{i=1}^n{h}_{\beta }\left(X_i\right)+{\parallel h_{\beta }\parallel }^2+2{\sum }_{j=1}^W{\omega }_j\left|{\beta }_j\right|+c{\sum }_{j=1}^W{\beta }_j^2.$ Then

$$\begin{array}{c}\hfill {\parallel h_{\widehat{\beta }}\parallel }^2-{\parallel h_{\beta }\parallel }^2\le \frac{2}{n}\sum _{i=1}^n{h}_{\widehat{\beta }}\left(X_i\right)-\frac{2}{n}\sum _{i=1}^n{h}_{\beta }\left(X_i\right)+2\sum _{j=1}^W{\omega }_j|{\beta }_j|-2\sum _{j=1}^W{\omega }_j\left|{\widehat{\beta }}_j\right|+c\sum _{j=1}^W{\beta }_j^2-c\sum _{j=1}^W{\widehat{\beta }}_j^2.\end{array}$$

Note that

$$\begin{array}{cc}\hfill {\parallel h_{\widehat{\beta }}-h\parallel }^2& ={\parallel h_{\widehat{\beta }}-h_{\beta }+h_{\beta }-h\parallel }^2={\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2+{\parallel h_{\beta }-h\parallel }^2+2<h_{\beta }-h,h_{\widehat{\beta }}-h_{\beta }>\hfill \\ & ={\parallel h_{\beta }-h\parallel }^2-2<h,h_{\widehat{\beta }}-h_{\beta }>+2<h_{\beta },h_{\widehat{\beta }}-h_{\beta }>+{\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2\hfill \\ & ={\parallel h_{\beta }-h\parallel }^2-2<h,h_{\widehat{\beta }}-h_{\beta }>+{\parallel h_{\widehat{\beta }}\parallel }^2-{\parallel h_{\beta }\parallel }^2.\hfill \end{array}$$

Combining the two result above, we obtain

$$\begin{array}{cc}\hfill {\parallel h_{\widehat{\beta }}-h\parallel }^2& \le {\parallel h_{\beta }-h\parallel }^2+2\sum _{j=1}^W{\omega }_j|{\beta }_j|-2\sum _{j=1}^W{\omega }_j\left|{\widehat{\beta }}_j\right|+c\sum _{j=1}^W{\beta }_j^2-c\sum _{j=1}^W{\widehat{\beta }}_j^2\hfill \\ & -2<h,h_{\widehat{\beta }}-h_{\beta }>+\frac{2}{n}\sum _{i=1}^n{h}_{\widehat{\beta }}\left(X_i\right)-\frac{2}{n}\sum _{i=1}^n{h}_{\beta }\left(X_i\right).\hfill \end{array}$$

(A1)

According to the definition of $h_{\beta }\left(x\right)$, it gives $h_{\beta }\left(x\right)={\sum }_{j=1}^W{\beta }_j{h}_j\left(x\right)$ with $\beta =({\beta }_1,\cdots ,{\beta }_W)$. For the three terms in Equation (A1), we have

$$\begin{array}{cc}& -2<h,h_{\widehat{\beta }}-h_{\beta }>+\frac{2}{n}\sum _{i=1}^n{h}_{\widehat{\beta }}\left(X_i\right)-\frac{2}{n}\sum _{i=1}^n{h}_{\beta }\left(X_i\right)\hfill \\ \hfill =& 2·\frac{1}{n}\sum _{i=1}^n\left(\sum _{j=1}^W{\widehat{\beta }}_j{h}_j\left(X_i\right)-\sum _{j=1}^W{\beta }_j{h}_j\left(X_i\right)\right)-2E(h_{{\beta }^{\prime }}-h_{\beta })\left(X_i\right){|}_{{\beta }^{\prime }=\widehat{\beta }}\hfill \\ \hfill =& 2\sum _{j=1}^W\frac{1}{n}\sum _{i=1}^n{h}_j\left(X_i\right)({\widehat{\beta }}_j-{\beta }_j)-2\sum _{j=1}^{W}E\left[h_j\left(X_i\right)\right]({\widehat{\beta }}_j-{\beta }_j)\hfill \\ \hfill =& 2\sum _{j=1}^W\left(\frac{1}{n}\sum _{i=1}^n{h}_j\left(X_i\right)-E\left[h_j\left(X_i\right)\right]\right)({\widehat{\beta }}_j-{\beta }_j).\hfill \end{array}$$

Then

$$\begin{array}{cc}\hfill {\parallel h_{\widehat{\beta }}-h\parallel }^2& \le {\parallel h_{\beta }-h\parallel }^2+2\sum _{j=1}^W\left(\frac{1}{n}\sum _{i=1}^n{h}_j\left(X_i\right)-E\left[h_j\left(X_i\right)\right]\right)({\widehat{\beta }}_j-{\beta }_j)\hfill \\ & +2\sum _{j=1}^W{\omega }_j|{\beta }_j|-2\sum _{j=1}^W{\omega }_j\left|{\widehat{\beta }}_j\right|+c\sum _{j=1}^W{\beta }_j^2-c\sum _{j=1}^W{\widehat{\beta }}_j^2.\hfill \end{array}$$

Conditioning on $\mathcal{E}$, we have ${\parallel h_{\widehat{\beta }}-h\parallel }^2\le {\parallel h_{\beta }-h\parallel }^2+{\sum }_{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+2{\sum }_{j=1}^W{\omega }_j\left(\right|{\beta }_j|-|{\widehat{\beta }}_j\left|\right)+c{\sum }_{j=1}^W({\beta }_j^2-{\widehat{\beta }}_j^2).$ We add ${\sum }_{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+c{\sum }_{j=1}^W{({\beta }_j-{\widehat{\beta }}_j)}^2$ to both sides of the inequality, it gives

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+c\sum _{j=1}^W{({\beta }_j-{\widehat{\beta }}_j)}^2\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+2\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+2\sum _{j=1}^W{\omega }_j\left(\right|{\beta }_j|-|{\widehat{\beta }}_j\left|\right)+c\sum _{j=1}^W({\beta }_j^2-{\widehat{\beta }}_j^2)+c\sum _{j=1}^W{({\beta }_j-{\widehat{\beta }}_j)}^2.\hfill \end{array}$$

Note that

$$\begin{array}{cc}& c[\sum _{j=1}^W({\beta }_j^2-{\widehat{\beta }}_j^2)+\sum _{j=1}^W{({\beta }_j-{\widehat{\beta }}_j)}^2]=c\left[\sum _{j=1}^W({\beta }_j^2-{\widehat{\beta }}_j^2+{\beta }_j^2-2{\beta }_j{\widehat{\beta }}_j+{\widehat{\beta }}_j^2)\right]\hfill \\ \hfill =& 2c\sum _{j=1}^W{\beta }_j({\beta }_j-{\widehat{\beta }}_j)=2c\sum _{j\in I\left(\beta \right)}{\beta }_j({\beta }_j-{\widehat{\beta }}_j)\le 2cB\sum _{j\in I\left(\beta \right)}|{\beta }_j-{\widehat{\beta }}_j|\le 2\sum _{j\in I\left(\beta \right)}{\tilde{\omega }}_j|{\beta }_j-{\widehat{\beta }}_j|,\hfill \end{array}$$

where the last inequality is due to the assumption $c=\frac{{\min }_{1\le j\le W}\left\{{\tilde{\omega }}_j\right\}}{B}\le \frac{{\tilde{\omega }}_j}{B}$. Thus, we obtain

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+c\sum _{j=1}^W{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+2\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+2\sum _{j=1}^W{\omega }_j\left(\right|{\beta }_j|-|{\widehat{\beta }}_j\left|\right)+2\sum _{j\in I\left(\beta \right)}{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+2\sum _{j=1}^W{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|+2\sum _{j=1}^W{\omega }_j\left(\right|{\beta }_j|-|{\widehat{\beta }}_j\left|\right)+2\sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|,\hfill \end{array}$$

where the last inequality follows from ${\tilde{\omega }}_j\le {\omega }_j$ for all j.

We know ${\beta }_j\ne 0$ if $j\in I\left(\beta \right)$, and ${\beta }_j=0$ if $j\notin I\left(\beta \right)$. Considering $|{\beta }_j|-|{\widehat{\beta }}_j|\le |{\widehat{\beta }}_j-{\beta }_j|$ for all j, we have $2{\sum }_{j=1}^W{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|+2{\sum }_{j=1}^W{\omega }_j\left(\right|{\beta }_j|-|{\widehat{\beta }}_j\left|\right)\le 4{\sum }_{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|.$ Then

$$\begin{array}{cc}\hfill & ||h_{\widehat{\beta }}{-h\parallel }^2+\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+c\sum _{j=1}^W{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill & \le \parallel h_{\beta }{-h\parallel }^2+4\sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|+2\sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|\hfill \\ \hfill & =\parallel h_{\beta }{-h\parallel }^2+6\sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|.\hfill \end{array}$$

Appendix A.2. Proof of Theorems

According to ${\tilde{\omega }}_j=2\sqrt{2}{L}_j\sqrt{\frac{1}{n}\log \frac{2W}{\delta }}$ in Equation (6), the sum of the independent random variables ${\zeta }_{ij}=h_j\left(X_i\right)-E{h}_j\left(X_i\right)$ is determined by Hoeffding’s inequality, and $|h_j\left(X_i\right)|\le 2L_j$. We obtain

$$\begin{array}{cc}\hfill P\left({\mathcal{E}}^c\right)=P\left(\bigcup _{j=1}^W\left\{2\right|M_j|>{\tilde{\omega }}_j\}\right)& \le \sum _{j=1}^{W}P\left(2\right|M_j|>{\tilde{\omega }}_j)\le 2\sum _{j=1}^W\exp \left(-\frac{2n^2·{\tilde{\omega }}_j^2/4}{4n{L}_j^2}\right)\hfill \\ & =2\sum _{j=1}^W\exp \left(-\log \frac{2W}{\delta }\right)=2W·\frac{\delta }{2W}=\delta .\hfill \end{array}$$

Appendix A.3. Proof of Theorem 1

By Lemma A1, we need an upper bound on ${\sum }_{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|$. For easy notation, let $q_j={\widehat{\beta }}_j-{\beta }_j,Q\left(\beta \right)={\sum }_{j\in I\left(\beta \right)}|q_j|\parallel h_j\parallel ,Q={\sum }_{j=1}^W|q_j|\parallel h_j\parallel .$ According to the definition of $H\left(\beta \right)$, that is, $H\left(\beta \right)={\max }_{j\in I\left(\beta \right)}\frac{{\omega }_j}{v(\delta /2)\parallel h_j\parallel }$, we have

$$\sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|\le v(\delta /2)H\left(\beta \right)Q\left(\beta \right).$$

(A2)

Let $Q_{*}\left(\beta \right):=\sqrt{{\sum }_{j\in I\left(\beta \right)}{q}_j^2{\parallel h_j\parallel }^2}$. Using the definition of $h_{\beta }\left(x\right)$, we obtain $Q_{*}^2\left(\beta \right)={\sum }_{j\in I\left(\beta \right)}{q}_j^2{\parallel h_j\parallel }^2={\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2-{\sum }_{i,j\notin I\left(\beta \right)}{q}_i{q}_j<h_i,h_j>-(2{\sum }_{i\notin I\left(\beta \right)}{\sum }_{j\in I\left(\beta \right)}{q}_i{q}_j<h_i,h_j>+\underset{i,j\in I\left(\beta \right),i\ne j}{\sum \sum }{q}_i{q}_j<h_i,h_j>).$ As $i,j\notin I\left(\beta \right)$, ${\beta }_i={\beta }_j=0$, it is easy to see $\underset{i,j\notin I\left(\beta \right)}{\sum \sum }<h_i,h_j>q_i{q}_j\ge 0$. Observe that

$$\begin{array}{cc}& 2\sum _{i\notin I\left(\beta \right)}\sum _{j\in I\left(\beta \right)}{q}_i{q}_j<h_i,h_j>+\underset{i,j\in I\left(\beta \right),i\ne j}{\sum \sum }{q}_i{q}_j<h_i,h_j>\hfill \\ \hfill =& 2\sum _{i\notin I\left(\beta \right)}\sum _{j\in I\left(\beta \right)}{q}_i{q}_j<h_i,h_j>+2\underset{i,j\in I\left(\beta \right),j>i}{\sum \sum }{q}_i{q}_j<h_i,h_j>=2\underset{i\in I\left(\beta \right),j>i}{\sum \sum }{q}_i{q}_j<h_i,h_j>.\hfill \end{array}$$

By the definitions of ${\rho }_W(i,j)$ and ${\rho }_{*}\left(\beta \right)$, then

$$\begin{array}{cc}\hfill Q_{*}^2\left(\beta \right)& \le {\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2+2\underset{i\in I\left(\beta \right),j>i}{\sum \sum }|q_i\left|\right|q_j|\parallel h_i\parallel \parallel h_j\parallel \frac{<h_i,h_j>}{\parallel h_i\parallel \parallel h_j\parallel }\hfill \\ & \le {\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2+2{\rho }_{*}\left(\beta \right)\underset{i\in I\left(\beta \right),j>i}{\max }|q_i|\parallel h_i\parallel |q_j|\parallel h_j\parallel .\hfill \end{array}$$

By ${\max }_{i\in I\left(\beta \right)}|q_i|\parallel h_i\parallel \le \sqrt{{\sum }_{j\in I\left(\beta \right)}{q}_j^2{\parallel h_j\parallel }^2}=Q_{*}\left(\beta \right),{\max }_{i\in I\left(\beta \right),j>i}|q_j|\parallel h_j\parallel \le {\sum }_{j=1}^W|q_j|\parallel h_j\parallel$,

$$\begin{array}{cc}\hfill Q_{*}^2\left(\beta \right)& \le {\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2+2{\rho }_{*}\left(\beta \right)Q_{*}\left(\beta \right)\sum _{j=1}^W|q_j|\parallel h_j\parallel ={\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2+2{\rho }_{*}\left(\beta \right)Q_{*}\left(\beta \right)Q.\hfill \end{array}$$

(A3)

By Equation (A3), we can obtain $Q_{*}^2\left(\beta \right)-2{\rho }_{*}\left(\beta \right)Q_{*}\left(\beta \right)Q-{\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2\le 0.$

To find the upper bound of $Q_{*}\left(\beta \right)$, applying the properties of the quadratic inequality to the above formula, we obtain that

$$\begin{array}{cc}\hfill Q_{*}\left(\beta \right)\le {\rho }_{*}\left(\beta \right)Q+\sqrt{{\rho }_{*}^2\left(\beta \right)Q^2+{\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2}& \le {\rho }_{*}\left(\beta \right)Q+[{\rho }_{*}\left(\beta \right)Q+\parallel h_{\widehat{\beta }}-h_{\beta }\parallel ]\hfill \\ & \le 2{\rho }_{*}\left(\beta \right)Q+\parallel h_{\widehat{\beta }}-h_{\beta }\parallel .\hfill \end{array}$$

(A4)

Note that $W\left(\beta \right)=\left|I\left(\beta \right)\right|={\sum }_{j=1}^{W}I({\beta }_j\ne 0)$, employing the Cauchy–Schwarz inequalities, we have

$$\begin{array}{cc}\hfill W\left(\beta \right)\sum _{j\in I\left(\beta \right)}{\left|q_j\right|}^2{\parallel h_j\parallel }^2& =\sum _{j\in I\left(\beta \right)}{I}^2(j\in I\left(\beta \right))\sum _{j\in I\left(\beta \right)}{\left|q_j\right|}^2{\parallel h_j\parallel }^2\hfill \\ & \ge {(\sum _{j\in I\left(\beta \right)}I\left(\right\{j\in I\left(\beta \right))|q_j|\parallel h_j\parallel )}^2=Q^2\left(\beta \right).\hfill \end{array}$$

Then, $Q_{*}^2\left(\beta \right)={\sum }_{j\in I\left(\beta \right)}{\left|q_j\right|}^2{\parallel h_j\parallel }^2\ge Q^2\left(\beta \right)/W\left(\beta \right).$ In combination with Equation (A4), we can obtain $Q\left(\beta \right)/\sqrt{W\left(\beta \right)}\le Q_{*}\left(\beta \right)\le 2{\rho }_{*}\left(\beta \right)Q+\parallel h_{\widehat{\beta }}-h_{\beta }\parallel$. Therefore,

$$\begin{array}{c}\hfill Q\left(\beta \right)\le 2{\rho }_{*}\left(\beta \right)\sqrt{W\left(\beta \right)}Q+\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel .\end{array}$$

(A5)

By Lemma A1 and Equation (A2), we have the following inequality with probability exceeding $1-\delta$,

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\le {\parallel h_{\beta }-h\parallel }^2+6\sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+6v(\delta /2)H\left(\beta \right)Q\left(\beta \right)\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+6v(\delta /2)H\left(\beta \right)[2{\rho }_{*}\left(\beta \right)\sqrt{W\left(\beta \right)}\sum _{j=1}^W|q_j|\parallel h_j\parallel +\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel ]\left(\mathrm{by}\left(\mathrm{A}5\right)\right)\hfill \\ \hfill =& {∥h_{\beta }-h∥}^2+12v(\delta /2)H\left(\beta \right){\rho }_{*}\left(\beta \right)\sqrt{W\left(\beta \right)}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|\frac{∥h_j∥}{{\tilde{\omega }}_j}\hfill \\ \hfill +& 6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel \hfill \\ \hfill \le & {∥h_{\beta }-h∥}^2+12FH\left(\beta \right){\rho }_{*}\left(\beta \right)\sqrt{W\left(\beta \right)}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel \hfill \\ \hfill \le & {∥h_{\beta }-h∥}^2+\gamma \sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel ,\hfill \end{array}$$

where the second last inequality follows from the definition of $F:=\underset{1\le j\le W}{\max }\frac{v(\delta /2)∥h_j∥}{{\tilde{\omega }}_j}$, and the last inequality is derived by the assumption $12FH\left(\beta \right){\rho }_{*}\left(\beta \right)\sqrt{W\left(\beta \right)}\le \gamma ,(0<\gamma \le 1).$

Further, we can find that, with probability at least $1-\delta$,

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+(1-\gamma )\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel \hfill \\ \hfill =& {\parallel h_{\beta }-h\parallel }^2+6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h+h-h_{\beta }\parallel \hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h_{\widehat{\beta }}-h\parallel +6v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\parallel h-h_{\beta }\parallel .\hfill \end{array}$$

Using the elementary inequality $2st\le s^2/\alpha +\alpha t^2(s,t\in \mathbb{R},\alpha >1)$ to the last two terms of the above inequality, it yields

$$\begin{array}{c}2\left\{3v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\right\}\parallel h_{\widehat{\beta }}-h\parallel \le \alpha ·9v^2(\delta /2)H^2\left(\beta \right)W\left(\beta \right)+{\parallel h_{\widehat{\beta }}-h\parallel }^2/\alpha ,\hfill \\ 2\left\{3v(\delta /2)H\left(\beta \right)\sqrt{W\left(\beta \right)}\right\}\parallel h_{\beta }-h\parallel \le \alpha ·9v^2(\delta /2)H^2\left(\beta \right)W\left(\beta \right)+{\parallel h_{\beta }-h\parallel }^2/\alpha .\hfill \end{array}$$

Thus,

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+(1-\gamma )\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill \le & {\parallel h_{\beta }-h\parallel }^2+18\alpha v^2(\delta /2)H^2\left(\beta \right)W\left(\beta \right)+{\parallel h_{\widehat{\beta }}-h\parallel }^2/\alpha +{\parallel h_{\beta }-h\parallel }^2/\alpha .\hfill \end{array}$$

Simplifying, we have

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+\frac{\alpha (1-\gamma )}{(\alpha -1)}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\frac{\alpha }{\alpha -1}\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \hfill & \le \frac{\alpha +1}{\alpha -1}{\parallel h_{\beta }-h\parallel }^2+\frac{18{\alpha }^2}{\alpha -1}{H}^2\left(\beta \right)v^2(\delta /2)W\left(\beta \right),\alpha >1,0<\gamma \le 1.\hfill \end{array}$$

(A6)

Optimizing $\alpha$ to obtain the sharp upper bounds for the above oracle inequality

$$\begin{array}{cc}\hfill {\alpha }_{opt1}:& =\underset{\alpha >1}{\arg \min }\left\{\frac{\alpha +1}{\alpha -1}{\parallel h_{\beta }-h\parallel }^2+\frac{18{\alpha }^2}{\alpha -1}{H}^2\left(\beta \right)v^2(\delta /2)W\left(\beta \right)\right\}\hfill \\ & =1+\sqrt{1+\frac{{\parallel h_{\beta }-h\parallel }^2}{9H^2\left(\beta \right)v^2(\delta /2)W\left(\beta \right)}}\hfill \end{array}$$

by the first order condition. To date, Theorem 1 is proved by substituting ${\alpha }_{opt1}$ into Equation (A6).

Appendix A.4. Proof of Theorem 2

By the minimal eigenvalue assumption for ${\psi }_W$, we have

$$\begin{array}{c}\hfill {\parallel h_{\beta }\parallel }^2={||\sum _{j=1}^W{\beta }_j{h}_j\left(x\right)||}^2={\beta }^T{\psi }_W\beta \ge {\lambda }_W{\parallel \beta \parallel }^2\ge {\lambda }_W\sum _{j\in I\left(\beta \right)}{\beta }_j^2.\end{array}$$

(A7)

Using the definition of ${\omega }_j$ and assumption $L_{\min }:=\underset{1\le j\le W}{\min }{L}_j>0$,

$${\omega }_j=2L_j\left(\sqrt{\frac{2\log (2W/\delta )}{n}}+\frac{cB}{2L_j}\right)\le 2L_j\left(\sqrt{\frac{2\log (2W/\delta )}{n}}+\frac{cB}{2L_{\min }}\right).$$

Since $cB={\tilde{\omega }}_{\min }=2\sqrt{2}{L}_{\min }v(\delta /2)$ and $v(\delta /2)=\sqrt{\frac{\log (2W/\delta )}{n}}$, we have ${\omega }_j\le 4\sqrt{2}{L}_{j}v(\delta /2).$

Let $G\left(\beta \right)={\sum }_{j\in I\left(\beta \right)}{L}_j^2$, by the Cauchy–Schwartz inequality, we obtain

$$\begin{array}{cc}\hfill 6& \sum _{j\in I\left(\beta \right)}{\omega }_j|{\widehat{\beta }}_j-{\beta }_j|\le 24\sqrt{2}v(\delta /2)\sum _{j\in I\left(\beta \right)}{L}_j|{\widehat{\beta }}_j-{\beta }_j|\hfill \\ & \le 24\sqrt{2}v(\delta /2)\sqrt{{\sum }_{j\in I\left(\beta \right)}{L}_j^2}\sqrt{{\sum }_{j\in I\left(\beta \right)}{({\widehat{\beta }}_j-{\beta }_j)}^2}\le 24\sqrt{2}v(\delta /2)\sqrt{\frac{G\left(\beta \right)}{{\lambda }_W}}\parallel h_{\widehat{\beta }}-h_{\beta }\parallel ,\hfill \end{array}$$

(A8)

where the last inequality above is from Equation (A7) due to

$${\parallel h_{\widehat{\beta }}-h_{\beta }\parallel }^2=\underset{1\le i,j\le W}{\sum \sum }({\widehat{\beta }}_i-{\beta }_i)({\widehat{\beta }}_j-{\beta }_j)<h_i,h_j>\ge {\lambda }_W\sum _{j\in I\left(\beta \right)}{({\widehat{\beta }}_j-{\beta }_j)}^2.$$

Let $b\left(\beta \right):=12\sqrt{2}v(\delta /2)\sqrt{\frac{G\left(\beta \right)}{{\lambda }_W}}$, Lemma 2 implies

$$\begin{array}{cc}& {\parallel h_{\widehat{\beta }}-h\parallel }^2+\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\le {\parallel h_{\beta }-h\parallel }^2+2b\left(\beta \right)\parallel h_{\widehat{\beta }}-h_{\beta }\parallel \hfill \\ \hfill =& {\parallel h_{\beta }-h\parallel }^2+2b\left(\beta \right)(\parallel h_{\widehat{\beta }}-h+h-h_{\beta }\parallel )\le {\parallel h_{\beta }-h\parallel }^2+2b\left(\beta \right)\parallel h_{\widehat{\beta }}-h\parallel +2b\left(\beta \right)\parallel h_{\beta }-h\parallel .\hfill \end{array}$$

Using the inequality $2st\le s^2/\alpha +\alpha t^2$$(s,t\in R,\alpha >1)$ for the last two terms on the right side of the above inequality, we find

$$\begin{array}{cc}\hfill 2b\left(\beta \right)\parallel h_{\widehat{\beta }}-h\parallel +2b\left(\beta \right)\parallel h_{\beta }-h\parallel & \le {\parallel h_{\widehat{\beta }}-h\parallel }^2/\alpha +b^2\left(\beta \right)\alpha +{\parallel h_{\beta }-h\parallel }^2/\alpha +b^2\left(\beta \right)\alpha \hfill \\ & ={\parallel h_{\widehat{\beta }}-h\parallel }^2/\alpha +{\parallel h_{\beta }-h\parallel }^2/\alpha +2b^2\left(\beta \right)\alpha .\hfill \end{array}$$

Thus,

$$\begin{array}{cc}\hfill {\parallel h_{\widehat{\beta }}-h\parallel }^2+\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2& \le {\parallel h_{\beta }-h\parallel }^2+{\parallel h_{\widehat{\beta }}-h\parallel }^2/\alpha \hfill \\ & +{\parallel h_{\beta }-h\parallel }^2/\alpha +2b^2\left(\beta \right)\alpha \hfill \end{array}$$

gives $\frac{\alpha -1}{\alpha }{\parallel h_{\widehat{\beta }}-h\parallel }^2+{\sum }_{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+{\sum }_{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\le \frac{\alpha +1}{\alpha }{\parallel h_{\beta }-h\parallel }^2+2\alpha b^2\left(\beta \right)$. Therefore,

$$\begin{array}{c}{\parallel h_{\widehat{\beta }}-h\parallel }^2+\frac{\alpha }{\alpha -1}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j|+\frac{\alpha }{\alpha -1}\sum _{j=1}^{W}c{({\widehat{\beta }}_j-{\beta }_j)}^2\hfill \\ \le \frac{\alpha +1}{\alpha -1}{\parallel h_{\beta }-h\parallel }^2+\frac{2{\alpha }^2}{\alpha -1}{b}^2\left(\beta \right)\hfill \\ =\frac{\alpha +1}{\alpha -1}{\parallel h_{\beta }-h\parallel }^2+\frac{576{\alpha }^2}{\alpha -1}\frac{G\left(\beta \right)}{{\lambda }_W}{v}^2(\delta /2).\hfill \end{array}$$

To obtain the sharp upper bounds for the above oracle inequality, we optimize $\alpha$

$$\begin{array}{cc}\hfill {\alpha }_{opt2}:& =\underset{\alpha >1}{\arg \min }\left\{\frac{\alpha +1}{\alpha -1}{\parallel h_{\beta }-h\parallel }^2+\frac{576{\alpha }^2}{\alpha -1}\frac{G\left(\beta \right)}{{\lambda }_W}{v}^2(\delta /2)\right\}=1+\sqrt{1+\frac{{\parallel h_{\beta }-h\parallel }^2}{288\frac{G\left(\beta \right)}{{\lambda }_W}{v}^2(\delta /2)}},\hfill \end{array}$$

by the first-order condition. This completes the proof of Theorem 2.

Appendix A.5. Proof of Corollory 1

Let ${\tilde{\omega }}_{\min }:={\min }_{1\le j\le W}{\tilde{\omega }}_j$. We replace $v(\delta /2)$ in Theorem 1 by the larger value $v(\delta /2W)$. Substituting $\beta ={\beta }^{*}$ in Theorem 1, we have

$$\begin{array}{c}\hfill \frac{{\alpha }_{opt1}(1-\gamma )}{{\alpha }_{opt1}-1}\sum _{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j^{*}|\le \frac{18{\alpha }_{opt1}^2}{{\alpha }_{opt1}-1}{H}^2\left({\beta }^{*}\right)v^2(\delta /2W)W\left({\beta }^{*}\right)\end{array}$$

by $h=h_{{\beta }^{*}}$. Since ${\tilde{\omega }}_j\ge {\tilde{\omega }}_{\min }$ for all j, we obtain

$$\begin{array}{c}\hfill \sum _{j=1}^W|{\widehat{\beta }}_j-{\beta }_j^{*}|\le \frac{18{\alpha }_{opt1}}{1-\gamma }·\frac{1}{{\tilde{\omega }}_{min}}·\underset{j\in I\left(\beta \right)}{max}\frac{{\omega }_j^2}{{\parallel h_j\parallel }^2}·W\left({\beta }^{*}\right).\end{array}$$

In this case, ${\alpha }_{opt1}=2$, and $\parallel h_j\parallel =1$; thus,

$$\begin{array}{cc}\hfill \parallel \widehat{\beta }-{\beta }^{*}\parallel & \le \frac{36}{1-\gamma }·\underset{j\in I\left(\beta \right)}{\max }\frac{{\omega }_j^2}{{\tilde{\omega }}_{\min }}·W\left({\beta }^{*}\right)\hfill \\ & =\frac{72\sqrt{2}v(\delta /2W)W\left({\beta }^{*}\right)}{1-\gamma }\underset{j\in I\left(\beta \right)}{\max }\frac{{(L_j+L_{\min })}^2}{L_{\min }}\le \frac{72\sqrt{2}v(\delta /2W)W\left({\beta }^{*}\right)}{1-\gamma }\frac{{(L+L_{\min })}^2}{L_{\min }}\hfill \end{array}$$

from ${\tilde{\omega }}_{\min }=2\sqrt{2}v(\delta /2W)L_{\min }$ and

${\omega }_j^2={\left[2\sqrt{2}v(\delta /2W)\right]}^2{\left[L_j+\frac{{\tilde{\omega }}_{\min }}{2\sqrt{2}v(\delta /2W)}\right]}^2={\left[2\sqrt{2}v(\delta /2W)\right]}^2{[L_j+L_{\min }]}^2.$ This completes the proof of Corollary 1.

Appendix A.6. Proof of Corollary 2

Let $\beta ={\beta }^{*}$ in Theorem 2, with ${\alpha }_{opt2}=2$, we replace $v(\delta /2)$ in Theorem 2 by the larger value $v(\delta /2W)$, then ${\sum }_{j=1}^W{\tilde{\omega }}_{min}|{\widehat{\beta }}_j-{\beta }_j^{*}|\le {\sum }_{j=1}^W{\tilde{\omega }}_j|{\widehat{\beta }}_j-{\beta }_j^{*}|\le \frac{576{\alpha }_{opt2}{G}^{*}}{{\lambda }_W}{v}^2(\delta /2W).$ By the definition of ${\tilde{\omega }}_{\min }$, we can obtain

$$\sum _{j=1}^W|{\widehat{\beta }}_j-{\beta }_j^{*}|\le \frac{576{\alpha }_{opt2}{G}^{*}{v}^2(\delta /2)}{{\tilde{\omega }}_{\min }{\lambda }_W}=\frac{576·2G^{*}{v}^2(\delta /2W)}{2\sqrt{2}v(\delta /2W)L_{\min }{\lambda }_W}=\frac{288\sqrt{2}{G}^{*}v(\delta /2W)}{L_{\min }{\lambda }_W}.$$

This concludes the proof of Corollary 2.

Appendix A.7. Proof of Theorem 3

The following lemma is by virtue of the KKT conditions. It derives a bound of $P\left(I_{*}\cancel{⫅}\widehat{I}\right)$, which is easily analyzed.

Lemma A2

(Proposition 3.3 in [33]). $P\left(I_{*}\cancel{⫅}\widehat{I}\right)\le W\left({\beta }^{*}\right){\max }_{k\in I_{*}}P({\widehat{\beta }}_k=0\mathrm{and}{\beta }_k^{*}\ne 0).$

To present the proof of Theorem 3, we first notice that $P(\widehat{I}\ne I_{*})\le P\left(I_{*}\cancel{⫅}\widehat{I}\right)+P\left(\widehat{I}\cancel{⫅}{I}_{*}\right).$ Next, we control the probability on the right side of the above inequality.

For the control of $P\left(I_{*}\cancel{⫅}\widehat{I}\right)$, by Lemma A2, it remains to bound $P({\widehat{\beta }}_k=0\mathrm{and}{\beta }_k^{*}\ne 0)$.

Below, we will use the conclusion of Lemma 2 (KKT conditions). Recall that $E\left[h_k\left(Z_1\right)\right]={\sum }_{j\in I_{*}}{\beta }_j^{*}<h_k,h_j>={\sum }_{j=1}^W{\beta }_j^{*}<h_k,h_j>$. Since we assume that the density of $Z_1$ is the mixture density $h_{{\beta }^{*}}={\sum }_{j\in I_{*}}{\beta }_j^{*}{h}_j$. Therefore, for $k\in I_{*}$, we have,

$$\begin{array}{cc}& P({\widehat{\beta }}_k=0\mathrm{and}{\beta }_k^{*}\ne 0)=P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-\sum _{j=1}^W{\widehat{\beta }}_j<h_j,h_k>\right|\le 2\sqrt{2}v(\delta /2W)L_k;{\beta }_k^{*}\ne 0\right)\hfill \\ \hfill =& P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]+E\left[h_k\left(Z_1\right)\right]-\sum _{j=1}^W{\widehat{\beta }}_j<h_j,h_k>\right|\le 2\sqrt{2}v(\delta /2W)L_k;{\beta }_k^{*}\ne 0\right)\hfill \\ \hfill =& P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]-\sum _{j=1}^W({\widehat{\beta }}_j-{\beta }_j^{*})<h_j,h_k>\right|\le 2\sqrt{2}v(\delta /2W)L_k;{\beta }_k^{*}\ne 0\right)\hfill \\ \hfill =& P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]-\sum _{j\ne k}^W({\widehat{\beta }}_j-{\beta }_j^{*})<h_j,h_k>+{\beta }_k^{*}{\parallel h_k\parallel }^2\right|\le 2\sqrt{2}v(\delta /2W)L_k\right)\hfill \\ \hfill \le & P\left(|{\beta }_k^{*}{\parallel h_k\parallel }^2-2\sqrt{2}v(\delta /2W)L_k\le \left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]\right|+\left|\sum _{j\ne k}^W({\widehat{\beta }}_j-{\beta }_j^{*})<h_j,h_k>\right|\right)\hfill \\ \hfill \mathrm{(A9)}& \hfill \le & P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]\right|\ge \frac{|{\beta }_k^{*}|{\parallel h_k\parallel }^2}{2}-\sqrt{2}v(\delta /2W)L_k\right)\hfill \hfill \mathrm{(A10)}& \hfill +& P\left(\left|\sum _{j\ne k}^W({\widehat{\beta }}_j-{\beta }_j^{*})<h_j,h_k>\right|\ge \frac{|{\beta }_k^{*}|{\parallel h_k\parallel }^2}{2}-\sqrt{2}v(\delta /2W)L_k\right).\hfill \end{array}$$

Similar to Lemma 2, for Equation (A9), we use Hoeffding’s inequality. Since $\parallel h_k\parallel =1$ for all k. Put ${ϵ}_k:=|E\left[h_k\left(X_1\right)\right]-E\left[h_k\left(Z_1\right)\right]|$. Consider Condition (B), ${\min }_{k\in I_{*}}\left|{\beta }_k^{*}\right|\ge 4\sqrt{2}v(\delta /2W)L$ and $L\ge {\max }_{1\le k\le W}{L}_k$, then we have

$$\begin{array}{cc}\hfill P& \left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]\right|\ge \frac{|{\beta }_k^{*}|{\parallel h_k\parallel }^2}{2}-\sqrt{2}v(\delta /2W)L_k\right)\hfill \\ & =P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(X_1\right)\right]+E\left[h_k\left(X_1\right)\right]-E\left[h_k\left(Z_1\right)\right]\right|\ge \frac{|{\beta }_k^{*}|{\parallel h_k\parallel }^2}{2}-\sqrt{2}v(\delta /2W)L_k\right)\hfill \\ & \le P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(X_1\right)\right]\right|\ge \frac{|{\beta }_k^{*}|}{2}-\sqrt{2}v(\delta /2W)L-{ϵ}_k\right)\hfill \\ & \le P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(X_1\right)\right]\right|\ge 2\sqrt{2}v(\delta /2W)L-\sqrt{2}v(\delta /2W)L-{ϵ}_k\right)\hfill \\ & =P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(X_1\right)\right]\right|\ge \sqrt{2}v(\delta /2W)L-{ϵ}_k\right)\hfill \\ & =P\left(\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(X_1\right)\right]\right|\ge \sqrt{2}v(\delta /2W)L(1-{ϵ}_k^{*})\right)(\mathrm{let}{ϵ}_k^{*}={ϵ}_k/\sqrt{2}v(\delta /2W)L)\hfill \\ & \le 2\exp \left\{-\frac{4n^2{v}^2(\delta /2W)L^2{(1-{ϵ}_k^{*})}^2}{4n{L}^2}\right\}\hfill \\ & =2\exp \left\{-n{(1-{ϵ}_k^{*})}^2\frac{\log (2W^2/\delta )}{n}\right\}=2{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}.\hfill \end{array}$$

(A11)

For the upper bound of Equation (A10), using Condition (A) and Condition (B), by the definitions of ${\rho }_{*}\left({\beta }^{*}\right)$ and $W\left({\beta }^{*}\right)$, we obtain

$$\begin{array}{cc}& P\left(\left|\sum _{j\ne k}^W({\widehat{\beta }}_j-{\beta }_j^{*})<h_j,h_k>\right|\ge \frac{|{\beta }_k^{*}|{\parallel h_k\parallel }^2}{2}-\sqrt{2}v(\delta /2W)L_k\right)\hfill \\ \hfill =& P\left(\left|\sum _{j\ne k}^W({\widehat{\beta }}_j-{\beta }_j^{*})<h_j,h_k>\right|\ge \frac{|{\beta }_k^{*}|}{2}-\sqrt{2}v(\delta /2W)L_k\right)\hfill \\ \hfill \le & P\left(\left|\sum _{j\ne k}^W({\widehat{\beta }}_j-{\beta }_j^{*})\frac{<h_j,h_k>}{\parallel h_j\parallel \parallel h_k\parallel }·\parallel h_j\parallel \parallel h_k\parallel \right|\ge 2\sqrt{2}v(\delta /2W)L-\sqrt{2}v(\delta /2W)L\right)\hfill \\ \hfill \le & P\left({\rho }_{*}\left({\beta }^{*}\right)\sum _{j\ne k}^W\left|{\widehat{\beta }}_j-{\beta }_j^{*}\right|\ge \sqrt{2}v(\delta /2W)L\right)\le P\left(\sum _{j=1}^W\left|{\widehat{\beta }}_j-{\beta }_j^{*}\right|\ge \frac{\sqrt{2}v(\delta /2W)L}{{\rho }_{*}\left({\beta }^{*}\right)}\right)\hfill \\ \hfill \le & P\left(\sum _{j=1}^W\left|{\widehat{\beta }}_j-{\beta }_j^{*}\right|\ge \frac{288\sqrt{2}{G}^{*}v(\delta /2W)}{L_{\min }{\lambda }_W}\right)\le \frac{\delta }{W}.\hfill \end{array}$$

where the second last inequality is by Condition (A), and the last inequality above is by using the ${\ell }_1$-estimation oracle inequality in Corollary 2.

Therefore, by the definition of $W\left({\beta }^{*}\right)$, $W\left({\beta }^{*}\right)=\left|I_{*}\right|\le W$, we find

$$\begin{array}{cc}\hfill P\left(I_{*}\cancel{⫅}\widehat{I}\right)\le W\left({\beta }^{*}\right)\underset{k\in I_{*}}{\max }P({\widehat{\beta }}_k=0)& \le W\left({\beta }^{*}\right)2{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+W\left({\beta }^{*}\right)\frac{\delta }{W}\hfill \\ & \le 2W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+W\frac{\delta }{W}=2W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+\delta .\hfill \end{array}$$

For the control of $P\left(\widehat{I}\cancel{⫅}{I}_{*}\right)$, let

$$\begin{array}{c}\hfill \tilde{\eta }=\underset{\eta \in {\mathbb{R}}^{W\left({\beta }^{*}\right)}}{\arg \min }z\left(\eta \right),\end{array}$$

(A12)

where $z\left(\eta \right)=-\frac{2}{n}{\sum }_{i=1}^n{\sum }_{j\in I_{*}}{\eta }_j{h}_j\left(X_i\right)+{\parallel {\sum }_{j\in I_{*}}{\eta }_j{h}_j\parallel }^2+{\sum }_{j\in I_{*}}(4\sqrt{2}v(\delta /2)L_j+2cB)\left|{\eta }_j\right|+c{\sum }_{j\in I_{*}}{\eta }_j^2.$ Consider the following random event

$$\begin{array}{cc}\hfill \bigcap _{k\notin I_{*}}& \left\{\left|-\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)+\sum _{j\in I_{*}}{\tilde{\eta }}_j<h_j,h_k>\right|\le 2\sqrt{2}v(\delta /2)L_k\right\}\hfill \\ & \subseteq \bigcap _{k\notin I_{*}}\left\{\left|-\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)+\sum _{j\in I_{*}}{\tilde{\eta }}_j<h_j,h_k>\right|\le 2\sqrt{2}v(\delta /2W)L\right\}:=\mathsf{\Psi }.\hfill \end{array}$$

(A13)

Let $\overline{\eta }\in {\mathbb{R}}^W$ be a vector corresponding to the component of the index set $I_{*}$ having $\tilde{\eta }$ given by Equation (A12), and the component at other corresponding positions is 0. By Lemma 1, we know that $\overline{\eta }\in {\mathbb{R}}^W$ is a solution of Equation (3) on the event $\mathsf{\Psi }$. It is recalled that $\widehat{\beta }\in {\mathbb{R}}^W$, which is also a solution of Equation (3). Through the definition of the indicator set $\widehat{I}$, we have ${\widehat{\beta }}_k\ne 0$ for $k\in \widehat{I}$. By construction, we obtain ${\tilde{\eta }}_k\ne 0$ for some subset $T⫅I_{*}$. The KKT conditions indicate that any two solutions have non-zero components at the same positions. Therefore, $\widehat{I}=T⫅I_{*}$ on the event $\mathsf{\Psi }$. Further, we can write

$$\begin{array}{cc}& P(\widehat{I}⊈I_{*})\le P\left({\mathsf{\Psi }}^c\right)=P\left(\bigcup _{k\notin I_{*}}\left\{\left|-\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)+\sum _{j\in I_{*}}{\tilde{\eta }}_j<h_j,h_k>\right|\ge 2\sqrt{2}v(\delta /2W)L\right\}\right)\hfill \\ \hfill \le & \sum _{k\notin I_{*}}P\left\{\left|-\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)+\sum _{j\in I_{*}}{\tilde{\eta }}_j<h_j,h_k>\right|\ge 2\sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill =& \sum _{k\notin I_{*}}P\left\{\left|-\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)+E\left[h_k\left(Z_1\right)\right]-E\left[h_k\left(Z_1\right)\right]+\sum _{j\in I_{*}}{\tilde{\eta }}_j<h_j,h_k>\right|\ge 2\sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill =& \sum _{k\notin I_{*}}P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]-\sum _{j\in I_{*}}({\tilde{\eta }}_j-{\beta }_j^{*})<h_j,h_k>\right|\ge 2\sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill \mathrm{(A14)}& \hfill \le & \sum _{k\notin I_{*}}P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E\left[h_k\left(Z_1\right)\right]\right|\ge \sqrt{2}v(\delta /2W)L\right\}\hfill \hfill \mathrm{(A15)}& & +\sum _{k\notin I_{*}}P\left\{\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|\left|<h_j,h_k>\right|\ge \sqrt{2}v(\delta /2W)L\right\}.\hfill \end{array}$$

According to the previously proven Formula (A11), we find

$$\begin{array}{cc}& \sum _{k\notin I_{*}}P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(Z_i\right)-E{h}_k\left(Z_1\right)\right|\ge \sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill \le & \sum _{k\notin I_{*}}P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E{h}_k\left(X_1\right)\right|\ge \sqrt{2}v(\delta /2W)L-{ϵ}_k\right\}\hfill \\ \hfill =& \sum _{k=1}^{W}P\left\{\left|\frac{1}{n}\sum _{i=1}^n{h}_k\left(X_i\right)-E{h}_k\left(X_1\right)\right|\ge \sqrt{2}v(\delta /2W)L(1-{ϵ}_k^{*})\right\}\le 2W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}.\hfill \end{array}$$

For the upper bound of Equation (A15), observe Theorem 2, we can use a larger $v(\delta /2W)$ instead of $v(\delta /2)$. Consider the construction of $\tilde{\eta }$ in Equation (A12), we obtain

$$\begin{array}{c}\hfill P\left(\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|\ge \frac{288\sqrt{2}{G}^{*}v(\delta /2W)}{L_{min}{\lambda }_W}\right)\le \frac{\delta }{W}.\end{array}$$

Similarly, we have

$$\begin{array}{cc}& \sum _{k\notin I_{*}}P\left\{\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|\left|<h_j,h_k>\right|\ge \sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill \le & \sum _{k=1}^{W}P\left\{\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|\left|\frac{<h_j,h_k>}{\parallel h_j\parallel \parallel h_k\parallel }\parallel h_j\parallel \parallel h_k\parallel \right|\ge \sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill \le & \sum _{k=1}^{W}P\left\{\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|{\rho }_{*}\left({\beta }^{*}\right)\ge \sqrt{2}v(\delta /2W)L\right\}\hfill \\ \hfill =& \sum _{k=1}^{W}P\left\{\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|\ge \frac{\sqrt{2}v(\delta /2W)L}{{\rho }_{*}\left({\beta }^{*}\right)}\right\}\hfill \\ \hfill \left(\mathrm{using}\mathrm{Condition}\left(\mathrm{A}\right)\right)& \le \sum _{k=1}^{W}P\left\{\sum _{j\in I_{*}}|{\tilde{\eta }}_j-{\beta }_j^{*}|\ge \frac{288\sqrt{2}{G}^{*}v(\delta /2W)}{L_{\min }{\lambda }_W}\right\}\le \sum _{k=1}^W\frac{\delta }{W}=\delta .\hfill \end{array}$$

Combining all the bounds above, we can obtain

$$\begin{array}{cc}\hfill P(\widehat{I}\ne I_{*})\le P\left(I_{*}\cancel{⫅}\widehat{I}\right)+P\left(\widehat{I}\cancel{⫅}{I}_{*}\right)& \le 2W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+\delta +2W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+\delta \hfill \\ & =4W{\left(\frac{\delta }{2W^2}\right)}^{{(1-{ϵ}_k^{*})}^2}+2\delta .\hfill \end{array}$$

This completes the proof of Theorem 3.