The Prediction Performance Analysis of the Lasso Model with Convex Non-Convex Sparse Regularization

Abstract

The incorporation of ${\mathsf{\ell }}_1$ regularization in Lasso regression plays a crucial role by inducing convexity to the objective function, thereby facilitating its minimization; when compared to non-convex regularization, the utilization of ${\mathsf{\ell }}_1$ regularization introduces bias through artificial coefficient shrinkage towards zero. Recently, the convex non-convex (CNC) regularization framework has emerged as a powerful technique that enables the incorporation of non-convex regularization terms while maintaining the overall convexity of the optimization problem. Although this method has shown remarkable performance in various empirical studies, its theoretical understanding is still relatively limited. In this paper, we provide a theoretical investigation into the prediction performance of the Lasso model with CNC sparse regularization. By leveraging oracle inequalities, we establish a tighter upper bound on prediction performance compared to the traditional ${\mathsf{\ell }}_1$ regularizer. Additionally, we propose an alternating direction method of multipliers (ADMM) algorithm to efficiently solve the proposed model and rigorously analyze its convergence property. Our numerical results, evaluated on both synthetic data and real-world magnetic resonance imaging (MRI) reconstruction tasks, confirm the superior effectiveness of our proposed approach.

1. Introduction

The Lasso model performs automatic feature selection by shrinking some coefficients to exactly zero. This mechanism prunes irrelevant features while retaining statistically significant predictors, thereby constructing a streamlined model with enhanced interpretability [1]. Consequently, the Lasso model has emerged as one of the most widely adopted tools for linear regression and finds applications across diverse domains including statistical and machine learning [2,3,4,5,6,7], image and signal processing [8,9,10,11].

Specifically, for n random observations $y_1,y_2,\dots ,y_n\in \mathbb{R}$ and p fixed covariates $x_1,\dots ,x_p\in {\mathbb{R}}^n$. This high-dimensional linear model is defined as

$$\begin{array}{c}\hfill y=X{\beta }^{*}+ϵ,ϵ\sim {\sigma }^{*}{\mathcal{N}}_n(0,I_n),\end{array}$$

(1)

where $y:={(y_1,y_2\dots ,y_n)}^{\top }\in {\mathbb{R}}^n$ is the vector of observations, ${\beta }^{*}\in {\mathbb{R}}^p$ is the unknown parameter of interest, $X:=(x^1,\dots ,x^p)\in {\mathbb{R}}^{n\times p}$ is a deterministic design matrix (for without loss of generality, we assume $\parallel x_j{\parallel }_2^2 \le n$ for all $j\in (1,\dots ,p)$, $ϵ\in {\mathbb{R}}^n$ is the noise vector with identically and independently distributed (i.i.d.) Gaussian entries with variance ${{\sigma }^{*}}^2$, and $I_n$ denotes the identity matrix in ${\mathbb{R}}^{n\times n}$. In general, we assume $p>n$, and our goal is to accurately estimate ${\beta }^{*}\in {\mathbb{R}}^p$.

The Lasso model is a type of linear regression model utilized for estimating $\beta$ and can be formulated as the following optimization problem [12]

$$\begin{array}{c}\hfill {\widehat{\beta }}_{\lambda }^{L1}\in arg\underset{\beta }{min}{\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2+\lambda {\parallel \beta \parallel }_1,\end{array}$$

(2)

where ${\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2$ is the mean squared error (MSE) loss and the ${\mathsf{\ell }}_1$ norm ${\parallel \beta \parallel }_1$ is the regularizer, $\lambda >0$ is a predefined tuning parameter that controls the level of regularization. The prediction performance of the Lasso model refers to how well it predicts outcomes or responses based on input features and depends on its ability to strike a balance between regularization and feature selection. The prediction performance of the Lasso model plays a crucial role in understanding its effectiveness [12,13,14].

The ${\mathsf{\ell }}_1$ norm in Equation (2) is the most popular regularization because of its outstanding ability to induce sparsity among convex regularization methods. However, it has been observed that this formal approach often underestimates the high-amplitude components of $\beta$ [15]. In contrast, non-convex regularizations in the Lasso model have also made significant progress [6,10,16,17]. For instance, the smoothly clipped absolute deviation (SCAD) penalty [18] and the minimax concave penalty (MCP) [15], as well as ${\mathsf{\ell }}_q(0<q<1)$ and other non-convex regularization terms, can more accurately estimate high-amplitude components. However, due to their non-convex nature, the objective function is prone to getting stuck in local optima, which poses additional challenges to the solution process.

To utilize the advantages of both non-convex regularization and convex optimization methods, Selesnick et al. introduced the CNC strategy, which involves constructing a non-convex regularizer by subtracting the smooth variation from its convex sparse counterpart [19,20,21,22]. Under specific conditions, the proposed regularization ensures global convexity of the objective function. Due to its global convexity, CNC sparse regularization can effectively avoid local optima and overcome the biased estimation associated with nonconvex sparse regularization. As a result, it has gained widespread usage in image processing and machine learning applications [23,24,25,26,27,28]. However, to the best of the authors’ knowledge, most of the research on CNC sparse regularization focuses on algorithm design and applications and the theoretical analysis is lacking. This motivates us to conduct an analysis on the prediction performance of the Lasso model with CNC sparse regularization, thereby substantiating that CNC sparse regularization outperforms ${\mathsf{\ell }}_1$ regularization.

In this paper, we consider the following Lasso model with CNC sparse regularization

$$\begin{array}{c}\hfill {\widehat{\beta }}_{\lambda }^{CNC}\in arg\underset{\beta }{min}{\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2+\lambda {\Psi }_B^{CNC}\left(\beta \right),\end{array}$$

(3)

where the non-convex regularization term ${\Psi }_B^{CNC}\left(\beta \right)$ is parameterized by a matrix B, and the global convexity of the objective function in (3) can also be guaranteed by adjusting B.

Through rigorous theoretical analysis and comprehensive experimental evaluations, we demonstrate that the utilization of non-convex regularization significantly enhances the prediction performance of the Lasso model, enabling accurate estimation of unknown variables of interest. Our contributions can be summarized as follows

• Theoretically, for the Lasso model with a specific CNC sparse regularization, we establish the conditions necessary to ensure global convexity of the objective function. Subsequently, by leveraging oracle inequality, we derive an improved upper bound on prediction performance compared to that of the Lasso model with ${\mathsf{\ell }}_1$ regularization.
• Algorithmically, we derive an ADMM algorithm that ensures convergence to a critical point for the proposed Lasso model with CNC sparse regularization.
• Empirically, we demonstrate that the proposed Lasso model with generalized minimax concave (GMC) regularization outperforms ${\mathsf{\ell }}_1$ regularization in both synthetic data and MRI reconstruction experiments, owing to its utilization of non-convex regularization.

The subsequent sections of this paper are arranged as follows. Section 2 presents some preliminary and related work concerning CNC sparse regularization, predictive performance results, and convex optimization. In Section 3, we give theoretical analysis on the prediction performance of the Lasso model with CNC sparse regularization and further propose an effective ADMM algorithm to solve it. In Section 4, we verified the superiority of the proposed model through synthetic datasets and real datasets. Finally, the conclusions are summarized in Section 5.

Notation

Throughout this paper, the ${\mathsf{\ell }}_1$, ${\mathsf{\ell }}_2$, ${\mathsf{\ell }}_q$ and ${\mathsf{\ell }}_{\infty }$ norms of $\beta \in {\mathbb{R}}^p$ are defined ${\parallel \beta \parallel }_1={\mathrm{\Sigma }}_i^p\left|{\beta }_i\right|$, ${\parallel \beta \parallel }_2=({\mathrm{\Sigma }}_i^p|{\beta }_i{{|}^2)}^{{\displaystyle \frac{1}{2}}}$, ${\parallel \beta \parallel }_q=({\mathrm{\Sigma }}_i^p|{\beta }_i{{|}^q)}^{{\displaystyle \frac{1}{q}}}$ and ${\parallel \beta \parallel }_{\infty }={max}_{i\in p}\left|{\beta }_i\right|$; meanwhile, ${\parallel \beta \parallel }_0$ represents the number of non-zero elements in $\beta$ and $\mathrm{Supp}\left(\beta \right)$ is the support of $\beta$. For any given set T, we use $T^c$ and $\left|T\right|$ to respectively represent its complement $p\setminus T$ and cardinality $\left|T\right|$. For matrix A, we denote its maximum eigenvalue and minimum eigenvalue as ${\sigma }_{max}$ and ${\sigma }_{min}$, respectively. The transpose and the pseudo-inverse of a matrix X are defined as $X^{\top }$ and $X^{†}$ respectively. For any given subset T of p in a matrix $X\in {\mathbb{R}}^{n\times p}$, we denote $X_T$ as the resulting matrix obtained by excluding all columns belonging to the complement of T from X. For the design matrix X, we denote the orthogonal projection onto $X_T$ by ${\Pi }_T$. In addition, for the sake of convenience, we denote the prediction loss ${\displaystyle \frac{1}{n}}{\parallel X(\widehat{\beta }-{\beta }^{*})\parallel }_2^2$ of two vectors $\widehat{\beta },{\beta }^{*}\in {\mathbb{R}}^p$ by ${\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})$.

2. Preliminaries and Related Works

In this section, we present essential background information and related literature that will be crucial for the subsequent sections of the paper.

2.1. Convex Non-Convex Sparse Regularization

To address the limitation of ${\mathsf{\ell }}_1$ regularization, ${\mathsf{\ell }}_q(0<q<1)$ regularization has been proposed to enhance the accuracy of estimation. However, the utilization of ${\mathsf{\ell }}_q$ regularization renders the objective function in problem (2) non-convex and leads to suboptimal local solutions [18,29,30,31].

To leverage the benefits of non-convex regularization and convex optimization techniques, Selesnick et al. introduced the CNC strategy, which constructs a non-convex sparse regularization but at the same time guarantees the global convexity of the objective function [20,21,22,23,24,25].

Common CNC sparse regularizations include logarithmic regularization [32], exponential regularization [33], arctangent regularization [32], minimax-convex (MC) regularization [21], etc. Formally, these regularizations can be represented by the class of additively separable functions

$${\psi }_b\left(\beta \right)=\sum _{i=1}^p{\psi }_b\left({\beta }_i\right),$$

(4)

where ${\psi }_b:\mathbb{R}\to \mathbb{R}$ is a non-convex function parameterized by a scalar parameter b, which controls the non-convexity of ${\psi }_b$. Take the MC regularization as an example, for each ${\beta }_i$, the ${\psi }_b^{MC}\left({\beta }_i\right)$ can be expressed as

$$\begin{array}{cc}\hfill {\psi }_b^{MC}\left({\beta }_i\right)& =\left\{\begin{array}{cc}|{\beta }_i|-{\displaystyle \frac{b^2}{2}},\hfill & |{\beta }_i|\le {\displaystyle \frac{1}{b^2}},\hfill \\ {\displaystyle \frac{1}{2b^2}},\hfill & |{\beta }_i|\ge {\displaystyle \frac{1}{b^2}}.\hfill \end{array}\right.\hfill \end{array}$$

(5)

Mathematically, ${\psi }_b^{MC}\left({\beta }_i\right)$ can also be written as

$${\psi }_b^{MC}\left({\beta }_i\right)=\left|{\beta }_i\right|-\underset{v}{min}\left\{\left|v\right|+{\displaystyle \frac{b^2}{2}}{({\beta }_i-v)}^2\right\},$$

(6)

where $\underset{v}{min}\left\{\right|v|+{\displaystyle \frac{b^2}{2}}{({\beta }_i-v)}^2\}$ is the scalar Huber function with parameter b. Noting that the scalar Huber function is a common smooth version of the ${\mathsf{\ell }}_1$ norm, the fundamental construction strategy for CNC involves subtracting its corresponding smooth version from the sparse regularization.

As highlighted in [19,24,34], the separable (additive) CNC sparse regularizations mentioned above necessitate the full column rank of the covariate matrix X to ensure the global convexity of (3). However, in many important applications, the  covariate matrix X is wide ($p>n$), then the CNC strategy with separable sparse regularizations cannot be used. This strongly motivated the development of non-separable sparse regularizations.

Again, starting with MC sparse regularization, the non-separable generalized MC (GMC) regularization is defined as

$${\Psi }_B^{GMC}\left(\beta \right)={\parallel \beta \parallel }_1-\underset{v}{min}\left\{{\parallel v\parallel }_1+{\displaystyle \frac{1}{2}}{\parallel B(\beta -v)\parallel }_2^2\right\},$$

(7)

where $B\in {\mathbb{R}}^{n\times p}$ is the non-convex control matrix parameter [19,20,21].

The GMC regularization ${\Psi }_B^{GMC}\left(\beta \right)$, in general, cannot be expressed in the separable form of (4) due to the arbitrariness of the matrix parameter B. Specifically, if $B^{\top }B$ is a diagonal matrix, i.e., $B^{\top }B=(b_1^2,b_2^2,\cdots ,b_n^2)$, then ${\Psi }_B^{GMC}\left(\beta \right)$ is separable, and if $B=0$ the ${\Psi }_B^{GMC}\left(\beta \right)$ reduces to the ${\mathsf{\ell }}_1$ norm. The key advantage of non-separable non-convex sparse regularization lies in its ability to ensure the global convexity of the objective function, even when the covariate matrix X is not full column rank. This feature is crucial as it enables the application of CNC sparse regularization to any linear image inverse problem.

2.2. Results of the Predicted Performance

The prediction performance of the Lasso ${\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})={\displaystyle \frac{1}{n}}{\parallel X(\widehat{\beta }-{\beta }^{*})\parallel }_2^2$ refers to how well the Lasso model predicts outcomes or responses based on input features and depends on its ability to strike a balance between regularization and feature selection.

In particular, Bunea et al. derive sparsity oracle inequalities pertaining to the prediction loss and highlight their implications for minimax estimation within the traditional nonparametric regression framework [35,36,37]. These inequalities enable us to establish bounds on the discrepancy between the prediction errors and the optimal sparse approximation, as determined by an oracle with full knowledge but constrained by sparsity. However, these results assumes a strict condition on the Gram matrix ${\displaystyle \frac{1}{n}}{X}^{\top }X$, requiring it to be positive definite or subject to mutually coherent constraints. Bickel  et al. proposed a more general oracle inequality by constraining the eigenvalues of the Gram matrix [38], but the inequality is not sharp because the constant in front of its infimum (usually referred to as the dominant constant of the oracle inequality) is not equal to 1 [12,39]. Subsequently, Koltchinskii et al. obtained the first sharp oracle inequality by constraining the diagonal elements of the Gram matrix to be no larger than 1. Sun et al. improved the predictive performance of the Lasso model by utilizing the scaling effect and further relaxing the constraints on the covariate matrix X [40]. Dalalyan et al. present novel findings regarding the predictive accuracy of the Lasso model, even when minimal assumptions are made about the relationship between the covariates [12]. More specifically, we give the definition of the compatibility factor as follows:

Definition 1 

(Compatibility factor). The compatibility factor ${\kappa }_T$ of a set $T\subseteq \{1,2,\dots ,p$} is defined as ${\kappa }_{\varnothing }=1$, and for any nonempty set T,

$$\begin{array}{c}\hfill {\kappa }_T=\underset{\beta }{inf}{\displaystyle \frac{\sqrt{\left|T\right|}·{\parallel X\beta \parallel }_2^2}{\sqrt{n}{\parallel \beta \parallel }_1}}.\end{array}$$

(8)

The basic error bound of ${\mathsf{\ell }}_1$ Lasso minimizer in [12] can be rewritten as Theorem 1.

Theorem 1 

([12], Theorem 2). Let $\delta \in (0,1)$ be a fixed tolerance level and $\gamma >1$. For (2), let $\widehat{\beta }$ be a stationary point. Set the regularization parameter as $\lambda =\gamma {\sigma }^{*}\sqrt{{\displaystyle \frac{2log(p/\delta )}{n}}}$ then

$$\begin{array}{c}\hfill {\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})\le \underset{\overline{\beta }\in {\mathbb{R}}^p,T\subset p}{inf}\left\{{\mathsf{\ell }}_n(\overline{\beta },{\beta }^{*})+4{\parallel {\overline{\beta }}_{T^c}\parallel }_1\right\}+{\displaystyle \frac{{\sigma }^{*2}}{n}}\left(\left|T\right|+2log\left({\displaystyle \frac{1}{\delta }}\right)+{\displaystyle \frac{8{\gamma }^{2}log(p/\delta )}{{\kappa }_T^2}}\right),\end{array}$$

(9)

on the estimation error with probability at least $1-2\delta$ for any $T\subset (1,2,\cdots ,p)$.

The result obtained from Theorem 1 differs slightly from the one given in [12]; however, it can be easily derived from the proof provided in [12]. The proof of Theorem 1 is provided in the Appendix A.1.

2.3. Convex Optimization

For the following optimization problem

$$\begin{array}{c}\hfill \underset{\beta }{min}\{f\left(\beta \right)+g\left(\beta \right)\},\end{array}$$

(10)

where $f\left(\beta \right)$ is a convex and differentiable, $g\left(\beta \right)$ is proper closed, convex but non-smooth regularization term. Proximal gradient descent (PGD) algorithm perform well in solving such optimization model [41]. The iterative formula utilized for the PGD is

$$\begin{array}{c}\hfill {\beta }^{k+1}={\mathrm{prox}}_{\alpha g}({\beta }^k-\alpha \nabla f\left({\beta }^k\right)),\end{array}$$

(11)

where $\alpha >0$ is the step size of each iteration and $\nabla f\left({\beta }^k\right)$ is the gradient of $f\left({\beta }^k\right)$. ${\mathrm{prox}}_g(·)$ is the proximal operator of g, which is defined as

$$\begin{array}{c}\hfill {\mathrm{prox}}_g\left(\beta \right)=arg\underset{v}{min}\left\{{\displaystyle \frac{1}{2}}{\parallel \beta -v\parallel }_2^2+g\left(v\right)\right\}.\end{array}$$

(12)

As a special case, if $g\left(\beta \right)={\lambda \parallel \beta \parallel }_1$, then its proximal operator ${\mathrm{prox}}_{{\lambda \parallel ·\parallel }_1}$ is the element-wise soft thresholding function

$${\mathrm{prox}}_{{\lambda \parallel ·\parallel }_1}\left({\beta }_i\right)=\mathrm{sgn}\left({\beta }_i\right)max\left\{\right|{\beta }_i|-\lambda ,0\}=\left\{\begin{array}{cc}{\beta }_i-\lambda ,& {\beta }_i\ge \lambda ,\hfill \\ 0,& |{\beta }_i| <\lambda ,\hfill \\ {\beta }_i+\lambda ,& {\beta }_i\le -\lambda .\hfill \end{array}\right.$$

(13)

where ${\beta }_i$ is the i-th element of $\beta$.

3. Lasso with CNC Sparse Regularization

In this section, we will analyze the Lasso model with CNC sparse regular terms both theoretically and algorithmically. In particular, we will consider the following Lasso model with the non-separable non-convex GMC regularization

$$\begin{array}{c}\hfill {\widehat{\beta }}_{\lambda }^{GMC}\in arg\underset{\beta }{min}\left\{{\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2+\lambda {\Psi }_B^{GMC}\left(\beta \right)\right\},\end{array}$$

(14)

where the GMC regularization is defined as (7), and the matrix parameter B can influence the non-convexity of ${\Psi }_B^{GMC}\left(\beta \right)$.

3.1. Convex Condition

In this subsection, we explore the adjustment of GMC regularization to preserve the overall convexity of the Lasso model (14).

Theorem 2. 

Let $y\in {\mathbb{R}}^n$, $X\in {\mathbb{R}}^{n\times p}$, and $\lambda >0$. Define the objection function of (14) as

$$F_B^{GMC}\left(\beta \right)={\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2+\lambda {\Psi }_B^{GMC}\left(\beta \right),$$

(15)

then $F_B^{GMC}\left(\beta \right)$ is convex if $B^{\top }B⪯{\displaystyle \frac{1}{n\lambda }}{X}^{\top }X$ and is strictly convex if $B^{\top }B\prec {\displaystyle \frac{1}{n\lambda }}{X}^{\top }X$.

The proof of Theorem 2 is provided in Appendix A.2.

By constructing the matrix parameter B, we can ensure that the convexity conditions mentioned above are maintained. Similar to [23], when X and $\lambda$ are specified, a straightforward method to determine the parameter B is

$$\begin{array}{c}\hfill B=\sqrt{{\displaystyle \frac{\omega }{n\lambda }}}X,0\le \omega \le 1.\end{array}$$

(16)

Then, $B^{\top }B={\displaystyle \frac{\omega }{n\lambda }}{X}^{\top }X$ which satisfies Theorem 2 when $\omega \le 1$. The  parameter $\omega$ controls the non-convexity of the ${\Psi }_B^{GMC}$. When $\omega =0$, $B=0$ and the penalty reduces to the ${\mathsf{\ell }}_1$ norm. When $\omega =1$, (15) is satisfied with equality, resulting in a ’maximally’ non-convex penalty.

According to Theorem 2, we can deduce the following corollary, which is useful in our proof of algorithm convergence.

Corollary 1. 

For all $\mu \ge \lambda {\sigma }_{max}$, ${\Psi }_B^{GMC}\left(\beta \right)+{\displaystyle \frac{\mu }{2}}{\parallel \beta \parallel }_2^2$ is convex, where ${\sigma }_{max}$ is the largest eigenvalue of $B^{\top }B$.

By replacing ${\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2$ with ${\displaystyle \frac{\mu }{2}}{\parallel \beta \parallel }_2^2$ in (A14), we can draw Corollary 1 from the proof of the Theorem 2.

3.2. Prediction Performance

In this subsection, we analyze the prediction performance of the Lasso model with GMC regularization (14) and use the oracle inequality to obtain an upper bound on the prediction performance that is better than the upper bound of Lasso model with ${\mathsf{\ell }}_1$ regularization.

We consider a general Lasso model with GMC regularization (14), that is, the GMC regularization term is non-convex and does not necessarily satisfy the convexity condition given by Theorem 2. Due to the non-convexity, it is possible that the global minimum of the proposed Lasso model cannot be achieved. Hence, it becomes crucial to find a stable point that satisfies the conditions. We define $\widehat{\beta }\in {\mathbb{R}}^p$ as a stationary point of $F_B^{GMC}\left(\beta \right)$ when it fulfills

$$\begin{array}{c}\hfill 0\in {\nabla }_{\beta }{F}_B^{GMC}{\left(\beta \right)|}_{\beta =\widehat{\beta }}.\end{array}$$

Based on Definition 1, the compatibility factor enables us to establish a subsequent oracle inequality, which is applicable to the stationary points of the Lasso estimator with GMC regularization.

Theorem 3 

(Oracle inequality of the Lasso model with GMC regularization). Assume $\mu <{\displaystyle \frac{1}{n\parallel X^{†}{\parallel }_2^2}}$. Let $\delta \in (0,1)$ be a fixed tolerance level and $\gamma >1$. Let $\widehat{\beta }$ be a stationary point of (14) and ${\sigma }_{min}$ is the smallest eigenvalue of $B^{\top }B$. Set the regularization parameter as $\lambda =\gamma {\sigma }^{*}\sqrt{{\displaystyle \frac{2log(p/\delta )}{n}}}$; then, the estimation error satisfies

$$\begin{array}{cc}\hfill {\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})& \le \underset{\overline{\beta }\in {\mathbb{R}}^p,T\subset p}{inf}\left\{{\mathsf{\ell }}_n(\overline{\beta },{\beta }^{*})+4{\Psi }_{\sqrt{{\sigma }_{min}I}}^{GMC}\left({\left(\overline{\beta }\right)}_{T^c}\right)\right\}\hfill \\ \hfill & +{\displaystyle \frac{2{\sigma }^{*2}}{n(1-n\mu \parallel X^{†}{\parallel }_2^2)}}\left(\left|T\right|+2log(1/\delta )+{\displaystyle \frac{8{\gamma }^{2}log(p/\delta )\left|T\right|}{{\kappa }_T^2}}\right),\hfill \end{array}$$

(17)

with probability at least $1-2\delta$ for any $T\subseteq (1,2,\cdots ,p)$.

The proof of Theorem 3 is given in the Appendix A.3.

The oracle inequality holds for any $\widehat{\beta }$ that meets the first-order optimality condition, regardless of whether the regularization is non-convex. This mild condition imposed on $\widehat{\beta }$ represents a significant divergence from previous findings, such as Theorem 2 of [12], which is suitable for global minimization but challenging to ensure when employing non-convex regularizations.

Theorem 3 enables the optimization of the error bounds on the right-hand side of (17) by allowing the selection of suitable values for $\overline{\beta }$ and T. For instance, choose $\overline{\beta }={\beta }^{*}$ in (17) (hence an ‘oracle’), then ${\mathsf{\ell }}_n(\overline{\beta },{\beta }^{*})=0$; therefore, the (17) can be rewritten as

$$\begin{array}{c}\hfill {\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})\le 4{\Psi }_{\sqrt{{\sigma }_{min}I}}^{GMC}\left({\left({\beta }^{*}\right)}_{T^c}\right)+{\displaystyle \frac{2{\sigma }^{*2}}{n(1-n\mu \parallel X^{†}{\parallel }_2^2)}}\left(\left|T\right|+2log(1/\delta )+{\displaystyle \frac{8{\gamma }^{2}log(p/\delta )\left|T\right|}{{\kappa }_T^2}}\right).\end{array}$$

(18)

Furthermore, if  T is set as the empty set, then

$$\begin{array}{c}\hfill {\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})\le 4{\Psi }_{\sqrt{{\sigma }_{min}I}}^{GMC}\left({\beta }^{*}\right)+{\displaystyle \frac{4log(1/\delta ){\sigma }^{*2}}{n(1-n\mu \parallel X^{†}{\parallel }_2^2)}}.\end{array}$$

(19)

In contrast, if  T is set as the support of $\beta$, then

$$\begin{array}{c}\hfill {\mathsf{\ell }}_n(\widehat{\beta },{\beta }^{*})\le {\displaystyle \frac{2{\sigma }^{*2}}{n(1-n\mu \parallel X^{†}{\parallel }_2^2)}}\left({\parallel \beta \parallel }_0+2log(1/\delta )+{\displaystyle \frac{8{\gamma }^{2}log(p/\delta ){\parallel \beta \parallel }_0}{{\kappa }_T^2}}\right),\end{array}$$

(20)

which increases linearly with the sparsity level ${\parallel \beta \parallel }_0$.

We compare the prediction error bounds of Theorem 3 with Theorem 1, which is obtained for a Lasso model with the ${\mathsf{\ell }}_1$ regularization. We note that the upper bounds of terms within the lower bound in inequality (17) are limited by the upper bounds of terms within the lower bound in Theorem 1. When $\beta$ contains larger coefficients, non-convex regularization leads to more stringent bound, hence ${\Psi }_{\sqrt{{\sigma }_{min}I}}^{GMC}\left(\overline{\beta }\right)\ll {\parallel \overline{\beta }\parallel }_1$. Furthermore, the error bound in Equation (17) includes $n(1-n\mu \parallel X^{†}{\parallel }_2^2)$ in the denominator, which makes it an upper bound for Theorem 1. This gap can be mitigated by adjusting the matrix parameter B in ${\Psi }_B^{GMC}\left(\beta \right)$, specifically by selecting a smaller value for $\omega$ in (16). However, when $\omega \to 0$, the non-convex GMC regularization ${\Psi }_B^{GMC}\left(\beta \right)$ also tends to ${\mathsf{\ell }}_1$, eliminating the improvement brought by using a non-convex regularization in the first term of the bound. This means that the overall error bound can be adjusted by tuning $\omega$, thereby enabling a trade-off with the non-convexity of the regularization.

In summary, despite the non-convex of the regularization term, we can ensure that any stationary point of the proposed Lasso model with GMC regularization possesses a strong statistical guarantee.

3.3. ADMM Algorithm

In this subsection, we will consider using the ADMM algorithm to solve the Lasso model with GMC regularization (14). Boyd et al. introduced a generalized ADMM framework for addressing sparse optimization problems [42]. The basic idea is to transform unconstrained optimization problems into constrained ones by splitting variables and then iteratively solve them. ADMM proves to be particularly efficient when the sparse regularization term possesses a closed-form proximal operator. The main challenge in solving (14) by ADMM lies in how to obtain the proximal operator of the GMC regularization.

The augmented Lagrange form of (14) can be written as follows by setting $z=\beta$.

$$\begin{array}{c}\hfill L(\beta ,z,u)={\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2+\lambda {\Psi }_B^{GMC}\left(z\right)+u^{\top }(z-\beta )+{\displaystyle \frac{\rho }{2}}{\parallel z-\beta \parallel }_2^2,\end{array}$$

(21)

where u represents the Lagrange multiplier and $\rho >0$ denotes the penalty parameter. We then update $\beta , z$ and u in an alternating manner.

Step 1. Update ${\beta }^{k+1}$:

$$\begin{array}{c}\hfill {\beta }^{k+1}=arg\underset{\beta }{min}\left\{{\displaystyle \frac{1}{2n}}{\parallel y-X\beta \parallel }_2^2-{\left(u^k\right)}^{\top }\beta +{\displaystyle \frac{\rho }{2}}{\parallel z^k-\beta \parallel }_2^2\right\}.\end{array}$$

(22)

The objective function in Equation (22) is differentiable with respect to $\beta$. As a result, ${\beta }^{k+1}$ can be obtained by utilizing the first-order optimality condition

$$\begin{array}{c}\hfill {\beta }^{k+1}={({\displaystyle \frac{1}{n}}{X}^{\top }X+\rho I)}^{-1}(y^{\top }X+u^k+\rho z^k).\end{array}$$

(23)

Step 2. Update $z^{k+1}$:

$$\begin{array}{cc}\hfill z^{k+1}& =arg\underset{z^k}{min}\left\{\lambda {\Psi }_B^{GMC}\left(z^k\right)+{\left(u^k\right)}^{\top }{z}^k+{\displaystyle \frac{\rho }{2}}{\parallel z^k-{\beta }^{k+1}\parallel }_2^2\right\}\hfill \\ \hfill & =arg\underset{z^k}{min}\left\{\lambda {\Psi }_B^{GMC}\left(z^k\right)+{\displaystyle \frac{\rho }{2}}{\parallel z^k-{\beta }^{k+1}+{\displaystyle \frac{u^k}{\rho }}\parallel }_2^2\right\}.\hfill \end{array}$$

(24)

Equation (24) serves as the proximal operator for ${\Psi }_B^{GMC}\left(z\right)$, although it lacks a closed-form solution. Despite this limitation, the PGD algorithm can be employed to iteratively address Equation (24).

The substitution of ${\Psi }_B^{GMC}\left(\beta \right)$ with (7) allows us to reformulate Equation (24) as

$$\begin{array}{c}\hfill z^{k+1}=arg\underset{z^k}{min}\left\{\lambda \parallel z^k{\parallel }_1-\lambda \underset{v}{min}{\{\parallel v\parallel }_1+{\displaystyle \frac{1}{2}}\parallel B(z^k-v){\parallel }_2^2\}+{\displaystyle \frac{\rho }{2}}{\parallel z^k-{\beta }^{k+1}+{\displaystyle \frac{u^k}{\rho }}\parallel }_2^2\right\}.\end{array}$$

(25)

Let $f\left(z^k\right)={\displaystyle \frac{\rho }{2}}\parallel z^k-({\beta }^{k+1}-{\displaystyle \frac{u^k}{\rho }}){\parallel }_2^2-\lambda \underset{v}{min}{\{\parallel v\parallel }_1+{\displaystyle \frac{1}{2}}{\parallel B(z^k-v)\parallel }_2^2$ and $g\left(z^k\right)=\lambda {\parallel z^k\parallel }_1$. The update of $z^{k+1}$ of (25) can be obtained through PGD algorithm as follows:

$$\begin{array}{cc}\hfill z^{k+1}& ={\mathrm{prox}}_{\alpha g}(z^k-\alpha \nabla f\left(z^k\right))\hfill \\ \hfill & ={\mathrm{prox}}_{{\alpha \lambda \parallel ·\parallel }_1}(z^k-\alpha \nabla f\left(z^k\right)),\hfill \end{array}$$

(26)

where $\alpha$ is the iteration step size. The main challenge of (26) is solving the gradient $\nabla f\left(z^k\right)$.

According to Lemma 3 in [24], the last term of $f\left(z^k\right)$ is a differentiable with respect to $z^k$. Furthermore, the gradient of $f\left(z^k\right)$ can be expressed as

$$\begin{array}{c}\hfill \nabla f\left(z^k\right)=\rho (z^k-{\beta }^{k+1}+{\displaystyle \frac{u^k}{\rho }})-\lambda \left(B^{\top }B(z^k-arg\underset{v}{min}\{{\displaystyle \frac{1}{2}}\parallel B(z^k-v){\parallel }_2^2+{\parallel v\parallel }_1\left\}\right)\right).\end{array}$$

(27)

Note that (27) contains an ${\mathsf{\ell }}_1$-regularization problem $v^{k+1}=arg\underset{v}{min}\{{\displaystyle \frac{1}{2}}{\parallel B(z-v)\parallel }_2^2+{\parallel v\parallel }_1\}$, which can be viewed as a proximal operator associated with the ${\parallel ·\parallel }_1$, i.e.,

$$\begin{array}{c}\hfill v^{k+1}={\mathrm{prox}}_{{\parallel ·\parallel }_1}(v^k+B(z^k-v^k)),\end{array}$$

(28)

where ${\mathrm{prox}}_{{\parallel ·\parallel }_1}$ can be easily addressed by employing the soft-thresholding function (13).

Then, by substituting (28) and (27) to (26) and sorting them out, the update of $z^{k+1}$ can be summarized as follows

$$\left\{\begin{array}{cc}{v}^{k+1}=& {\mathrm{prox}}_{{\parallel ·\parallel }_1}(v^k+B(z^k-v^k)),\hfill \\ s^{k+1}=& (1-\alpha \rho +\alpha \lambda B^{\top }B)z^k+\alpha \rho {\beta }^{k+1}-\alpha u^k-\alpha \lambda B^{\top }B{v}^{k+1},\hfill \\ z^{k+1}=& {\mathrm{prox}}_{{\alpha \lambda \parallel ·\parallel }_1}\left(s^{k+1}\right).\hfill \end{array}\right.$$

(29)

Step 3. update $u^{k+1}$:

$$\begin{array}{c}\hfill u^{k+1}=u^k+({\beta }^{k+1}-z^{k+1}).\end{array}$$

(30)

Finally, the ADMM algorithm for the Lasso model with convex non-convex sparse regularization can be derived by integrating Equations (23), (29), and (30). This derivation is presented as Algorithm 1.

Algorithm 1 ADMM for solving (14)

Require:  $y,X,{\beta }^0,z^0,u^0,B,\lambda ,\rho ,\alpha$. Ensure:  $\beta$. while “stopping criterion is not met” do    ${\beta }^{k+1}={({\displaystyle \frac{1}{n}}{X}^{\top }X+\rho I)}^{-1}(y^{\top }X+u^k+\rho z^k)$;    $v^{k+1}={\mathrm{prox}}_{{\parallel ·\parallel }_1}(v^k+B(z^k-v^k);$    $s^{k+1}=(1-\alpha \rho +\alpha \lambda B^{\top }B)z^k+\alpha \rho {\beta }^{k+1}-\alpha u^k-\alpha \lambda B^{\top }B{v}^{k+1}$;    $z^{k+1}={\mathrm{prox}}_{{\alpha \lambda \parallel ·\parallel }_1}\left(s^{k+1}\right);$    $u^{k+1}=u^k+({\beta }^{k+1}-z^{k+1})$. end while

Furthermore, we provide the following convergence guarantee for Algorithm 1.

Theorem 4. 

Through proper selection of the penalty parameter ρ, the primal residual $r^{\left(k\right)}={\parallel \beta -z^{\left(k\right)}\parallel }_2$, and the dual residual $s^{(k+1)}={\parallel \rho (z^{(m+1)}-z^{\left(m\right)})\parallel }_2$, Algorithm 1 satisfies $\underset{k\to \infty }{lim}{r}^{\left(k\right)}=0$ and $\underset{k\to \infty }{lim}{s}^{\left(k\right)}=0$.

The proof of Theorem 4 is given in Appendix A.4.

Theorem 4 illustrates that Algorithm 1 ultimately converges to satisfy both primal and dual feasibility conditions. Additionally, it confirms the equivalence between the augmented Lagrangian formulation (21) with constant z and u values and the original Lasso problem (14). During each iteration of Algorithm 1, a stationary point $\beta$ is generated for the augmented Lagrangian formulation (21) when z and u are fixed, which indicates that ${\beta }^{*}$ also serves as a stationary point for (14).

4. Numerical Experiment

In this section, we present the efficacy of the proposed Lasso model that incorporates GMC sparse regularization through numerical experiments conducted on synthetic and real-world data. All experiments are demonstrated on MATLAB R2020a on a PC equipped with a 2.5 GHz CPU and 16 GB memory.

4.1. Synthetic Data

The data in this experiment are simulated with $n=4000$ samples and $p=8000$ features, where the correlation between features j and $j^{\prime }$ is equal to $0.6|j-j^{\prime }|$. The true vector ${\beta }^{*}$ consists of 800 non-zero entries, all equal to 1. The observations are obtained through the linear model $y=X{\beta }^{*}+ϵ$, where $ϵ$ is Gaussian noise with variance such that $\parallel X\beta \parallel /\parallel ϵ\parallel =5$ ($SNR=5$). The parameter $\lambda$ is expressed as a fraction of ${\lambda }_{max}={\parallel X^{\top }y\parallel }_{\infty }/n$. The objective function incorporates a maximum penalty ${\lambda }_{max}$ for regularization, which will give the maximum sparse solution, i.e., $\beta =0$. Then, the range of $\lambda$ values can be set to vary between $(0,{\lambda }_{max})$ in order to obtain a regular path graph. For the GMC regularization, it is necessary to explicitly specify matrix B we set it by using (16) and $\omega =0.8$.

In this experiment, F1-score and root mean square error (RMSE) were chosen as evaluation metrics to assess the predictive performance of model (14).

The F1 score is the harmonic mean of precision and recall, offers a more comprehensive perspective for assessing model performance in comparison to solely considering precision and recall metrics. Specifically, the precision and recall are defined as

$$\begin{array}{c}\hfill \mathrm{Precision}={\displaystyle \frac{|\mathrm{Supp}\left(\widehat{\beta }\right)\cap \mathrm{Supp}\left({\beta }^{*}\right)|}{\left|\mathrm{Supp}\right(\widehat{\beta }\left)\right|}},\mathrm{Recall}={\displaystyle \frac{|\mathrm{Supp}\left(\widehat{\beta }\right)\cap \mathrm{Supp}\left({\beta }^{*}\right)|}{\left|\mathrm{Supp}\right({\beta }^{*}\left)\right|}},\end{array}$$

(31)

where $\widehat{\beta }$ and ${\beta }^{*}$ respectively represent the estimated value and the true value. Then, the F1 score is defined as

$$\mathrm{F}1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}.$$

(32)

The range of F1 score is between 0 and 1, where an F1 score of 1 indicates complete support for recovery, meaning the model accurately estimates sparse vectors; while an F1 score of 0 means no support for recovery, indicating zero estimation capability. Thus, a higher F1 score signifies stronger predictive performance demonstrated by the model.

The RMSE is an important indicator in regression model evaluation, which can be used to measure the magnitude of prediction errors. Its definition is as follows

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{({\beta }_i-{\beta }_i^{*})}^2},$$

(33)

where ${\beta }_i$ is the i-th predicted value, ${\beta }_i^{*}$ is the i-th actual observed value, n is the number of observations. A smaller RMSE value indicates a higher predictive capability and better model fit.

We compared the GMC sparse regularization (14) with the traditional ${\mathsf{\ell }}_1$ and ${\mathsf{\ell }}_{1/2}$ regularizations. To obtain typical conclusions, we conducted 100 Monte Carlo experiments using random noise with $SNR=5$ and calculated the average F1 and RMSE values. As shown in Figure 1a, when an appropriate $\lambda$ value is selected, the F1 score of the GMC sparse regularization reaches the maximum value of 1. In contrast, the F1 scores of the ${\mathsf{\ell }}_1$ and ${\mathsf{\ell }}_{1/2}$ regularizations are approximately 0.7 and 0.95, respectively, both lower than the performance of the GMC regularization. Additionally, it can be clearly observed from Figure 1b that in the sparse vector prediction task, the minimum RMSE of the GMC sparse regularization is significantly lower than that of the ${\mathsf{\ell }}_1$ and ${\mathsf{\ell }}_{1/2}$ regularizations. This result further validates the superior accuracy of the GMC sparse regularization in sparse vector prediction. On the other hand, Figure 1 shows that when estimating sparse vectors, the difference between the $\lambda$ values corresponding to the maximum F1 score and the minimum RMSE in the Lasso model based on GMC sparse regularization is relatively small. In contrast, this difference is larger in the ${\mathsf{\ell }}_1$ regularization, indicating that GMC sparse regularization (14) outperforms the ${\mathsf{\ell }}_1$ and ${\mathsf{\ell }}_{1/2}$ regularizations in terms of prediction performance when choosing an appropriate $\lambda$ value.

The conclusion of our study highlights the exceptional predictive performance of our model (2) in estimating sparse vectors, surpassing the ${\mathsf{\ell }}_1$ regularization approach employed by the Lasso model. Additionally, although the Lasso model with ${\mathsf{\ell }}_{1/2}$ regularization also performs well in prediction tasks, its non-convexity poses challenges for the algorithms solving the objective function. In contrast, our model not only has excellent predictive ability but also ensures the overall convexity of the objective function, thereby providing convenient conditions for the design and implementation of optimization algorithms. In summary, the model proposed in this study has significant advantages in both predictive performance and optimization characteristics, effectively avoiding the computational difficulties brought by non-convex regularization methods.

4.2. MRI Reconstruction

The utilization of high-dimensional sparse linear regression serves as the foundation for numerous signal and image processing techniques, including MRI reconstruction [43,44,45]. The MRI is a powerful medical imaging technique, featuring high soft tissue contrast and the advantage of being radiation-free. It is widely used in clinical diagnosis and scientific research fields. In this experimental, we employ GMC sparse regularization (15) for MRI reconstruction and compare its performance against ${\mathsf{\ell }}_1$ and ${\mathsf{\ell }}_{1/2}$ regularization.

To make the statement clear, we define the design matrix $X=R\times F$, where $R\in {\mathbb{R}}^{n\times p}$ is the sampling template, and $F\in {\mathbb{R}}^{p\times p}$ is the sparse Fourier operator. We tested different reconstruction models on three MRI images with variable density sampling and Cartesian sampling templates, respectively. For comparison, all images are set to $256\times 256$ pixels, with grayscale values ranging from 0 to 255. The parameters were $\lambda =0.001$, $\rho =150$, the iteration step size $\alpha$ was chosen $\alpha =1$. For the GMC regularization, it is necessary to explicitly specify matrix B and set it using (16) with $\omega =0.9$.

In this experiment, we selected relative error (RE) and peak signal-to-noise ratio (PSNR) as evaluation metrics to quantitatively assess the quality and accuracy of reconstructed images.

The RE is defined as

$$\mathrm{RE}={\displaystyle \frac{\parallel \widehat{\beta }-{\beta }^{*}{\parallel }_2}{\parallel {\beta }^{*}{\parallel }_2}},$$

(34)

where ${\beta }^{*}$ and $\widehat{\beta }$ are the original and reconstructed images, respectively.

The PSNR is defined as

$$\mathrm{PSNR}=10·lg{\displaystyle \frac{{\mathrm{MAX}}^2}{\frac{1}{N}{\parallel \widehat{\beta }-{\beta }^{*}\parallel }_2^2}},$$

(35)

where $N=256\times 256$, $\mathrm{MAX}=255$, and $\widehat{\beta }$, ${\beta }^{*}$ are the reconstructed, and the original image, respectively.

For these two types of quantitative evaluation criteria, the lower the RE value, the more superior the reconstruction effect, while for PSNR, an increase in numerical value indicates a more outstanding reconstruction ability.

To further enhance the visual contrast effect, we calculated the difference between the reconstructed image and the original image. Furthermore, we magnified a small portion of the local image to show more details. The reconstruction results of three regularizations are shown in Figure 2, Figure 3 and Figure 4. It can be found that the reconstructed images based on ${\mathsf{\ell }}_1$ regularization have problems such as blurred edges and residual shadows, and the ${\mathsf{\ell }}_{1/2}$ regularization reconstruction model also shows similar phenomena. In contrast, the reconstructed images based on GMC sparse regularization are closer to the original images and exhibit higher reconstruction quality.

Additionally, we conducted a quantitative evaluation of the reconstruction results in

Table 1. Quantitative results of different regularizations. The best results are highlighted in bold.

Template	Image	Model	RE	PSNR (dB)
Variable Density Sampling	Image1	${\mathsf{\ell }}_1$	0.0458	35.6637
		${\mathsf{\ell }}_{1/2}$	0.0356	37.8626
		GMC	0.0277	40.0481
Variable Density Sampling	Image2	${\mathsf{\ell }}_1$	0.0912	31.3563
		${\mathsf{\ell }}_{1/2}$	0.0797	32.5274
		GMC	0.0648	34.3238
Cartesian Sampling	Image3	${\mathsf{\ell }}_1$	0.1076	29.8075
		${\mathsf{\ell }}_{1/2}$	0.0952	30.8666
		GMC	0.0866	31.6867

. It is obvious that compared with ${\mathsf{\ell }}_1$ and ${\mathsf{\ell }}_{1/2}$ regularization, GMC sparse regularization has the best performance in MRI reconstruction and can obtain the lowest RE and highest PSNR.

5. Conclusions

In this paper, we propose CNC sparse regularization as a valuable alternative to the ${\mathsf{\ell }}_1$ regularization used in Lasso regression. This approach effectively addresses the issue of underestimation for high-amplitude components while ensuring the global convexity of the objective function. Our theoretical analysis demonstrates that the prediction error bound associated with CNC sparse regularization is smaller than that of ${\mathsf{\ell }}_1$ regularization, which provides theoretical support for the practical application of the CNC regularization. Additionally, we demonstrate that the Lasso model with CNC regularization exhibits superior performance on both synthetic and real-world datasets. These findings suggest its potential significance in future applications such as image denoising, image reconstruction, seismic reflection analysis, etc.

Additionally, given that the oracle inequality of the Lasso model relies on the restricted eigenvalue condition, future research directions may include exploring theoretical guarantees under more relaxed assumptions, such as the oracle inequality based on weak correlation or unconstrained design matrices, and experimentally verifying the practical effectiveness of the restricted eigenvalue condition (such as calculating the specific value of ${\kappa }_T$), and further analyzing its impact on model performance.