Sparse Adaptive Iteratively-Weighted Thresholding Algorithm (SAITA) for L p -Regularization Using the Multiple Sub-Dictionary Representation

Abstract

Both $L_{1/2}$ and $L_{2/3}$ are two typical non-convex regularizations of $L_p$ ($0<p<1$), which can be employed to obtain a sparser solution than the $L_1$ regularization. Recently, the multiple-state sparse transformation strategy has been developed to exploit the sparsity in $L_1$ regularization for sparse signal recovery, which combines the iterative reweighted algorithms. To further exploit the sparse structure of signal and image, this paper adopts multiple dictionary sparse transform strategies for the two typical cases $p\in \left\{1/2,2/3\right\}$ based on an iterative $L_p$ thresholding algorithm and then proposes a sparse adaptive iterative-weighted $L_p$ thresholding algorithm (SAITA). Moreover, a simple yet effective regularization parameter is proposed to weight each sub-dictionary-based $L_p$ regularizer. Simulation results have shown that the proposed SAITA not only performs better than the corresponding $L_1$ algorithms but can also obtain a better recovery performance and achieve faster convergence than the conventional single-dictionary sparse transform-based $L_p$ case. Moreover, we conduct some applications about sparse image recovery and obtain good results by comparison with relative work.

1. Introduction

Compressed sensing (CS) [1,2] and sparse representation [3,4] have been widely used in the field of wireless communications [5,6,7] and image processing [8,9,10]. CS implies that it is possible to reconstruct the sparse signal/image from incomplete data if some prior knowledge and reconstruction constraints are satisfied. Mathematically, the unconstrained $L_0$ minimization is the optimal model to obtain the sparsest solution ${\widehat{\mathbf{x}}}_{l_0}$:

$${\widehat{\mathbf{x}}}_{l_0}=\arg \underset{\widehat{\mathbf{x}}}{\min }\{\gamma {\parallel \mathbf{y}-\mathsf{\Phi }\mathbf{x}\parallel }_2^2+\lambda {\parallel \mathbf{x}\parallel }_0\},$$

(1)

where $\parallel \mathbf{x}{\parallel }_0$ denotes the zero-norm function to find the number of nonzero elements in $\mathbf{x}$; $\mathsf{\Phi }\in R^{M\times N}$ denotes the down-sampling measurement matrix; $\mathbf{y}$ and $\mathbf{x}$ represent the observed vector and the unknown sparse image, respectively; $\lambda$ is the regularization parameter to balance between the fidelity of the image and the sparsity; and $\gamma >0$ is a small positive constant, e.g., $\gamma =1/2$. Unfortunately, this problem (1) is an NP (non-deterministic) problem, and thus, it is difficult to efficiently solve. When the matrix $\mathsf{\Phi }$ satisfies some necessary conditions [11], an alternative convex relaxation method are developed using the $L_1$ regularization method as:

$${\widehat{\mathbf{x}}}_{l_p}=\arg \underset{\widehat{\mathbf{x}}}{\min }\{\gamma {\parallel \mathbf{y}-\mathsf{\Phi }\mathbf{x}\parallel }_2^2+\lambda {\parallel \mathbf{x}\parallel }_1\},$$

(2)

where $\parallel \mathbf{x}{\parallel }_1={{\displaystyle \sum }}_{i=1}^n\left|x_i\right|$ denotes the $L_1$-norm. Then, the NP problem (1) is converted into problem (2), which is a typical a convex optimization problem and can be solved efficiently, such as with the alternating direction method of multipliers (ADMM) [12,13], fast iterative shrinkage-thresholding algorithm (FISTA) [14], Nesterov’s algorithm (NESTA) [15], and approximate message passing (AMP) [16]. However, the method of $L_1$ regularization can only obtain a suboptimal solution and usually requires much more measurements. Theoretical analysis of CS implies that better performance can be obtained by taking advantage of sparser information in many systems, especially in the presence of strong noise interference.

1.1. The Non-Convex Penalties

Many state-of-the-art algorithms have been proposed to improve the performance of the $L_1$ regularization algorithms. The non-convex penalty regularization algorithms are among the most effective algorithms for sparse recovery problems. Research shows that the non-convex penalty-based optimization methods can more closely approximate the sparsest solution over the $L_1$-norm penalty in problem (2), which requires a weaker incoherent condition and fewer measurement data [17]. There have been many non-convex functions proposed as relaxations of the $L_0$-norm penalty, such as the smoothly clipped absolute deviation penalty (SCAD) [18], the $L_p,(0<p<1)$-norm penalty [17] and the minimax concave penalty (MCP) [19]. By replacing the $L_1$-norm with the $L_p$-norm, the non-convex $L_p$-norm regularization optimization method is described as:

$${\widehat{\mathbf{x}}}_{l_p}=\arg \underset{\widehat{\mathbf{x}}}{\min }\{\gamma {\parallel \mathbf{y}-\mathsf{\Phi }\mathbf{x}\parallel }_2^2+\lambda {\parallel \mathbf{x}\parallel }_p^p\},0<p<1,$$

(3)

where $\parallel \mathbf{x}{\parallel }_p^p={\displaystyle \sum }_{i=1}^n{\left|x_i\right|}^p$. Unfortunately, when $0<p<1$, problem (3) becomes a non-convex, non-smooth, and non-Lipschitz optimization problem. Thus, the $L_p$-norm optimization is always difficult to efficiently address.

1.2. The Iterative Thresholding Algorithm of $L_p$ Regularization

There are two main classes of algorithms to solve the non-convex $L_p$-norm optimization problem. One is the iteratively reweighted algorithm [20], and the other is the iterative thresholding algorithm (ITA). As one of the most effective and efficient methods, the ITA has been employed for many sparse recovery optimization problems due to its low computational complexities, including the iterative hard thresholding for $L_0$ regularization [21], the iterative soft thresholding for $L_1$ regularization [22] and the iterative $L_p$ thresholding for $L_p$ regularization [23]. $L_{1/2}$ and $L_{2/3}$ regularizations are two special and important cases, not only their solutions can be expressed in closed-forms, but also their importance on sparse modeling. Recent studies show that $L_{1/2}$ regularizer can be taken as the most representative $L_p$ regularizer [24] and $L_{2/3}$ regularization is more effective in image deconvolution problem [25]. Hence, in this paper, we focus on the $L_{1/2}$ and $L_{2/3}$ regularizations, which is described in Equation (4) and (5):

$${\widehat{\mathbf{x}}}_{l_{1/2}}=\arg \underset{\widehat{\mathbf{x}}}{\min }\{\gamma {\parallel \mathbf{y}-\mathsf{\Phi }\mathbf{x}\parallel }_2^2+\lambda {\parallel \mathbf{x}\parallel }_{1/2}^{1/2}\},$$

(4)

$${\widehat{\mathbf{x}}}_{l_{2/3}}=\arg \underset{\widehat{\mathbf{x}}}{\min }\{\gamma {\parallel \mathbf{y}-\mathsf{\Phi }\mathbf{x}\parallel }_2^2+\lambda {\parallel \mathbf{x}\parallel }_{2/3}^{2/3}\},$$

(5)

1.3. New Multiple-State Sparse Transform Based $L_1$ Regularization Algorithm

Recently, some new multiple-state sparse transform based algorithms were proposed to exploit more a priori knowledge of the signal/image by employing some new sparsifying transform strategies. A shearlet-based multiple level sparse representation algorithmic framework was proposed in [26,27] for the unconstrained $L_1$ regularization by adaptively incorporating the iteratively reweighted shrinkage step. To enhance the sparsity, the algorithm [27] is specifically adapted to the sparse structure of the multiscale coefficients based on the ADMM [12,13]. Similarly, considering the fact that the sparsity of a certain signal/image will change under different sparsifying transform dictionaries, a sparsity-induced composite regularization algorithm was proposed for the unconstrained $L_1$ regularization problem (Co-L1) [28]. The novel Co-L1 method is described as:

$${\widehat{\mathbf{x}}}_{l_{d,1}}^t=\arg \underset{\widehat{\mathbf{x}}}{\min }\left\{{\parallel \mathsf{\Phi }\mathbf{x}-\mathbf{y}\parallel }_2^2+{{\displaystyle \sum }}_{d=1}^D{\lambda }_{d,1}{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_1\right\},$$

(6)

where ${\mathsf{\Psi }}_d=[{\mathbf{\psi }}_1|{\mathbf{\psi }}_2|\cdots |{\mathbf{\psi }}_{N_d}]\in R^{N_d\times N},\left(d=1,\cdots ,D\right)$ are different dictionaries, $d$ denotes the number of ${\mathsf{\Psi }}_d$, and $N_d$ represents the dictionary size. The regularization parameters ${\lambda }_{d,1}=\frac{N_d}{\epsilon +{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_1}$ play the roles of weighting the $L_1$-norm of the sparsifying transform coefficients ${\mathsf{\Psi }}_d\mathbf{x}$. Multiplying by the weighting parameter ${\mathbf{\lambda }}_{d,1}^t={\left[\frac{N_1}{\epsilon +{\parallel {\mathsf{\Psi }}_1\mathbf{x}\parallel }_1},\cdots ,\frac{N_d}{\epsilon +{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_1}\right]}^T$, the regularizer ${{\displaystyle \sum }}_{d=1}^D{\lambda }_{d,1}{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_1^1$ is indeed a composition of multiple dictionary based regularizers. The algorithm [28] can significantly improve the image reconstruction performance over the fast iterative shrinkage-thresholding algorithm (FISTA) [29] by iteratively and adaptively weighting the composite regularizer. We define these new emerged sparsifying transforms as the “multiple-state” transform. The common property of these methods is how to exploit the prior information from the multiple-state sparsifying transform to improve the conditioning of the sparse recovery problems.

In this paper, benefiting from the improvement of the $L_{\mathrm{p}}$ regularization algorithm [24,25,30], and motivated by recent advances in the iterative reweighted algorithms, we propose a new iteratively-weighted algorithm framework for $L_p,p\in \left\{1/2,2/3\right\}$, norm minimization using the multiple-state sparsifying transform, i.e., multiple sub-dictionary sparse representation [28]. The contributions of this paper are summarized as follows. (1) Based on the multiple sub-dictionary sparse representation, we develop a new iteratively-weighted $L_p\left(p\in \left\{1/2,2/3\right\}\right)$ thresholding algorithm, which is called as SAITA-$L_p\left(p\in \left\{1/2,2/3\right\}\right)$. (2) By comparison with the existing iteratively-reweighted parameter scheme, we propose an updating regularization parameter for weighting the sub-dictionary. (3) $L_{1/2}$ regularization and $L_{2/3}$ regularization are special and important on sparse modeling, particularly on sparse recovery. However, related studies are rare, this paper also extend the applications to sparse image recovery and Magnetic resonance imaging (MRI) and get good results.

The organization of the rest of the paper is as follows: in Section 2, we propose the multiple sub-dictionary $L_p$-regularization in the SAITA-$L_p$ algorithm, including the multiple sub-dictionary sparsifying transforms and the iterative reweighted scheme for the SAITA-$L_p$ regularizer. Then, in Section 3, we develop a new $L_p$ norm minimization, iteratively thresholding algorithm SAITA-$L_p$. To confirm the proposed algorithm, we conduct simulations and applications in image restorations in Section 4. In Section 5, we further validate our proposed algorithm using three applications. Finally, conclusions are given in Section 6.

2. The Proposed Multiple Sub-Dictionary-Based ${\mathit{L}}_{\mathit{p}}$ Regularization

The multiple dictionary sparsifying transform method for the $L_1$ regularization optimization problem was proposed in [28], which extends the well-known Lasso problem into a composite regularization problem. Motivated by the composite regularization method for the $L_1$-norm, this paper employs the multiple sub-dictionary method for the $L_p$ regularization problem.

Suppose $\mathbf{x}\in {\mathrm{R}}^{N\times 1}$ is the non-sparse, raw signal. We can obtain the sparse coefficients $\mathsf{\Psi }\mathbf{x}$ through an analysis dictionary $\mathsf{\Psi }\in {\mathrm{R}}^{N_1\times N}$. The shearlet transform [31] and the wavelet transform [32] are two typical sparsifying transforms. We choose the wavelet transform as the ideal transform because of its effectiveness to compress natural images. The main steps to design the multiple sub-dictionary sparsifying transform based $L_p$ regularization method are:

First, we construct an $\left(DN\right)\times N$ over-complete sparsifying transform matrix $\mathsf{\Psi }$ by:

$$\mathsf{\Psi }=\left[\begin{array}{c}{\mathsf{\Psi }}_1\\ ⋮\\ {\mathsf{\Psi }}_d\\ ⋮\\ {\mathsf{\Psi }}_D\end{array}\right]\in R^{\left(DN\right)\times N},$$

(7)

and:

$${\mathsf{\Psi }}_d=\left[\begin{array}{c}{\mathbf{\psi }}_1\\ {\mathbf{\psi }}_2\\ ⋮\\ {\mathbf{\psi }}_{N_d}\end{array}\right]\in R^{N_d\times N},$$

(8)

where $D$ denotes the number of sub-dictionaries ${\mathsf{\Psi }}_d$ in the over-complete sparsifying transform matrix $\mathsf{\Psi }$, the $N_d\times N$ sub-dictionary matrix ${\mathsf{\Psi }}_d,\left(d=1,\cdots ,D\right)$ is acquired by a collection of row vectors ${\left\{{\mathbf{\psi }}_i\right\}}_{i=1}^{N_d}$, such as the “dbN” wavelet basis [33], and $N_d$ represents the number of column in ${\mathbf{\psi }}_i$. From Equation (8) we can see that each ${\mathsf{\Psi }}_d$ is composed of a set of rows from the $\left(DN\right)\times N$ over-complete sparsifying transform matrix $\mathsf{\Psi }$, and:

$$N_1+N_2+\cdots +N_d=DN,$$

(9)

By splitting the matrix $\mathsf{\Psi }$ into several sub-dictionaries ${\mathsf{\Psi }}_d$, we convert the sparsifying transform $\mathsf{\Psi }\mathbf{x}$ to a composition of several ${\mathsf{\Psi }}_d\mathbf{x},d=1,2,\cdots ,D$ with different sparse structures. Intuitively, we can utilize these differences to improve the recovery performance in sparse recovery problems. In this paper, we choose $N_1=N_2=\cdots =N_d=N$, so ${\mathsf{\Psi }}_d\in R^{N\times N}$, which are orthogonal matrixes.

Then, a new multiple non-convex $L_p$ regularization method is proposed:

$${\widehat{\mathbf{x}}}_{l_{d,p}}^t=\arg \underset{\widehat{\mathbf{x}}}{\min }\left\{f_d\left(\mathbf{x}\right)=\mathsf{\gamma }{\parallel \mathsf{\Phi }\mathbf{x}-\mathbf{y}\parallel }_2^2+R_{SAITA}\right\},$$

(10)

where $R_{SAITA}$ is a linearly weighted combination of multiple sub-dictionary based $L_p$ regularizers ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$:

$$R_{SAITA}={{\displaystyle \sum }}_{d=1}^D{\lambda }_{d,p}^t{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p={\lambda }_{1,p}^t{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p+{\lambda }_{2,p}^t{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p+\cdots +{\lambda }_{D,p}^t{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p,$$

(11)

the ${\lambda }_{d,p},d=1,2,\cdots D$ denotes the iterative weighted regularization parameter. Hence, the contribution of each sub-dictionary is controlled adaptively and iteratively with the weighted parameter ${\lambda }_{d,p}$, and the regularizer ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$ will vary across the sub-dictionary index $d$. Intuitively, the variation of each sub-dictionary based regularizer is best weighted by the parameter ${\lambda }_{d,p}$ to improve the sparse recovery problem.

3. The Proposed SAITA-${\mathit{L}}_{\mathit{p}}$ Algorithm

The major disadvantage of the $L_p(0<p<1)$ minimization is that it is nonconvex, making it difficult to efficiently solve. In this section, we first introduce the iteratively thresholding representation theory for the conventional $L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm according to the existing series of algorithms in [25,34]. Then, we deduce the SAITA-$L_p$ algorithm combined with the proposed weighted scheme ${\lambda }_{d,p}$.

3.1. The Relationship of the SAITA-$L_p$ and Conventional $L_p$ Methods

Considering the multiple sub-dictionary $L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ regularization problem in Equation (10), when $\gamma =1$, we have:

$${\widehat{\mathbf{x}}}_{l_{d,p}}=\arg \underset{\widehat{\mathbf{x}}}{\min }\left\{f_d\left(\mathbf{x}\right)={\parallel \mathsf{\Phi }\mathbf{x}-\mathbf{y}\parallel }_2^2+{{\displaystyle \sum }}_{d=1}^D{\lambda }_{d,p}{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p\right\},$$

(12)

where the $f_d\left(\mathbf{x}\right)$ denotes the objective functions. Correspondingly, the conventional single dictionary ${\mathsf{\Psi }}^{\prime }\in R^{N\times N}$ based analysis $L_p$-norm minimization problem is as follows:

$${\widehat{\mathbf{x}}}_{l_p}^t=\arg \underset{\widehat{\mathbf{x}}}{\min }\left\{f\left(\mathbf{x}\right)={\parallel \mathsf{\Phi }\mathbf{x}-\mathbf{y}\parallel }_2^2+{\lambda }_p{\parallel {\mathsf{\Psi }}^{\prime }\mathbf{x}\parallel }_p^p\right\},$$

(13)

The proposed SAITA-$L_p\left(p\in \left\{1/2,2/3\right\}\right)$ methods of (12) and the conventional method (13) are nearly identical, and the major difference is how to weight the contribution of the $L_p$-norm by the regularization parameter [28]. Compared with the conventional method, the SAITA method can exploit more prior knowledge of the sparse signal/image for reconstruction. Figure 1 depicts the relationship between the two methods. In the case of (A), the number of sub-dictionaries $d$ is reduced to 1, and the multiple dictionaries ${\mathsf{\Psi }}_d$. convert into a single $\mathsf{\Psi }$. Then, the proposed SAITA-$L_p$ algorithm converts to the conventional single dictionary $L_p$ method [24,25]. In case (B), with the increase of the number $d$, the conventional single dictionary $L_p$ method converts to the proposed SAITA-$L_p$ method.

3.2. The Thresholding Representation Theory for SAITA-L_p

According to the relationship between the proposed SAITA-$L_p$ method and the conventional $L_p$ method shown in Figure 1, we first consider the conventional single dictionary analysis $L_p$ problem (13). The first order optimality condition of $\mathbf{x}$ is described as:

$$\nabla f\left(\mathbf{x}\right)=2{\mathsf{\Psi }}^{\prime }{\mathsf{\Phi }}^{\mathrm{T}}\left(\mathsf{\Phi }\mathbf{x}-\mathbf{y}\right)+\lambda \nabla ({\parallel {\mathsf{\Psi }}^{\prime }\mathbf{x}\parallel }_p^p)$$

(14)

in which the operator $\nabla \left(·\right)$ denotes the gradient. Letting $\nabla f\left(\mathbf{x}\right)=0$, we have:

$${\mathsf{\Psi }}^{\prime }{\mathsf{\Phi }}^{\mathrm{T}}\left(\mathbf{y}-\mathsf{\Phi }\mathbf{x}\right)=\frac{1}{2}(\lambda \nabla ({\parallel {\mathsf{\Psi }}^{\prime }\mathbf{x}\parallel }_p^p)),$$

(15)

Multiplying by any parameter $\tau$ to control the step size and adding ${\mathsf{\Psi }}^{\prime }\mathbf{x}$ in both sides of Equation (15):

$${\mathsf{\Psi }}^{\prime }\mathbf{x}+\tau {\mathsf{\Psi }}^{\prime }{\mathsf{\Phi }}^{\mathrm{T}}\left(\mathbf{y}-\mathsf{\Phi }\mathbf{x}\right)={\mathsf{\Psi }}^{\prime }\mathbf{x}+\frac{1}{2}(\lambda \tau \nabla ({\parallel {\mathsf{\Psi }}^{\prime }\mathbf{x}\parallel }_p^p)),$$

(16)

Then, we can immediately obtain:

$$\left(\mathbf{I}+\frac{\lambda \tau }{2}\nabla ({\parallel ·\parallel }_p^p\right){\mathsf{\Psi }}^{\prime }\mathbf{x}={\mathsf{\Psi }}^{\prime }\mathbf{x}+\tau {\mathsf{\Psi }}^{\prime }{\mathsf{\Phi }}^{\mathrm{T}}\left(\mathbf{y}-\mathsf{\Phi }\mathbf{x}\right)$$

(17)

That is:

$${\mathsf{\Psi }}^{\prime }\mathbf{x}={\left(\mathbf{I}+\frac{\lambda \tau }{2}\nabla ({\parallel ·\parallel }_p^p\right)}^{-1}{\mathsf{\Psi }}^{\prime }\left(\mathbf{x}+\tau {\mathsf{\Phi }}^{\mathrm{T}}\left(\mathbf{y}-\mathsf{\Phi }\mathbf{x}\right)\right)$$

(18)

To this end, when $p\in \left\{1/2,2/3\right\}$, the resolvent operator [24,25,30] is denoted as:

$${\mathrm{H}}_{\mathsf{\lambda },p}\left(·\right)={\left(\mathbf{I}+\frac{\lambda \tau }{2}\nabla ({\parallel ·\parallel }_p^p\right)}^{-1},$$

(19)

where $\lambda$ and $\tau$ are the regularization parameter and the step tunning parameter, respectively. Then:

$${\mathsf{\Psi }}^{\prime }\mathbf{x}={\mathrm{H}}_{\mathsf{\lambda },p}\left({\mathsf{\Psi }}^{\prime }\left(\mathbf{x}+\tau \mathsf{\Phi }{\mathbf{x}}^{\mathrm{T}}\left(\mathbf{y}-\mathsf{\Phi }\mathbf{x}\right)\right)\right)$$

(20)

where $\tau >0$ (e.g., $\tau =\frac{0.99}{{\parallel \mathsf{\Phi }\parallel }_2^2}$, or $\tau =\frac{0.99}{{\parallel \mathsf{\Phi }\parallel }_2}$) controls the step size in each iteration.

According to the Equation (20), we immediately imply:

$${\mathbf{x}}^{n+1}={\left({\mathsf{\Psi }}^{\prime }\right)}^{-1}{\mathrm{H}}_{\mathsf{\lambda },p}\left(\mathbf{\theta }\left({\mathbf{x}}^n\right)\right),$$

(21)

in which:

$$\mathbf{\theta }\left({\mathbf{x}}^n\right)={\mathsf{\Psi }}^{\prime }\left({\mathbf{x}}^n+\tau {\mathsf{\Phi }}^{\mathrm{T}}\left(\mathbf{y}-\mathsf{\Phi }{\mathbf{x}}^n\right)\right),$$

(22)

where ${\mathbf{x}}^n$ represents the $n$-th iterative solution. The resolvent operator ${\mathrm{H}}_{\lambda ,p}\left(·\right)$ is defined as:

$${\mathrm{H}}_{\lambda ,p}\left(x\right)={\left(h_{\lambda ,p}\left(x_1\right),h_{\lambda ,p}\left(x_2\right),\cdots ,h_{\lambda ,p}\left(x_N\right)\right)}^T,$$

(23)

where the $h_{\lambda ,p}\left(x_i\right)$ is the nonlinear function defined by:

$$h_{\lambda ,p}\left(x_i\right)=\{\begin{array}{c}{\phi }_{{\lambda }_{d,p}}\left(x_i\right),\left|x_i\right|>T\\ 0,otherwise\end{array},$$

(24)

when $p=\frac{1}{2}$; $T=\frac{3\sqrt[3]{2}}{4}{\left({\lambda }_{d,1/2}\tau \right)}^{2/3}$ is the threshold value, and [24]:

$${\phi }_{\lambda ,1/2}\left(x_i\right)=\frac{2}{3}{x}_i(1+\cos (\frac{2\pi }{3}-\frac{2}{3}co{s}^{-1}(\frac{{\lambda }_{d,1/2}\tau }{8}{(\frac{\left|x_i\right|}{3})}^{-\frac{3}{2}}))),$$

(25)

when $p=\frac{2}{3}$; $T=\frac{2\sqrt[4]{3}}{3}{\left({\lambda }_{d,2/3}\tau \right)}^{3/4}$ is the related threshold value, and [25]:

$${\phi }_{\lambda ,2/3}\left(x_i\right)={\left(\frac{{\vartheta }_{\lambda ,2/3}\left(x_i\right)+\sqrt{\frac{2\left|\mathbf{x}\right|}{\left|{\vartheta }_{\lambda ,2/3}\left(x_i\right)\right|}-{\left|{\vartheta }_{{\lambda }_{d,2/3}}\left(x_i\right)\right|}^2}}{2}\right)}^3·\mathrm{sgn}\left(t\right),$$

(26)

where $\mathrm{sgn}(\cdot )$ denotes the sign function and

$${\vartheta }_{{\lambda }_{d,2/3}}(x_i)=\frac{2}{\sqrt{3}}{({\lambda }_{d,2/3}\tau )}^{1/4}{(\cos \mathrm{h}(\frac{1}{3}arccosh(\frac{27}{16}{({\lambda }_{d,2/3}\tau )}^{-3/2}{x}_i{}^2)))}^{1/2}$$

(27)

3.3. The Proposed SAITA-L_p Algorithm

As mentioned above, the iteratively weighted parameter plays a key role during the optimization process for the sparse recovery problem. For the proposed SAITA-L_p method, the iteratively weighted parameter ${\lambda }_{d,p}$ mainly plays two roles. One is the role of controlling the tradeoff of the fidelity and the prior knowledge between the quadratic term and the regularizer, and the other role is controlling the contribution of each regularizer. Unfortunately, it is not clear how to do this because setting an ideal parameter is not straightforward. In [28], a iteratively updating parameter was reduced by applying a Maximization-Minimization algorithm shown as:

$${\lambda }_{d,1}=\frac{N_d}{ϵ+{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_1^1}$$

(28)

where $N_d$ controls the sub-dictionary size, $ϵ>0$ is a small constant which prevent the denominator form zero. From Equation (28) we can obtain some useful information about setting a proper regularization parameter. Firstly, the contribution of denominator in Equation (28) is to counterweigh each sub-dictionary based regularizer; Secondly, the numerator $N_d$ control the size of each sub-dictionary. Inspired by the above insights, in this paper, we design the important iteratively weighted parameter ${\lambda }_{d,p}$ as:

$${\lambda }_{d,p}=\frac{N_d}{{(ϵ+{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_2^2)}^{\alpha }},$$

(29)

where $N_d$ controls the sub-dictionary size as showed in [28], $\alpha \in \left(0,2\right)$ is a small constant to tune it that is determined from the following experimental results. Then the parameter ${\lambda }_{d,p}^t$ plays the role of weighting each $L_p$-norm regularizer adaptively.

The following are the analytical justifications. (i) When signal $\mathbf{x}$ is sparser under any dictionary of ${\mathsf{\Psi }}_d$ than others, the value of each regularizer ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$ is smaller. Hence, the dictionary of ${\mathsf{\Psi }}_d$ is more appropriate, and on the other hand, a smaller regularizer ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$ will be beneficial to minimizing the objective function. Thus, the weight of the regularizer should be enhanced. (ii) When the signal $\mathbf{x}$ may not be sparse enough under another dictionary ${\mathsf{\Psi }}_d$, that is, the dictionary of ${\mathsf{\Psi }}_d$ is not ideal, the value of ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$ will be larger. The larger ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$ will not be helpful to minimizing the objective function; thus the weight of the ${\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$ should be smaller to counterweigh the regularizer.

For the main comparison, the conventional single-dictionary based $L_p,p\in \left\{1/2,2/3\right\}$-regularization method in problem (13) will be considered, and the tradeoff parameter ${\lambda }_p$ is a fixed constant, which is shown as:

$${\lambda }_p=\frac{1}{{(ϵ+{\parallel \mathsf{\Psi }\mathbf{x}\parallel }_2^2)}^{\alpha }}.$$

(30)

From Equation (30), we find that the conventional single-dictionary based parameter ${\lambda }_p$ is a constant and will not vary.

Moreover, we employ the forward-backward linear strategy to accelerate the convergence of the proposed algorithm as [14]:

$${\mu }^{n+1}=\frac{1+\sqrt{1+4{\left({\mu }^n\right)}^2}}{2},$$

(31)

and:

$${\mathbf{x}}^{n+1}={\mathbf{x}}^n+\frac{{\mu }^n-1}{{\mu }^{n+1}}\left({\mathbf{x}}^n-{\mathbf{x}}^{n-1}\right),$$

(32)

The proposed iteratively-weighted SAITA-$L_p$ algorithm can be described in Algorithm 1.

Algorithm 1: The proposed SAITA-$L_p$ algorithm.

Problem: ${\mathbf{x}}^{n+1}=\arg \min \gamma {\parallel \mathsf{\Phi }\mathbf{x}-\mathbf{y}\parallel }_2^2+{{\displaystyle \sum }}_{d=1}^D{\lambda }_{d,p}{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_p^p$;   1: Input:${\left\{{\mathsf{\Psi }}_d\right\}}_{d=1}^D$, $\mathbf{y}$, $\mathsf{\Phi }$; $L_d$; $\gamma =1$; $ϵ>0$;   2: Initialization: $n=0$; $\epsilon =0.01$; $\tau =\frac{1-\epsilon }{{\parallel \mathsf{\Phi }\parallel }^2}$; ${\lambda }_{d,1/2}^{\left(1\right)}=1$; $\alpha$.   3: for $\mathit{n}=1,2,3,\cdots$   4: Calling the conventional analysis ${\mathit{L}}_{\mathit{p}}$ algorithm in (13):    While not converged do     Step 1: Compute $\mathbf{\theta }\left({\mathbf{x}}^n\right)={\mathsf{\Psi }}^{\prime }({\mathbf{x}}^n+{\mathsf{\Phi }}^{\mathrm{T}}(\mathbf{y}-\mathsf{\Phi }{\mathbf{x}}^n))$ in Equation (22);     Step 2: Compute ${\mathbf{x}}^{n+1}={\left({\mathsf{\Psi }}^{\prime }\right)}^{-1}{\mathrm{H}}_{\mathsf{\lambda },p}\left(\mathbf{\theta }\left({\mathbf{x}}^n\right)\right)$ in Equation (21);     Step 3: Update the value of $\mu$ using ${\mu }^{n+1}=\frac{1+\sqrt{1+4{\left({\mu }^n\right)}^2}}{2}$ in Equation (31);     Step 4: Update the solution ${\mathbf{x}}^{n+1}={\mathbf{x}}^n+\frac{{\mu }^n-1}{{\mu }^{n+1}}({\mathbf{x}}^n-{\mathbf{x}}^{n-1})$ in Equation (32);    End   5: Updating: ${\lambda }_{d,p}^{\left(n+1\right)}\leftarrow \frac{N_d}{{(ϵ+{\parallel {\mathsf{\Psi }}_d{\mathbf{x}}^{\left(n\right)}\parallel }_2^2)}^{\alpha }}$ in Equation (29);   6: end   7: Output ${\mathbf{x}}^t$

4. Performance Analysis and Discussion

We first conduct some experiments to determine the value of $\alpha$ and verify the performance of the proposed ${\lambda }_{d,p}^t$, then we evaluate the superiority of the proposed SAITA algorithm compared with the conventional single dictionary analysis $L_p$ iterative thresholding algorithms [24,25] and the Co-L1 [28]. All the experiments in this paper were conducted on a personal computer (2.21 GHz, 16 GB RAM) in a MATLAB (R2014a) platform.

Assuming the clean image $\mathbf{x}$, we construct a measurement matrix $\mathsf{\Phi }$ using the “Spread Spectrum” operator [35], so the measurement image shows: $\mathbf{y}=\mathsf{\Phi }\mathbf{x}+\mathbf{n}$, where n denotes the additive noise. Since the wavelet is known to compress natural raw images very efficiently, we choose the wavelet transform as the sparsifying transform operator. Then, we construct the sparsifying transform matrix $\mathsf{\Psi }\in {\mathrm{R}}^{8N\times N}$ by concatenating the undecimated ‘db1’ and ‘db2’ wavelet transform with 2-levels of decomposition. Thus, we can obtain the sub-dictionaries: ${\mathsf{\Psi }}_d\in R^{N\times N},d=1,2,\cdots 8$.

The SNR measurement is adopted to measure the noise level, which is defined as $mSNR=\frac{{\parallel \mathbf{y}\parallel }_2^2}{M{\sigma }^2}$, where $M$ and ${\sigma }^2$ denote the number of $\mathbf{y}$ and the variance of the white Gaussian noise, respectively. The higher the value of mSNR, the weaker of the noise level is. We evaluate the performance by the popular recovery SNR: $RSNR=-20\log (\frac{{\parallel \mathbf{x}-\widehat{\mathbf{x}}\parallel }_2}{{\parallel \mathbf{x}\parallel }_2})$, where $\widehat{\mathbf{x}}$ denotes the estimated sparse image. The higher the value of $RSNR$, the better the performance. We conduct the experiments by utilizing the well-known figure of “Cameraman” with $\mathrm{mSNR}=\mathrm{M}/\mathrm{N}=40\mathrm{dB}$, which is shown in Figure 2a. To reduce the computation time, we choose only a part of the figure, shown in Figure 2b.

4.1. The Value Range of $\alpha$ in ${\lambda }_{d,p}$

In Section 4.1, we first determine the value range for $\alpha$ in ${\lambda }_{d,p}$ by evaluating the performances with different values of $\alpha$. The results are shown in Figure 3 and Figure 4. From the results, we can find that when $\mathsf{\alpha }\in \left(0,p\right)$, the proposed algorithms perform well and enjoy strong robustness. With the increase of $\mathsf{\alpha }$, the performances deteriorate rapidly. Therefore, we estimate that the value of $\mathsf{\alpha }$ should be $\left[0,p\right]$ to obtain the adaptive weighting. We specially set:

$${\lambda }_{d,p}^t=\frac{N_d}{{(ϵ+{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_2^2)}^{\frac{1-p}{2}}},$$

(33)

4.2. The Recovery SNR Performances Versus Sampling Ratio

In Section 4.2, we evaluate the robustness of the proposed algorithm by considering the RSNR versus the sampling ratio. Specifically, we set three mSNR levels to $25\mathrm{dB},30\mathrm{dB}\mathrm{and}35\mathrm{dB}$. Figure 5 depicts the RSNR versus the sampling ratio. Based on the results, the proposed SAITA-$L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm performs better than the conventional single dictionary based algorithm, and the robustness of the proposed algorithm is good.

4.3. The Recovery SNR Performance Versus Measurement SNR

For our third experiment, we investigate the influence of different noise levels on the proposed algorithm and compare the $L_p$ algorithm and Co-L1 [28]. Similarly, we consider three sampling ratio levels of $M/N\in \left\{0.15,0.20,0.25\right\}$. We evaluate the performance by the RSNR versus the lower mSNR (20 dB~40 dB) of the image, and the results are presented in Figure 6.

From the results, we can find that the proposed SAITA-$L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm can obtain a higher recovery SNR and has better robustness and fidelity than the Co-L1. The robustness and fidelity of the corresponding $L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm will deteriorate with the increase of the signal measurement SNR.

4.4. The Relative Error Performances Versus the Number of Iterations

We study the convergence and the reconstruction error by the relative error performances versus the number of iterations. Choosing the relative error as the second quality measurement, the formula is given:

$$RelativeError=\frac{{\parallel \mathbf{x}-\widehat{x}\parallel }_2}{{\parallel \mathbf{x}\parallel }_2}$$

(34)

Considering the proposed SAITA-$L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ and the corresponding $L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm from the result shown in Figure 7, when the sampling ratio is 0.2 (shown in (a)), the relative errors of the proposed SAITA-$L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm are significant smaller, and converge faster than the corresponding $L_p,\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm. When the sampling ratio is 0.5 (shown in (b)), though the final relative errors are close, the convergence speed of the proposed algorithm is higher (the number of iterations are approximately 15 and 7, respectively). While compared to Co-L1 [28], our proposed SAITA-$L_p$ algorithm can obtain a markedly lower relative error when the sampling ratio is $M/N\in \left\{0.2,0.5\right\}$. In addition, one can observe that the relative error is slightly smaller than for $p=2/3$, while the convergence rate is faster than the $p=1/2$.

5. Practical Experiments

The proposed algorithms can be applied in many practical applications [36,37,38,39,40,41,42]. In this section, we conduct some typical applications about sparse image recovery and medical imaging to extend the applications of $L_{1/2}$ and $L_{2/3}$ regularizations and illustrate the excellent robustness and adaptation of the proposed SAITA-$L_p$,$\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm. The conventional single dictionary analysis $L_p$ iterative thresholding algorithms [24,25] and the Co-L1 [28] as the standard algorithms for comparison.

5.1. Application 1: Image Sparse Restoration

In the first application, the proposed SAITA-$L_p$ algorithm is applied to restoring the “Cameraman” image shown in Figure 2 versus sampling ratio $\mathrm{M}/\mathrm{N}$. We use the reduced $N=96\times 104$ cameraman image as the objective image. Figure 8a,c show the recovery results using the conventional single dictionary algorithm of $L_1$ and $L_p,\left(p\in \left\{1/2,2/3\right\}\right)$, respectively. Figure 8b,d show the recovery images using the corresponding multiple sub-dictionary algorithm of $L_1$ and $L_p\left(p\in \left\{1/2,2/3\right\}\right)$, respectively. The experiments show that all the algorithms can recover the images, and the proposed multiple sub-dictionary algorithms significantly outperform the conventional single-dictionary algorithms. In Figure 9, we depict the RSNR of four algorithms vs. different sampling ratios. When $p=1/2$ and $p=2/3$, it can be observed that the proposed SAITA algorithm can obtain a lager RSNR compared with the conventional $L_p$ algorithms. There is no obvious improvement between the two cases of $p=\frac{1}{2}$ and $p=\frac{2}{3}$, and the SAITA-$L_{1/2}$ algorithm outperforms the SAITA-$L_{2/3}$ algorithm with a weak advantage.

5.2. Application 2: Medical Imaging

In the application 2, we extend the applications of our proposed algorithm to solve typical medical construction problems. We first consider the well-known Shepp-Logan phantom, and then we construct a high-quality dMRI cardiac cine [8,28].

5.2.1. Test 1 for the Shepp-Logan Model

In the test 1, we first consider the well-known Shepp-Logan phantom of $N=96\times 96$ with an $mSNR=40\mathrm{dB}$. We construct the compressed noisy measurement signal $\mathbf{y}$ by utilizing the “Spread Spectrum” operator as the measurement matrix $\mathsf{\Phi }$ [35], and we conduct a sparsifying transform with the constructed operator $\mathsf{\Psi }\in R^{\left(4N\right)\times N}$ (‘db3’, $N=1$).

From the experimental results shown in Figure 10, we find that the proposed SAITA-${\mathit{L}}_{\mathit{p}}$ algorithm can recover the images perfectly, as shown in Figure 10b,d, while the conventional single dictionary algorithms failed to recover the image, which is shown in Figure 10a,c. The proposed SAITA-${\mathit{L}}_{\mathit{p}}$ algorithm of $\mathit{p}=2/3$ can obtain the best effect compared with the other algorithms, and the next best is the SAITA-${\mathit{L}}_{\mathit{p}}$ algorithm of $\mathit{p}=1/2$. In Figure 11, we depict the RSNR of the respective algorithms versus different sampling ratios $\mathit{M}/\mathit{N}$. When $\mathit{p}=1/2$ and $\mathit{p}=2/3$, it can be observed that the proposed SAITA-${\mathit{L}}_{\mathit{p}}$ algorithms can obtain a larger RSNR than the ${\mathit{L}}_{\mathit{p}}$ algorithms with $\mathit{M}/\mathit{N}\in \left(0.1,0.2\right)$ significantly.

5.2.2. Test 2 for Real-World Data (2D MRI)

MRI is a typical medical inverse problem that can be solved well by CS [8]. In this experiment, we apply the proposed algorithm to real-world medical data. We investigate a simplified “dynamic MRI” problem [8] and use the high-quality MRI cardiac cine as the ground truth and select a spatio-temporal slice of $144\times 48$ [28]. We construct the sparse matrix $\mathsf{\Psi }\in R^{3N\times N}$ with a vertical concatenation of ‘db1’ and ‘db2’ orthogonal discrete wavelet bases with two levels of decomposition (‘db1’, ‘db2’, and $N=2$). Figure 12 presents the constructed MRI images using the SAITA-$L_p\left(p\in \left\{1/2,2/3\right\}\right)$ algorithm and $L_p\left(p\in \left\{1/2,2/3\right\}\right)$ algorithms. From the experiment results, we can see that the proposed SAITA algorithm can reconstruct the images perfectly, as shown in Figure 12b,d, while the conventional algorithms failed to restore the image in Figure 12a,c. The proposed multiple sub-dictionaries algorithm of $p=2/3$ obtained the best effect and the corresponding recovery SNR is 23.1872 dB. Next is the proposed algorithm of $p=1/2$ with recovery SNRs of 21.0189 dB. In Figure 13, we depict the RSNR of four algorithms versus different sampling ratios $M/N$. When $p=1/2$ and $p=2/3$, it can be observed that the proposed SAITA-$L_p$ algorithms can obtain a lager RSNR compared with the conventional single dictionary $L_p$ algorithms. The results also show that the algorithms of $p=2/3$ outperform the methods of $p=1/2$. That is to say, the $L_{2/3}$ regularization can exploit more prior knowledge than $L_{1/2}$ regularization in MRI.

6. Conclusions

In this paper, we propose a novel adaptive iteratively weighted thresholding algorithm (SAITA-$L_p$) based on the conventional $L_{1/2}$ and $L_{2/3}$ thresholding algorithm by incorporating the multiple sub-dictionary sparse representation strategy. We make the following conclusions from the above experiments:

(1)

Using the proposed multiple sub-dictionary sparsifying transforms strategy, we construct a multiple sub-dictionary based $L_p$ regularization method to exploit more priori knowledge of images for the sparse image recovery problem. By multiplying by the proposed adaptive weighting parameter ${\lambda }_{d,p}^t=\frac{N_d}{{\left(ϵ+{\parallel {\mathsf{\Psi }}_d\mathbf{x}\parallel }_2^2\right)}^{\frac{1-p}{2}}}$, we can gain more control of weighting the contribution of each sub-dictionary based regularizer. Experiments show that the proposed algorithms appear to perform better than the conventional single-dictionary algorithms, especially when the sampling rate is very low (e.g., 0.1~0.3);

(2)

Compared with the $L_1$-norm regularization based work, the nonconvex $L_p(0<p<1)$-norm penalty can more closely approximate the $L_0$-norm minimization over the $L_1$-norm, which gives a sparser solution and needs fewer measurement data.

(3)

In our experiments, we find that the recovery performances between the $L_p\left(p=1/2\right)$ and $L_p\left(p=2/3\right)$ are close, even when the corresponding $p=2/3$ algorithm can obtain a better performance over the $p=1/2$. Hence, a proper $p$ need to be selected in practical applications.

(4)

Moreover, the proposed SAITA-$L_p$ method also indicates that it is feasible to improve the recovery performance by exploiting the signal inner sparse structure and designing a proper sparse representation dictionary. Thus, it is beneficial to exploit the signal sparse structure with a dictionary learning method, which will be the subject of future work.

(5)

The proposed SAITA-$L_p$ algorithm can be extended to other non-convex penalties include smoothly clipped absolute deviation (SCAD) and minimax concave penalty (MCP).