Sparse Coding Algorithm with Negentropy and Weighted ℓ₁-Norm for Signal Reconstruction

Abstract

Compressive sensing theory has attracted widespread attention in recent years and sparse signal reconstruction has been widely used in signal processing and communication. This paper addresses the problem of sparse signal recovery especially with non-Gaussian noise. The main contribution of this paper is the proposal of an algorithm where the negentropy and reweighted schemes represent the core of an approach to the solution of the problem. The signal reconstruction problem is formalized as a constrained minimization problem, where the objective function is the sum of a measurement of error statistical characteristic term, the negentropy, and a sparse regularization term, ${\ell }_p$-norm, for 0 < p < 1. The ${\ell }_p$-norm, however, leads to a non-convex optimization problem which is difficult to solve efficiently. Herein we treat the ${\ell }_p$-norm as a serious of weighted ${\ell }_1$-norms so that the sub-problems become convex. We propose an optimized algorithm that combines forward-backward splitting. The algorithm is fast and succeeds in exactly recovering sparse signals with Gaussian and non-Gaussian noise. Several numerical experiments and comparisons demonstrate the superiority of the proposed algorithm.

1. Introduction

Sparse signal reconstruction, or compressed sensing, is an emerging field in signal processing and communication [1,2,3,4,5,6]. The problem of recovering a sparse signal from a very low number of linear measurements arises in many real application fields, ranging from error correction and lost data recovery, to image acquisition and reconstruction. In general, an $N$-dimensional sparse signal can be described by $M<N$ significant coefficients in an appropriate transform domain. In some cases, a signal which is non-sparse in time domain can be classified as a sparse one, as it shows sparsity in spatial domain or some appropriate transform domain, such as the frequency domain and Gabor transformed domain.

In this paper, the signals will be treated as real-valued functions having domains that are either continuous or discrete, and either infinite or finite. We will typically be concerned with normed vector spaces. In the case of a discrete, finite domain, the signals can be viewed as vectors in an $n$-dimensional Euclidean space, denoted by ${ℝ}^n$. According to the compressed sensing theory, sparse signal reconstruction problem can be formalized as a constrained minimization problem, where the objective function defines the sparsity. The basic mathematical model is:

$$x=As+e$$

(1)

where $A\in {ℝ}^{M\times N}$ is a known measurement matrix with $M<N$ and needs to satisfy the RIP condition. Any $M$ columns vectors are linearly independent [7]. $x\in {ℝ}^{M\times 1}$ is an available measurement vector. $e\in {ℝ}^{M\times 1}$ is an unknown noise vector. The sparse reconstruction problem can be cast as: given the $M\times N$ measurement matrix $A$, find the vector $s\in {ℝ}^{N\times 1}$, whose nonzero components are only , with $K\ll N$ (such a vector will be called $K$-sparse vector), from the measurements $x$, by solving:

$$\underset{s\in {ℝ}^{N\times 1}}{\min }F(s)\mathrm{subject}\mathrm{to}x=As+e$$

(2)

The most natural choice for $F(s)$ is given by $F(s)={\Vert s\Vert }_0$, where ${\Vert s\Vert }_0$ denotes the ${\ell }_0$-norm, which counts the number of non-zero components in $s$. With $F(s)={\Vert s\Vert }_0$, however, the problem in (2) becomes a combinatorial optimization and is proven to be non-deterministic polynomial (NP)-hard [8]. Recent results have shown that the use of different sparsity inducing functions allows us to exactly and efficiently recover a sparse signal from a lower number of measurements [9,10,11]. For instance, ${\ell }_p$-norm is introduced as a relaxation of the ${\ell }_0$-norm, and the problem can be formulated by applying constraints on the signal model and introducing a cost function:

$$\underset{s\in {ℝ}^{N\times 1}}{\min }{\Vert x-As\Vert }_2+\lambda {\Vert s\Vert }_p$$

(3)

where ${\Vert s\Vert }_p={\left({\displaystyle \sum _{i=1}^M{\left|s_i\right|}^p}\right)}^{1/p}$ is the ${\ell }_p$-norm of $s$, with $0<p<1$. $\lambda >0$ is the regularizing parameter which controls the sparsity.

Many algorithms have been developed to solve the problem in (3) in the literature [12,13,14,15,16,17,18,19,20], where the mean square error (MSE) [21] criterion based on second-order statistics has been employed for these algorithms, which show their optimality when $e$ is Gaussian noise. In practical applications, however the transmitted signals are distorted by not only Gaussian noise, but also other kinds of noise, such as burst noise and high noise. Burst noise is a type of discrete noise and consists of sudden interruptions. High noise has high energy, frequency or power. These noises have non-Gaussian characteristics. In such cases, the MSE criterion becomes less robust.

In this work, we focus on the problem of sparse signal reconstruction especially with non-Gaussian noise. This problem is formalized as a constrained minimization problem and verified by simulation. We propose a sparse coding algorithm in which the sparse signal is recovered by applying the negentropy [22] as the error measurement and ${\ell }_p$-norm as the sparsity regularization. The ${\ell }_p$-norm as sparsity constraint is not very commonly used due to its non-convexity. To solve the corresponding non-convex minimization problems, we treat ${\ell }_p$-norm as a serious of weighted ${\ell }_1$-norm and convert it to a series of convex optimizations. The negentropy, rather than the MSE, is used because of two folds of reasons. First, in the square error (SE) case, the minimization is on the cost function of the form MSE + ${\ell }_p$-norm like Equation (3). Since SE is required to be as small as possible while the ${\ell }_1$-norm is lower bounded by a finite value, the optimization cannot reach a very small SE value, so that leads a biased solution. In our case, the cost is negentropy + (weighted) ${\ell }_1$-norm, the negentropy is required to be maximized and have finite value in the optimal state, so that an unbiased solution can be obtained. Second, negentropy can tolerate bigger, non-Gaussian noise on the zero-valued components so that the estimation for non-zero-valued components becomes more accurate. For efficient optimization, we propose an algorithm with two main steps: first we use the gradient-based maximization only to the negentropy; then we find a sparser solution within the neighborhood of what has been obtained in the gradient-based maximization. Such a strategy was termed in the literature as the forward-backward splitting (FOBOS) algorithm [23].

The proposed algorithm is distinguished from other related works in that we present a novel objective function for sparse signal recovery. The sparse signals can be estimated by applying the negentropy as the error measurement and weighted ${\ell }_1$-norm as the sparsity regularization. An effective algorithm is provided, which has improved accuracy and convergence rate.

This paper is organized as follows: in Section 2 we recall the least absolute shrinkage and selection operator (LASSO) algorithm which is developed to solve the problem in Equation (3) and we propose a new one. The formulation of our algorithm is presented and the details about the targeted minimization problem are described. Then we propose to use negentropy and weighted ${\ell }_1$-norm as the core of our algorithm. Numerical results which show the effectiveness of the proposed algorithm are presented in Section 3. Section 4 concludes this paper.

2. Materials and Methods

2.1. Least Absolute Shrinkage and Selection Operator

Although the ${\ell }_p$-norm, with $0<p<1$, seem to be the natural choice for sparsity regularization, the fact that it is not convex makes the optimization process hard. The ${\ell }_1$-norm is the one that is “closest” to it yet ${\ell }_1$-norm retains the computationally attractive property of convexity and has been used for such problem in Equation (3) for a long time. Here, the LASSO, which is the hot research topic in statistical society, is discussed to solve the problem. We intend to recover the original signal $s$ from (1). $\tilde{s}$ is the sparse reconstruction of the original signal by optimizing the following function as the LASSO [24,25]:

$$\tilde{s}=\underset{s\in {ℝ}^{N\times 1}}{\arg \min }\{\frac{1}{2}{\Vert x-\mathrm{As}\Vert }_2^2+\lambda {\Vert s\Vert }_1\}$$

(4)

One cause is the loss function based on MSE criterion, which makes it necessary to reduce the rest error as much as possible. The other clause is a regularization function which uses the ${\ell }_1$-norm to obtain a sparse solution. This algorithm is likely to be sparse to coefficient vector obtained as the estimation results by use as constraints of the ${\ell }_1$-norm. Therefore, only the main part of the signal can be approximated by a linear combination of the basis set, $\tilde{s}$ can be reconstructed by sparse coding and the noise can be removed. However, the MSE criterion has little robustness when the signal reconstruction process involves non-Gaussian noise. We want to improve Equation (4) in order to tolerate bigger, non-Gaussian noise and obtain the sparser solution.

2.2. Proposed Minimization Formulation

In this section we describe a novel and effective proposal for the constrained minimization problem in Equation (3). With complex noise and more sparsity, a challenging optimization problem has to be solved. The proposed algorithm is based on the signal model in Equation (1) presented in Figure 1, where $s={\left[s(0),s(1),\cdots ,s(N-1)\right]}^T\in {ℝ}^{N\times 1}$ is sparse enough so that the non-zero components, $K$, in $s$ is less than $M$. For the model, the measurement matrix $A$ and measurement vector $x$ are known, while the vector $s$ is to be estimated and reconstructed, under the assumption that $s$ is $K$-sparse. $\tilde{s}={\left[\tilde{s}(0),\tilde{s}(1),\cdots ,\tilde{s}(N-1)\right]}^T\in {ℝ}^{N\times 1}$ denotes the estimated sparse vector.

For quantizing the fidelity of the estimation, one can measure a kind of distance between $x$ and $\tilde{x}=A\tilde{s}$. In the most situations, the root of mean square error is used usually for such purpose. If the error is Gaussian distributed, MSE is optimal, since the first and second moments incorporate all statistical characteristics. However, if the error is not Gaussian distributed, such as audio signal, image, communication signal, MSE will not be optimal. Considering the application to such cases, in this paper, we use the non-Gaussianity of error. As a sparse measure we use ${\ell }_p$-norm. Therefore, the sparse vector $s$ can be estimated by minimizing the following objective function and formulate the optimization problem as follows

$$\begin{array}{l}\tilde{s}=\underset{s\in {ℝ}^{N\times 1}}{\arg \min }J(s)\\ =\underset{s\in {ℝ}^{N\times 1}}{\arg \min }\{-J_n(x-As)+\lambda {\Vert s\Vert }_p^p\}\\ =\underset{s\in {ℝ}^{N\times 1}}{\arg \min }\{-J_n(e)+\lambda {\Vert s\Vert }_p^p\}\end{array}$$

(5)

where $J_n(·)$ denotes the negentropy, ${\Vert ·\Vert }_p^{}$ is ${\ell }_p$-norm and $\lambda$ is the regularizing parameter. Note that, algorithms based on MSE criterion have also proven to be weaker in the performances of convergence and accuracy.

Here, we apply the negentropy and ${\ell }_p$-norm to construct the objective function, where the measurement of $e$ is negentropy and the sparsity constraint is for sparse-promoting. The small error on the zero components can be tolerated in exchange for high accuracy of the non-zero components in order to retain all significantly large components, so $\tilde{s}$ can be more exact and the performance of sparse signal reconstruction can be observably improved.

2.3 Algorithm

2.3.1. Negentropy Maximization

In the information theory, Gaussian variables have maximum entropy in all random variables which have the same variance. Therefore, one can use entropy to measure non-Gaussian noise. A modified form of entropy is negentropy which is used as the error measurement in the proposed algorithm. Negentropy can tolerate non-Gaussian noise on the signal so that the estimation of sparse signal becomes more accurate. The negentropy is defined as $J_n(e)=H(e_{gauss})-H(e)$, where $H(·)$ is the differential entropy and $H(e)=-{\displaystyle \int p_e(\xi )}\log p_e(\xi )d\xi$, $e_{gauss}$ is the Gaussian random variable which has the same variance with $e$. When $e$ is Gaussian, $J_n(e)=0$. $J_n(e)$ gets larger while $e$ becomes more non-Gaussian. $J_n(e)$ reflects the non-Gaussian level of $e$. When we apply the negentropy to the sparse signal reconstruction, the probability density distribution of variables is considered to be known. It is unpractical for the noise $e$. Here, approximately, the negentropy is given by ${[E\{G(e)\}-E\{G(e_{gauss})\}]}^2$. Assume $e$ is super-Gaussian, which is non-Gaussian and the kurtosis value is greater than zero. This is true in most realistic situations, we can just use:

$$J_n(e)=E\{G(e)\}-E\{G(e_{gauss})\}$$

(6)

where $E\{·\}$ denotes the statistical expectation. $G(·)$ is the nonlinear function, such as $G(e)=e^3$, $G(e)=e·\exp (-e^2/2)$ and $G(e)=\tanh (c·e)$ [22]. In this paper, as an instance, the function is as follows:

$$G(e)=\frac{1}{c}\log (\cosh (ce))$$

(7)

where $c$ is constant. This nonlinear function is smooth and differentiable. When $c\gg 1$, $G(e)$ is approximately equal to $\left|e\right|$. Then, the gradient of $J_n(e)$ can be calculated in order to maximize the negentropy so that $-J_n(e)$ can be minimized. It is an effective method to solve the optimization problem. The gradient of $J_n(e)$ is calculated as:

$$\frac{\partial J_n(e)}{\partial s}=-A^{T}g(e)$$

(8)

where $g(e)$ is the derivative of $G(e)$ and $g(e)=\tanh (ce)$.

The sparse signal $s$ can be iteratively updated as follows:

$$s^{k+\frac{1}{2}}\leftarrow s^k+{\mu }_n\frac{\partial J_n(e)}{\partial s^k}$$

(9)

where $k$ represents the number of iterations and ${\mu }_n\ge 0$ is the step size.

2.3.2. Weighted ${\ell }_1$-norm and FOBOS

Considering the ${\ell }_p$ regulation, we find a sparse solution in the neighborhood of $s^{k+\frac{1}{2}}$. That is we solve the problem as follows:

$$s^{k+1}=\arg \underset{s\in R^{N\times 1}}{\min }\{\frac{1}{2}{\Vert s-s^{k+\frac{1}{2}}\Vert }_2^2+\lambda {\Vert s\Vert }_p^p\}$$

(10)

where $s^{k+\frac{1}{2}}$ is the updated vector from (9). Based on the understanding that the sparsity constraint ${\Vert s\Vert }_p^p$ resulting in sparser results when $p$ is closer to zero, we tend to choose value of $p$ between 0 and 1. However, this choice makes the optimization problem a concave one. To reform a convex sparsity constraint, we propose to approximate the ${\ell }_p$-norm sparsity constraint with a weighted ${\ell }_1$-norm. Thus, Equation (10) is rewritten as follows:

$$\begin{array}{lll}{s}^{k+1}& =& \underset{s\in R^{N\times 1}}{\arg \min }\{\frac{1}{2}{\Vert s-s^{k+\frac{1}{2}}\Vert }_2^2+\lambda w{\Vert s\Vert }_1\}\\ & =& \underset{}{\arg {\displaystyle \sum _i\underset{s_i\in R}{\min }\{\frac{1}{2}{\Vert s_i-s_i{}^{k+\frac{1}{2}}\Vert }_2^2+\lambda w_i\left|s_i\right|\}}}\end{array}$$

(11)

where $w\in R^{N\times 1}$ is weight. Since ${\Vert s\Vert }_p^p={{\displaystyle {\sum }_{i=1}^N\left|s_i\right|}}^p\approx {\displaystyle {\sum }_{i=1}^N{(\left|s_i\right|+\delta )}^{p-1}\left|s_i\right|}$ and $\delta$ is close to 0, $w_i{}^{k+1}={(\left|s_i^k\right|+\delta )}^{p-1}$. Then, we update each component of $s^{k+\frac{1}{2}}$ by a FOBOS-like algorithm as:

$$s_i{}^{k+1}=\mathrm{sign}(s_i{}^{k+\frac{1}{2}})\max \{0,(\left|s_i{}^{k+\frac{1}{2}}\right|-\lambda w_i{}^{k+\frac{1}{2}})\}$$

(12)

where $s_i{}^{k+\frac{1}{2}}$ denotes the $i\mathrm{th}$ component of $s^{k+\frac{1}{2}}$. When $\left|s_i{}^{k+\frac{1}{2}}\right|\le \lambda w_i^{k+\frac{1}{2}}$, $s_i{}^{k+1}=0$.

To summarize, the method proposed in this paper is presented in Algorithm 1.

Algorithm 1: Proposed algorithm for sparse signal reconstruction with negentropy and weighted ${\ell }_1$-norm

Task: Estimate the sparse signal $s$ by minimizing $J(s)=-J_n(e)+\lambda {\Vert s\Vert }_p^p$  Initialization: input signal matrix $A$, system noise $e$ and proper $c,\lambda ,{\mu }_n,p$ and $\delta$. Initialize $s^0$ and $w^0$.  Main iteration: Calculate gradient of $J_n(e)$ and update $s$, $s^{k+\frac{1}{2}}\leftarrow s^k+{\mu }_n\frac{\partial J_n(e)}{\partial s^k}$ Update each component of $s^{k+1}$ by weighted ${\ell }_1$ regulation in a FOBOS style, $s_i{}^{k+1}=\mathrm{sign}(s_i{}^{k+\frac{1}{2}})\max \{0,(\left|s_i{}^{k+\frac{1}{2}}\right|-\lambda w_i{}^{k+\frac{1}{2}})\}$ If the stop condition is satisfied, algorithm ends, otherwise goes to step 1.  Output: $\tilde{s}$

The stop condition can be the desired accuracy, convergence or a maximal iteration number.

3. Results and Discussions

In this section, we analyze the performance of the proposed algorithm for sparse signal reconstruction with negentropy and weighted ${\ell }_1$-norm described in Section 2. In order to perform the numerical analysis, we present our experiment results to show whether the algorithm can recover the true sparse signal $s_{orig}$ or not. The original sparse signal $s_{orig}$ was sized as a 100 × 1 vector and generated by drawing value randomly from a normal distribution $N(0,1)$. In each sparse vector, there were several non-zero values, which were also picked and located randomly. The measurement matrix $A$ was sized as 40 × 100. Correspondingly, $x_{orig}$ had 40 sample size and was generated by $A$ and $s_{orig}$ using the equation $x_{orig}=A{s}_{orig}+e$. The noise vectors $e$ were added based on Gaussian and non-Gaussian random entries with various signal to noise ratios (SNR). The other parameters were set as below, $c=7,\lambda =1.57\times {10}^{-2},{\mu }_n=6.1\times {10}^{-3},p=0.9,\delta ={10}^{-7}$. In order to evaluate the performance of the proposed algorithm, we use the normalized ${\ell }_2$-error as a criteria to measure reconstruction accuracy for sparse signals, where the normalized ${\ell }_2$-error is defined as ${\Vert \tilde{s}-s_{org}\Vert }_2^2/{\Vert s\Vert }_2^2$.

3.1. The Reconstructed Signal Comparison between MSE Criterion and Proposed Algorithms

As discussed in Section 2.1, in the algorithm based on the MSE criterion, the objective function is:

$$J(s)=E\{{\Vert x-\mathrm{As}\Vert }_2^2\}+\lambda {\Vert s\Vert }_1=E\{{\Vert e\Vert }_2^2\}+\lambda {\Vert s\Vert }_1$$

(13)

The measurement of $e$ in Equation (13) is a Frobenius norm which is the common model of error [26]. As described in Section 2, we apply the negentropy and weighted ${\ell }_1$-norm to form the objective function, where the measurement of $e$ is negentropy and the sparsity constraint can result in sparser results. Here, we compare the proposed algorithm with the MSE algorithm from the recovery and original signals images.

Figure 2 shows the reconstructed and original signal comparison between MSE and proposed algorithm. From Figure 2, it is clearly that MSE algorithm does not work well since it does not exploit the sparsity availably and is based on the assumption that noise has Gaussian characteristic. The zero components of $s_{orig}$ is estimated exactly with the proposed algorithm, but is partly not zero with MSE algorithm. Furthermore, the non-zero components of $s_{orig}$ is recovered more precisely with the proposed algorithm, where $\tilde{s}$ has better sparse characteristics.

3.2. The Accuracy Performance Comparison of the Algorithms

For a more complete description on how the performance is affected by noise, the normalized ${\ell }_2$-error of $\tilde{s}$ is plotted under different SNR in Figure 3. The results demonstrate the significant reduction of normalized ${\ell }_2$-error as SNR is increased. More specifically, the proposed algorithm has better performance that is at 10⁻² order, compared with the MSE algorithm that is at 10⁻¹ order. Therefore, the proposed algorithm has higher accuracy as expected and is more suitable for sparse signal reconstruction with complex noise.

3.3. The Convergence Performance Comparison of the Algorithms

To illustrate the convergence of the algorithm, we present the performance of normalized ${\ell }_2$-error with the iterations in Figure 4 in the same non-Gaussian noise circumstance. As the number of iterations increases, the relative error of reconstructed sparse signal decreases, and becomes stable at a certain error value where the convergence is reached. Figure 4 shows that the number of iterations for convergence of the proposed algorithm is smaller than that of MSE. It can be seen that the proposed algorithm converged faster. Novel loss function and FOBOS-like algorithm lead to the better performance in convergence rate. Besides, the stable normalized ${\ell }_2$-error is lower compared to that of MSE seen in Figure 4. With respect to the performance of reconstructing sparse signals, the recovery ratios are shown in Figure 5. The recovery ratio of the proposed algorithm can reach 100% with the smaller number of iterations than that of MSE. Therefore, the proposed algorithm has significant advantages in computational complexity and specially in recovery ratio.

Figure 6 presents the normalized ${\ell }_2$-error with the sparsity level which is the number of non-zero components in all components. We consider a randomly generated $K$-sparse signal $s$ of length $N=100$ and $K=5,10,15,20$ non-zero components. It is apparent that for different sparse signals, the normalized ${\ell }_2$-error is reduced through decreasing the number of non-zero components. Therefore, when implementing the proposed strategy in practice, it is important to consider the sparsity of signals.

4. Conclusions

We have presented an effective algorithm for reconstructing sparse signals, which is based on a convex optimization problem upon the proposed objective function. The objective function includes the negentropy as the fitting error measurement and ${\ell }_p$-norm, which is treated as the weighted ${\ell }_1$-norm, as the sparse-promotion. Furthermore, the proposed algorithm includes two main steps in each iteration: (1) gradient based maximization to the negentroy; (2) a soft thresholding to the result of (1), for sparsity promotion. Experiments show that the proposed algorithm has improved accuracy and convergence rate, especially when the noise is non-Gaussian in the information transmission. In our future research, the algorithm will be optimized for complexity and the real-world applications in signal processing and communication fields will also be considered.