A Lagrange Programming Neural Network Approach with an ℓ₀-Norm Sparsity Measurement for Sparse Recovery and Its Circuit Realization

Abstract

Many analog neural network approaches for sparse recovery were based on using ${\ell }_1$-norm as the surrogate of ${\ell }_0$-norm. This paper proposes an analog neural network model, namely the Lagrange programming neural network with ${\ell }_p$ objective and quadratic constraint (LPNN-LPQC), with an ${\ell }_0$-norm sparsity measurement for solving the constrained basis pursuit denoise (CBPDN) problem. As the ${\ell }_0$-norm is non-differentiable, we first use a differentiable ${\ell }_p$-norm-like function to approximate the ${\ell }_0$-norm. However, this ${\ell }_p$-norm-like function does not have an explicit expression and, thus, we use the locally competitive algorithm (LCA) concept to handle the nonexistence of the explicit expression. With the LCA approach, the dynamics are defined by the internal state vector. In the proposed model, the thresholding elements are not conventional analog elements in analog optimization. This paper also proposes a circuit realization for the thresholding elements. In the theoretical side, we prove that the equilibrium points of our proposed method satisfy Karush Kuhn Tucker (KKT) conditions of the approximated CBPDN problem, and that the equilibrium points of our proposed method are asymptotically stable. We perform a large scale simulation on various algorithms and analog models. Simulation results show that the proposed algorithm is better than or comparable to several state-of-art numerical algorithms, and that it is better than state-of-art analog neural models.

1. Introduction

1.1. Background

The last few decades have seen increasingly rapid advances in analog neural networks for solving optimization problems. In an analog neural network, the state transitions of neurons are governed by some differential equations. After the dynamics of the network converge to an equilibrium point, the solution of the problem is obtained from the state of the neurons. From many neural pioneers [1,2,3,4,5,6], this approach is very attractive when real-time solutions are required.

The research of the analog neural network approach could be dated back to the 1980s [2]. One of the earliest analog networks is the Hopfield model [2]. Early applications of the Hopfield model are analog-to-digital conversion and the traveling salesman problem. Later, many analog models [3,4,5,6,7] for various optimization problems were proposed. Over the last decade, many new applications of the analog neural modes were investigated, including image processing, sparse approximation [5,8], mobile target localization [9,10], and feature selection [6]. Recently, several analog techniques [5,8,11,12,13] were designed for solving sparse recovery problems.

In sparse recovery [8,14,15,16,17,18], the aim is to recover an unknown sparse vector $\mathit{x}\in {\mathcal{R}}^n$ from an observation vector $\mathit{b}\in {\mathcal{R}}^m$. For many real life signals, their internal representations are with the sparse property [19]. For example, audio signals are approximately sparse in the time-frequency domain [20]. Sparse recovery techniques can be in many signal processing applications. For instance, we use can sparse recovery techniques for image restoration [18,21,22]. Additionally, they can be used to remove the stripes in hyperspectral images [23]. In the inverse synthetic aperture radar (ISAR) application [24,25], a high-quality image of an object can be obtained from the Fourier transformed signal based on sparse recovery techniques. Another application of sparsity recovery is to process electrocardiogram (ECG) signal for classification of various heart diseases [26].

One of sparse recovery problems is the following ${\ell }_0$-norm optimization problem:

$$\underset{\mathit{x}}{min}{\parallel \mathit{x}\parallel }_0,\mathrm{subject}\mathrm{to}\mathit{b}=\mathbf{\Phi }\mathit{x},$$

(1)

where $\mathbf{\Phi }\in {\mathcal{R}}^{m\times n}$ is the measurement matrix. When there are some measurement noise in $\mathit{b}$, the problem becomes the constrained basis pursuit denoise (CBPDN) problem:

$$\underset{\mathit{x}}{min}{\parallel \mathit{x}\parallel }_0,\mathrm{subject}\mathrm{to}\left|\right|\mathit{b}-{\mathbf{\Phi }\mathit{x}\left|\right|}_2^2\le m{\theta }^2,$$

(2)

where $\theta >0$ is the standard deviation of observation noise. Since the ${\ell }_0$-norm is difficult to handle, we usually use the ${\ell }_1$-norm to replace the ${\ell }_0$-norm. The problems, stated in (1) and (2), become

$$\underset{\mathit{x}}{min}{\parallel \mathit{x}\parallel }_1,\mathrm{subject}\mathrm{to}\mathit{b}=\mathbf{\Phi }\mathit{x},\mathrm{and}\underset{\mathit{x}}{min}{\parallel \mathit{x}\parallel }_1,\mathrm{subject}\mathrm{to}\left|\right|\mathit{b}-{\mathbf{\Phi }\mathit{x}\left|\right|}_2^2\le m{\theta }^2,$$

(3)

respectively. In the last two decades, many ${\ell }_1$-norm based numerical algorithms were proposed, such as BPDN-interior [27] and Homotopy [28]. In addition, elegant implementation packages [29,30] are available, such as SPGL1 [30].

Although the aforementioned ${\ell }_1$-norm relaxation approaches were well studied, they have some drawbacks. For instance, in the BPDN-interior algorithm, the solution vector contain many small non-zero elements [31]. As mentioned in [32,33], ${\ell }_p$-norm ($0<p<1$) is a better choice for replacing ${\ell }_0$-norm. However, ${\ell }_p$-norm is a non-convex function, which introduces complex behaviours in the problem-solving process. Therefore, we can use some approximation functions to replace the ${\ell }_p$-norm term, such as minimax concave penalty (MCP) function [34,35]. In addition, there are other methods, which directly handle the ${\ell }_0$-norm. They are normalized iterative hard threshold (NIHT) method [36], approximate message passing (AMP) [37], ${\ell }_0$-norm zero attraction projection ($l_0$-ZAP) [38], ${\ell }_0$-norm alternating direction method of multipliers (${\ell }_0$-ADMM) [39,40], and expectation-conditional maximization either (ECME) [41]. All the mentioned ${\ell }_p$-norm or ${\ell }_0$-norm techniques in this paragraph are digital numerical algorithms.

1.2. Motivation

Apart from using the numerical methods to solve the sparse recovery problem, we can consider using the analog neural approach for solving sparse recovery problems [5,8,11,12,13,21]. However, those analog models in [5,8,11,12,13,21] were developed based on the ${\ell }_1$-norm relaxation techniques. Since the ${\ell }_1$-norm is a surrogate function of the ${\ell }_0$-norm only, directly using the ${\ell }_0$-norm or a ${\ell }_0$-like norm usually leads to a better performance. There is indeed a ${\ell }_0$-norm-based analog model, namely local competition algorithm (LCA) [31]. However, it was designed for unconstrained sparse recovery problems only.

As directly working with the ${\ell }_p$-norm or ${\ell }_0$-norm usually results in better performance, it is interesting to develop some ${\ell }_p$-norm or ${\ell }_0$-norm analog models for sparse recovery problems with constraints. Another shortcoming of existing ${\ell }_1$-norm relaxation neural models [8,11,12,13] is that the corresponding circuit realizations, especially the circuit for projection and thresholding operations, were not discussed.

1.3. Contribution and Organization

This paper focuses on using the analog technique to solve the CBPDN problem with the ${\ell }_0$-norm objective, stated in (2). Strictly speaking, the ${\ell }_0$-norm is not a norm and is not differentiable. These properties create difficulties for constructing the analog model for the CBPDN problem.

The paper proposes a ${\ell }_p$-norm-like function for representing the sparsity measurement. The proposed ${\ell }_p$-norm-like function is differentiable. We then apply the Lagrange programming neural network (LPNN) framework [8,42] for solving the CBPDN problem. The proposed ${\ell }_p$-norm-like function does not have a simple expression, but its derivative has. Hence, we borrow the internal state concept from the LCA [31] to construct a LPNN model for the CBPDN problem. We call our model “LPNN with ${\ell }_p$ objective and quadratic constraint (LPNN-LPQC)”.

In developing an analog neural model, one of difficulties is to analyze the behaviour of the analog neural network dynamics, especially, for non-convex objective function without an explicit expression. The paper discusses the stability of the proposed LPNN-LPQC model. We theoretically prove that the equilibrium points of our proposed LPNN-LPQC satisfy Karush Kuhn Tucker (KKT) conditions of the approximated CBPDN problem. We use the term “approximated” because we have applied an approximation for the ${\ell }_0$-norm. In addition, we prove that the equilibrium points of our proposed method are asymptotically stable.

Unlike some existing analog neural results which do not discuss the circuit realization [8,11,12,13], this paper also discusses the circuit realization of the proposed model. In particular, the detailed circuit design for the thresholding element is given. We then use the MATLAB Simulink to verify our design. In the verification, we find that the MATLAB Simulink results are nearly the same as the corresponding results obtained from the discretized dynamic equations.

This paper also presents a large scale simulation. The simulation result shows that the proposed LPNN-LPQC is better than or comparable to several state-of-art digital numerical algorithms, and that is better than state-of-art analog neural networks.

The rest of this paper is organized as follows. Backgrounds on the LPNN and LCA models are described in Section 2. In Section 3, the proposed LPNN-LPQC is developed. Section 4 discusses the circuit realization of the thresholding element. Section 5 discusses the LPNN-LPQC’s stability. Simulation results and comparisons with state-of-art numerical algorithms are provided in Section 6. Finally, conclusions are drawn in Section 7.

2. LPNN Framework and LCA

2.1. LPNN

The LPNN technique [42] can be used in many applications, such as locating a target in a radar system [9] and ${\ell }_1$-norm-based sparse recovery [8], and ellipse fitting [43]. It was developed for solving a general non-linear constrained optimization problem:

$$\underset{\mathit{x}}{min}\varphi \left(\mathit{x}\right),\mathrm{subject} \mathrm{to} \mathit{h}\left(\mathit{x}\right)=\mathit{0},$$

(4)

where $\varphi :{\mathcal{R}}^n\to \mathcal{R}$ is the objective, and $\mathit{h}:{\mathcal{R}}^n\to {\mathcal{R}}^m$ ($m<n$) represents m equality constraints. In the LPNN approach, we first set up a Lagrangian function, given by

$${\mathcal{L}}_{eq}=\varphi \left(\mathit{x}\right)+{\mathit{\lambda }}^{\top }\mathit{h}\left(\mathit{x}\right),$$

(5)

where $\mathit{\lambda }={[{\lambda }_1,\cdots ,{\lambda }_m]}^{\top }$ is the Lagrange multiplier vector. An LPNN has two classes of neurons: variable neurons for holding $\mathit{x}$ and Lagrange neurons for holding $\mathit{\lambda }$. Its neural dynamics are

$$\mu \frac{d\mathit{x}}{dt}=-\frac{\partial {\mathcal{L}}_{eq}}{\partial \mathit{x}}, \mathrm{and} \mu \frac{d\mathit{\lambda }}{dt}=\frac{\partial {\mathcal{L}}_{eq}}{\partial \mathit{\lambda }},$$

(6)

where $\mu$ is the characteristic time constant. Without loss of generality, we consider that $\mu$ is equal to 1. With (6), when some mild conditions [42] are held, the network settles down at a stable state. The main restriction of using the LPNN framework is that $\varphi \left(\mathit{x}\right)$ and $\mathit{h}\left(\mathit{x}\right)$ should be differentiable.

2.2. LCA

The LCA [31,44] aims at solving the following unconstrained optimization problem:

$$\underset{\mathit{x}}{min}\mathcal{L}\left(\mathit{x}\right)=\frac{1}{2}\left|\right|\mathit{b}-{\mathbf{\Phi }\mathit{x}\left|\right|}_2^2+\kappa \sum _{i=1}^n{S}_{\alpha ,\gamma ,\kappa }\left(x_i\right),$$

(7)

where $\kappa {\sum }_{i=1}^n{S}_{\alpha ,\gamma ,\kappa }\left(x_i\right)$ is a penalty term to improve the sparsity of the resultant $\mathit{x}$. The function $\kappa S_{\alpha ,\gamma ,\kappa }\left(x\right)$ does not have an exact expression. Instead, it is defined by introducing an internal state vector $\mathit{u}$. To define $\kappa S_{\alpha ,\gamma ,\kappa }\left(x\right)$, a thresholding function on $\mathit{u}$ is first introduced

$$x_i=T_{\alpha ,\gamma ,\kappa }\left(u_i\right)=\mathrm{sign}\left(u_i\right)\frac{|u_i|-\alpha \kappa }{1+exp(-\gamma (|u_i|-\kappa ))},$$

(8)

where $\kappa >0$ is the threshold and is related to the magnitude of the non-zero elements, $\gamma \in (0,+\infty )$ controls the threshold transition rate or slope around threshold, and $\alpha \in [0,1]$ indicates adjustment fraction after the internal neuron across threshold. Figure 1a illustrates the shape of $T_{\alpha ,\gamma ,\kappa }\left(u_i\right)$ under various settings. With $(\alpha ,\gamma ,\kappa )=(1,\infty ,\kappa )$, the threshold function $T_{\alpha ,\gamma ,\kappa }(·)$ is the well-known soft threshold function. Given a threshold function, the penalty function $S_{\alpha ,\gamma ,\kappa }\left(x_i\right)$ is then defined by

$$\begin{array}{ccc}\hfill \kappa \partial S_{\alpha ,\gamma ,\kappa }\left(x_i\right)& =& u_i-x_i=u_i-T_{\alpha ,\gamma ,\kappa }\left(u_i\right),\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill S_{\alpha ,\gamma ,\kappa }\left(0\right)& =& 0,\hfill \end{array}$$

(10)

where $\partial S_{\alpha ,\gamma ,\kappa }\left(x_i\right)$ is the gradient or sub-gradient of $S_{\alpha ,\gamma ,\kappa }\left(x_i\right)$.

In the vector form, (9) is written as

$$\kappa \partial \left(\sum _{i=1}^{n}S\alpha ,\gamma ,\kappa \left(x_i\right)\right)=\mathit{u}-\mathit{x}=\mathit{u}-{\mathit{T}}_{\alpha ,\gamma ,\kappa }\left(\mathit{u}\right),$$

(11)

where ${\mathit{T}}_{\alpha ,\gamma ,\kappa }\left(\mathit{u}\right)={[T_{\alpha ,\gamma ,\kappa }\left(u_1\right),\cdots ,T_{\alpha ,\gamma ,\kappa }\left(u_n\right)]}^{\top }$.

From (9), $S_{\alpha ,\gamma ,\kappa }\left(x_i\right)$ is defined by the derivative. Hence, in general, there is no explicit expression for $S_{\alpha ,\gamma ,\kappa }\left(x_i\right)$. To visualize them, we should use numerical integration. Figure 1b shows the sparsity measure function $S_{\alpha ,\gamma ,\kappa }\left(x_i\right)$ under various parameter settings. As shown in the figure, in some cases, such as a large $\gamma$ value, the sparsity measure function ${\mathit{S}}_{0,\gamma ,\kappa }\left(\mathit{x}\right)$ is closed to ${\ell }_p$-norm ($0<p<1$). Hence, ${\mathit{S}}_{0,\gamma ,\kappa }\left(\mathit{x}\right)$ is called the ${\ell }_p$-norm-like sparsity function.

With the internal state concept and (9), LCA defines the dynamics on $\mathit{u}$ (rather than on $\mathit{x}$) as

$$\frac{d\mathit{u}}{dt}=-\frac{\partial {\mathcal{L}}_{eq}}{\partial \mathit{x}}=-\kappa \partial \left(\sum _{i=1}^{n}S\alpha ,\gamma ,\kappa \left(x_i\right)\right)+{\mathbf{\Phi }}^{\top }(\mathit{b}-\mathbf{\Phi }\mathit{x})=-\mathit{u}+\mathit{x}+{\mathbf{\Phi }}^{\top }(\mathit{b}-\mathbf{\Phi }\mathit{x}).$$

(12)

3. LPNN-LPQC Model

For the proposed model, $\alpha =0$, $\kappa =\frac{1}{2}$, and $\gamma$ is a large positive number. The meaning of $\kappa =\frac{1}{2}$ is that the magnitude of the non-zero elements in the resultant $\mathit{x}$ should be greater than $\frac{1}{2}$.

For simplicity, we use notation $S\left(x\right)$ to replace $S_{0,\gamma ,\frac{1}{2}}\left(x\right)$. We consider the following LPQC problem:

$$\underset{\mathit{x}}{min}\frac{1}{2}\sum _{i=1}^{n}S\left(x_i\right),\mathrm{subject}\mathrm{to}\left|\right|\mathit{b}-{\mathbf{\Phi }\mathit{x}\left|\right|}_2^2\le m{\theta }^2,$$

(13)

We first discuss some properties of $S\left(x\right)$. Afterwards, we derive the LPNN-LPQC model and perform the stability analysis on the proposed model.

3.1. ${\ell }_p$-Norm-like Sparsity Measure Function

Recall that from Figure 1b, for large $\gamma$, $S\left(x\right)$ is similar to the ${\ell }_p$-norm. When $\alpha =0$ and $\kappa =\frac{1}{2}$, the threshold function becomes

$$x=T\left(u\right)=\mathrm{sign}\left(u\right)\frac{\left|u\right|}{1+exp(-\gamma (\left|u\right|-\frac{1}{2}))}.$$

(14)

Sparsity measurement function $S\left(x\right)$ is defined by its derivative:

$$\begin{array}{ccc}\hfill \frac{1}{2}\partial S\left(x\right)& =& u-x=u-T\left(u\right),\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill S\left(0\right)& =& 0.\hfill \end{array}$$

(16)

Before discussing the properties of $\frac{1}{2}S\left(x\right)$ and $T\left(x\right)$, we would like to make some remarks. First, the explicit expression of $\frac{1}{2}S\left(x\right)$ and $T^{-1}(·)$ are not available, but the value of $\frac{1}{2}\partial S\left(x\right)$ is easily obtained from u based on (14). Second, as shown in the rest of this paper, in the implementation of the proposed model, we need to implement $T\left(u\right)$ rather than $\frac{1}{2}S\left(x\right)$.

We first list a number of properties of $T\left(u\right)$ and $\frac{1}{2}S\left(x\right)$. From (14) and basic mathematics, we have Property 1.

Property 1. 

• P1.a:$T\left(u\right)$ is a continuous odd function.
• P1.b:$T\left(u\right)$ is strictly monotonically increasing.
• P1.c:Inverse of $T\left(u\right)$ exists.
• P1.d:$T\left(u\right)$ is differentiable at everywhere for all real u.

Since $T\left(u\right)$ is an odd function and monotonically increasing, from basic mathematics, $S\left(x\right)$ has the following properties.

Property 2. 

• P2.a:$\frac{1}{2}S\left(x\right)$ is an even function.
• P2.b:$\frac{1}{2}S\left(\right|x\left|\right)$ is monotonically increasing with respect to $\left|x\right|$.

Additionally, $T\left(u\right)$ has the following properties.

Property 3. 

Define $f\left(u\right)=\frac{1}{1+exp(-\gamma (\left|u\right|-\frac{1}{2}\left)\right)}$. Thus, we have $T\left(u\right)=uf\left(u\right)$. For small positive ϵ, we have the following properties:

• P3.a:If $\left|u\right|\ge \frac{1}{2}+\frac{1}{\gamma }log\frac{1}{ϵ}$, then $\frac{1}{1+ϵ}\le f\left(u\right)<1$.
• P3.b:If $\left|u\right|\le \frac{1}{2}-\frac{1}{\gamma }log\frac{1}{ϵ}$, then $0<f\left(u\right)\le \frac{ϵ}{1+ϵ}$.
• P3.c:If $\left|u\right|\ge \frac{1}{2}+\frac{1}{\gamma }log\frac{\gamma }{ϵ}$, then $1<\frac{dT\left(u\right)}{du}\le 1+\frac{3}{2}ϵ$;
• P3.d:If $0\le \left|u\right|\le \frac{1}{2}-\frac{1}{\gamma }log\frac{\gamma }{ϵ}$, then $0<\frac{dT\left(u\right)}{du}\le \frac{3}{2}ϵ$.

Proof. 

The proof is exhibited in Appendix A. □

Remark 1. 

P3.aandP3.bmean that in some regions of u, $T\left(u\right)$ is approximately equal to either u or 0. On the contrary, the regions of $|\frac{1}{2}-\frac{1}{\gamma }log\frac{1}{ϵ}|<|u|<|\frac{1}{2}+\frac{1}{\gamma }log\frac{1}{ϵ}|$ are uncertain regions. However, the uncertain regions can be arbitrarily small by choosing a sufficient large γ.

With Properties 1–3, we can prove that $\frac{1}{2}S\left(x\right)$ is differentiable at everywhere.

Property 4. 

$\frac{1}{2}S\left(x\right)$ is a continuously differentiable function.

Proof. 

The proof is exhibited in Appendix B. □

Since $\frac{1}{2}S\left(x\right)$ is differentiable, in the rest of this paper, we replace “∂” with “∇” to indicate the partial derivative of $\frac{1}{2}{\sum }_{i=1}^{n}S\left(x_i\right)$, i.e., $\frac{1}{2}\nabla \left({\sum }_{i=1}^{n}S\left(x_i\right)\right)=\mathit{u}-\mathit{x}$. In addition, we discuss two features of $\frac{1}{2}S(·)$, which make $\frac{1}{2}S(·)$ to be a good sparsity measure function.

Property 5. 

For a given ϵ and sufficient large γ,

• Boundness: for $\left|u\right|\ge \frac{1}{2}+\frac{1}{\gamma }log\frac{1}{ϵ}$, $\frac{1}{2}S\left(x\right)\approx 1/8$.
• Sparsity: for $\left|u\right|\le \frac{1}{2}-\frac{1}{\gamma }log\frac{1}{ϵ}$, $\frac{1}{2}S\left(x\right)\approx 0$.

Proof. 

The proof is exhibited in Appendix C. □

Note that when $\gamma$ tends to $+\infty$, $S\left(x\right)$ becomes the ${\ell }_0$-norm and $T\left(u\right)$ is the hard threshold function.

3.2. Properties of LPQC Problem

In order to analyze the properties of the LPQC problem, stated in (13), we review the KKT necessary conditions for general constrained optimization problems.

Lemma 1. 

Consider the following non-linear optimization problem:

$$\underset{\mathit{x}}{min}\varphi \left(\mathit{x}\right),\mathrm{subject} \mathrm{to} h\left(\mathit{x}\right)\le 0,$$

(17)

where $\varphi :{\mathcal{R}}^n\to \mathcal{R}$ is the objective, $h:{\mathcal{R}}^n\to \mathcal{R}$ defines the inequality constraint, ϕ and h are continuously differentiable. If ${\mathit{x}}^{*}$ is a local optimum, then there exists a constant ${\lambda }^{*}$, called Lagrange multiplier, such that

• Stationarity:$\nabla \varphi \left({\mathit{x}}^{*}\right)+{\lambda }^{*}\nabla h\left({\mathit{x}}^{*}\right)=\mathit{0}$,
• Primal feasibility:$h\left({\mathit{x}}^{*}\right)\le 0$,
• Dual feasibility:${\lambda }^{*}\ge 0$,
• Complementary slackness:${\lambda }^{*}h\left({\mathit{x}}^{*}\right)=0$.

With the arm with Lemma 1 and some reasonable assumptions, we could simplify the KKT conditions of the LPQC, stated in (13). The result is summarized in Theorem 1.

Theorem 1. 

Given that ${\left|\right|\mathit{b}\left|\right|}_2^2>m{\theta }^2$, and that ${\mathit{x}}^{*}$ is a local optimum of the optimization problem (13), the KKT conditions become

$$\begin{array}{ccc}& & {\mathit{u}}^{*}-{\mathit{x}}^{*}-{\lambda }^{*}{\mathbf{\Phi }}^{\top }(\mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*})=\mathit{0},\hfill \end{array}$$

(18a)

$$\begin{array}{ccc}& & \left|\right|\mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*}{\left|\right|}_2^2-m{\theta }^2=0,\hfill \end{array}$$

(18b)

$$\begin{array}{ccc}& & {\lambda }^{*}>0.\hfill \end{array}$$

(18c)

Note that $\nabla \left({\sum }_{i=1}^{n}S\left(x_i^{*}\right)\right)={\mathit{u}}^{*}-{\mathit{x}}^{*}$.

Proof. 

From Lemma 1, the KKT conditions of (13) are

$$\begin{array}{cc}& {\mathit{u}}^{*}-{\mathit{x}}^{*}-{\lambda }^{*}{\mathbf{\Phi }}^{\top }(\mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*})=\mathit{0},\end{array}$$

(19a)

$$\begin{array}{cc}& \left|\right|\mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*}{\left|\right|}_2^2-m{\theta }^2\le 0,\end{array}$$

(19b)

$$\begin{array}{cc}& {\lambda }^{*}\ge 0,\end{array}$$

(19c)

$$\begin{array}{cc}& {\lambda }^{*}\left(\right||\mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*}{\left|\right|}_2^2-m{\theta }^2)=0.\end{array}$$

(19d)

According to (19c), ${\lambda }^{*}$ is either greater than or equal to 0. The proof of ${\lambda }^{*}>0$ is by contradiction. Assume that ${\lambda }^{*}=0$. From (19a), we have $\nabla \left({\sum }_{i=1}^{n}S\left(x_i^{*}\right)\right)=\mathit{0}$. Furthermore, from $\frac{1}{2}\nabla \left({\sum }_{i=1}^{n}S\left(x_i^{*}\right)\right)={\mathit{u}}^{*}-{\mathit{x}}^{*}$ and (14), it is easy to prove that ${\mathit{u}}^{*}={\mathit{x}}^{*}$, and that ${\mathit{u}}^{*}={\mathit{x}}^{*}=\mathit{0}$. On the other hand, from (19b), when ${\mathit{u}}^{*}={\mathit{x}}^{*}=\mathit{0}$, we have ${\left|\right|\mathit{b}\left|\right|}_2^2-m{\theta }^2\le 0$, which contradicts our earlier assumption ${\left|\right|\mathit{b}\left|\right|}_2^2>m{\theta }^2$. Hence, ${\lambda }^{*}$ must be greater than 0. In other words, the KKT conditions of (13) become (18). The proof is completed. □

3.3. Dynamics of LPNN-LPQC

In the analog neural approach, we need to design the neural dynamics such that the equilibrium points of the dynamics fulfill the KKT conditions of the problem.

From the LPNN framework, we set up a Lagrangian function:

$${\mathcal{L}}_{LPQC}=\frac{1}{2}\left(\sum _{i=1}^{n}S\left(x_i^{*}\right)\right)+{\alpha }^2\left(\right||\mathit{b}-{\mathbf{\Phi }\mathit{x}\left|\right|}_2^2-m{\theta }^2),$$

(20)

where ${\alpha }^2:=\lambda$, which is used to simplify the proof of the Lagrange multiplier $\lambda$ being greater than zero. Based on the concepts of LPNN and LCA, we propose the dynamics of the LPNN-LPQC as

$$\begin{array}{ccc}\hfill \frac{d\mathit{u}}{dt}& =& -\frac{\partial {\mathcal{L}}_{LPQC}}{\partial \mathit{x}}=-\frac{1}{2}\nabla \left(\sum _{i=1}^{n}S\left(x_i\right)\right)+2{\alpha }^2{\mathbf{\Phi }}^{\top }(\mathit{b}-\mathbf{\Phi }\mathit{x})\hfill \end{array}$$

$$\begin{array}{ccc}& =& -\mathit{u}+\mathit{x}+2{\alpha }^2{\mathbf{\Phi }}^{\top }(\mathit{b}-\mathbf{\Phi }\mathit{x}),\hfill \end{array}$$

(21a)

$$\begin{array}{ccc}\hfill \frac{d\alpha }{dt}& =& \frac{\partial {\mathcal{L}}_{LPQC}}{\partial \alpha }={2\alpha (\parallel \mathit{b}-\mathbf{\Phi }\mathit{x}\parallel }_2^2-m{\theta }^2).\hfill \end{array}$$

(21b)

4. Circuit Realization

In the concept of analog neural models for optimization process, one important issue is whether the operations in the dynamic equation can be implemented with analog circuits or not. This section addresses this issue.

4.1. Thresholding Element

From the dynamic equations, stated in (21), there are many conventional analog operations, such as adders [45], integrators [45], multipliers [46,47] and square circuits [48,49]. The circuit realization of those common operations are well discussed in [45,46,47,48,49].

There is an unconventional element in the dynamic equations. It is the thresholding function:

$$x=T\left(u\right)=\mathrm{sign}\left(u\right)\frac{\left|u\right|}{1+exp(-\gamma (\left|u\right|-\frac{1}{2}))}.$$

Figure 2a shows a generalized realization of the thresholding element for large $\gamma$. In this generalized realization, the thresholding level is $V_{ref}$, where $V_{ref}$ is a positive number. The detailed function is given by

$$x=T\left(u\right)=\mathrm{sign}\left(u\right)\frac{\left|u\right|}{1+exp(-\gamma (\left|u\right|-V_{ref}))}.$$

The two MOSFETs in Figure 2a control the thresholding mode.

If the magnitude $\left|u\right|$ of the input is greater than $V_{ref}$, then one of the two MOSFETs is on and the circuit in Figure 2a becomes an equivalent one shown in Figure 2b. Clearly, for this equivalent circuit, the output is equal to input.

On the other hand, in Figure 2a, if the magnitude $\left|u\right|$ of the input is less than or equal to $V_{ref}$, then the two MOSFETs are off and we obtain another equivalent circuit shown in Figure 2c. In this case, the inputs of the OP-AMP1 are clamped at $u/2$. Since the two current values, $I_u$ and $I_l$, of the upper and lower paths are equal, the output $u_o$ of the OP-AMP1 is zero.

In Figure 3, we show the transfer function of our threshold function for $V_{ref}=1$. The transfer function is obtained from a circuit simulator. In the simulator, the open loop gain of OP-AMP1 is set to ${10}^8$ and the open loop gain of OP-AMP2-to-OP-AMP4 is set to ${10}^5$. For the two MOSFETs, they are n-type with the following parameters: channel width = 100 $\mathsf{\mu }$m, channel length = 200 nm, transconductance = 118 $\mathsf{\mu }$ A/V${}^2$, zero-bias threshold voltage = 430 mV and channel-length modulation = 60 mV${}^{-1}$. From the figure, the transfer function quite matches our theoretical model with a large value of $\gamma$.

4.2. Circuit Structure

Figure 4 shows the analog realization of (21) in the block diagram level. In the realization, there are n blocks to compute $\frac{d{u}_i}{dt}$’s. Inside each block, there are adders, multipliers, and square circuits. Additionally, there is a block to compute $\frac{d\alpha }{dt}$. The time derivatives $\frac{d{u}_i}{dt}$’s and $\frac{d\alpha }{dt}$ are then fed to the integrators to obtain internal variables $u_i\left(t\right)$ and $\alpha$. In order to obtain the decision variables $x_i\left(t\right)$’s, the internal variables are fed to n thresholding elements (Figure 2), where $V_{ref}=1/2$.

4.3. Circuit Simulation

In this subsection, we use a small scale problem to verify our approach based on the Matlab Simulink platform. The problem details are:

• $n=8$ and $m=6$.
• Measurement matrix:

  $$\mathbf{\Phi }=\frac{1}{\sqrt{8}}\left(\begin{array}{cccccccc}\hfill 1& \hfill -1& \hfill -1& \hfill -1& \hfill 1& \hfill -1& \hfill 1& \hfill 1\\ \hfill 1& \hfill 1& \hfill -1& \hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill -1\\ \hfill -1& \hfill 1& \hfill -1& \hfill -1& \hfill -1& \hfill 1& \hfill 1& \hfill 1\\ \hfill -1& \hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill -1& \hfill 1& \hfill 1\\ \hfill 1& \hfill 1& \hfill 1& \hfill -1& \hfill 1& \hfill 1& \hfill -1& \hfill 1\\ \hfill -1& \hfill -1& \hfill -1& \hfill 1& \hfill 1& \hfill 1& \hfill -1& \hfill 1\end{array}\right).$$
• Real sparse vector $\mathit{x}={[-2,0,0,0,0,0,2,0]}^{\mathrm{T}}$ and noisy observation vector
  $\mathit{b}={[-0.0001,0.0038,1.4149,1.4145,-1.4140,-0.0035]}^{\mathrm{T}}$.
• Noise tolerant parameter $\theta =0.001$.

We build the Simulink file for our model, shown in Figure 4. We implement the thresholding function based on Figure 2. Other blocks are based on Simulink’s functional blocks. To verify our Simulink result, we also consider the discrete time simulation on (21). For digitization of (21), the discrete time simulation equations are

$$\begin{array}{ccc}\hfill \mathit{u}(t+\Delta t)& =& \mathit{u}\left(t\right)+\Delta t\frac{d\mathit{u}}{dt}\hfill \end{array}$$

(22a)

$$\begin{array}{ccc}\hfill \alpha (t+\Delta t)& =& \alpha \left(t\right)+\Delta t\frac{d\alpha }{dt}.\hfill \end{array}$$

(22b)

where $\Delta t=0.001$. Figure 5 shows the dynamics from the Simulink results and discrete time simulation. From the figure, the dynamics from the two approaches are nearly the same. The final outputs $\mathit{x}$ are

$$\begin{array}{cc}\hfill \mathrm{Simulink}:& \mathit{x}={[-1.9970,0,0,0,0,0,2.0037,0]}^{\top }\hfill \end{array}$$

(23a)

$$\begin{array}{cc}\hfill \mathrm{Discrete}\mathrm{time}:& \mathit{x}={[-1.9965,0,0,0,0,0,2.0032,0]}^{\top }.\hfill \end{array}$$

(23b)

Clearly, from Figure 5 and (23), both approaches produce the similar results.

5. Properties of the Dynamics

The first issue that we need to address is that equilibrium points of the LPNN-LPQC model should fulfill the KKT conditions of the LPQC problem, stated in (13). Otherwise, the LPNN-LPQC model cannot find out local/global minimums of the LPQC problem. Note that since the LPQC problem or the original ${\ell }_0$-norm CBPDN problem, stated in (2), are non-convex, few algorithms can ensure that their solutions are the global minimum.

The relationship between the equilibrium points of the LPNN-LPQC model and the KKT conditions are summarized in the following theorem.

Theorem 2. 

Given that $\{{\mathit{u}}^{*},{\alpha }^{*}\}$ with ${\alpha }^{*}\ne 0$ is an equilibrium point of the LPNN-LPQC model, this point corresponds to the KKT conditions of the LPQC problem. Note that ${\mathit{x}}^{*}=T\left({\mathit{u}}^{*}\right)$.

Proof. 

From (21), when $\{{\mathit{u}}^{*},{\alpha }^{*}\}$ is an equilibrium point, we

$$\begin{array}{c}\hfill -{\mathit{u}}^{*}+{\mathit{x}}^{*}+2{{\alpha }^{*}}^2{\mathbf{\Phi }}^{\mathrm{T}}(\mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*})=\mathit{0},\end{array}$$

(24a)

$$\begin{array}{c}\hfill 2{\alpha }^{*}(\parallel \mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*}{\parallel }_2^2-m{\theta }^2)=0.\end{array}$$

(24b)

Clearly, (24a) is the same as (19a) with ${{\alpha }^{*}}^2={\lambda }^{*}$. Additionally, when ${\alpha }^{*}\ne 0$, from (24b) becomes $\parallel \mathit{b}-\mathbf{\Phi }{\mathit{x}}^{*}{\parallel }_2^2-m{\theta }^2=0$ which is the same (19b). The proof is completed. □

Theorem 2 tells us that equilibrium points of the LPNN-LPQC model corresponds to the KKT condition of the LPQC problem. Another concern is the stability of the equilibrium points of the LPNN-LPQC model. Theorem 3 presents the stability of the equilibrium points of the LPNN-LPQC model.

Theorem 3. 

Given a positive ϵ and for a sufficient large positive γ, if $({\mathit{u}}^{*},{\alpha }^{*})$ with $\alpha \ne 0$ is an equilibrium point of the LPNN-LPQC model and for all i either $|u_i^{*}|>\frac{1}{2}+\frac{1}{\gamma }log\frac{\gamma }{ϵ}$ or $|u_i^{*}|<\frac{1}{2}-\frac{1}{\gamma }log\frac{\gamma }{ϵ}$, then the equilibrium point is an asymptotically stable point.

Proof. 

The condition of either $|u_i|>\frac{1}{2}+\frac{1}{\gamma }log\frac{\gamma }{ϵ}$ or $|u_i|<\frac{1}{2}-\frac{1}{\gamma }log\frac{\gamma }{ϵ}$ is equivalent to for all i, $|u_i|\notin [\frac{1}{2}-\zeta ,\frac{1}{2}+\zeta ]$, where $\zeta =\frac{1}{\gamma }log\frac{\gamma }{ϵ}$. With the condition, we define two index sets, ${\Gamma }_{\mathit{u}}$ and ${\Gamma }_{\mathit{u}}^c$, given by

• active set${\Gamma }_{\mathit{u}}$: If $|u_i|>1+\zeta$, then $i\in {\Gamma }_{\mathit{u}}$.
• inactive set${\Gamma }_{\mathit{u}}^c$: If $|u_i|<1-\zeta$, then $i\in {\Gamma }_{\mathit{u}}^c$.

Obviously, for a given $ϵ$ and a sufficient large $\gamma$, $\zeta$ tends to zero and the value of $\zeta$ can be arbitrary small. Thus, the inactive neurons have nearly no effect on the dynamics, i.e., $T\left(u_i\right)$ tends to 0 for sufficient large $\gamma$ (see Property 3).

For a given $\mathit{a}$, we define ${\mathit{a}}_{{\Gamma }_{\mathit{u}}}$ as the vector composed of the elements of $\mathit{a}$ indexed by ${\Gamma }_{\mathit{u}}$ and ${\mathit{a}}_{{\Gamma }_{\mathit{u}}^c}$ as the vector composed of the elements of $\mathit{a}$ indexed by ${\Gamma }_{\mathit{u}}^c$. Similarly, given a matrix $\mathbf{\Phi }$, we define ${\mathbf{\Phi }}_{{\Gamma }_{\mathit{u}}}$ as the matrix composed of the columns of $\mathbf{\Phi }$ indexed by ${\Gamma }_{\mathit{u}}$ and ${\mathbf{\Phi }}_{{\Gamma }_{\mathit{u}}^c}$ as the matrix composed of the columns of $\mathbf{\Phi }$ indexed by ${\Gamma }_{\mathit{u}}^c$.

Now, we consider the dynamics near ${\mathit{u}}^{*}$ and ${\alpha }^{*}$. We have two index sets ${\Gamma }_{{\mathit{u}}^{*}}$ and ${\Gamma }_{{\mathit{u}}^{*}}^c$ As mentioned in the above, for inactive states ${\mathit{x}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}\to \mathit{0}$ for sufficient large $\gamma$. The dynamics given in (21) can be rewritten as:

$$\begin{array}{cc}\hfill \frac{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}}{dt}& =-{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}+{\mathit{x}}_{{\Gamma }_{{\mathit{u}}^{*}}}+2{\alpha }^2{\mathbf{\Phi }}_{{\Gamma }_{{\mathit{u}}^{*}}}^{\top }(\mathit{b}-{\mathbf{\Phi }}_{{\Gamma }_{{\mathit{u}}^{*}}}{\mathit{x}}_{{\Gamma }_{{\mathit{u}}^{*}}}),\hfill \end{array}$$

(25a)

$$\begin{array}{cc}\hfill \frac{d\alpha }{dt}& =2\alpha (\parallel \mathit{b}-{\mathbf{\Phi }}_{{\Gamma }_{{\mathit{u}}^{*}}}{\mathit{x}}_{{\Gamma }_{{\mathit{u}}^{*}}}{\parallel }_2^2-m{\theta }^2),\hfill \end{array}$$

(25b)

$$\begin{array}{cc}\hfill \frac{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}{dt}& =-{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}+2{\alpha }^2{\mathbf{\Phi }}_{{\Gamma }_{{\mathit{u}}^{*}}^c}^{\top }\left(\mathit{b}-{\mathbf{\Phi }}_{{\Gamma }_{{\mathit{u}}^{*}}}{\mathit{x}}_{{\Gamma }_{{\mathit{u}}^{*}}}\right).\hfill \end{array}$$

(25c)

Furthermore, the linearization of (25) around the equilibrium point $({\mathit{u}}^{*},{\alpha }^{*})$ is

$$\left[\begin{array}{c}\frac{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}}{dt}\\ \frac{d\alpha }{dt}\\ \frac{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}{dt}\end{array}\right]=-\mathit{H}\left[\begin{array}{c}{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}-{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}^{*}\\ \alpha -{\alpha }^{*}\\ {\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}-{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}^{*}\end{array}\right],$$

(26)

where “$-\mathit{H}$” is the Jacobian matrix at $({\mathit{u}}^{*},{\alpha }^{*})$ and it is given by

$$-\mathit{H}=[{\left.\begin{array}{ccc}\hfill \frac{d{\dot{\mathit{u}}}_{{\Gamma }_{{\mathit{u}}^{*}}}}{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}}\hfill & \hfill \frac{d{\dot{\mathit{u}}}_{{\Gamma }_{{\mathit{u}}^{*}}}}{d\alpha }\hfill & \hfill \frac{d{\dot{\mathit{u}}}_{{\Gamma }_{{\mathit{u}}^{*}}}}{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}\hfill \\ \hfill \frac{d\dot{\alpha }}{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}}\hfill & \hfill \frac{d\dot{\alpha }}{d\alpha }\hfill & \hfill \frac{d\dot{\alpha }}{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}\hfill \\ \hfill \frac{d{\dot{\mathit{u}}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}}}\hfill & \hfill \frac{d{\dot{\mathit{u}}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}{d\alpha }\hfill & \hfill \frac{d{\dot{\mathit{u}}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}{d{\mathit{u}}_{{\Gamma }_{{\mathit{u}}^{*}}^c}}\hfill \end{array}]\right|}_{(\mathit{u},\alpha )=({\mathit{u}}^{*},{\alpha }^{*})}.$$

(27)

From the given condition, for active nodes, we have $d{x}_i/d{u}_i\approx 1$, and for inactive nodes $d{x}_i/d{u}_i\approx 0$. After deriving the sub-matrices in (27), we obtain

$$\mathit{H}=\left[\begin{array}{ccc}\hfill 2{\overline{\alpha }}^2{\mathbf{\Phi }}_{\overline{\Gamma }}^{\top }{\mathbf{\Phi }}_{\overline{\Gamma }}\hfill & -4\overline{\alpha }{\mathbf{\Phi }}_{\overline{\Gamma }}^{\top }(\mathit{b}-{\mathbf{\Phi }}_{\overline{\Gamma }}{\overline{\mathit{x}}}_{\overline{\Gamma }})& \hfill \varnothing \hfill \\ \hfill 4\overline{\alpha }{(\mathit{b}-{\mathbf{\Phi }}_{\overline{\Gamma }}{\overline{\mathit{x}}}_{\overline{\Gamma }})}^{\top }{\mathbf{\Phi }}_{\overline{\Gamma }}\hfill & 0& \hfill \varnothing \hfill \\ \hfill 2{\overline{\alpha }}^2{\mathbf{\Phi }}_{\overline{{\Gamma }^c}}^{\top }{\mathbf{\Phi }}_{\overline{\Gamma }}\hfill & -4\overline{\alpha }{\mathbf{\Phi }}_{\overline{{\Gamma }^c}}^{\top }(\mathit{b}-{\mathbf{\Phi }}_{\overline{\Gamma }}{\overline{\mathit{x}}}_{\overline{\Gamma }})& \hfill \mathit{I}\hfill \end{array}\right],$$

(28)

where ∅ denotes a matrix of zero with appropriate size. Following the proof logic of Theorem 5 in [8], one can find that all eigenvalues of $\mathit{H}$ are with positive real part. Therefore, according to the classical control theory, the corresponding equilibrium point $({\mathit{u}}^{*},{\alpha }^{*})$ is asymptotically stable. The proof is completed. □

6. Experiment Results

6.1. Comparison Algorithms and Settings

This section compares our proposed LPNN-LPQC model with a number of digital numerical methods and analog neural methods. The comparison numerical methods are SPGL1 [30,50], MCP [35], AMP [37], ${\ell }_0$-ADMM [39,40], ${\ell }_0$-ZAP [38], NIHT [36], and ECME [41]. The comparison analog methods are IPNNSR [11] and PNN-${\ell }_1$ [13].

The seven numerical algorithms are described as follows. The SPGL1 [30,50] is a standard ${\ell }_1$-norm approach. The MCP [35] is based on the approximation of the ${\ell }_0$-norm function and its turning parameters are selected by cross validation. The NIHT [36], ECME [41], and AMP [37] are iterative algorithms, and they handle the ${\ell }_0$-norm term by the hard thresholding concept. The ${\ell }_0$-ADMM [39,40] uses the frameworks of ADMM and hard thresholding. The ${\ell }_0$ ZAP [38] utilizes the idea of adaptive filter and projection. The two analog comparison models are IPNNSR [11] and PNN-${\ell }_1$ [13], and are developed based the projection concepts.

The setting of our experiment follows that of the experiment in [51]. The measurement matrix $\mathbf{\Phi }$ is a random $\pm 1/\sqrt{m}$ matrix. The dimension n of the sparse signal $\mathit{x}$ is 4096. A sparse signal contains k non-zero elements which locations are randomly chosen with uniform distribution. Their corresponding value are random $\pm 1$. In our experiment we set $k=\{75,100,125\}$.

6.2. Parameter Settings

To conduct the experiment, we need to select some parameters for the proposed method and comparison methods. In our proposed analog method, there are two tuning parameters which are $\gamma$ and $\kappa$. To make the approximation ${\ell }_p$-norm close to the ${\ell }_0$ norm, we should use a large $\gamma$. In our experiment, we set $\gamma =10,000$. Parameter $\kappa$ is used to control the minimum value of the magnitudes of the decision variables $x_i$. In our experiment, we set $\kappa =\frac{1}{2}$ and $\Delta t=0.0001$.

The SPGL1 package is used to solve

$$\underset{\mathit{x}}{min}{\parallel \mathit{b}-\mathbf{\Phi }\mathit{x}\parallel }_2^2\mathrm{subject}\mathrm{to}{\parallel \mathit{x}\parallel }_1\le k,$$

where k is the number of non-zero elements in $\mathit{x}$. In the experiment, we set the maximum number of iterations to 100,000.

The MCP algorithm is used to solve

$$\underset{\mathit{x}}{min}{\parallel \mathit{b}-\mathbf{\Phi }\mathit{x}\parallel }_2^2+P_{(\lambda ,\gamma )}\left(\mathit{x}\right),$$

where $P_{(\lambda ,\gamma )}\left(\mathit{x}\right)$ is a penalty function to control the sparsity of the solution, and $\lambda$ and $\gamma$ are parameters in $P_{(\lambda ,\gamma )}\left(\mathit{x}\right)$. We set $\gamma =1.5$ and use a linear search to obtain the best value of $\lambda$. We set the maximum number of iterations to 100,000.

The AMP, NIHT, and ECME algorithms are used to solve

$$\underset{\mathit{x}}{min}{\parallel \mathit{b}-\mathbf{\Phi }\mathit{x}\parallel }_2^2\mathrm{subject}\mathrm{to}{\parallel \mathit{x}\parallel }_0\le k,$$

where k is the number of non-zero elements in $\mathit{x}$. We set the maximum number of iterations to 100,000.

The ${\ell }_0$-ADMM algorithm is used to solve

$$\underset{\mathit{x}}{min}{\parallel \mathit{b}-\mathbf{\Phi }\mathit{x}\parallel }_2^2\mathrm{subject}\mathrm{to}{\parallel \mathit{x}\parallel }_0\le k,$$

where k is the number of non-zero elements in $\mathit{x}$. In ${\ell }_0$-ADMM, there is an augmented Lagrangian parameter $\rho$. It is set to $0.5$. The maximum number of iterations to 100,000.

The ${\ell }_0$-ZAP algorithm is used to solve

$$\underset{\mathit{x}}{min}{\parallel \mathit{b}-\mathbf{\Phi }\mathit{x}\parallel }_2^2+\gamma {\parallel \mathit{x}\parallel }_0,$$

where $\gamma$ is used to control the sparsity of the solution vector. We use a linear search to obtain the best value of $\gamma$. In addition, there are two parameter $\kappa$ and $\alpha$. In the experiment, $\kappa =0.001$ and $\alpha =2$. Additionally, the maximum number of iterations is 100,000.

The IPNNSR and PNN-${\ell }_1$ analog models are used to solve

$$\underset{\mathit{x}}{min}{\parallel \mathit{x}\parallel }_1\mathrm{subject}\mathrm{to}\mathit{b}=\mathbf{\Phi }\mathit{x}.$$

For IPNNSR, we set $\Delta t=1$. For PNN-${\ell }_1$, we set $\Delta t=0.01$.

6.3. Convergence

The proposed algorithm is an analog neural network. Hence, one important issue is the time to reach the equilibrium. Here, we conduct an experiment to empirically study the convergent time. Some typical dynamics are given in Figure 6. In the figure, the first row is the case of $k=75$ and $m=500$, the second row is the case of $k=100$ and $m=600$, and the third row is the case of $k=125$ and $m=700$. Since there are 4096 elements in the decision variable vector $\mathit{x}$, the legibility of the figure will be very poor, if we plot all $u_i\left(t\right)$’s and $x_i\left(t\right)$’s in the figure. Therefore, we only plot the dynamics of the $u_i\left(t\right)$’s and $x_i\left(t\right)$’s whose original $x_i$ values are non-zero. From the figure, it can be seen that within 20–40 characteristic time units, the dynamics of our proposed analog neural network settle down.

6.4. Comparison with Other Algorithms

To further analyze the performance of our proposed method, we compare it with seven numerical algorithms and two analog models.

The observation vectors are generated by the following noisy model:

$$\mathit{b}=\mathbf{\Phi }\mathit{x}+\mathit{e}$$

(29)

where $\mathit{e}$ is a zero mean Gaussian noise with standard deviation $\sigma =\{0.001,0.005,0.01\}$. The experiments repeat 100 times with different measurement matrix, initial states, and sparse signals. For each instance we declare that it is successful if

$$\frac{\parallel {\mathit{x}}_0-\widehat{\mathit{x}}{\parallel }_2}{\parallel {\mathit{x}}_0{\parallel }_2}\le \mathit{tol},$$

(30)

where ${\mathit{x}}_0$ denotes the true signal, $\widehat{\mathit{x}}$ is the recovered signal, and tol is the tolerant value. In our experiment, we set $tol=0.01$.

The successful rate results are shown in Figure 7. For all the algorithms, their performances are improved with the increasing numbers of measurements. From the figure, all ${\ell }_0$-norm and ${\ell }_p$-norm models are superior to the ${\ell }_1$ models, including SPGL1, IPNNSR, and PNN-${\ell }_1$. In addition, comparing with the seven numerical methods, our proposed method usually needs less measurements to obtain the same probability of exact reconstruction.

Comparing with the two analog models, IPNNSR and PNN-${\ell }_1$, our LPNN-LPQC model is better. For example, with 125 non-zero elements and noise level equal to 0.01, the IPNNSR and PNN-${\ell }_1$ need around 650 measurement vectors, while our LPNN-LPQC needs around 575 to 600 measurement vectors only. The rationale of the improvement is that our model uses a ${\ell }_0$-norm approach, while IPNNSR and PNN-${\ell }_1$ are based on the ${\ell }_0$-norm.

6.5. Comparison with Other Analog Models

As the paper proposed an analog model, namely LPNN-LPQC, for sparse recovery, this subsection performs a deep discussion on the comparison among our proposed LPNN-LPQC, IPNNSR [11], and PNN-${\ell }_1$ [13]. We discuss three different aspects: probability of exact reconstruction and reconstruction error. In IPNNSR [11] and PNN-${\ell }_1$ [13], the circuit realization of thresholding operator and the projection operator were not addressed.

6.5.1. Successful Rate of Recall

Since Figure 7 shows the successful rates of all algorithms, it may not be easy to see the difference among the three analog models. In Figure 8, we only present the results of three analog models. From the figure, it can be seen that the performance of our LPNN-LPQC is better than that of IPNNSR and PNN-${\ell }_1$. In particular, the performance improvement is significant when the number of non-zero elements is large and the noise level is high. For instance, with 125 non-zero elements and noise level = 0.01, the IPNNSR and PNN-${\ell }_1$ need around 650 measurements for high successful rates of recall, while our LPNN-LPQC needs around 575 to 600 measurements only.

6.5.2. MSE of Recall

The successful rate results concern about whether the estimated $\mathit{x}$ has the correct non-zero positions. Here, in Figure 9, we present the MSE versus the number of measurements used. From Figure 9, it can be seen that in terms of MSE, the performance of our LPNN-LPQC is much better than that of IPNNSR and PNN-${\ell }_1$. In most cases, the MSE values of our proposed model are less than those of IPNNSR and PNN-${\ell }_1$ in one or two orders of magnitude. For example, for $k=75$, $\sigma =0.005$, and $M=450$, the MSE values of the IPNNSR and PNN-${\ell }_1$ are around ${10}^{-3}$, while the MSE value of the proposed model is around ${10}^{-5}$ only.

7. Conclusions

This paper proposed a LPNN-LPQC method for sparse recovery under noise environment. The proposed algorithm is an analog neural network. We showed that the equilibrium points of the model satisfy the KKT conditions of the LPQC problem, and that the equilibrium points of the model are asymptotically stable. From the simulation results, we can see that the performance of the proposed algorithm is comparable to, even superior to many state-of-art ${\ell }_0$-norm or ${\ell }_p$-norm numerical methods. In addition, our proposed algorithm is superior to two analog models. We also presented the circuit realization for the thresholding element and performed circuit simulation for verifying our realization based on the MATLAB Simulink.

In our analysis, we assume that the realization of analog circuit does not have any time delay or synchronization problem. In fact, time delay and mis-synchronization may affect the stability of analog circuits [52,53]. Hence, one of future works is to investigate the behaviour of the proposed analog model with time delay or mis-synchronization.