A Lagrange Programming Neural Network Approach with an ℓ 0 -Norm Sparsity Measurement for Sparse Recovery and Its Circuit Realization

: Many analog neural network approaches for sparse recovery were based on using (cid:96) 1 -norm as the surrogate of (cid:96) 0 -norm. This paper proposes an analog neural network model, namely the Lagrange programming neural network with (cid:96) p objective and quadratic constraint (LPNN-LPQC), with an (cid:96) 0 -norm sparsity measurement for solving the constrained basis pursuit denoise (CBPDN) problem. As the (cid:96) 0 -norm is non-differentiable, we ﬁrst use a differentiable (cid:96) p -norm-like function to approximate the (cid:96) 0 -norm. However, this (cid:96) p -norm-like function does not have an explicit expression and, thus, we use the locally competitive algorithm (LCA) concept to handle the nonexistence of the explicit expression. With the LCA approach, the dynamics are deﬁned by the internal state vector. In the proposed model, the thresholding elements are not conventional analog elements in analog optimization. This paper also proposes a circuit realization for the thresholding elements. In the theoretical side, we prove that the equilibrium points of our proposed method satisfy Karush Kuhn Tucker (KKT) conditions of the approximated CBPDN problem, and that the equilibrium points of our proposed method are asymptotically stable. We perform a large scale simulation on various algorithms and analog models. Simulation results show that the proposed algorithm is better than or comparable to several state-of-art numerical algorithms, and that it is better than state-of-art analog neural models.


Introduction 1.Background
The last few decades have seen increasingly rapid advances in analog neural networks for solving optimization problems.In an analog neural network, the state transitions of neurons are governed by some differential equations.After the dynamics of the network converge to an equilibrium point, the solution of the problem is obtained from the state of the neurons.From many neural pioneers [1][2][3][4][5][6], this approach is very attractive when real-time solutions are required.
The research of the analog neural network approach could be dated back to the 1980s [2].One of the earliest analog networks is the Hopfield model [2].Early applications of the Hopfield model are analog-to-digital conversion and the traveling salesman problem.Later, many analog models [3][4][5][6][7] for various optimization problems were proposed.Over the last decade, many new applications of the analog neural modes were investigated, including image processing, sparse approximation [5,8], mobile target localization [9,10], and feature selection [6].Recently, several analog techniques [5,8,[11][12][13] were designed for solving sparse recovery problems.
In sparse recovery [8,[14][15][16][17][18], the aim is to recover an unknown sparse vector x ∈ R n from an observation vector b ∈ R m .For many real life signals, their internal representations are with the sparse property [19].For example, audio signals are approximately sparse in the time-frequency domain [20].Sparse recovery techniques can be in many signal processing applications.For instance, we use can sparse recovery techniques for image restoration [18,21,22].Additionally, they can be used to remove the stripes in hyperspectral images [23].In the inverse synthetic aperture radar (ISAR) application [24,25], a highquality image of an object can be obtained from the Fourier transformed signal based on sparse recovery techniques.Another application of sparsity recovery is to process electrocardiogram (ECG) signal for classification of various heart diseases [26].
One of sparse recovery problems is the following 0 -norm optimization problem: where Φ ∈ R m×n is the measurement matrix.When there are some measurement noise in b, the problem becomes the constrained basis pursuit denoise (CBPDN) problem: where θ > 0 is the standard deviation of observation noise.Since the 0 -norm is difficult to handle, we usually use the 1 -norm to replace the 0 -norm.The problems, stated in ( 1) and ( 2), become min x x 1 , subject to b = Φx, and min respectively.In the last two decades, many 1 -norm based numerical algorithms were proposed, such as BPDN-interior [27] and Homotopy [28].In addition, elegant implementation packages [29,30] are available, such as SPGL1 [30].

Motivation
Apart from using the numerical methods to solve the sparse recovery problem, we can consider using the analog neural approach for solving sparse recovery problems [5,8,[11][12][13]21].However, those analog models in [5,8,[11][12][13]21] were developed based on the 1 -norm relaxation techniques.Since the 1 -norm is a surrogate function of the 0 -norm only, directly using the 0 -norm or a 0 -like norm usually leads to a better performance.There is indeed a 0 -norm-based analog model, namely local competition algorithm (LCA) [31].However, it was designed for unconstrained sparse recovery problems only.
As directly working with the p -norm or 0 -norm usually results in better performance, it is interesting to develop some p -norm or 0 -norm analog models for sparse recovery problems with constraints.Another shortcoming of existing 1 -norm relaxation neural models [8,[11][12][13] is that the corresponding circuit realizations, especially the circuit for projection and thresholding operations, were not discussed.

Contribution and Organization
This paper focuses on using the analog technique to solve the CBPDN problem with the 0 -norm objective, stated in (2).Strictly speaking, the 0 -norm is not a norm and is not differentiable.These properties create difficulties for constructing the analog model for the CBPDN problem.
The paper proposes a p -norm-like function for representing the sparsity measurement.The proposed p -norm-like function is differentiable.We then apply the Lagrange programming neural network (LPNN) framework [8,42] for solving the CBPDN problem.The proposed p -norm-like function does not have a simple expression, but its derivative has.Hence, we borrow the internal state concept from the LCA [31] to construct a LPNN model for the CBPDN problem.We call our model "LPNN with p objective and quadratic constraint (LPNN-LPQC)".
In developing an analog neural model, one of difficulties is to analyze the behaviour of the analog neural network dynamics, especially, for non-convex objective function without an explicit expression.The paper discusses the stability of the proposed LPNN-LPQC model.We theoretically prove that the equilibrium points of our proposed LPNN-LPQC satisfy Karush Kuhn Tucker (KKT) conditions of the approximated CBPDN problem.We use the term "approximated" because we have applied an approximation for the 0norm.In addition, we prove that the equilibrium points of our proposed method are asymptotically stable.
Unlike some existing analog neural results which do not discuss the circuit realization [8,[11][12][13], this paper also discusses the circuit realization of the proposed model.In particular, the detailed circuit design for the thresholding element is given.We then use the MATLAB Simulink to verify our design.In the verification, we find that the MAT-LAB Simulink results are nearly the same as the corresponding results obtained from the discretized dynamic equations.
This paper also presents a large scale simulation.The simulation result shows that the proposed LPNN-LPQC is better than or comparable to several state-of-art digital numerical algorithms, and that is better than state-of-art analog neural networks.
The rest of this paper is organized as follows.Backgrounds on the LPNN and LCA models are described in Section 2. In Section 3, the proposed LPNN-LPQC is developed.Section 4 discusses the circuit realization of the thresholding element.Section 5 discusses the LPNN-LPQC's stability.Simulation results and comparisons with state-of-art numerical algorithms are provided in Section 6.Finally, conclusions are drawn in Section 7.

LPNN
The LPNN technique [42] can be used in many applications, such as locating a target in a radar system [9] and 1 -norm-based sparse recovery [8], and ellipse fitting [43].It was developed for solving a general non-linear constrained optimization problem: where φ : R n → R is the objective, and h : R n → R m (m < n) represents m equality constraints.In the LPNN approach, we first set up a Lagrangian function, given by where λ = [λ 1 , where µ is the characteristic time constant.Without loss of generality, we consider that µ is equal to 1.With (6), when some mild conditions [42] are held, the network settles down at a stable state.The main restriction of using the LPNN framework is that φ(x) and h(x) should be differentiable.

LCA
The LCA [31,44] aims at solving the following unconstrained optimization problem: where κ ∑ n i=1 S α,γ,κ (x i ) is a penalty term to improve the sparsity of the resultant x.The function κS α,γ,κ (x) does not have an exact expression.Instead, it is defined by introducing an internal state vector u.To define κS α,γ,κ (x), a thresholding function on u is first introduced where κ > 0 is the threshold and is related to the magnitude of the non-zero elements, γ ∈ (0, +∞) controls the threshold transition rate or slope around threshold, and α ∈ [0, 1] indicates adjustment fraction after the internal neuron across threshold.Figure 1a illustrates the shape of T α,γ,κ (u i ) under various settings.With (α, γ, κ) = (1, ∞, κ), the threshold function T α,γ,κ (•) is the well-known soft threshold function.Given a threshold function, the penalty function S α,γ,κ (x i ) is then defined by where ∂S α,γ,κ (x i ) is the gradient or sub-gradient of S α,γ,κ (x i ).For α = 0, γ → ∞ and κ = 1, the value of x cannot be in the range of (0, 1) based on the property of the ideal thresholding function T 0,∞,1 (u).
In the vector form, ( 9) is written as where 9), S α,γ,κ (x i ) is defined by the derivative.Hence, in general, there is no explicit expression for S α,γ,κ (x i ).To visualize them, we should use numerical integration.Figure 1b shows the sparsity measure function S α,γ,κ (x i ) under various parameter settings.As shown in the figure, in some cases, such as a large γ value, the sparsity measure function S 0,γ,κ (x) is closed to p -norm (0 < p < 1).Hence, S 0,γ,κ (x) is called the p -norm-like sparsity function.
With the internal state concept and ( 9), LCA defines the dynamics on u (rather than on x) as

LPNN-LPQC Model
For the proposed model, α = 0, κ = 1 2 , and γ is a large positive number.The meaning of κ = 1 2 is that the magnitude of the non-zero elements in the resultant x should be greater than 1  2 .
For simplicity, we use notation S(x) to replace S 0,γ, 1 2 (x).We consider the following LPQC problem: We first discuss some properties of S(x).Afterwards, we derive the LPNN-LPQC model and perform the stability analysis on the proposed model.

p -Norm-like Sparsity Measure Function
Recall that from Figure 1b, for large γ, S(x) is similar to the p -norm.When α = 0 and κ = 1 2 , the threshold function becomes Sparsity measurement function S(x) is defined by its derivative: Before discussing the properties of 1 2 S(x) and T(x), we would like to make some remarks.First, the explicit expression of 1  2 S(x) and T −1 (•) are not available, but the value of 1  2 ∂S(x) is easily obtained from u based on (14).Second, as shown in the rest of this paper, in the implementation of the proposed model, we need to implement T(u) rather than 1  2 S(x).
We first list a number of properties of T(u) and 1 2 S(x).From ( 14) and basic mathematics, we have Property 1. Property 1.

P1.a: T(u) is a continuous odd function.
P1.b: T(u) is strictly monotonically increasing.

P1.d: T(u) is differentiable at everywhere for all real u.
Since T(u) is an odd function and monotonically increasing, from basic mathematics, S(x) has the following properties.
. Thus, we have T(u) = u f (u).For small positive , we have the following properties: Proof.The proof is exhibited in Appendix A.
Remark 1. P3.a and P3.b mean that in some regions of u, T(u) is approximately equal to either u or 0. On the contrary, the regions of | are uncertain regions.However, the uncertain regions can be arbitrarily small by choosing a sufficient large γ.
With Properties 1-3, we can prove that 1  2 S(x) is differentiable at everywhere.Property 4. 1  2 S(x) is a continuously differentiable function.
Proof.The proof is exhibited in Appendix B.
Since 1 2 S(x) is differentiable, in the rest of this paper, we replace "∂" with "∇" to indicate the partial derivative of 1  2 In addition, we discuss two features of 1  2 S(•), which make 1 2 S(•) to be a good sparsity measure function.
Property 5.For a given and sufficient large γ, Note that when γ tends to +∞, S(x) becomes the 0 -norm and T(u) is the hard threshold function.

Properties of LPQC Problem
In order to analyze the properties of the LPQC problem, stated in (13), we review the KKT necessary conditions for general constrained optimization problems.Lemma 1.Consider the following non-linear optimization problem: where φ : R n → R is the objective, h : R n → R defines the inequality constraint, φ and h are continuously differentiable.If x * is a local optimum, then there exists a constant λ * , called Lagrange multiplier, such that With the arm with Lemma 1 and some reasonable assumptions, we could simplify the KKT conditions of the LPQC, stated in (13).The result is summarized in Theorem 1.

Dynamics of LPNN-LPQC
In the analog neural approach, we need to design the neural dynamics such that the equilibrium points of the dynamics fulfill the KKT conditions of the problem.
From the LPNN framework, we set up a Lagrangian function: where α 2 := λ, which is used to simplify the proof of the Lagrange multiplier λ being greater than zero.Based on the concepts of LPNN and LCA, we propose the dynamics of the LPNN-LPQC as

Circuit Realization
In the concept of analog neural models for optimization process, one important issue is whether the operations in the dynamic equation can be implemented with analog circuits or not.This section addresses this issue.
There is an unconventional element in the dynamic equations.It is the thresholding function: .
Figure 2a shows a generalized realization of the thresholding element for large γ.In this generalized realization, the thresholding level is V re f , where V re f is a positive number.The detailed function is given by .
The two MOSFETs in Figure 2a control the thresholding mode.
If the magnitude |u| of the input is greater than V re f , then one of the two MOS-FETs is on and the circuit in Figure 2a becomes an equivalent one shown in Figure 2b.Clearly, for this equivalent circuit, the output is equal to input.uV xV On the other hand, in Figure 2a, if the magnitude |u| of the input is less than or equal to V re f , then the two MOSFETs are off and we obtain another equivalent circuit shown in Figure 2c.In this case, the inputs of the OP-AMP1 are clamped at u/2.Since the two current values, I u and I l , of the upper and lower paths are equal, the output u o of the OP-AMP1 is zero.
In Figure 3, we show the transfer function of our threshold function for V re f = 1.The transfer function is obtained from a circuit simulator.In the simulator, the open loop gain of OP-AMP1 is set to 10 8 and the open loop gain of OP-AMP2-to-OP-AMP4 is set to 10 5 .For the two MOSFETs, they are n-type with the following parameters: channel width = 100 µm, channel length = 200 nm, transconductance = 118 µ A/V 2 , zero-bias threshold voltage = 430 mV and channel-length modulation = 60 mV −1 .From the figure, the transfer function quite matches our theoretical model with a large value of γ.

Circuit Structure
Figure 4 shows the analog realization of (21) in the block diagram level.In the realization, there are n blocks to compute du i dt 's.Inside each block, there are adders, multipliers, and square circuits.Additionally, there is a block to compute dα dt .The time derivatives du i dt 's and dα dt are then fed to the integrators to obtain internal variables u i (t) and α.In order to obtain the decision variables x i (t)'s, the internal variables are fed to n thresholding elements (Figure 2), where V re f = 1/2.
Clearly, from Figures 5 and ( 23), both approaches produce the similar results.

Properties of the Dynamics
The first issue that we need to address is that equilibrium points of the LPNN-LPQC model should fulfill the KKT conditions of the LPQC problem, stated in (13).Otherwise, the LPNN-LPQC model cannot find out local/global minimums of the LPQC problem.Note that since the LPQC problem or the original 0 -norm CBPDN problem, stated in (2), are non-convex, few algorithms can ensure that their solutions are the global minimum.
The relationship between the equilibrium points of the LPNN-LPQC model and the KKT conditions are summarized in the following theorem.Theorem 2. Given that {u * , α * } with α * = 0 is an equilibrium point of the LPNN-LPQC model, this point corresponds to the KKT conditions of the LPQC problem.Note that x * = T(u * ).
Theorem 2 tells us that equilibrium points of the LPNN-LPQC model corresponds to the KKT condition of the LPQC problem.Another concern is the stability of the equilibrium points of the LPNN-LPQC model.Theorem 3 presents the stability of the equilibrium points of the LPNN-LPQC model.Theorem 3. Given a positive and for a sufficient large positive γ, if (u * , α * ) with α = 0 is an equilibrium point of the LPNN-LPQC model and for all i either |u * then the equilibrium point is an asymptotically stable point.

Proof. The condition of either |u
where ζ = 1 γ log γ .With the condition, we define two index sets, Γ u and Γ c u , given by active set Γ u : Obviously, for a given and a sufficient large γ, ζ tends to zero and the value of ζ can be arbitrary small.Thus, the inactive neurons have nearly no effect on the dynamics, i.e., T(u i ) tends to 0 for sufficient large γ (see Property 3).
For a given a, we define a Γ u as the vector composed of the elements of a indexed by Γ u and a Γ c u as the vector composed of the elements of a indexed by Γ c u .Similarly, given a matrix Φ, we define Φ Γ u as the matrix composed of the columns of Φ indexed by Γ u and Φ Γ c u as the matrix composed of the columns of Φ indexed by Γ c u .Now, we consider the dynamics near u * and α * .We have two index sets Γ u * and Γ c u * As mentioned in the above, for inactive states x Γ c u * → 0 for sufficient large γ.The dynamics given in ( 21) can be rewritten as: Furthermore, the linearization of ( 25) around the equilibrium point where "−H" is the Jacobian matrix at (u * , α * ) and it is given by From the given condition, for active nodes, we have dx i /du i ≈ 1, and for inactive nodes dx i /du i ≈ 0. After deriving the sub-matrices in (27), we obtain where ∅ denotes a matrix of zero with appropriate size.Following the proof logic of Theorem 5 in [8], one can find that all eigenvalues of H are with positive real part.Therefore, according to the classical control theory, the corresponding equilibrium point (u * , α * ) is asymptotically stable.The proof is completed.
The seven numerical algorithms are described as follows.The SPGL1 [30,50] is a standard 1 -norm approach.The MCP [35] is based on the approximation of the 0 -norm function and its turning parameters are selected by cross validation.The NIHT [36], ECME [41], and AMP [37] are iterative algorithms, and they handle the 0 -norm term by the hard thresholding concept.The 0 -ADMM [39,40] uses the frameworks of ADMM and hard thresholding.The 0 ZAP [38] utilizes the idea of adaptive filter and projection.The two analog comparison models are IPNNSR [11] and PNN-1 [13], and are developed based the projection concepts.
The setting of our experiment follows that of the experiment in [51].The measurement matrix Φ is a random ±1/ √ m matrix.The dimension n of the sparse signal x is 4096.A sparse signal contains k non-zero elements which locations are randomly chosen with uniform distribution.Their corresponding value are random ±1.In our experiment we set k = {75, 100, 125}.

Parameter Settings
To conduct the experiment, we need to select some parameters for the proposed method and comparison methods.In our proposed analog method, there are two tuning parameters which are γ and κ.To make the approximation p -norm close to the 0 norm, we should use a large γ.In our experiment, we set γ = 10, 000.Parameter κ is used to control the minimum value of the magnitudes of the decision variables x i .In our experiment, we set κ = 1 2 and ∆t = 0.0001.The SPGL1 package is used to solve min where k is the number of non-zero elements in x.In the experiment, we set the maximum number of iterations to 100,000.
The MCP algorithm is used to solve min where P (λ,γ) (x) is a penalty function to control the sparsity of the solution, and λ and γ are parameters in P (λ,γ) (x).We set γ = 1.5 and use a linear search to obtain the best value of λ.
We set the maximum number of iterations to 100,000.The AMP, NIHT, and ECME algorithms are used to solve where k is the number of non-zero elements in x.We set the maximum number of iterations to 100,000.The 0 -ADMM algorithm is used to solve min where k is the number of non-zero elements in x.In 0 -ADMM, there is an augmented Lagrangian parameter ρ.It is set to 0.5.The maximum number of iterations to 100,000.
The 0 -ZAP algorithm is used to solve min where γ is used to control the sparsity of the solution vector.We use a linear search to obtain the best value of γ.In addition, there are two parameter κ and α.In the experiment, κ = 0.001 and α = 2. Additionally, the maximum number of iterations is 100,000.
The IPNNSR and PNN-1 analog models are used to solve min x x 1 subject to b = Φx.

Convergence
The proposed algorithm is an analog neural network.Hence, one important issue is the time to reach the equilibrium.Here, we conduct an experiment to empirically study the convergent time.Some typical dynamics are given in Figure 6.In the figure, the first row is the case of k = 75 and m = 500, the second row is the case of k = 100 and m = 600, and the third row is the case of k = 125 and m = 700.Since there are 4096 elements in the decision variable vector x, the legibility of the figure will be very poor, if we plot all u i (t)'s and x i (t)'s in the figure.Therefore, we only plot the dynamics of the u i (t)'s and x i (t)'s whose original x i values are non-zero.From the figure, it can be seen that within 20-40 characteristic time units, the dynamics of our proposed analog neural network settle down.

Comparison with Other Algorithms
To further analyze the performance of our proposed method, we compare it with seven numerical algorithms and two analog models.
The observation vectors are generated by the following noisy model: e is a zero mean Gaussian noise with standard deviation σ = {0.001,0.005, 0.01}.The experiments repeat 100 times with different measurement matrix, initial states, and sparse signals.For each instance we declare that it is successful if where x 0 denotes the true signal, x is the recovered signal, and tol is the tolerant value.
In our experiment, we set tol = 0.01.The successful rate results are shown in Figure 7.For all the algorithms, their performances are improved with the increasing numbers of measurements.From the figure, all 0 -norm and p -norm models are superior to the 1 models, including SPGL1, IPNNSR, and PNN-1 .In addition, comparing with the seven numerical methods, our proposed method usually needs less measurements to obtain the same probability of exact reconstruction.
Comparing with the two analog models, IPNNSR and PNN-1 , our LPNN-LPQC model is better.For example, with 125 non-zero elements and noise level equal to 0.01, the IPNNSR and PNN-1 need around 650 measurement vectors, while our LPNN-LPQC needs around 575 to 600 measurement vectors only.The rationale of the improvement is that our model uses a 0 -norm approach, while IPNNSR and PNN-1 are based on the 0 -norm.

Comparison with Other Analog Models
As the paper proposed an analog model, namely LPNN-LPQC, for sparse recovery, this subsection performs a deep discussion on the comparison among our proposed LPNN-LPQC, IPNNSR [11], and PNN-1 [13].We discuss three different aspects: probability of exact reconstruction and reconstruction error.In IPNNSR [11] and PNN-1 [13], the circuit realization of thresholding operator and the projection operator were not addressed.

Successful Rate of Recall
Since Figure 7 shows the successful rates of all algorithms, it may not be easy to see the difference among the three analog models.In Figure 8, we only present the results of three analog models.From the figure, it can be seen that the performance of our LPNN-LPQC is better than that of IPNNSR and PNN-1 .In particular, the performance improvement is significant when the number of non-zero elements is large and the noise level is high.For instance, with 125 non-zero elements and noise level = 0.01, the IPNNSR and PNN-1 need around 650 measurements for high successful rates of recall, while our LPNN-LPQC needs around 575 to 600 measurements only.

MSE of Recall
The successful rate results concern about whether the estimated x has the correct nonzero positions.Here, in Figure 9, we present the MSE versus the number of measurements used.From Figure 9, it can be seen that in terms of MSE, the performance of our LPNN-LPQC is much better than that of IPNNSR and PNN-1 .In most cases, the MSE values of our proposed model are less than those of IPNNSR and PNN-1 in one or two orders of magnitude.For example, for k = 75, σ = 0.005, and M = 450, the MSE values of the IPNNSR and PNN-1 are around 10 −3 , while the MSE value of the proposed model is around 10 −5 only.

P3.d
The proof consists of two parts.First, we show the monotonic properties of du is an even function, we only need to consider the region of u < 1 2 − 1 γ log γ .For 0 < u < 1 2 − 1 γ log γ , sign(u) > 0, g(u) is monotonic decreasing, and exp(γ) > g(u) > γ .In this region, the second derivative of T(u) is

Figure 1 .
Figure 1.Threshold function and sparsity measure function. (a) The shape of T α,γ,κ (u i ) under various settings.(b) Sparsity measure function.For α = 0, γ → ∞ and κ = 1, the value of x cannot be in the range of (0, 1) based on the property of the ideal thresholding function T 0,∞,1 (u).

Figure 2 .
Figure 2. (a) Circuit for the thresholding function T(x).(b) Equivalent circuit of the thresholding function T(x) when the magnitude of input is greater than V re f .(c) Equivalent circuit of the thresholding function T(x) when the magnitude of input is less than or equal to V re f .

Figure 3 .
Figure 3. Thresholding function T(x) obtained from the circuit simulation of Figure 2 with V re f = 1.

Figure 4 .
Figure 4. Analog realization of (21) in the block diagram level.

8 Figure 5 .
Figure 5. Dynamics obtained from Simulink and discrete time simulation.

Figure 6 .
Figure 6.Typical dynamics of u and x for the LPNN-LPQC model, where n = 4096.The first column: k = 75, m = 500.The second column: k = 100, m = 600.The third column, k = 125, m = 700.In all sub-figures, we only show the dynamics of the u i 's and x i 's whose original x i values are non-zero, because there are 4096 curves in each sub-figure when we show all the dynamics for u i 's and x i 's.

Figure 7 .
Figure 7. Simulation results of different algorithms, where n = 4096.For the first column, k = 75.For the second column, k = 100.For the third column, k = 125.The three rows are based on three different noise levels.The experiments are repeated 100 times using different settings.

Figure 8 .
Figure 8.Comparison with other analog models on probability of reconstruction, where n = 4096.For the first column, k = 75.For the second column, k = 100.For the third column, k = 125.The three rows are based on three different noise levels.The experiments are repeated 100 times using different settings.

1 γ
log γ .Second, the upper and lower bounds of dT(u) du are established.Since T(u) is an odd function and dT(u)