Least-Square-Based Three-Term Conjugate Gradient Projection Method for ℓ₁-Norm Problems with Application to Compressed Sensing

Abstract

In this paper, we propose, analyze, and test an alternative method for solving the ${\ell }_1$-norm regularization problem for recovering sparse signals and blurred images in compressive sensing. The method is motivated by the recent proposed nonlinear conjugate gradient method of Tang, Li and Cui [Journal of Inequalities and Applications, 2020(1), 27] designed based on the least-squares technique. The proposed method aims to minimize a non-smooth minimization problem consisting of a least-squares data fitting term and an ${\ell }_1$-norm regularization term. The search directions generated by the proposed method are descent directions. In addition, under the monotonicity and Lipschitz continuity assumption, we establish the global convergence of the method. Preliminary numerical results are reported to show the efficiency of the proposed method in practical computation.

1. Introduction

Discrete ill-posed problems are systems of linear equations arising from the discretization of ill-posed problems. Consider the linear system

$$b=At,$$

(1)

where $t\in R^n$ is an original signal, $A\in R^{m\times n}(m<n)$ is a linear map, and $b\in R^m$ is an observed data. In particular, the original signal t is assumed to be sparse, that is, it has very few non-zero coefficients. Since $m<n,$ the linear system (1) is usually referred to as ill-conditioned or under-determined problems. In compressive sensing (CS), it is possible to regain the sparse signal t from the linear system (1), by finding the solution of the ${\ell }_0$-regularized problem

$$\underset{t}{min}{\{\parallel t\parallel }_0:At=b\},$$

(2)

where ${\parallel t\parallel }_0$ denotes the nonzero components in t. Unfortunately, the ${\ell }_0$-norm is NP-hard in general. Based on this, researchers have developed alternative model by replacing the ${\ell }_0$-norm with ${\ell }_1$-norm. Thus, they solved the following problem:

$$\underset{t}{min}{\{\parallel t\parallel }_1:At=b\},$$

(3)

Under some mild assumptions, Donoho [1] proved that solution(s) of problem (2) also solves (3). In most applications, the observed value b usually contains some noise, thus problem (3) can be relaxed to the penalized least squares problem

$$\underset{t}{min}{\tau \parallel t\parallel }_1+\frac{1}{2}{\parallel At-b\parallel }_2^2,$$

(4)

where $\tau >0$, balancing the trade-off between sparsity and residual error. Problems of the form (4) have become familiar over the past three decades, particularly in compressive sensing contexts. Interested readers may refer to the recent papers (Refs. [2,3]) for more details.

In order to address problem (4), several numerical algorithms have been proposed, for instance, Daubechies, Defrise, and Demol [4] proposed the iterative shrinkage thresholding (IST) algorithm; thereafter, Beck and Teboulle [5] proposed the fast iterative shrinkage thresholding algorithm (FISTA). These algorithms are well known due to their simplicity and efficiency. Likewise, Hale, Yin, and Zhang [6] proposed the fixed-point continuous search method [6], and an acceleration technique was introduced by nonmonotone line search with the Barzilai–Borwein stepsize [7]. He, Chang, and Osher [8] introduced the unconstrained formulation of ${\ell }_1$-regularization problem, where the Bregman iterative approach was used to obtain the solutions to problem (4). The proximal forward backward splitting technique is another efficient technique for solving (4). This technique is based on the proximal operator introduced by Moreau [9]. Another type of method for solving the problem (4) is by using the gradient descent method. Quite recently, Figueiredo, Nowak, and Wright [10] first developed a gradient projection method to solve the sparse reconstruction problem. Thereafter, Xiao and Zhu [11,12] proposed a conjugate gradient projection method and a spectral gradient method to solve problem (4), respectively. Unlike IST and FISTA, in order to solve problem (4), the problem was first transformed into a monotone operator equation; see Section 2. Thereafter, an algorithm is developed to solve these systems of equations.

With the approximate equivalence between problem (4) and a system of monotone operator equations, one of the methods for solving the systems of nonlinear monotone operator equations is the conjugate gradient method. Considering the importance of the method, several extensions of this method have been proposed. The three-term conjugate gradient method happens to be one such extension. The first two three-term conjugate gradient method was introduced by Beale [13] and Nazareth [14] to weaken the condition of global convergence of the two-term conjugate gradient method. It is clear that, due to the existence of an additional parameter in the three-term conjugate gradient schemes, establishing the sufficient descent property is more accessible than the two-term conjugate gradient ones. To this end, the three-term conjugate gradient methods are presented, analyzed, and extensively studied in several references because of their advantages in the descent property and the computational performance. The references [15,16,17] have proposed different three-term conjugate gradient methods, and shown their specific properties, global convergence, and numerical performance. Tang, Li and Cui [18] presented a new three-term conjugate gradient approach based on the technique of the least-squares. Their approach incorporates the advantage of two existing efficient conjugate gradient approaches and also generates sufficient descent direction without the aid of a line search procedure. Preliminary numerical tests indicate that their method is efficient. Due to the simplicity and low storage requirements of the conjugate gradient method, numerous researchers have recently extended a number of conjugate gradient algorithms designed to solve unconstrained optimization problem to solve large-scale nonlinear equations. Using the popular CG_DESCENT method [19], Xiao and Zhu [12] recently constructed a conjugate gradient method (CGD) based on the projection scheme of Solodov and Svaiter [20] to solve monotone nonlinear operator equations with convex constraints. The method was also successfully used to solve the sparse signal in compressive sensing. Interested readers may refer to the following articles [21,22,23,24,25,26,27] for an overview of algorithms used for solving monotone operator equations.

Inspired by the work of Xiao and Zhu [12], the least-squares-based three-term conjugate gradient method (LSTT) for solving unconstrained optimization problems by Tang, Li, and Cui [18] and the projection technique of Solodov and Svaiter [20], we further study, analyze, and construct a derivative-free least-square-based three-term conjugate gradient method to solve the ${\ell }_1$-norm problem arising from the reconstruction of sparse signal and image in compressive sensing. The method can be viewed as an extension of the LSTT method for solving unconstrained optimization problem and a projection technique. Under the monotonicity and Lipchitz continuity assumption, the global convergence of the proposed method is established using the backtracking line search. Computational experiments are carried out to reconstruct sparse signal and image in compressive sensing. The numerical results indicate that the proposed method is more efficient and robust.

The rest of the paper is organized as follows. In Section 2, we give a review of the reformulation of problem (4) into a convex quadratic program problem by Figueiredo et al. [10]. In Section 3, we present the motivation and general algorithm of the proposed method. The global convergence of the proposed algorithm is presented in Section 4. In Section 5, numerical experiments are presented to illustrate the efficiency of our algorithm. Unless otherwise stated, throughout this paper, the symbol $\parallel ·\parallel$ denotes the Euclidean norm. Furthermore, the projection map denoted as $P_S,$ which is a mapping from $R^n$ onto the non-empty, closed and convex subset $S\subseteq R^n,$ that is,

$$P_S\left(t\right):=argmin\{\parallel t-y\parallel |y\in S\},$$

which has the well known nonexpansive property, that is,

$$\parallel P_S\left(h\right)-P_S\left(g\right)\parallel \le \parallel h-g\parallel ,\forall h,g\in R^n.$$

(5)

2. Reformulation of the Model

Figuredo, Nowak, and Wright [10] gave the reformulation of the minimization problem (4) into a quadratic programming problem as follows. Consider any vector $t\in R^n$, t can be rewritten as follows:

$$\begin{array}{c}\hfill t=u-v,u\ge 0,v\ge 0,\end{array}$$

where $u\in R^n,v\in R^n$ and $u_i={\left(t_i\right)}_{+},$ $v_i={(-t_i)}_{+}$ for all $i\in [1,n]$ with ${(·)}_{+}=max\{0,·\}.$ Therefore, the ${\ell }_1$-norm could be represented as ${\parallel t\parallel }_1=e_n^{T}u+e_n^{T}v,$ where $e_n$ is an n-dimensional vector with all elements one. Thus, (4) was rewritten as

$$\underset{u,v}{min}\{\frac{1}{2}{\parallel b-A(u-v)\parallel }^2+\tau e_n^{T}u+\tau e_n^{T}v:u,v\ge 0\}.$$

(6)

Moreover, from [10], with no difficulty, (6) can be rewritten as the quadratic program problem with box constraints. That is,

$$\underset{z}{min}\frac{1}{2}{z}^{T}Hz+c^{T}z,z\ge 0,$$

(7)

where $z=\left[\begin{array}{c}u\\ v\end{array}\right]$, $c=\tau e_{2n}+\left[\begin{array}{c}-y\\ y\end{array}\right]$, $b=A^{T}y$, $H=\left[\begin{array}{cc}{A}^{T}A& -A^{T}A\\ -A^{T}A& A^{T}A\end{array}\right]$.

Simple calculation shows that H is a semi-definite positive matrix. Hence, (7) is a convex quadratic program problem, and it is equivalent to

$$F\left(z\right)=min\{z,Hz+c:z\in S\}=0.$$

(8)

The function F is a vector-valued and the “min” interpreted as componentwise minimum. From ([28], Lemma 3) and ([11], Lemma 2.2), we know that the mapping F is Lipschitz continuous and monotone. Hence, the algorithm for solving problem (8) can be equally and effectively used to solve problem (4).

3. Algorithm

Recently, Chunning, Shuangyu, and Zengru [18] proposed a new three-term conjugate gradient method based on the least-squares technique for solving the following unconstrained optimization problem:

$$min\{f\left(t\right):t\in R^n\}$$

where the function f is continuously differentiable from $R^n$ into R and the gradient $\nabla f\left(t\right)$ is available. Similar to most conjugate gradient methods, the iterative scheme of the conjugate gradient method developed in [18] generates a sequence of iterates by letting

$$t_{k+1}=t_k+{\alpha }_k{d}_k,k\ge 0$$

where ${\alpha }_k$ is the steplength and the search direction $d_k$ is updated by

$$d_k:=\left\{\begin{array}{cc}-\nabla f\left(t_k\right)+\widetilde{{\beta }_k}{d}_{k-1}-{\theta }_k{\overline{y}}_{k-1}\hfill & \mathrm{if}k>0,\hfill \\ -\nabla f\left(t_k\right)\hfill & \mathrm{if}k=0,\hfill \end{array}\right.$$

(9)

where ${\overline{y}}_{k-1}=\nabla f\left(t_k\right)-\nabla f\left(t_{k-1}\right),$ $\widetilde{{\beta }_k}$ and ${\theta }_k$ are scalar computed as

$$\begin{array}{cc}\hfill \widetilde{{\beta }_k}& :={\beta }_k^{MHS}({\tau }_k^{\ast }):={\beta }_k^{HS}-\frac{\nabla f{\left(t_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}:=\frac{\nabla f{\left(t_k\right)}^T{\overline{y}}_{k-1}}{{\overline{y}}_{k-1}^T{d}_{k-1}}-\frac{\nabla f{\left(t_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2},\hfill \\ \hfill {\theta }_k& :=\frac{\nabla f{\left(t_k\right)}^T{d}_{k-1}}{{\overline{y}}_{k-1}^T{d}_{k-1}}.\hfill \end{array}$$

A valid approach for solving (8) is by using a derivative-free line search to ensure the step-size ${\alpha }_k$ [29]. To this end, we present the following derivative-free least-square three term conjugate gradient projection algorithm (Algorithm 1).

Algorithm 1 DF-LSTT

Input. Choose any arbitrary initial point $t_0\in S$, the positive constants: $Tol\in (0,1)\xi \in (0,2)$, $\varrho \in (0,1),\beta >0,\varsigma >0.$ Step 0. Let $d_0=-F\left(t_0\right)$ and $k:=0.$ Step 1. Determine the step-size ${\alpha }_k=\beta {\varrho }^i$, where i is the smallest non-negative integer such that the following line search is satisfied: $$-F{(t_k+{\alpha }_k{d}_k)}^T{d}_k\ge \varsigma {\alpha }_k{\parallel d_k\parallel }^2.$$ (10) Step 2. Compute $$u_k:=t_k+{\alpha }_k{d}_k.$$ (11) Step 3. If $u_k\in S$ and $F\left(u_k\right)=0,$ stop. Otherwise, compute the next iterate by $$\begin{array}{c}\hfill t_{k+1}:=P_S[t_k-{\delta }_k\xi F\left(u_k\right)],\end{array}$$ (12) where $${\delta }_k:=\frac{F{\left(u_k\right)}^T(t_k-u_k)}{\parallel F\left(u_k\right){\parallel }^2}$$ (13) Step 4. If the stopping criterion is satisfied, that is, if $\parallel F\left(t_k\right)\parallel \le Tol,$ stop. Otherwise, compute the next search direction $d_k$ by $$d_k:=-F\left(t_k\right)+{\beta }_k\left(N_k^{\ast }\right)d_{k-1}-v_k{y}_{k-1}$$ (14) where $$y_{k-1}:=F\left(t_k\right)-F\left(t_{k-1}\right),v_k:=\frac{F{\left(t_k\right)}^T{d}_{k-1}}{{\tilde{y}}_{k-1}^T{d}_{k-1}},$$ (15) $$\begin{array}{cc}\hfill {\beta }_k\left(N_k^{\ast }\right)& :={\beta }_k^{MHS}-\frac{F{\left(t_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}:=\frac{y_{k-1}^{T}F\left(t_k\right)}{{\tilde{y}}_{k-1}^T{d}_{k-1}}-\frac{F{\left(t_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}\hfill \end{array}$$ $${\tilde{y}}_{k-1}:=y_{k-1}+j_{k-1}{d}_{k-1},j_{k-1}:=1+max\left\{0,-\frac{y_{k-1}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}\right\}$$ (16) Step 5. Finally, we set $k:=k+1$ and return to step 1.

Remark 1.

In order to ensure that the parameters $\widetilde{{\beta }_k}$ and ${\theta }_k$ are well defined when extending them to solve (4), we re-modify their denominator using the scalar ${\tilde{y}}_{k-1}^T{d}_{k-1}$. Furthermore, to guarantee the boundness of the search direction, we assume the operator under consideration to be monotone rather than uniformly monotone as assumed in [18].

Lemma 1.

The search direction $d_k$ generated by DF-LSTT algorithm is a descent direction. That is,

$$F{\left(t_k\right)}^T{d}_k=-{\parallel F\left(t_k\right)\parallel }^2,\forall k\ge 0.$$

(17)

Proof. 

Now, by direct computation, we have

$$\begin{array}{cc}\hfill F{\left(t_k\right)}^T{d}_k& =-\parallel F\left(t_k\right){\parallel }^2+\frac{y_{k-1}^{T}F\left(t_k\right)}{{\tilde{y}}_{k-1}^T{d}_{k-1}}F{\left(t_k\right)}^T{d}_{k-1}-\frac{F{\left(t_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}F{\left(t_k\right)}^T{d}_{k-1}+\frac{F{\left(t_k\right)}^T{d}_{k-1}}{{\tilde{y}}_{k-1}^T{d}_{k-1}}F{\left(t_k\right)}^T{y}_{k-1}\hfill \\ \hfill & =-\parallel F\left(t_k\right){\parallel }^2-\frac{{(F{(t_k)}^T{d}_{k-1})}^2}{\parallel d_{k-1}{\parallel }^2}\hfill \\ \hfill & \le -\parallel F\left(t_k\right){\parallel }^2.\hfill \end{array}$$

This completes the proof. □

4. Global Convergence

In this section, we investigate the global convergence property of DF-LSTT algorithm for solving (8). For this purpose, we make the following assumptions.

Assumption 1.

A1. 

The mapping F is Lipschitz continuous, that is, there exists a constant $L>0$ such that

$$\parallel F\left(t\right)-F\left(y\right)\parallel \le L\parallel t-y\parallel ,\forall t,y\in R^n$$

(18)

A2. 

Xiao et al. [11] proved that, for the problem (8), F is monotone. That is,

$${(F\left(t\right)-F\left(y\right))}^T(t-y)\ge 0,\forall t,y\in R^n.$$

(19)

Lemma 2.

Suppose that Assumption 1 holds. Let $\left\{u_k\right\}$ and $\left\{t_k\right\}$ be sequences generated by (11) and (12) in the DF-LSTT algorithm. Then, the following statements hold:

1. 

$\left\{t_k\right\}$ and $\left\{u_k\right\}$ are bounded.

2. 

${lim}_{k\to \infty }\parallel u_k-t_k\parallel =0.$

3. 

${lim}_{k\to \infty }\parallel t_{k+1}-t_k\parallel =0.$

Proof. 

Since F is monotone, then, for any solution $t_{\ast }$ of problem (8), we have

$$\begin{array}{cc}\hfill F{\left(u_k\right)}^T(t_k-t_{\ast })& =F{\left(u_k\right)}^T(t_k-u_k)+F{\left(u_k\right)}^T(u_k-t_{\ast })\hfill \\ \hfill & \ge F{\left(u_k\right)}^T(t_k-u_k)+F{\left(t_{\ast }\right)}^T(u_k-t_{\ast })\hfill \\ \hfill & =F{\left(u_k\right)}^T(t_k-u_k)\hfill \\ \hfill & \ge \varsigma {\alpha }_k^2·{\parallel d_k\parallel }^2\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & =\varsigma \parallel t_k-u_k{\parallel }^2.\hfill \end{array}$$

(21)

Note that Equation (20) is obtained from the line search. From (5), we get

$$\begin{array}{cc}\hfill \parallel t_{k+1}-t_{\ast }{\parallel }^2& = \parallel P_S[t_k-{\delta }_k\xi F\left(u_k\right)]-t_{\ast }{\parallel }^2\hfill \\ \hfill & \le \parallel t_k-{\delta }_k\xi F\left(u_k\right)-t_{\ast }{\parallel }^2\hfill \\ \hfill & = \parallel t_k-t_{\ast }{\parallel }^2-2{\delta }_k\xi F{\left(u_k\right)}^T(t_k-t_{\ast })+{\delta }_k^2{\xi }^2{\parallel F\left(u_k\right)\parallel }^2\hfill \\ \hfill & \le \parallel t_k-t_{\ast }{\parallel }^2-2{\delta }_k\xi F{\left(u_k\right)}^T(t_k-u_k)+{\delta }_k^2{\xi }^2{\parallel F\left(u_k\right)\parallel }^2\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & = \parallel t_k-t_{\ast }{\parallel }^2-\xi (2-\xi )\frac{{(F{(u_k)}^T(t_k-u_k))}^2}{\parallel F(u_k){\parallel }^2}\hfill \\ \hfill & \le \parallel t_k-t_{\ast }{\parallel }^2-\xi (2-\xi )\frac{{\varsigma }^2{\parallel t_k-u_k\parallel }^4}{\parallel F\left(u_k\right){\parallel }^2}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & \le \parallel t_k-t_{\ast }{\parallel }^2\hfill \end{array}$$

(24)

where (22) and (23) follow from (21).

From (24), it is easy to see that the sequence $\left\{t_k\right\}$ is bounded. That is,

$$\parallel t_k\parallel \le \gamma ,\forall k\ge 0.$$

(25)

Furthermore, by (18), we have

$$\parallel F\left(t_k\right)\parallel =\parallel F\left(t_k\right)-F\left(t_{\ast }\right)\parallel \le L\parallel t_k-t_{\ast }\parallel \le L\parallel t_0-t_{\ast }\parallel .$$

(26)

Letting $M=L\parallel t_0-t_{\ast }\parallel$, then

$$\parallel F\left(t_k\right)\parallel \le M,\forall k\ge 0$$

(27)

By the monotonicity of property given in (19), we know that

$${(F\left(t_k\right)-F\left(u_k\right))}^T(t_k-u_k)\ge 0.$$

Therefore, by Cauchy–Schwarz inequality, we have

$$\parallel F\left(t_k\right)\parallel \parallel t_k-u_k\parallel \ge F{\left(t_k\right)}^T(t_k-u_k)\ge F{\left(u_k\right)}^T(t_k-u_k)\ge \varsigma {\parallel t_k-u_k\parallel }^2,$$

where the last inequality can be implied from (21). Thus, it is easy to obtain that

$$\varsigma \parallel t_k-u_k\parallel \le \parallel F\left(t_k\right)\parallel \le M,\forall k\ge 0$$

which implies that $\left\{u_k\right\}$ is bounded. Using the continuity of F, we know that there exists a constant $M_1>0$, such that

$$\parallel F\left(u_k\right)\parallel \le M_1\forall k\ge 0.$$

(28)

It follows from (23) that

$$\xi (2-\xi )\frac{{\varsigma }^2{\parallel t_k-u_k\parallel }^4}{\parallel F\left(u_k\right){\parallel }^2}\le \parallel t_k-t_{\ast }{\parallel }^2-{\parallel t_{k+1}-t_{\ast }\parallel }^2.$$

(29)

Adding (29) for $k\ge 0$, we obtain

$$\xi (2-\xi )\frac{{\varsigma }^2}{M_1^2}\sum _{k=0}^{\infty }\parallel t_k-u_k{\parallel }^4\le \sum _{k=0}^{\infty }(\parallel t_k-t_{\ast }{\parallel }^2-\parallel t_{k+1}-t_{\ast }{\parallel }^2)\le \parallel t_0-t_{\ast }{\parallel }^2<\infty$$

(30)

Equation (30) implies that

$$\underset{k\to \infty }{lim}\parallel t_k-u_k\parallel =0.$$

(31)

Hence, the second assertion holds.

From (5), we have

$$\parallel t_{k+1}-t_k\parallel =\parallel P_S[t_k-{\delta }_{k}F\left(u_k\right)]-t_k\parallel \le \parallel {\delta }_k\xi F\left(u_k\right)\parallel ,$$

then by (13) and Cauchy–Schwartz inequality, we obtain

$$\parallel t_{k+1}-t_k\parallel \le \xi \parallel u_k-t_k\parallel ,$$

which shows that the third assertion holds. □

Theorem 1.

Suppose that Assumption 1 holds. Let the sequence $u_k$ and $t_k$ be generated by (11) and (12) in the DF-LSTT algorithm. Then,

$$\underset{k\to \infty }{lim\; inf}\parallel F\left(t_k\right)\parallel =0.$$

(32)

Proof. 

Suppose that conclusion (32) does not hold, that is, there exists ${\kappa }_0\ge 0$ such that

$$\parallel F\left(t_k\right)\parallel \ge {\kappa }_0,\forall k\ge 0.$$

(33)

Equation (33) together with (17) imply that

$$\parallel d_k\parallel \ge \phi ,\forall k\ge 0.$$

We note that the sequences $\left\{t_k\right\},$ $\left\{u_k\right\},$ $\left\{F\left(t_k\right)\right\}$, and $\left\{F\left(u_k\right)\right\}$ are bounded from (25), (31), (27), and (28), respectively. In addition, from the Lipchitz continuity and (25), we have

$$\begin{array}{cc}\hfill \parallel y_{k-1}\parallel & =\parallel F\left(t_k\right)-F\left(t_{k-1}\right)\parallel \hfill \\ \hfill & \le L\parallel t_k-t_{k-1}\parallel \hfill \\ \hfill & =L(\parallel t_k\parallel +\parallel t_{k-1}\parallel )\hfill \\ \hfill & \le 2L\gamma .\hfill \end{array}$$

On the other hand, from the definition of ${\tilde{y}}_{k-1},$ it holds that

$${\tilde{y}}_{k-1}^T{d}_{k-1}\ge y_{k-1}^T{d}_{k-1}+\parallel d_{k-1}{\parallel }^2-y_k^T{d}_{k-1}={\parallel d_{k-1}\parallel }^2.$$

(34)

Thus, by Cauchy–Schwartz inequality, we obtain

$$\begin{array}{cc}\hfill |{\beta }_k\left(N_k^{\ast }\right)|& \le \frac{|y_{k-1}^{T}F\left(t_k\right)|}{{\tilde{y}}_{k-1}^T{d}_{k-1}}+\frac{|F{\left(t_k\right)}^T{d}_{k-1}|}{\parallel d_{k-1}{\parallel }^2}\le \frac{\parallel y_{k-1}\parallel \parallel F\left(t_k\right)\parallel }{\parallel d_{k-1}{\parallel }^2}+\frac{\parallel F\left(t_k\right)\parallel }{\parallel d_{k-1}\parallel }\hfill \\ \hfill & \le \frac{2LM\gamma }{{\phi }^2}+\frac{M}{\phi }\hfill \\ \hfill & =\frac{M}{\phi }\left(\frac{2L\gamma }{\phi }+1\right)\hfill \end{array}$$

(35)

Note that the last inequality in (35) can be easily obtained using (34).

Therefore, from (14), it follows that

$$\begin{array}{cc}\hfill \parallel d_k\parallel & \le \parallel F\left(t_k\right)\parallel +|{\beta }_k\left(N_k^{\ast }\right)|\parallel d_{k-1}\parallel +\frac{\parallel F\left(t_k\right)\parallel \parallel d_{k-1}\parallel }{\parallel d_{k-1}{\parallel }^2}\parallel y_{k-1}\parallel \hfill \\ \hfill & \le M+\left(\frac{2ML\gamma }{\phi }+M\right)+\frac{2ML\gamma }{\phi }\hfill \\ \hfill & =M+\frac{\phi M+4ML\gamma }{\phi }\triangleq \chi \hfill \end{array}$$

From (22), ${\alpha }_k{\varrho }^{-1}$ does not satisfy (10). Thus, we have

$$-F{(t_k+{\alpha }_k{\varrho }^{-1}{d}_k)}^T{d}_k<\varsigma {\alpha }_k{\varrho }^{-1}{\parallel d_k\parallel }^2.$$

It follows from Lemma 1 that

$$\begin{array}{cc}\hfill \parallel F(t_k){\parallel }^2& \le -F{(t_k)}^T{d}_k\hfill \\ \hfill & \le {(F(t_k+{\alpha }_k{\varrho }^{-1}{d}_k)-F(t_k))}^T{d}_k-F{(t_k+{\alpha }_k{\varrho }^{-1}{d}_k)}^T{d}_k\hfill \\ \hfill & \le L{\alpha }_k{\varrho }^{-1}\parallel d_k{\parallel }^2+\varsigma {\alpha }_k{\varrho }^{-1}{\parallel d_k\parallel }^2\hfill \\ \hfill & \le {\alpha }_k{\varrho }^{-1}(L+\varsigma ){\parallel d_k\parallel }^2.\hfill \end{array}$$

(36)

Equation (36) is obtained by using Cauchy–Schwartz inequality and Lipchitz continuity (18). Therefore, it holds that

$${\alpha }_k{\parallel d_k\parallel }^2\ge \frac{\varrho \parallel F\left(t_k\right){\parallel }^2}{(L+\varsigma )}\ge \frac{\varrho {\kappa }_0^2}{(L+\varsigma )}.$$

Note, from (33), the last inequality is obtained which implies that

$$\underset{k\to \infty }{lim}\parallel t_k-u_k\parallel >0$$

holds, which contradicts (31). Thus, (32) holds. □

5. Numerical Experiment

We present numerical experiments to show the efficiency of the DF-LSTT method. The experiments presented here are of two types. The first experiment involves applying the DF-LSTT method to solve the ${\ell }_1$-norm regularization problem arising in compressive sensing. The second involves testing DF-LSTT in solving some given convex constrained nonlinear equations with different initial points and various dimensions. The implementations were performed using Matlab R2019b Update 1 (9.7.0.1216025, Mathwork Inc, Massachusetts, USA) on a HP PC (Hewlett-Packard, California, USA) with CPU 2.4 GHz, 8.0 GB RAM with the Windows 10 operating system.

5.1. Experiments on the ${\ell }_1$-Norm Regularization Problem in Compressive Sensing

We begin by considering a typical compressive sensing scenario where the ultimate goal is to reconstruct a length-n sparse signal from m observations ($m<<n$) with Gaussian noise, where the number of samples is dramatically smaller than the size of the original signal. We compare DF-LSTT with the CGD conjugate gradient method [12] and the PCG projection method [30] designed to solve nonlinear equations with convex constrained and signal recovery problems.

As a consequence of the limited memory of our PC, in this test, we considered a small size of signal with $n=2^{11}$, $m=2^9$ and the original t contains $2^6$ randomly non-zero elements. Similar to [11,12,31], the quality of the restored signal is assessed by the mean of squared error (MSE) to the original signal t, that is,

$$MSE:=\frac{1}{n}{\parallel t-t^{\ast }\parallel }^2$$

where $t^{\ast }$ is the restored signal. In this test, we generate the random matrix A using the Matlab command rand(n,k). In addition, noise is appropriately added to the observed data computed by

$$b=At+\eta$$

(37)

where $\eta$ is the Gaussian noise with $N(0,{10}^{-4}).$ The DF-LSTT algorithm is implemented with the following parameters:$\beta =10,\varrho =0.55,\varsigma ={10}^{-4}.$

For the compared methods, their parameters are set as reported in their respective papers. In line with [32], we chose the parameter $\tau$ in the merit function $f\left(t\right)=\tau \parallel t\parallel +\frac{1}{2}{\parallel b-At\parallel }^2$ as $\tau =0.008\parallel A^T{b\parallel }_{\infty }$ and the initial point for all algorithms also starts at $t_0=A^{T}b$. The process terminates when

$$Tol:=\frac{\parallel f_k-f_{k-1}\parallel }{\parallel f_{k-1}\parallel }<{10}^{-5},$$

where $f_k$ denotes the function value at $t_k.$ Note, for this test, we only observe the convergence behavior of each method to obtain a similar accuracy solution.

In view of the plots depicted in Figure 1, DF-LSTT wins in decoding sparse signals in compressive sensing in terms of number of iterations for around 116 and around 0.69s. Table 1 contains the number of iterations, mean of Squared error (MSE), and time of the decoding of sparse signal for over 10 runs for the algorithms tested. The reported results in Figure 2 shows that, in decoding sparse signal in compressive sensing, DF-LSTT is faster than CGD and PCG with a clear lowest number of iterations.

Figure 1. Illustration of the sparse signal recovery. From the top to the bottom is the original signal (First plot), the measurement (Second plot), and the reconstructed signals by DF-LSTT (Third plot), CGD (Fourth plot), and PCG (Fifth plot).

Table 1. Results for sparse signal recovery.

	DF-LSTT			CGD			PCG
	ITER	MSE	TIME	ITER	MSE	Time	ITER	MSE	Time
	137	7.39 $\times {10}^{-6}$	1.31	259	4.51 $\times {10}^{-5}$	1.7	184	1.50 $\times {10}^{-5}$	0.88
	88	9.18 $\times {10}^{-6}$	0.73	187	7.04 $\times {10}^{-5}$	1.38	144	1.63 $\times {10}^{-5}$	1.42
	90	1.40 $\times {10}^{-5}$	0.66	231	3.78 $\times {10}^{-5}$	1.73	144	1.99 $\times {10}^{-5}$	1.61
	86	1.26 $\times {10}^{-5}$	0.78	228	2.15 $\times {10}^{-5}$	1.41	82	5.32 $\times {10}^{-4}$	0.78
	97	1.22 $\times {10}^{-5}$	0.7	245	6.75 $\times {10}^{-5}$	1.55	118	5.99 $\times {10}^{-5}$	0.78
	87	9.72 $\times {10}^{-6}$	0.64	199	5.13 $\times {10}^{-5}$	1.33	82	4.50 $\times {10}^{-4}$	0.83
	115	5.39 $\times {10}^{-6}$	0.84	211	3.67 $\times {10}^{-5}$	1.64	152	9.68 $\times {10}^{-6}$	0.97
	89	1.31 $\times {10}^{-5}$	1.27	158	1.56 $\times {10}^{-4}$	3.14	150	1.16 $\times {10}^{-5}$	1.59
	105	1.35 $\times {10}^{-5}$	0.63	280	4.82 $\times {10}^{-5}$	1.89	152	2.23 $\times {10}^{-5}$	0.86
	97	5.22 $\times {10}^{-6}$	0.63	228	3.89 $\times {10}^{-5}$	1.45	154	8.00 $\times {10}^{-6}$	2.3
Average	99.1	1.02 $\times {10}^{-5}$	0.819	222.6	5.73 $\times {10}^{-5}$	1.722	136.2	1.14 $\times {10}^{-4}$	1.202

Figure 2. Comparison results of the DF-LSTT, CGD, and PCG algorithms with the signal parameter chosen as $n=2048,m=512,\tau =0.00396321.$ The x-axes represent the number of iterations (top left and bottom left) and the CPU time in seconds (top right and bottom right). The y-axes represent the MSE (top left and top right) and the function values (bottom left and right).

Next, the effectiveness and robustness of DF-LSTT algorithm is illustrated in an image de-blurring problem. We carried out our experiment using some widely used test images obtained from http://hlevkin.com/06testimages.htm. DF-LSTT is compared with the state-of-art methods CGD proposed by Xiaoh and Zhu [12], SGCS [11], and MFR [33].

The performance evaluation criteria to measure the quality of restoration by the methods are measured by the signal-to-noise ratio (SNR) defined as

$$SNR:=20\times {log}_{10}\left(\frac{\parallel t\parallel }{\parallel t^{\ast }-t\parallel }\right),$$

the peak signal-to-noise ratio (PSNR) [34], and the structural similarity index (SSIM) [35]. For fairness in comparing the algorithms, the iteration process of all algorithms started at $t_0=A^{T}b$ and terminates when $Tol<{10}^{-5}$. For the image de-blurring experiment, the following parameters were used in our implementation: $\beta =0.001,\varrho =0.6,\varsigma ={10}^{-4}$. We tested several images including Tiffany $(512\times 512)$, Lena $(512\times 512)$, Barbara $(720\times 576),$ degraded by Gaussian blur and 10% Gaussian noise.

In Table 2, we report the performance for SNR, PSNR, and SSIM for DF-LSTT, CGD, SGCS, and the MFRM method in recovery blurred and noisy images. We can see that the SNR, PSNR, and SSIM of the test images calculated by the DF-LSTT algorithm are a bit higher than CGD, SGCS, and MFRM. The higher value of SNR, PSNR, and SSIM reflects better quality of restoration.

Table 2. Numerical results of DF-LSTT, CGD, SGCS, and MFRM methods in image restoration.

	DF-LSTT			CGD			SGCS			MFRM
Image	SNR	PSNR	SSIM	SNR	PSNR	SSIM	SNR	PSNR	SSIM	SNR	PSNR	SSIM
Tiffany	21.25	23.08	0.9204	21.20	23.04	0.9193	21.24	23.07	0.9202	20.87	22.70	0.9128
Lenna	16.98	22.31	0.9176	16.93	22.26	0.9166	16.96	22.29	0.9173	16.60	21.94	0.9104
Barbara	13.81	20.23	0.6377	13.77	20.19	0.6355	13.80	20.22	0.6373	13.57	19.99	0.6231
Average	17.35	21.87	0.8252	17.30	21.83	0.8238	17.33	21.86	0.8249	17.01	21.54	0.8154

Based on the performance reported in Table 2, the DF-LSTT algorithm restores blurred and noisy images quite well and obtain better quality reconstructed images in an efficient manner. Figure 3 and Figure 4 show the original and blurred images while Figure 5 shows the restored images by each method.

Figure 3. The original test images: From (left), Tiffany, Lenna (middle), and Barbara (right).

Figure 4. Blurred and noisy Barbara and Lenna test images.

Figure 5. Restored images by DF-LSTT (left column), CGD (left middle column), SGCS (right middle column), and MFRM (right column).

5.2. Experiments on Some Large-Scaled Monotone Nonlinear Equations

In this subsection, we evaluate the performance of the proposed conjugate gradient method in solving nonlinear equations with convex constraints. We compare the proposed method with CGD [12] and PCG [30]. For each test problem, the stopping condition employed is

$$\parallel F\left(t_k\right)\parallel \le {10}^{-6}.$$

(38)

We also stop the algorithms when the iterations exceed 1000 without achieving convergence. The algorithms are tested using seven different initial points, one of which is randomly generated in $R^n$. We ran the algorithms for several dimensions ranging from $n=$ 1000 to 100,000. The parameters chosen for the proposed algorithm are; $\varrho =0.75$, $\beta =1$, $\varsigma ={10}^{-4}$, $\xi =1.2$. For the CGD and PCG algorithm, all parameters are chosen as in [12,30], respectively. We use the following well-known benchmark test functions where the mapping F is taken as

$$F\left(t\right)={(f_1\left(t\right),f_2\left(t\right),\cdots ,f_n\left(t\right))}^T,$$

where the associated initial points for these problems are

$$t_1={(0.1,0.1,\cdots ,0.1)}^T,t_2={(0.2,0.2,\cdots ,0.2)}^T,t_3={(0.5,0.5,\cdots ,0.5)}^T,t_4={(1.2,1.2,\cdots ,1.2)}^T,$$

$$t_5={(1.5,1.5,\cdots 1.5)}^T,t_6={(2,2,\cdots ,2)}^T,t_7=rand(n,1).$$

Problem 1.

This problem is the Exponential function [36] with constraint set $S=R_{+}^n$, that is,

$$\begin{array}{c}{f}_1\left(t\right)=e^{t_1}-1,\hfill \\ f_i\left(t\right)=e^{t_i}+t_i-1,\mathrm{for}i=2,3,\dots ,n.\hfill \end{array}$$

Problem 2.

Modified Logarithmic function [36] with constraint set $S=\{t\in R^n:{\sum }_{i=1}^n{t}_i\le n,t_i>-1,i=1,2,\dots ,n\},$ that is,

$$\begin{array}{c}\hfill f_i\left(t\right)=ln(t_i+1)-\frac{t_i}{n},i=2,3,\dots ,n.\end{array}$$

Problem 3.

The function $f_i\left(x\right)$ [37] with $S=R_{+}^n$ defined by

$$f_i\left(t\right)=min\left(min\left(\right|t_i|,t_i^2),max(|t_i|,t_i^3)\right)\mathrm{for}i=2,3,\dots ,n.$$

Problem 4.

The Strictly convex function [38], with constraint set $S=R_{+}^n,$ that is,

$$f_i\left(t\right)=e^{t_i}-1,i=2,3,\cdots ,n.$$

Problem 5.

Strictly convex function II [38], with constraint set $S=R_{+}^n,$ that is,

$$f_i\left(x\right)=\frac{1}{n}{e}^{t_i}-1,i=2,3,\cdots ,n.$$

Problem 6.

Tridiagonal Exponential function [39] with constraint set $S=R_{+}^n,$ that is,

$$\begin{array}{cc}\hfill & f_1\left(t\right)=t_1-e^{cos\left(h(t_1+t_2)\right)},\hfill \\ \hfill & f_i\left(t\right)=t_i-e^{cos\left(h(t_{i-1}+t_i+t_{i+1})\right)},\mathit{for}2\le i\le n-1,\hfill \\ \hfill & f_n\left(t\right)=t_n-e^{cos\left(h(t_{n-1}+t_n)\right)},\mathit{where}h=\frac{1}{n+1}\hfill \end{array}$$

Problem 7.

Nonsmooth function [40] with constraint set $S=\{t\in R^n:{\sum }_{i=1}^n{t}_i\le n,t_i\ge -1,1\le i\le n\}:$

$$f_i\left(t\right)=t_i-sin|t_i-1|,i=2,3,\cdots ,n.$$

Problem 8.

The function $f_i\left(t\right)$ with $S=R_{+}^n$ defined by

$$f_i\left(t\right)=8^{\frac{1}{2}}{t}_i-1i=1,2,3,\dots ,n.$$

In Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9 and Table 10, we report the computation results obtained from the implementation of DF-LSTT, CGD, and PCG algorithm. We report the number of iterations (ITER), the number of function evaluations (FVAL), and the CPU time in seconds (TIME).

Table 3. Numerical test reports for the three tested methods for Problem 1.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	2	7	0.011683	0.00 $\times {10}^0$	42	125	0.032606	9.97 $\times {10}^{-7}$	18	71	0.020567	5.72 $\times {10}^{-6}$
	$t_2$	2	7	0.007157	0.00 $\times {10}^0$	45	134	0.020408	9.45 $\times {10}^{-7}$	18	71	0.017477	9.82 $\times {10}^{-6}$
	$t_3$	2	7	0.059349	0.00 $\times {10}^0$	48	143	0.025053	9.82 $\times {10}^{-7}$	19	75	0.010401	7.10 $\times {10}^{-6}$
	$t_4$	2	7	0.010097	0.00 $\times {10}^0$	50	149	0.020462	9.70 $\times {10}^{-7}$	18	71	0.014602	8.27 $\times {10}^{-6}$
	$t_5$	31	124	0.075653	7.60 $\times {10}^{-7}$	51	152	0.022198	8.17 $\times {10}^{-7}$	63	251	0.035587	9.58 $\times {10}^{-6}$
	$t_6$	14	55	0.041615	0.00 $\times {10}^0$	51	152	0.017499	8.56 $\times {10}^{-7}$	61	243	0.070899	9.15 $\times {10}^{-6}$
	$t_7$	44	175	0.052359	1.83 $\times {10}^{-12}$	38	113	0.013579	9.14 $\times {10}^{-7}$	18	71	0.009228	9.24 $\times {10}^{-6}$
5000	$t_1$	2	7	0.023049	0.00 $\times {10}^0$	41	122	0.045814	8.34 $\times {10}^{-7}$	18	71	0.037294	7.42 $\times {10}^{-6}$
	$t_2$	2	7	0.013178	0.00 $\times {10}^0$	43	128	0.051819	9.81 $\times {10}^{-7}$	19	75	0.030883	6.53 $\times {10}^{-6}$
	$t_3$	2	7	0.008287	0.00 $\times {10}^0$	47	140	0.056948	8.05 $\times {10}^{-7}$	20	79	0.040342	5.20 $\times {10}^{-6}$
	$t_4$	2	7	0.008819	0.00 $\times {10}^0$	48	143	0.060879	9.93 $\times {10}^{-7}$	19	75	0.03691	8.10 $\times {10}^{-6}$
	$t_5$	28	112	0.14831	6.87 $\times {10}^{-7}$	49	146	0.053006	8.36 $\times {10}^{-7}$	62	247	0.0775	9.53 $\times {10}^{-6}$
	$t_6$	28	112	0.17661	3.38 $\times {10}^{-7}$	49	146	0.061534	8.76 $\times {10}^{-7}$	60	239	0.081647	9.10 $\times {10}^{-6}$
	$t_7$	19	75	0.10719	0.00 $\times {10}^0$	40	119	0.052358	7.70 $\times {10}^{-7}$	19	75	0.042343	9.16 $\times {10}^{-6}$
10,000	$t_1$	2	7	0.017941	0.00 $\times {10}^0$	40	119	0.082014	8.97 $\times {10}^{-7}$	18	71	0.055224	9.50 $\times {10}^{-6}$
	$t_2$	2	7	0.025597	0.00 $\times {10}^0$	43	128	0.074301	8.26 $\times {10}^{-7}$	19	75	0.050823	8.15 $\times {10}^{-6}$
	$t_3$	2	7	0.015978	0.00 $\times {10}^0$	46	137	0.096062	8.46 $\times {10}^{-7}$	20	79	0.046486	6.74 $\times {10}^{-6}$
	$t_4$	2	7	0.24659	0.00 $\times {10}^0$	48	143	0.097356	8.30 $\times {10}^{-7}$	20	79	0.05297	5.11 $\times {10}^{-6}$
	$t_5$	39	156	0.86102	4.62 $\times {10}^{-7}$	48	143	0.11071	8.75 $\times {10}^{-7}$	62	247	0.19236	8.87 $\times {10}^{-6}$
	$t_6$	2	7	0.023214	0.00 $\times {10}^0$	48	143	0.087449	9.17 $\times {10}^{-7}$	59	235	0.16649	9.96 $\times {10}^{-6}$
	$t_7$	11	43	0.089553	0.00 $\times {10}^0$	37	110	0.072561	7.58 $\times {10}^{-7}$	20	79	0.055156	5.82 $\times {10}^{-6}$
50,000	$t_1$	2	7	0.046111	0.00 $\times {10}^0$	39	116	0.29127	8.43 $\times {10}^{-7}$	19	75	0.21501	8.80 $\times {10}^{-6}$
	$t_2$	2	7	0.088396	0.00 $\times {10}^0$	41	122	0.32975	9.37 $\times {10}^{-7}$	20	79	0.27646	7.39 $\times {10}^{-6}$
	$t_3$	2	7	0.050768	0.00 $\times {10}^0$	44	131	0.36349	9.16 $\times {10}^{-7}$	21	83	0.28492	6.31 $\times {10}^{-6}$
	$t_4$	2	7	0.052953	0.00 $\times {10}^0$	46	137	0.34443	8.84 $\times {10}^{-7}$	21	83	0.21931	5.10 $\times {10}^{-6}$
	$t_5$	34	136	1.336	2.22 $\times {10}^{-7}$	46	137	0.41225	9.34 $\times {10}^{-7}$	61	243	0.5975	8.85 $\times {10}^{-6}$
	$t_6$	2	7	0.10667	0.00 $\times {10}^0$	46	137	0.44581	9.78 $\times {10}^{-7}$	59	235	0.59777	8.50 $\times {10}^{-6}$
	$t_7$	64	256	2.2443	4.32 $\times {10}^{-13}$	46	137	0.44419	8.86 $\times {10}^{-7}$	21	83	0.2052	5.79 $\times {10}^{-6}$
100,000	$t_1$	2	7	0.092134	0.00 $\times {10}^0$	39	116	0.55718	7.72 $\times {10}^{-7}$	20	79	0.37816	5.52 $\times {10}^{-6}$
	$t_2$	2	7	0.14977	0.00 $\times {10}^0$	41	122	0.57738	8.33 $\times {10}^{-7}$	21	83	0.45721	4.62 $\times {10}^{-6}$
	$t_3$	2	7	0.093678	0.00 $\times {10}^0$	44	131	0.62707	7.92 $\times {10}^{-7}$	21	83	0.54042	8.78 $\times {10}^{-6}$
	$t_4$	2	7	0.10506	0.00 $\times {10}^0$	45	134	0.63408	9.66 $\times {10}^{-7}$	21	83	0.53141	7.21 $\times {10}^{-6}$
	$t_5$	39	156	9.4738	9.14 $\times {10}^{-7}$	46	137	0.66002	7.99 $\times {10}^{-7}$	60	239	1.1271	9.73 $\times {10}^{-6}$
	$t_6$	2	7	0.32882	0.00 $\times {10}^0$	46	137	0.68921	8.38 $\times {10}^{-7}$	58	231	1.1519	9.42 $\times {10}^{-6}$
	$t_7$	61	244	13.7075	7.48 $\times {10}^{-7}$	41	122	0.58348	9.17 $\times {10}^{-7}$	21	83	0.38529	8.20 $\times {10}^{-6}$

Table 4. Numerical test reports for the three tested methods for Problem 2.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	6	23	0.018184	1.73 $\times {10}^{-7}$	55	163	0.032169	8.99 $\times {10}^{-7}$	15	58	0.011849	8.59 $\times {10}^{-6}$
	$t_2$	6	23	0.014572	2.51 $\times {10}^{-7}$	61	181	0.0357	8.88 $\times {10}^{-7}$	11	41	0.010006	9.07 $\times {10}^{-6}$
	$t_3$	6	23	0.019846	5.60 $\times {10}^{-7}$	69	205	0.032634	8.33 $\times {10}^{-7}$	17	65	0.012605	6.44 $\times {10}^{-6}$
	$t_4$	6	23	0.057022	1.19 $\times {10}^{-7}$	76	226	0.028091	9.17 $\times {10}^{-7}$	18	68	0.014979	6.00 $\times {10}^{-6}$
	$t_5$	8	31	0.030773	1.98 $\times {10}^{-7}$	78	232	0.039636	9.18 $\times {10}^{-7}$	13	47	0.009625	7.58 $\times {10}^{-6}$
	$t_6$	7	27	0.014646	2.53 $\times {10}^{-7}$	81	241	0.03899	8.64 $\times {10}^{-7}$	18	67	0.011159	5.40 $\times {10}^{-6}$
	$t_7$	14	55	0.026256	6.83 $\times {10}^{-7}$	72	214	0.032568	8.68 $\times {10}^{-7}$	19	73	0.016356	6.13 $\times {10}^{-6}$
5000	$t_1$	6	23	0.046357	4.66 $\times {10}^{-7}$	59	175	0.08852	8.02 $\times {10}^{-7}$	16	62	0.037321	9.35 $\times {10}^{-6}$
	$t_2$	6	23	0.07737	6.75 $\times {10}^{-7}$	64	190	0.10686	9.97 $\times {10}^{-7}$	12	45	0.030551	8.80 $\times {10}^{-6}$
	$t_3$	7	27	0.068471	2.49 $\times {10}^{-7}$	72	214	0.12192	9.37 $\times {10}^{-7}$	18	69	0.047498	6.98 $\times {10}^{-6}$
	$t_4$	7	27	0.1053	7.27 $\times {10}^{-7}$	80	238	0.12654	8.26 $\times {10}^{-7}$	19	72	0.039088	6.45 $\times {10}^{-6}$
	$t_5$	8	31	0.065362	5.25 $\times {10}^{-7}$	82	244	0.11871	8.26 $\times {10}^{-7}$	14	51	0.03502	6.71 $\times {10}^{-6}$
	$t_6$	7	27	0.062662	7.28 $\times {10}^{-7}$	84	250	0.13106	9.71 $\times {10}^{-7}$	19	71	0.040604	5.71 $\times {10}^{-6}$
	$t_7$	23	91	0.45265	1.77 $\times {10}^{-7}$	75	223	0.16438	9.73 $\times {10}^{-7}$	20	77	0.059424	6.86 $\times {10}^{-6}$
10,000	$t_1$	6	23	0.081017	6.73 $\times {10}^{-7}$	60	178	0.16239	9.04 $\times {10}^{-7}$	17	66	0.075512	6.60 $\times {10}^{-6}$
	$t_2$	6	23	0.15334	9.76 $\times {10}^{-7}$	66	196	0.16906	9.00 $\times {10}^{-7}$	13	49	0.041324	6.11 $\times {10}^{-6}$
	$t_3$	7	27	0.11084	3.61 $\times {10}^{-7}$	74	220	0.19362	8.46 $\times {10}^{-7}$	18	69	0.076734	9.83 $\times {10}^{-6}$
	$t_4$	5	19	0.076488	6.03 $\times {10}^{-7}$	81	241	0.23663	9.32 $\times {10}^{-7}$	19	72	0.064377	9.07 $\times {10}^{-6}$
	$t_5$	8	31	0.25185	7.57 $\times {10}^{-7}$	83	247	0.21784	9.33 $\times {10}^{-7}$	14	51	0.050394	9.18 $\times {10}^{-6}$
	$t_6$	8	31	0.15676	1.66 $\times {10}^{-7}$	86	256	0.23307	8.77 $\times {10}^{-7}$	19	71	0.075462	8.02 $\times {10}^{-6}$
	$t_7$	27	107	1.1061	4.66 $\times {10}^{-7}$	77	229	0.27924	8.81 $\times {10}^{-7}$	20	77	0.088795	9.69 $\times {10}^{-6}$
50,000	$t_1$	7	27	0.5849	2.40 $\times {10}^{-7}$	64	190	0.70165	8.26 $\times {10}^{-7}$	18	70	0.25185	7.37 $\times {10}^{-6}$
	$t_2$	7	27	0.48516	3.47 $\times {10}^{-7}$	70	208	0.75749	8.23 $\times {10}^{-7}$	14	53	0.27213	6.74 $\times {10}^{-6}$
	$t_3$	7	27	1.0522	8.23 $\times {10}^{-7}$	77	229	0.81368	9.67 $\times {10}^{-7}$	20	77	0.31128	5.50 $\times {10}^{-6}$
	$t_4$	7	27	0.34532	2.05 $\times {10}^{-7}$	85	253	0.93153	8.52 $\times {10}^{-7}$	21	80	0.28943	5.07 $\times {10}^{-6}$
	$t_5$	9	35	4.3347	2.69 $\times {10}^{-7}$	87	259	0.94105	8.53 $\times {10}^{-7}$	16	59	0.22366	5.02 $\times {10}^{-6}$
	$t_6$	8	31	0.92387	3.80 $\times {10}^{-7}$	90	268	1.0832	8.02 $\times {10}^{-7}$	20	75	0.27706	8.93 $\times {10}^{-6}$
	$t_7$	20	79	1.8206	8.21 $\times {10}^{-7}$	81	241	1.4008	8.03 $\times {10}^{-7}$	22	85	0.39411	5.41 $\times {10}^{-6}$
100,000	$t_1$	7	27	0.73406	3.39 $\times {10}^{-7}$	65	193	1.3721	9.34 $\times {10}^{-7}$	19	74	0.52829	5.22 $\times {10}^{-6}$
	$t_2$	7	27	0.49843	4.92 $\times {10}^{-7}$	71	211	1.5287	9.30 $\times {10}^{-7}$	14	53	0.37191	9.52 $\times {10}^{-6}$
	$t_3$	8	31	0.75063	1.82 $\times {10}^{-7}$	79	235	1.6725	8.75 $\times {10}^{-7}$	20	77	0.54637	7.78 $\times {10}^{-6}$
	$t_4$	7	27	0.50287	3.21 $\times {10}^{-7}$	86	256	1.8409	9.64 $\times {10}^{-7}$	21	80	0.57391	7.17 $\times {10}^{-6}$
	$t_5$	9	35	0.76034	3.81 $\times {10}^{-7}$	88	262	1.9085	9.64 $\times {10}^{-7}$	16	59	0.52223	7.07 $\times {10}^{-6}$
	$t_6$	8	31	0.48187	5.39 $\times {10}^{-7}$	91	271	1.9642	9.07 $\times {10}^{-7}$	21	79	0.72579	6.32 $\times {10}^{-6}$
	$t_7$	21	83	1.7606	1.06 $\times {10}^{-7}$	82	244	2.5148	9.07 $\times {10}^{-7}$	22	85	0.85567	7.66 $\times {10}^{-6}$

Table 5. Numerical test reports for the three tested methods for Problem 3.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	2	6	0.004359	0	1	2	0.029302	0	1	3	0.006887	0
	$t_2$	2	6	0.004097	0	1	2	0.003218	0	1	3	0.002975	0
	$t_3$	2	6	0.002719	0	1	2	0.002851	0	1	3	0.007665	0
	$t_4$	2	6	0.002236	0	1	3	0.003144	0	1	4	0.007096	0
	$t_5$	2	6	0.002477	0	1	3	0.004593	0	1	4	0.005078	0
	$t_6$	2	6	0.002744	0	1	3	0.003093	0	1	4	0.002427	0
	$t_7$	15	59	0.056123	7.18 $\times {10}^{-7}$	1	2	0.003046	0	1	3	0.006539	0
5000	$t_1$	2	6	0.008889	0	1	2	0.008198	0	1	3	0.008733	0
	$t_2$	2	6	0.012555	0	1	2	0.007947	0	1	3	0.008462	0
	$t_3$	2	6	0.008658	0	1	2	0.007763	0	1	3	0.008092	0
	$t_4$	2	6	0.00612	0	1	3	0.008812	0	1	4	0.009649	0
	$t_5$	2	6	0.005853	0	1	3	0.009794	0	1	4	0.00889	0
	$t_6$	2	6	0.006947	0	1	3	0.007755	0	1	4	0.007802	0
	$t_7$	18	71	0.31812	7.95 $\times {10}^{-7}$	1	2	0.010464	0	1	3	0.008169	0
10,000	$t_1$	2	6	0.015963	0	1	2	0.012685	0	1	3	0.012387	0
	$t_2$	2	6	0.016001	0	1	2	0.011649	0	1	3	0.01044	0
	$t_3$	2	6	0.015956	0	1	2	0.010229	0	1	3	0.010306	0
	$t_4$	2	6	0.018197	0	1	3	0.011267	0	1	4	0.01704	0
	$t_5$	2	6	0.011355	0	1	3	0.01182	0	1	4	0.010001	0
	$t_6$	2	6	0.018842	0	1	3	0.012149	0	1	4	0.012029	0
	$t_7$	18	71	0.38333	8.40 $\times {10}^{-7}$	1	2	0.010945	0	1	3	0.011036	0
50,000	$t_1$	2	6	0.069633	0	1	2	0.038998	0	1	3	0.038391	0
	$t_2$	2	6	0.058314	0	1	2	0.040189	0	1	3	0.04888	0
	$t_3$	2	6	0.095231	0	1	2	0.038922	0	1	3	0.036443	0
	$t_4$	2	6	0.046998	0	1	3	0.040942	0	1	4	0.043418	0
	$t_5$	2	6	0.044827	0	1	3	0.053045	0	1	4	0.040382	0
	$t_6$	2	6	0.091957	0	1	3	0.042239	0	1	4	0.041188	0
	$t_7$	18	71	1.2966	5.49 $\times {10}^{-7}$	1	2	0.046987	0	1	3	0.040932	0
100,000	$t_1$	2	6	0.11838	0	1	2	0.090727	0	1	3	0.077927	0
	$t_2$	2	6	0.11658	0	1	2	0.077024	0	1	3	0.074788	0
	$t_3$	2	6	0.12338	0	1	2	0.10601	0	1	3	0.082006	0
	$t_4$	2	6	0.16886	0	1	3	0.085545	0	1	4	0.08095	0
	$t_5$	2	6	0.086942	0	1	3	0.090827	0	1	4	0.12206	0
	$t_6$	2	6	0.11021	0	1	3	0.080531	0	1	4	0.080564	0
	$t_7$	20	79	3.0379	3.93 $\times {10}^{-7}$	1	2	0.078351	0	1	3	0.07851	0

Table 6. Numerical test reports for the three tested methods for Problem 4.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	3	11	0.004591	0	68	203	0.029809	8.60 $\times {10}^{-7}$	18	71	0.010585	9.93 $\times {10}^{-6}$
	$t_2$	3	11	0.002489	0	71	212	0.036826	8.29 $\times {10}^{-7}$	19	75	0.007156	8.75 $\times {10}^{-6}$
	$t_3$	2	7	0.002095	0	74	221	0.028185	8.89 $\times {10}^{-7}$	20	79	0.009969	7.15 $\times {10}^{-6}$
	$t_4$	3	11	0.002705	0	76	227	0.034306	9.28 $\times {10}^{-7}$	47	187	0.017485	7.83 $\times {10}^{-6}$
	$t_5$	2	7	0.002888	0	76	227	0.028661	9.95 $\times {10}^{-7}$	46	183	0.019829	9.76 $\times {10}^{-6}$
	$t_6$	3	11	0.009778	0	77	230	0.035383	8.34 $\times {10}^{-7}$	41	163	0.015464	8.77 $\times {10}^{-6}$
	$t_7$	57	228	0.088381	7.44 $\times {10}^{-7}$	74	221	0.027205	8.89 $\times {10}^{-7}$	20	79	0.010696	6.77 $\times {10}^{-6}$
5000	$t_1$	3	11	0.01231	0	71	212	0.077834	9.85 $\times {10}^{-7}$	20	79	0.024704	5.57 $\times {10}^{-6}$
	$t_2$	3	11	0.009414	0	74	221	0.065757	9.49 $\times {10}^{-7}$	20	79	0.029764	9.80 $\times {10}^{-6}$
	$t_3$	2	7	0.007662	0	78	233	0.10357	8.14 $\times {10}^{-7}$	21	83	0.026368	8.01 $\times {10}^{-6}$
	$t_4$	3	11	0.016453	0	80	239	0.086417	8.50 $\times {10}^{-7}$	49	195	0.057593	9.46 $\times {10}^{-6}$
	$t_5$	2	7	0.007345	0	80	239	0.07626	9.11 $\times {10}^{-7}$	49	195	0.067517	8.68 $\times {10}^{-6}$
	$t_6$	3	11	0.011993	0	80	239	0.076816	9.55 $\times {10}^{-7}$	44	175	0.07587	7.79 $\times {10}^{-6}$
	$t_7$	45	180	0.38386	2.35 $\times {10}^{-7}$	78	233	0.066987	8.20 $\times {10}^{-7}$	21	83	0.037799	7.86 $\times {10}^{-6}$
10,000	$t_1$	3	11	0.011882	0	73	218	0.12965	8.91 $\times {10}^{-7}$	20	79	0.076194	7.88 $\times {10}^{-6}$
	$t_2$	3	11	0.029407	0	76	227	0.11801	8.59 $\times {10}^{-7}$	21	83	0.053778	6.94 $\times {10}^{-6}$
	$t_3$	2	7	0.016882	0	79	236	0.13374	9.21 $\times {10}^{-7}$	22	87	0.046575	5.67 $\times {10}^{-6}$
	$t_4$	3	11	0.018218	0	81	242	0.19064	9.62 $\times {10}^{-7}$	50	199	0.11588	9.84 $\times {10}^{-6}$
	$t_5$	2	7	0.012436	0	82	245	0.12664	8.25 $\times {10}^{-7}$	50	199	0.10082	9.03 $\times {10}^{-6}$
	$t_6$	3	11	0.049629	0	82	245	0.12031	8.64 $\times {10}^{-7}$	45	179	0.088501	8.11 $\times {10}^{-6}$
	$t_7$	45	180	0.26301	3.61 $\times {10}^{-7}$	79	236	0.13346	9.24 $\times {10}^{-7}$	22	87	0.075082	5.55 $\times {10}^{-6}$
50,000	$t_1$	3	11	0.059552	0	77	230	0.53026	8.16 $\times {10}^{-7}$	21	83	0.17253	8.83 $\times {10}^{-6}$
	$t_2$	3	11	0.05224	0	79	236	0.51796	9.84 $\times {10}^{-7}$	22	87	0.16907	7.78 $\times {10}^{-6}$
	$t_3$	2	7	0.031386	0	83	248	0.55337	8.44 $\times {10}^{-7}$	23	91	0.18569	6.36 $\times {10}^{-6}$
	$t_4$	3	11	0.089099	0	85	254	0.56858	8.81 $\times {10}^{-7}$	53	211	0.46984	8.75 $\times {10}^{-6}$
	$t_5$	2	7	0.041153	0	85	254	0.63624	9.44 $\times {10}^{-7}$	53	211	0.44157	8.02 $\times {10}^{-6}$
	$t_6$	3	11	0.068645	0	85	254	0.75653	9.89 $\times {10}^{-7}$	47	187	0.5116	9.80 $\times {10}^{-6}$
	$t_7$	51	204	0.97251	2.43 $\times {10}^{-7}$	83	248	0.59375	8.48 $\times {10}^{-7}$	23	91	0.22912	6.16 $\times {10}^{-6}$
100,000	$t_1$	3	11	0.23207	0	78	233	1.1907	9.24 $\times {10}^{-7}$	22	87	0.32312	6.25 $\times {10}^{-6}$
	$t_2$	3	11	0.096102	0	81	242	0.95676	8.90 $\times {10}^{-7}$	23	91	0.36086	5.51 $\times {10}^{-6}$
	$t_3$	2	7	0.059522	0	84	251	0.99173	9.55 $\times {10}^{-7}$	23	91	0.36819	8.99 $\times {10}^{-6}$
	$t_4$	3	11	0.18868	0	86	257	1.0274	9.97 $\times {10}^{-7}$	54	215	0.82162	9.10 $\times {10}^{-6}$
	$t_5$	2	7	0.076178	0	87	260	1.3603	8.55 $\times {10}^{-7}$	54	215	0.85533	8.34 $\times {10}^{-6}$
	$t_6$	3	11	0.13387	0	87	260	1.0415	8.95 $\times {10}^{-7}$	49	195	0.7369	7.49 $\times {10}^{-6}$
	$t_7$	48	192	1.7777	7.92 $\times {10}^{-7}$	84	251	0.99015	9.59 $\times {10}^{-7}$	23	91	0.36125	8.67 $\times {10}^{-6}$

Table 7. Numerical test reports for the three tested methods for Problem 5.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	35	139	0.028177	4.01 $\times {10}^{-7}$	90	268	0.024747	8.03 $\times {10}^{-7}$	22	82	0.012429	7.48 $\times {10}^{-6}$
	$t_2$	21	83	0.017203	6.42 $\times {10}^{-7}$	89	265	0.02733	9.07 $\times {10}^{-7}$	23	87	0.012966	7.31 $\times {10}^{-6}$
	$t_3$	31	123	0.02123	8.88 $\times {10}^{-7}$	88	262	0.025983	8.96 $\times {10}^{-7}$	23	89	0.011284	9.31 $\times {10}^{-6}$
	$t_4$	30	120	0.032095	3.98 $\times {10}^{-7}$	89	266	0.040925	9.36 $\times {10}^{-7}$	49	195	0.026429	8.45 $\times {10}^{-6}$
	$t_5$	27	108	0.048348	4.09 $\times {10}^{-7}$	87	260	0.025181	9.16 $\times {10}^{-7}$	53	211	0.019872	8.38 $\times {10}^{-6}$
	$t_6$	26	104	0.019389	4.97 $\times {10}^{-7}$	84	251	0.031167	8.32 $\times {10}^{-7}$	46	183	0.027704	8.80 $\times {10}^{-6}$
	$t_7$	28	111	0.066312	5.41 $\times {10}^{-7}$	90	268	0.026267	9.86 $\times {10}^{-7}$	168	670	0.05767	9.42 $\times {10}^{-6}$
5000	$t_1$	30	119	0.099879	4.22 $\times {10}^{-7}$	97	289	0.099454	8.47 $\times {10}^{-7}$	24	90	0.036297	6.36 $\times {10}^{-6}$
	$t_2$	49	195	0.20684	2.06 $\times {10}^{-7}$	96	286	0.085712	9.58 $\times {10}^{-7}$	25	94	0.03958	6.24 $\times {10}^{-6}$
	$t_3$	27	107	0.091189	6.71 $\times {10}^{-7}$	95	283	0.12574	9.47 $\times {10}^{-7}$	25	97	0.034165	5.86 $\times {10}^{-6}$
	$t_4$	25	100	0.11071	7.95 $\times {10}^{-7}$	97	290	0.10252	8.05 $\times {10}^{-7}$	53	211	0.076152	9.11 $\times {10}^{-6}$
	$t_5$	41	164	0.13125	8.90 $\times {10}^{-7}$	94	281	0.092231	9.87 $\times {10}^{-7}$	58	231	0.08755	8.56 $\times {10}^{-6}$
	$t_6$	30	120	0.10232	7.65 $\times {10}^{-7}$	91	272	0.13952	8.88 $\times {10}^{-7}$	50	199	0.07417	7.65 $\times {10}^{-6}$
	$t_7$	35	139	0.15239	4.54 $\times {10}^{-7}$	97	289	0.094881	9.23 $\times {10}^{-7}$	316	1262	0.39534	9.96 $\times {10}^{-6}$
10,000	$t_1$	48	191	0.61255	5.75 $\times {10}^{-7}$	100	298	0.30348	8.69 $\times {10}^{-7}$	25	94	0.079057	5.40 $\times {10}^{-6}$
	$t_2$	37	147	0.32599	2.23 $\times {10}^{-7}$	99	295	0.18161	9.83 $\times {10}^{-7}$	25	94	0.079123	8.90 $\times {10}^{-6}$
	$t_3$	25	99	0.17968	5.85 $\times {10}^{-7}$	98	292	0.17487	9.72 $\times {10}^{-7}$	25	97	0.072827	8.64 $\times {10}^{-6}$
	$t_4$	24	96	0.12434	8.81 $\times {10}^{-7}$	100	299	0.18342	8.29 $\times {10}^{-7}$	55	219	0.15362	9.11 $\times {10}^{-6}$
	$t_5$	31	124	0.17139	2.16 $\times {10}^{-7}$	98	293	0.17674	8.14 $\times {10}^{-7}$	60	239	0.13688	9.01 $\times {10}^{-6}$
	$t_6$	33	132	0.25773	7.87 $\times {10}^{-7}$	94	281	0.16779	9.15 $\times {10}^{-7}$	51	203	0.15962	9.62 $\times {10}^{-6}$
	$t_7$	43	171	0.35695	3.19 $\times {10}^{-7}$	99	295	0.18399	8.84 $\times {10}^{-7}$	325	1298	0.7484	9.67 $\times {10}^{-6}$
50,000	$t_1$	99	395	6.631	5.43 $\times {10}^{-7}$	107	319	0.77898	9.16 $\times {10}^{-7}$	26	98	0.26672	6.75 $\times {10}^{-6}$
	$t_2$	78	311	4.9202	6.69 $\times {10}^{-7}$	107	319	0.79789	8.28 $\times {10}^{-7}$	27	102	0.23943	5.16 $\times {10}^{-6}$
	$t_3$	36	143	0.81487	4.78 $\times {10}^{-7}$	106	316	0.91174	8.19 $\times {10}^{-7}$	27	105	0.30807	5.28 $\times {10}^{-6}$
	$t_4$	29	116	0.66459	7.02 $\times {10}^{-7}$	107	320	1.1448	8.77 $\times {10}^{-7}$	60	239	0.5978	8.66 $\times {10}^{-6}$
	$t_5$	30	120	0.7737	6.25 $\times {10}^{-7}$	105	314	0.76987	8.61 $\times {10}^{-7}$	65	259	0.61224	9.05 $\times {10}^{-6}$
	$t_6$	35	140	0.97053	3.67 $\times {10}^{-7}$	101	302	0.75366	9.68 $\times {10}^{-7}$	56	223	0.53722	8.19 $\times {10}^{-6}$
	$t_7$	48	191	2.2774	4.52 $\times {10}^{-7}$	106	316	0.77381	8.74 $\times {10}^{-7}$	F	F	F	F
100,000	$t_1$	95	379	11.9112	9.12 $\times {10}^{-7}$	110	328	1.7098	9.39 $\times {10}^{-7}$	26	98	0.4346	9.73 $\times {10}^{-6}$
	$t_2$	95	379	10.7767	4.09 $\times {10}^{-7}$	110	328	1.4604	8.50 $\times {10}^{-7}$	27	102	0.44774	7.39 $\times {10}^{-6}$
	$t_3$	27	107	1.2642	4.65 $\times {10}^{-7}$	109	325	1.4436	8.40 $\times {10}^{-7}$	27	105	0.46265	7.77 $\times {10}^{-6}$
	$t_4$	30	120	1.5062	6.16 $\times {10}^{-7}$	110	329	1.7956	9.00 $\times {10}^{-7}$	62	247	1.0814	9.00 $\times {10}^{-6}$
	$t_5$	57	228	5.2267	9.73 $\times {10}^{-7}$	108	323	1.4142	8.84 $\times {10}^{-7}$	67	267	1.2447	9.50 $\times {10}^{-6}$
	$t_6$	27	107	1.4554	3.33 $\times {10}^{-7}$	104	311	1.386	9.94 $\times {10}^{-7}$	58	231	1.1897	8.32 $\times {10}^{-6}$
	$t_7$	119	475	16.6573	7.19 $\times {10}^{-7}$	109	325	1.7025	9.46 $\times {10}^{-7}$	F	F	F	F

Table 8. Numerical test reports for the three tested methods for Problem 6.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	12	48	0.03188	2.30 $\times {10}^{-7}$	83	248	0.031732	8.42 $\times {10}^{-7}$	23	91	0.014183	9.28 $\times {10}^{-6}$
	$t_2$	12	48	0.041971	2.23 $\times {10}^{-7}$	83	248	0.031901	8.09 $\times {10}^{-7}$	23	91	0.014925	8.92 $\times {10}^{-6}$
	$t_3$	12	48	0.040553	2.02 $\times {10}^{-7}$	82	245	0.031622	8.91 $\times {10}^{-7}$	23	91	0.020135	7.86 $\times {10}^{-6}$
	$t_4$	11	44	0.024141	6.97 $\times {10}^{-7}$	80	239	0.031387	9.53 $\times {10}^{-7}$	23	91	0.014473	5.38 $\times {10}^{-6}$
	$t_5$	11	44	0.038626	5.62 $\times {10}^{-7}$	79	236	0.043836	9.56 $\times {10}^{-7}$	22	87	0.016292	8.62 $\times {10}^{-6}$
	$t_6$	11	44	0.01937	3.35 $\times {10}^{-7}$	77	230	0.029442	8.81 $\times {10}^{-7}$	22	87	0.014627	5.08 $\times {10}^{-6}$
	$t_7$	12	48	0.041621	4.70 $\times {10}^{-7}$	82	245	0.031674	9.04 $\times {10}^{-7}$	23	91	0.026782	7.91 $\times {10}^{-6}$
5000	$t_1$	12	48	0.096723	4.01 $\times {10}^{-7}$	86	257	0.16263	9.65 $\times {10}^{-7}$	25	99	0.062377	5.22 $\times {10}^{-6}$
	$t_2$	12	48	0.14973	3.86 $\times {10}^{-7}$	86	257	0.20647	9.28 $\times {10}^{-7}$	25	99	0.063981	5.02 $\times {10}^{-6}$
	$t_3$	12	48	0.12093	3.40 $\times {10}^{-7}$	86	257	0.17205	8.17 $\times {10}^{-7}$	24	95	0.056084	8.82 $\times {10}^{-6}$
	$t_4$	12	48	0.10493	2.33 $\times {10}^{-7}$	84	251	0.17167	8.74 $\times {10}^{-7}$	24	95	0.059374	6.04 $\times {10}^{-6}$
	$t_5$	12	48	0.083136	1.87 $\times {10}^{-7}$	83	248	0.15447	8.77 $\times {10}^{-7}$	23	91	0.055294	9.67 $\times {10}^{-6}$
	$t_6$	11	44	0.082714	7.05 $\times {10}^{-7}$	81	242	0.15192	8.08 $\times {10}^{-7}$	23	91	0.055478	5.70 $\times {10}^{-6}$
	$t_7$	12	48	0.084104	3.48 $\times {10}^{-7}$	86	257	0.16641	8.25 $\times {10}^{-7}$	24	95	0.06338	8.87 $\times {10}^{-6}$
10,000	$t_1$	12	48	0.158	5.68 $\times {10}^{-7}$	88	263	0.34039	8.73 $\times {10}^{-7}$	25	99	0.098299	7.38 $\times {10}^{-6}$
	$t_2$	12	48	0.15694	5.46 $\times {10}^{-7}$	88	263	0.35149	8.40 $\times {10}^{-7}$	25	99	0.13644	7.09 $\times {10}^{-6}$
	$t_3$	12	48	0.2381	4.81 $\times {10}^{-7}$	87	260	0.34294	9.25 $\times {10}^{-7}$	25	99	0.11271	6.25 $\times {10}^{-6}$
	$t_4$	12	48	0.20746	3.29 $\times {10}^{-7}$	85	254	0.26475	9.89 $\times {10}^{-7}$	24	95	0.093885	8.54 $\times {10}^{-6}$
	$t_5$	12	48	0.15857	2.64 $\times {10}^{-7}$	84	251	0.2461	9.92 $\times {10}^{-7}$	24	95	0.10396	6.85 $\times {10}^{-6}$
	$t_6$	11	44	0.81101	9.97 $\times {10}^{-7}$	82	245	0.24381	9.14 $\times {10}^{-7}$	23	91	0.12403	8.06 $\times {10}^{-6}$
	$t_7$	12	48	0.16525	4.85 $\times {10}^{-7}$	87	260	0.26637	9.35 $\times {10}^{-7}$	25	99	0.10128	6.30 $\times {10}^{-6}$
50,000	$t_1$	13	52	1.0506	1.98 $\times {10}^{-7}$	91	272	1.2907	1.00 $\times {10}^{-6}$	26	103	0.40761	8.26 $\times {10}^{-6}$
	$t_2$	13	52	0.75114	1.91 $\times {10}^{-7}$	91	272	1.1071	9.61 $\times {10}^{-7}$	26	103	0.40356	7.95 $\times {10}^{-6}$
	$t_3$	13	52	0.98117	1.68 $\times {10}^{-7}$	91	272	1.0738	8.47 $\times {10}^{-7}$	26	103	0.41251	7.00 $\times {10}^{-6}$
	$t_4$	12	48	0.65997	7.36 $\times {10}^{-7}$	89	266	1.0542	9.06 $\times {10}^{-7}$	25	99	0.39262	9.56 $\times {10}^{-6}$
	$t_5$	12	48	0.46882	5.91 $\times {10}^{-7}$	88	263	1.5109	9.08 $\times {10}^{-7}$	25	99	0.39186	7.67 $\times {10}^{-6}$
	$t_6$	12	48	0.61393	3.48 $\times {10}^{-7}$	86	257	1.0452	8.37 $\times {10}^{-7}$	24	95	0.38423	9.03 $\times {10}^{-6}$
	$t_7$	13	52	0.54298	1.70 $\times {10}^{-7}$	91	272	1.0556	8.54 $\times {10}^{-7}$	26	103	0.4047	7.06 $\times {10}^{-6}$
100,000	$t_1$	13	52	1.4092	2.81 $\times {10}^{-7}$	93	278	2.7716	9.05 $\times {10}^{-7}$	27	107	1.067	5.86 $\times {10}^{-6}$
	$t_2$	13	52	1.3229	2.70 $\times {10}^{-7}$	93	278	2.3812	8.70 $\times {10}^{-7}$	27	107	1.2086	5.63 $\times {10}^{-6}$
	$t_3$	13	52	1.6913	2.38 $\times {10}^{-7}$	92	275	2.6594	9.58 $\times {10}^{-7}$	26	103	0.96952	9.90 $\times {10}^{-6}$
	$t_4$	13	52	1.3408	1.63 $\times {10}^{-7}$	91	272	2.3237	8.20 $\times {10}^{-7}$	26	103	0.94103	6.78 $\times {10}^{-6}$
	$t_5$	12	48	1.2425	8.35 $\times {10}^{-7}$	90	269	3.1173	8.22 $\times {10}^{-7}$	26	103	1.0502	5.44 $\times {10}^{-6}$
	$t_6$	12	48	1.3332	4.93 $\times {10}^{-7}$	87	260	2.4359	9.47 $\times {10}^{-7}$	25	99	0.88394	6.40 $\times {10}^{-6}$
	$t_7$	13	52	1.3846	2.40 $\times {10}^{-7}$	92	275	2.6667	9.66 $\times {10}^{-7}$	26	103	0.9198	9.98 $\times {10}^{-6}$

Table 9. Numerical test reports for the three tested methods for Problem 7.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	8	32	0.011558	5.30 $\times {10}^{-7}$	37	110	0.013147	9.48 $\times {10}^{-7}$	17	67	0.009313	6.98 $\times {10}^{-6}$
	$t_2$	9	36	0.013381	9.70 $\times {10}^{-7}$	37	110	0.013514	6.87 $\times {10}^{-7}$	15	59	0.010714	9.89 $\times {10}^{-6}$
	$t_3$	7	28	0.011551	3.38 $\times {10}^{-7}$	30	89	0.011236	6.51 $\times {10}^{-7}$	16	63	0.008948	5.79 $\times {10}^{-6}$
	$t_4$	9	36	0.01024	2.83 $\times {10}^{-7}$	38	113	0.013281	8.05 $\times {10}^{-7}$	16	63	0.008965	5.21 $\times {10}^{-6}$
	$t_5$	9	36	0.00978	8.28 $\times {10}^{-7}$	38	113	0.021489	8.05 $\times {10}^{-7}$	19	75	0.014196	4.95 $\times {10}^{-6}$
	$t_6$	10	39	0.012021	1.26 $\times {10}^{-7}$	37	109	0.011749	9.07 $\times {10}^{-7}$	18	70	0.009389	8.93 $\times {10}^{-6}$
	$t_7$	21	84	0.035658	5.40 $\times {10}^{-7}$	37	110	0.015765	6.61 $\times {10}^{-7}$	20	79	0.019759	8.71 $\times {10}^{-6}$
5000	$t_1$	9	36	0.042338	1.32 $\times {10}^{-7}$	39	116	0.052548	8.30 $\times {10}^{-7}$	18	71	0.037642	7.60 $\times {10}^{-6}$
	$t_2$	10	40	0.081106	2.42 $\times {10}^{-7}$	38	113	0.051834	9.60 $\times {10}^{-7}$	17	67	0.027158	5.25 $\times {10}^{-6}$
	$t_3$	7	28	0.039851	7.55 $\times {10}^{-7}$	31	92	0.041545	9.10 $\times {10}^{-7}$	17	67	0.028917	6.31 $\times {10}^{-6}$
	$t_4$	9	36	0.039967	6.33 $\times {10}^{-7}$	40	119	0.057437	7.05 $\times {10}^{-7}$	17	67	0.037275	5.68 $\times {10}^{-6}$
	$t_5$	10	40	0.046165	2.06 $\times {10}^{-7}$	40	119	0.052164	7.05 $\times {10}^{-7}$	20	79	0.030917	5.39 $\times {10}^{-6}$
	$t_6$	10	39	0.044881	2.82 $\times {10}^{-7}$	39	115	0.064438	7.94 $\times {10}^{-7}$	19	74	0.046654	9.73 $\times {10}^{-6}$
	$t_7$	24	96	0.18701	6.38 $\times {10}^{-7}$	38	113	0.074383	9.00 $\times {10}^{-7}$	21	83	0.048626	9.52 $\times {10}^{-6}$
10,000	$t_1$	9	36	0.084021	1.87 $\times {10}^{-7}$	40	119	0.1459	7.34 $\times {10}^{-7}$	19	75	0.060554	5.23 $\times {10}^{-6}$
	$t_2$	10	40	0.091788	3.42 $\times {10}^{-7}$	39	116	0.095325	8.50 $\times {10}^{-7}$	17	67	0.057649	7.42 $\times {10}^{-6}$
	$t_3$	8	32	0.082144	1.19 $\times {10}^{-7}$	32	95	0.074842	8.05 $\times {10}^{-7}$	17	67	0.046905	8.92 $\times {10}^{-6}$
	$t_4$	9	36	0.065787	8.95 $\times {10}^{-7}$	40	119	0.10024	9.96 $\times {10}^{-7}$	17	67	0.05047	8.03 $\times {10}^{-6}$
	$t_5$	10	40	0.069698	2.92 $\times {10}^{-7}$	40	119	0.10003	9.96 $\times {10}^{-7}$	20	79	0.069303	7.62 $\times {10}^{-6}$
	$t_6$	10	39	0.1457	3.99 $\times {10}^{-7}$	40	118	0.097749	7.02 $\times {10}^{-7}$	20	78	0.055323	6.70 $\times {10}^{-6}$
	$t_7$	21	84	0.14332	3.62 $\times {10}^{-7}$	39	116	0.12983	8.05 $\times {10}^{-7}$	22	87	0.078308	6.51 $\times {10}^{-6}$
50,000	$t_1$	9	36	0.25176	4.17 $\times {10}^{-7}$	42	125	0.47157	6.42 $\times {10}^{-7}$	20	79	0.29584	5.70 $\times {10}^{-6}$
	$t_2$	10	40	0.76304	7.64 $\times {10}^{-7}$	41	122	0.44664	7.43 $\times {10}^{-7}$	18	71	0.21246	8.08 $\times {10}^{-6}$
	$t_3$	8	32	0.22903	2.66 $\times {10}^{-7}$	34	101	0.30639	7.04 $\times {10}^{-7}$	18	71	0.22024	9.71 $\times {10}^{-6}$
	$t_4$	10	40	0.46107	2.23 $\times {10}^{-7}$	42	125	0.40534	8.72 $\times {10}^{-7}$	18	71	0.19871	8.75 $\times {10}^{-6}$
	$t_5$	10	40	0.39956	6.52 $\times {10}^{-7}$	42	125	0.46103	8.72 $\times {10}^{-7}$	21	83	0.22027	8.30 $\times {10}^{-6}$
	$t_6$	10	39	0.30315	8.92 $\times {10}^{-7}$	41	121	0.51651	9.82 $\times {10}^{-7}$	21	82	0.23139	7.30 $\times {10}^{-6}$
	$t_7$	21	84	0.88881	4.14 $\times {10}^{-7}$	41	122	0.73262	7.00 $\times {10}^{-7}$	23	91	0.37248	7.08 $\times {10}^{-6}$
100,000	$t_1$	9	36	0.49984	5.90 $\times {10}^{-7}$	42	125	0.90106	9.08 $\times {10}^{-7}$	20	79	0.57692	8.06 $\times {10}^{-6}$
	$t_2$	11	44	0.83437	1.20 $\times {10}^{-7}$	42	125	0.95004	6.58 $\times {10}^{-7}$	19	75	0.51407	5.57 $\times {10}^{-6}$
	$t_3$	8	32	0.63611	3.76 $\times {10}^{-7}$	34	101	0.74225	9.96 $\times {10}^{-7}$	19	75	0.42249	6.69 $\times {10}^{-6}$
	$t_4$	10	40	0.80033	3.15 $\times {10}^{-7}$	43	128	0.76591	7.71 $\times {10}^{-7}$	19	75	0.41102	6.03 $\times {10}^{-6}$
	$t_5$	10	40	0.53766	9.23 $\times {10}^{-7}$	43	128	0.79251	7.71 $\times {10}^{-7}$	22	87	0.45205	5.72 $\times {10}^{-6}$
	$t_6$	11	43	0.69393	1.41 $\times {10}^{-7}$	42	124	0.89477	8.69 $\times {10}^{-7}$	22	86	0.50902	5.03 $\times {10}^{-6}$
	$t_7$	21	84	1.3381	8.09 $\times {10}^{-7}$	41	122	1.451	9.90 $\times {10}^{-7}$	24	95	0.74006	4.89 $\times {10}^{-6}$

Table 10. Numerical test reports for the three tested methods for Problem 8.

		DF-LSTT				CGD				PCG
DIM	INP	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM	ITER	FVAL	TIME	NORM
1000	$t_1$	13	52	0.011502	4.53 $\times {10}^{-7}$	21	62	0.007103	9.28 $\times {10}^{-7}$	9	35	0.007659	2.15 $\times {10}^{-6}$
	$t_2$	13	52	0.010497	2.74 $\times {10}^{-7}$	21	62	0.007684	5.62 $\times {10}^{-7}$	8	31	0.005796	7.72 $\times {10}^{-6}$
	$t_3$	12	48	0.018922	7.51 $\times {10}^{-7}$	21	62	0.006208	5.36 $\times {10}^{-7}$	8	31	0.004682	7.36 $\times {10}^{-6}$
	$t_4$	14	56	0.011298	2.60 $\times {10}^{-7}$	23	68	0.00871	5.84 $\times {10}^{-7}$	9	35	0.005209	7.18 $\times {10}^{-6}$
	$t_5$	14	56	0.0119	3.53 $\times {10}^{-7}$	23	68	0.006241	7.91 $\times {10}^{-7}$	9	35	0.004898	9.73 $\times {10}^{-6}$
	$t_6$	14	56	0.020175	5.07 $\times {10}^{-7}$	24	71	0.006771	4.93 $\times {10}^{-7}$	10	39	0.005248	2.36 $\times {10}^{-6}$
	$t_7$	9	36	0.007601	9.21 $\times {10}^{-7}$	22	65	0.005934	5.11 $\times {10}^{-7}$	9	35	0.004781	2.82 $\times {10}^{-6}$
5000	$t_1$	14	56	0.044747	2.48 $\times {10}^{-7}$	22	65	0.030281	9.01 $\times {10}^{-7}$	9	35	0.014961	4.81 $\times {10}^{-6}$
	$t_2$	13	52	0.039991	6.13 $\times {10}^{-7}$	22	65	0.037367	5.46 $\times {10}^{-7}$	9	35	0.019975	2.91 $\times {10}^{-6}$
	$t_3$	13	52	0.061786	4.11 $\times {10}^{-7}$	22	65	0.024256	5.20 $\times {10}^{-7}$	9	35	0.012236	2.78 $\times {10}^{-6}$
	$t_4$	14	56	0.037006	5.82 $\times {10}^{-7}$	24	71	0.030963	5.67 $\times {10}^{-7}$	10	39	0.012613	2.71 $\times {10}^{-6}$
	$t_5$	14	56	0.042827	7.89 $\times {10}^{-7}$	24	71	0.027167	7.68 $\times {10}^{-7}$	10	39	0.012167	3.67 $\times {10}^{-6}$
	$t_6$	15	60	0.040744	2.77 $\times {10}^{-7}$	25	74	0.032369	4.79 $\times {10}^{-7}$	10	39	0.014305	5.27 $\times {10}^{-6}$
	$t_7$	10	40	0.031693	5.07 $\times {10}^{-7}$	23	68	0.017202	5.02 $\times {10}^{-7}$	9	35	0.011532	6.18 $\times {10}^{-6}$
10,000	$t_1$	14	56	0.1304	3.51 $\times {10}^{-7}$	23	68	0.031436	5.53 $\times {10}^{-7}$	9	35	0.015852	6.80 $\times {10}^{-6}$
	$t_2$	13	52	0.082982	8.67 $\times {10}^{-7}$	22	65	0.032629	7.71 $\times {10}^{-7}$	9	35	0.01903	4.12 $\times {10}^{-6}$
	$t_3$	13	52	0.10124	5.82 $\times {10}^{-7}$	22	65	0.031087	7.36 $\times {10}^{-7}$	9	35	0.025673	3.93 $\times {10}^{-6}$
	$t_4$	14	56	0.084077	8.24 $\times {10}^{-7}$	24	71	0.058121	8.02 $\times {10}^{-7}$	10	39	0.015808	3.83 $\times {10}^{-6}$
	$t_5$	15	60	0.13836	2.73 $\times {10}^{-7}$	25	74	0.034695	4.72 $\times {10}^{-7}$	10	39	0.021954	5.19 $\times {10}^{-6}$
	$t_6$	15	60	0.09151	3.92 $\times {10}^{-7}$	25	74	0.033872	6.78 $\times {10}^{-7}$	10	39	0.017664	7.45 $\times {10}^{-6}$
	$t_7$	10	40	0.074314	7.22 $\times {10}^{-7}$	23	68	0.04517	7.08 $\times {10}^{-7}$	9	35	0.0245	8.64 $\times {10}^{-6}$
50,000	$t_1$	14	56	0.48989	7.84 $\times {10}^{-7}$	24	71	0.15209	5.37 $\times {10}^{-7}$	10	39	0.070466	2.57 $\times {10}^{-6}$
	$t_2$	14	56	0.39664	4.75 $\times {10}^{-7}$	23	68	0.12867	7.49 $\times {10}^{-7}$	9	35	0.059796	9.21 $\times {10}^{-6}$
	$t_3$	14	56	1.1025	3.19 $\times {10}^{-7}$	23	68	0.26774	7.15 $\times {10}^{-7}$	9	35	0.055552	8.79 $\times {10}^{-6}$
	$t_4$	15	60	0.37756	4.51 $\times {10}^{-7}$	25	74	0.18066	7.79 $\times {10}^{-7}$	10	39	0.063615	8.57 $\times {10}^{-6}$
	$t_5$	15	60	0.48364	6.11 $\times {10}^{-7}$	26	77	0.20521	4.58 $\times {10}^{-7}$	11	43	0.066811	1.96 $\times {10}^{-6}$
	$t_6$	15	60	0.38215	8.77 $\times {10}^{-7}$	26	77	0.1468	6.58 $\times {10}^{-7}$	11	43	0.070802	2.81 $\times {10}^{-6}$
	$t_7$	11	44	0.29749	3.97 $\times {10}^{-7}$	24	71	0.15389	6.86 $\times {10}^{-7}$	10	39	0.11456	3.27 $\times {10}^{-6}$
100,000	$t_1$	15	60	0.76533	2.72 $\times {10}^{-7}$	24	71	0.28556	7.60 $\times {10}^{-7}$	10	39	0.15136	3.63 $\times {10}^{-6}$
	$t_2$	14	56	0.96545	6.72 $\times {10}^{-7}$	24	71	0.30656	4.60 $\times {10}^{-7}$	10	39	0.15471	2.20 $\times {10}^{-6}$
	$t_3$	14	56	0.71691	4.51 $\times {10}^{-7}$	24	71	0.27951	4.39 $\times {10}^{-7}$	10	39	0.2006	2.10 $\times {10}^{-6}$
	$t_4$	15	60	0.76153	6.38 $\times {10}^{-7}$	26	77	0.39044	4.78 $\times {10}^{-7}$	11	43	0.18627	2.04 $\times {10}^{-6}$
	$t_5$	15	60	0.79539	8.64 $\times {10}^{-7}$	26	77	0.57875	6.48 $\times {10}^{-7}$	11	43	0.13307	2.77 $\times {10}^{-6}$
	$t_6$	16	64	0.82574	3.04 $\times {10}^{-7}$	26	77	0.4141	9.31 $\times {10}^{-7}$	11	43	0.142	3.98 $\times {10}^{-6}$
	$t_7$	11	44	0.52429	5.60 $\times {10}^{-7}$	24	71	0.47424	9.71 $\times {10}^{-7}$	10	39	0.15694	4.64 $\times {10}^{-6}$

We employ the widely used performance profile metric of Dolan and More [41] to compare the performance of the methods. The profile metric measures the ratio of each method by its computational outcome versus the computational outcome of the best presented method. The performance profile metric operates in the following manner. Let $S_m$ and $P_m$ denote the set of methods and test problems, respectively. In this section, we see a problem with different dimensions or different initial points as another new problem. For $n_s$ methods and $n_p$ problems, the performance profile $\rho :R\to [0,1]$ is defined as follows: for each problem $p\in P_m$ and for each method $s\in S_m$, they define

$$t_{p,s}:=(\mathrm{computing}\mathrm{time}\mathrm{required}\mathrm{to}\mathrm{solve}\mathrm{problem}\mathrm{p}\mathrm{by}\mathrm{method}\mathrm{s})$$

The performance ratio is given by $r_{p,s}:=t_{p,s}/{min}_{s\in S_m}{t}_{p,s}.$ Then, the performance profile is defined by

$$\rho \left(\tau \right):=n_p^{-1}size\{p\in P_m|{log}_2\left(r_{p,s}\right)\le \tau \},\forall \tau \in R^{+},\mathrm{where}1\le s\le n_s.$$

We note that $\rho \left(\tau \right)$ is the probability for a method $s\in S_m$ that ${log}_2\left(r_{p,s}\right)$ is within a factor $\tau \in R^{+}$ of the best possible ratio. Obviously, when $\tau$ takes certain value, a method with high value of $\rho \left(\tau \right)$ is preferable or represents the best method. As usual, we obtain the number of iterations, number of function evaluations, and the computing time (CPU time) from the performance profile metric. Figure 6 and Figure 7 are the performance profile obtained based on the Dolan and Moré profile. Both figures indicate that the performances of the proposed method is competitive with the other methods. However, from both Figure 6 and Figure 7, DF-LSTT is more efficient because it was able to solve a higher percentage of the test problems with less number of iterations and function evaluations compared to CGD and PCG methods. However, with respect to CPU time, our algorithm did not perform well. This could be as a result of several computations involved in the method.

Figure 6. Performance of compared methods relative to the number of iterations.

Figure 7. Performance of compared methods relative to the number of function evaluations.

6. Conclusions

In this paper, we have presented a derivative-free conjugate gradient method for solving the ${\ell }_1$-norm regularization problem by combining the projection technique with the proposed direction in [18]. Unlike the uniform convexity assumption used in establishing the convergence of the method in [18], our convergence was established under the monotonicity and Lipchitz continuity assumption. Our numerical experiments in recovering sparse signals and blurred images indicate efficient and robust behavior of the proposed algorithm. For instance, in recovering sparse signals, the proposed algorithm is faster than the compared algorithms. In addition, it exhibits less iterations and least mean squared error. Moreover, our algorithm is able to restore blurred and noisy images with better quality. This is reflected in the values of the SNR, PSNR, and SSIM. Furthermore, numerical experiments on a set of monotone problems with different initial points and dimensions were reported. Although, applying the proposed method in solving monotone operator equations, the proposed method does not perform well in terms of CPU time. This could be as a result of several computations involved in the method.