A Family of Developed Hybrid Four-Term Conjugate Gradient Algorithms for Unconstrained Optimization with Applications in Image Restoration

Abstract

The most important advantage of conjugate gradient methods (CGs) is that these methods have low memory requirements and convergence speed. This paper contains two main parts that deal with two application problems, as follows. In the first part, three new parameters of the CG methods are designed and then combined by employing a convex combination. The search direction is a four-term hybrid form for modified classical CG methods with some newly proposed parameters. The result of this hybridization is the acquisition of a newly developed hybrid CGCG method containing four terms. The proposed CGCG has sufficient descent properties. The convergence analysis of the proposed method is considered under some reasonable conditions. A numerical investigation is carried out for an unconstrained optimization problem. The comparison between the newly suggested algorithm (CGCG) and five other classical CG algorithms shows that the new method is competitive with and in all statuses superior to the five methods in terms of efficiency reliability and effectiveness in solving large-scale, unconstrained optimization problems. The second main part of this paper discusses the image restoration problem. By using the adaptive median filter method, the noise in an image is detected, and then the corrupted pixels of the image are restored by using a new family of modified hybrid CG methods. This new family has four terms: the first is the negative gradient; the second one consists of either the HS-CG method or the HZ-CG method; and the third and fourth terms are taken from our proposed CGCG method. Additionally, a change in the size of the filter window plays a key role in improving the performance of this family of CG methods, according to the noise level. Four famous images (test problems) are used to examine the performance of the new family of modified hybrid CG methods. The outstanding clearness of the restored images indicates that the new family of modified hybrid CG methods has reliable efficiency and effectiveness in dealing with image restoration problems.

1. Introduction

CG parameters are widely utilized for dealing with a great variety of different optimization problems.

Many researchers have focused on modifying CG parameters in order to augment the corresponding CG strategies. Those newly proposed approaches have enhanced the performance of the conventional CG methods.

The CG algorithm has been applied numerous areas such as neural networks, image processing, medical science, operational research, engineering problems, etc.

Thus, the different problems generated in these applications can be formulated in various forms, such as unconstrained, constrained, multi-objective optimization problems, nonlinear systems, etc.

Therefore, this study concentrates on an unconstrained optimization problem defined as

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right),$$

(1)

where the function $f:{\mathbb{R}}^n\to \mathbb{R}$ is continuously differentiable. CG methods have very powerful convergence abilities and fair storage needs.

Among the most important advantages of the CG method are its low memory requirements and convergence speed [1].

Thus, many authors have analyzed CG methods in order to solve large minimization optimization problems and other applications [2].

Accordingly, many authors have suggested several modificationsof the classical methods that deal with optimization problems, e.g, the Newton method [3], the quasi-Newton method [4], the semi-gradient method [5], the hybrid gradient meta-heuristic algorithm [6,7], and the CG-method [8,9,10,11].

Different versions of the Newton method are widely employed to solve multiple different optimization problems because of their fast convergence rates, e.g., [12,13,14,15,16,17].

However, these methods are very expensive since they require a calculation of the exact or approximate Jacobian matrix for each iteration. Therefore, it is best to avoid utilizing these methods when solving large-scale optimization problems.

Hence, the use of the CG technique has been more widespread compared to other conventional methods because of its simplicity, lower storage requirements, efficiency, and optimal convergence features [1,18,19].

Accordingly, the reason for selecting this topic for analysis depends on the following bases.

Conjugate gradient methods (CGs) are associated with a very strong global convergence theory for a local minimizer and they have low memory requirements. Moreover, in practice, combining the CG method with a line search strategy showed merit in dealing with an unconstrained minimization problem [20,21,22,23,24,25].

Additionally, CG parameters have shown remarkable superiority in solving problems involving systems of nonlinear equations (see, for example, [26,27,28,29,30,31,32,33,34,35]). According to previous successful uses of CG techniques to solve different applications problems, many authors have adapted CG methods such that they are capable of dealing with image restoration problems (see, for example, [25,35,36,37,38,39,40,41,42,43]).

Many researchers have shown that the CG method can also be used as a mathematical tool for training deep neural networks. (see, for example, [44,45,46,47,48]).

Generally, the iterative formula used to generate the sequence solutions of Problem (1) is defined in the following form

$$x_{{}_{k+1}}=x_{{}_k}+{\alpha }_{{}_k}{d}_{{}_k},$$

(2)

where $x_{{}_k}$ is the current point, ${\alpha }_{{}_k}>0$ is the step size obtained using a line search technique, and $d_{{}_k}$ is the search direction decided by the conjugate parameter ${\beta }_{{}_k}$.

The sequence of the step size ${\alpha }_{{}_k}$ and the search direction $d_{{}_k}$ can be generated through various approaches. These approaches depend on many concepts that have been implemented for designing different formulas of ${\alpha }_{{}_k}$, $d_{{}_k}$, and ${\beta }_{{}_k}$.

The short and simple formula used to determinethe search direction is defined as follows:

$$d_{{}_k}=-g_{{}_k}+{\beta }_{{}_k}{d}_{{}_{k-1}},d_{{}_0}=-g_{{}_0},$$

(3)

where $g_{{}_k}=g\left(x_{{}_k}\right)$, ${\beta }_{{}_k}$ is known as the CG parameter and $g\left(x_{{}_k}\right)$ represents the gradient vector of the function f at $x_{{}_k}$.

A core difference between all the proposed CG methods is the form of the parameter ${\beta }_{{}_k}$.

The proposal of new CG parameters of the classical CG methods has led to improvements in said methods’ performance in dealing with many problems in different applications.

The classical CG methods include the HS-CG method proposed by the authors of [9]; the FR-CG parameter proposed by the authors of [8]; the PRP-CG method proposed by Polak and Ribiere [11], Polyak [49]; the LS-CG method proposed by Liu and Storey [10]; and the DY-CG method proposed by Dai and Yuan [50]. The ${\beta }_{{}_k}$ value of these CG algorithms contains one term.

These CG parameters are listed in the following formulas:

$${\beta }_{{}_k}^{HS}=\frac{y_{{}_k}^T{g}_{k+1}}{d_{{}_k}^T{y}_{{}_k}},$$

(4)

where $y_{{}_k}=g_{k+1}-g_{{}_k}$.

$${\beta }_{{}_k}^{FR}=\frac{\left|\right|g_{k+1}{\left|\right|}^2}{\left|\right|g_{{}_k}{\left|\right|}^2},$$

(5)

$${\beta }_{{}_k}^{PRP}=\frac{y_{{}_k}^T{g}_{k+1}}{\left|\right|g_{{}_k}{\left|\right|}^2},$$

(6)

$${\beta }_{{}_k}^{LS}=\frac{g_{k+1}^T{y}_{{}_k}}{-d_{{}_k}^T{g}_{{}_k}},$$

(7)

$${\beta }_{{}_k}^{DY}=\frac{{\parallel g_{k+1}\parallel }^2}{y_{{}_k}^T{d}_{{}_k}},$$

(8)

where $\parallel .\parallel$ is the Euclidean norm and the values of the parameter $y_{{}_k}$ are computed as follows: $y_{{}_k}=g_{k+1}-g_{{}_k}$.

Recently, many novel formulas for determining the parameter ${\beta }_{{}_k}$ have been suggested, with those corresponding to CG methods including two or three terms (see [21,22,25,51,52])

In several papers, CG methods studied using convergence analysis have been reported (see [42,53]).

For example, Hager and Zhang [54] presented the following CG-method, which contains three terms:

$${\beta }_{{}_k}^{H{Z}^{*}}=max\{{\beta }^{HZ},r_{{}_k}\}$$

(9)

where ${\beta }_{{}_k}^{HZ}=\frac{\left(y_{{}_k}^T{g}_k\right)\left(d_{k-1}^T{y}_{{}_k}\right)-2\left|\right|y_{{}_k}{\left|\right|}^2\left(d_{k-1}^T{g}_{{}_k}\right)}{{\left(d_{k-1}^T{y}_{{}_k}\right)}^2}$, $r_{{}_k}=\frac{-1}{\left|\right|d_{{}_{k-1}}\left|\right|min\{r,||g_{{}_{k-1}}\left|\right|\}}$.  

In their numerical experiments, the authors of [54] set $r=0.01$.

If employed appropriately, remarkable results can be when obtained using the CG method to solve the many different problems posed in several applications.

Hence, the modifications and additions to and recommendations for conventional CG techniques are undertaken to develop an updated version of the CG method or a novel technique with widespread applications.

These propositions and modifications have either one term or multiple terms.

For example, Jiang et al. [25] recently proposed a family of combination 3-term CG techniques for solving nonlinear optimization problems and aiding image restoration.

The authors of [25] combined the parameter ${\beta }_{{}_k}^{DY}$ with each parameter in the set $\{{\beta }_{{}_k}^{HS},{\beta }_{{}_k}^{PRP},{\beta }_{{}_k}^{LS}\}$ to obtain a family of CG methods. They define the direction as follows:

$$d_{{}_k}=\left\{\begin{array}{cc}-g_{{}_k}\hfill & \mathrm{for}k=1,\hfill \\ -g_{k+1}+(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{new}{d}_{{}_k}-\gamma {\lambda }_{{}_k}\frac{\left(g_{k+1}^T{g}_{{}_k}\right)}{{\parallel g_{{}_k}\parallel }^2}{g}_{{}_k}\hfill & \mathrm{Otherwise}.\hfill \end{array}\right.$$

(10)

where $0<\gamma <1$ and ${\lambda }_{{}_k}=\frac{|g_{k+1}^T{d}_{{}_k}|}{\parallel g_{{}_{k+1}}\parallel \parallel d_{{}_k}\parallel }$. The authors of [25] used the convex combination technique developed by [35]. In addition to their proposal in Formula (10), they have suggested a new CG parameter, which was defined as follows:

$${\beta }_{{}_k}^N=\frac{\left(g_{k+1}^T{y}_{{}_k}\right)}{{\parallel g_{{}_k}\parallel }^2-\xi \left(g_{{}_k}^T{d}_{{}_k}\right)},$$

(11)

where $0<\xi <1$. Then, they combined their new parameter ${\beta }_{{}_k}^N$ with the parameter ${\beta }_{{}_k}^{DY}$ to get a new algorithm that solves Problem (1) and can be used in image restoration; furthermore, they performed a convergence analysis of this family of combination 3-term CG methods.

Huo et al. [55] proposed a new CG method containing three parameters in order to solve Problem (1). Moreover, under reasonable assumptions, the author of [55] established the global convergence of their method.

Tian et al. [56] suggested a new hybrid three-term CG approach without using a line search method. The authors of [56] designed the new hybrid descent three-term direction in the following form:

$$d_{k+1}^N=-g_{k+1}+{\beta }_{k+1}^N{s}_{{}_k}-{\sigma }_{k+1}^N{y}_{{}_k},d_{{}_0}=g_{{}_0},$$

$${\beta }_{k+1}^N=\frac{g_{k+1}^T(y_{{}_k}-\gamma s_{{}_k})}{max\{y_{{}_k}^T{s}_{{}_k},\parallel g_{{}_k}{\parallel }^2\}},$$

$${\sigma }_{k+1}^N=\frac{g_{k+1}^T{s}_{{}_k}}{max\{y_{{}_k}^T{s}_{{}_k},\parallel g_{{}_k}{\parallel }^2\}},$$

where, when $y_{{}_k}^T{s}_{{}_k}={\parallel g_{{}_k}\parallel }^2=0$ is met, the stopping criterion is satisfied, and the optimal solution $x^{*}=x_{{}_k}$ is obtained. They proceeded to provide some remarks regarding their proposed directions. In addition, the authors of [56] successfully performed a convergence analysis of their newly proposed CG method (under certain conditions). The numerical outcomes demonstrate that the three-term CG method is more efficient and reliable in terms of solving large-scale unconstrained optimization problems compared to other methods.

Jian et al. [57] proposed a new spectral SCGM approach, i.e., the core study of the authors in [57], which consisted of designing a spectral parameter and a conjugate parameter using the SCGM method. They proved the global convergence of the suggested SCGM method. The approach proposed by the authors of [57] for yielding the spectral parameter is as follows:

$${\varphi }_{{}_k}^{JYJLL}=1+\frac{|g_{{}_k}^T{d}_{{}_{k-1}}|}{-g_{{}_{k-1}}^T{d}_{{}_{k-1}}}.$$

(12)

The authors clearly showed that ${\varphi }_{{}_k}^{JYLL}\ge 1$ when the prior $d_{{}_{k-1}}$ is in a descent.

In addition, they proposed a new conjugate parameter, which was defined as follows:

$${\beta }_{{}_k}^{JYJLL}=\frac{\parallel g_{{}_k}{\parallel }^2-{\left(g_{{}_k}^T{d}_{{}_{k-1}}\right)}^2}{max\{\parallel g_{{}_{k-1}}{\parallel }^2,d_{{}_{k-1}}^T(g_{{}_k}-g_{{}_{k-1}})\}}.$$

(13)

Subsequently, they presented a convergence-analysis-based proof of the SCGM method. Using the SCGM method, the authors of [57] solved an unconstrained optimization problem with many dimensions.

There are other studies that employ the same convex combination concept with different combination parameters (see, for example, [58]).

Yuan et al. [52] proposed a research direction of the CG method containing the following formula:

$$d_{{}_k}=-g_{{}_k}+{\beta }_{{}_k}^{MHZ}{d}_{k-1},d_{{}_0}=-g_{{}_0},$$

(14)

where the authors in [52] defined the parameter ${\beta }_{{}_k}^{MHZ}$ as follows:

$${\beta }_{{}_k}^{MHZ}=\frac{\left(y_{{}_k}^T{g}_{{}_k}\right)\left(d_{k-1}^T{y}_{{}_k}\right)-2\left|\right|y_{{}_k}{\left|\right|}^2\left(d_{k-1}^T{g}_{{}_k}\right)}{max\{\sigma \parallel y_{{}_k}{\parallel }^2\parallel d_{{}_k}{\parallel }^2,{\left(d_{k-1}^T{y}_{{}_k}\right)}^2\}},$$

(15)

where $\sigma >0.5$ is a constant. The authors of [52] presented a convergence analysis proof of ${\beta }_{{}_k}^{MHZ}$ with some conditions.

Abubakar et al. [20] proposed the following CG-method (a new modified LS method (NMLS)):

$$d_{{}_0}=g_{{}_0},d_{{}_k}=\left\{\begin{array}{cc}-g_{{}_k}\hfill & \mathrm{if}{g}_{{}_k}^T{y}_{{}_{k-1}}\le 0,\mathrm{for}k>0,\hfill \\ d_{{}_k}^{\prime }\hfill & \mathrm{otherwise},\mathrm{for}k>0,\hfill \end{array}\right.$$

(16)

where $d_{{}_k}^{\prime }$ is defined as follows

$$d_{{}_0}=g_{{}_0},d_{{}_k}^{\prime }=\left\{\begin{array}{cc}-{\gamma }_{{}_k}{g}_{{}_k}+{\beta }_{{}_k}^{MLS}{d}_{{}_k}\hfill & \mathrm{if}{g}_{{}_k}^T{d}_{{}_{k-1}}>0,\mathrm{for}k>0,\hfill \\ -g_{{}_k}+{\beta }_{{}_k}^{LS}{d}_{{}_k}\hfill & \mathrm{otherwise},\mathrm{for}k>0,\hfill \end{array}\right.$$

(17)

where $y_{{}_{k-1}}=g_{{}_k}-g_{{}_{k-1}}$, ${\gamma }_{{}_k}=1+{\beta }_{{}_k}^{LS}\frac{g_{{}_k}^T{d}_{{}_{k-1}}}{\parallel g_{{}_k}{\parallel }^2}$ and the parameter ${\beta }_{{}_k}^{LS}$ is defined by (7) and ${\beta }_{{}_k}^{MLS}$ is defined by

$${\beta }_{{}_k}^{MLS}=(1-\frac{g_{{}_k}^T{s}_{{}_{k-1}}}{-g_{{}_k}^T{y}_{{}_{k-1}}}){\beta }_{{}_k}^{LS}-t\frac{\parallel y_{k-1}{\parallel }^2}{\left(g_{{}_k}^T{y}_{{}_{k-1}}\right)},t\ge 0.$$

The convergence analysis of this method was carried out by Abubakar et al. [20]. In addition, the numerical results, which were obtained by solving the corresponding unconstrained minimization problems, were implemented in applications in motion control

Alhawarat et al. [59] proposed a new CG method that depends on a convex group between two various search directions of the CG method; then, they employed this method to solve an unconstrained optimization problem involving image restoration. Their proposed method presented incredible results. The numerical results of the method proposed in [59] show that the convex combination allows the new CG method to provide more efficient results than the other compared methods.

Therefore, the convex combination technique plays a critical role in improving the performance of any new version of a CG method. The above arguments indicate that merging two or more parameters of the classical CG methods offers promising results.

Accordingly, such promising results encouraged us to continue this pattern of improving the performance of the classical CG techniques.

Many modifications and performance enhancements of the CG parameters have recently been proposed by numerous authors. Such recently developed GC parameters include two terms: the first one denotes the negative gradient vector, while the second term indicates the parameter proposed by the cited authors (see, for example, [21,22,25].

A brief review of the ideas presented in the literature has inspired us to design a new convex hybrid CG technique. This new convex combined technique has four terms, the first of which is a negative gradient vector, while the other three terms are the proposed parameters multiplied by $d_{{}_k}$.

Therefore, the major contributions of this paper include the following aspects.

• The presentation of some suggestions for and modifications to some classical CG parameters with new parameters;
• Three new parameters ${\beta }_{{}_k}^{N^j}$ for CG techniques, which are are combined together, where $j=1,2,3$;
• Through the above procedures, a new suggested CG approach has been designed, which we dubbed “Convex Group Conjugated Gradient”, which has been shortened to CGCG;
• The performance of a convergence analysis of the new CGCG approach;
• Numerical investigations are offered, which were executed by solving a set of test minimization problems;
• The adaptation of the CGCG algorithm yielded a family of modified CG methods that can be used to deal with image restoration problems;
• By applying the adaptive median filter method, the noise in an image is detected;
• According to the noise level in the corrupted image, a change in the size of the filter window is considered for improving the performance of this family of CG methods;
• Four corrupted images (test problems) with salt-and-pepper noise (at 30–90% levels) are used to examine the performance of the new family of modified hybrid CG methods.

The rest of the current study is arranged as follows. In the next section, the CGCG method and convergence analysis proof are presented. Section 3 presents the numerical experiments concerning the set of optimization test problems solved using the CGCG method and five other traditional CG methods.

Section 4 offers a brief discussion about image processing, including some applications of the adaptive CGCG method (a member of the family of CG methods) in image restoration. Section 5 presents the numerical experiments conducted with the proposed family of CG methods, which concern the solution to an image restoration problem. The last section provides some concluding remarks.

2. CGCG Method

The proposed CGCG method involves modifications to and suggestions for several classical CG-methods, which are listed in (4)–(8). In addition, this CGCG method contains new parameters, which are presented in this paper.

Therefore, the details of the newly proposed CGCG-method are listed in Algorithm 1 and used to solve Problem (1). The convergence analysis proof of the CGCG algorithm is also presented in this section. In addition, the CGCG method is adapted and modified in order to yield the family of CG-methods listed in Algorithms 2–4. These algorithms can be used in image restoration problems, as we will see in Section 4.

Algorithm 1 A Convex Group Conjugated Gradient Method (CGCG).

Input:  $f:{\mathbb{R}}^n\to \mathbb{R}$, $f\in C^1$, $\delta \in (0,0.5)$ and $\sigma \in (\delta ,1)$, $k=0$, an initial point $x_{{}_k}\in {\mathbb{R}}^n$ and $\epsilon >0$. Output:  $x^{*}$ is the optimal solution.   1: Set $d_{{}_0}=-g_{{}_0}$ and $k:=0$.   2: while $\parallel g_{{}_{k+1}}\parallel >\epsilon$. do   3:    Compute ${\alpha }_{{}_k}$ to fulfill (31) and (32).   4:    Calculate a new point $x_{{}_{k+1}}=x_{{}_k}+{\alpha }_{{}_k}{d}_{{}_k}$.   5:    Compute $f_{{}_k}=f\left(x_{{}_{k+1}}\right)$, $g_{{}_{k+1}}=g(x_{{}_{k+1}})$   6:    Set $k=k+1$.   7: end while   8: return $x^{*}$, the optimal solution.

Algorithm 2 CGCG-HS2 Algorithm

Step 1: Inputs: Original image (O_img), Noise Ratio(N.R)$=\{30\%,50\%,70\%,90\%\}$, $\alpha =100$ and W_max $=\{3\times 3,5\times 5,7\times 7,9\times 9\}$. Step 2: Destroy the original (O_img) by using noise (salt and pepper) to obtain a noised image (N_img). Step 3: Apply adaptive median filter algorithm (A.M.F .A) for each level and W_max. Step 4: Detect pixels of the noised image (N_img). Step 5: Use Formulas (53) and (55) to remove the Noise from the corrupted pixels. Step 6: Output: Repaired Image (R_img) $=$(O_img)

Algorithm 3 CGCG-HS1 Algorithm

Step 1: Inputs: Original image (O_img), Noise Ratio(N.R)$=\{30\%,50\%,70\%,90\%\}$, $\alpha =\{500,350\}$ and W_max $=5\times 5$. Step 2: Destroy the original (O_img) using a high noise level (salt and pepper) to obtain a noised image (N_img). Step 2a: If N.R $<60\%$. Step 2b: Set $\alpha =500$. Step 2b: Otherwise, set $\alpha =350$. Step 3: Apply adaptive median filter algorithm (A.M.F .A) for each level W_max $=5\times 5$. Step 4: Detect noisy pixels in the noised image (N_img). Step 5: Use Formulas (53) and (55) to remove the noise from the corrupted pixels. Step 6: Output: Repaired Image (R_img) $=$(O_img)

Algorithm 4 CGCG-HZ Algorithm

Step 1: Inputs: Original image (O_img), Noise Ratio(N.R)$=\{30\%,50\%,70\%,90\%\}$, $\alpha =\{500,350\}$ and W_max $=5\times 5$. Step 2: Destroy the original (O_img) using a high noise level (salt and pepper) to obtain a noised image (N_img). Step 2a: If N.R $<60\%$. Step 2b: Set $\alpha =500$. Step 2b: Otherwise, set $\alpha =350$. Step 3: Apply adaptive median filter algorithm (A.M.F .A) for each level with W_max $=5\times 5$. Step 4: Noisy pixels of the Noise image (N_img) are detected. Step 5: Use Formulas (53) and (56) to remove the Noise from the corrupted pixels. Step 6: Output: Repaired Image (R_img) $=$(O_img)

Thus, to solve Problem (1), the next iterative form is utilized to generate a new candidate solution:

$$x_{{}_{k+1}}=x_{{}_k}+{\alpha }_{{}_k}{d}_{{}_k},$$

(18)

where $x_{{}_k}$ is the current point and ${\alpha }_{{}_k}$ is the step size in the search direction $d_{{}_k}$.

The new proposed CG approach is defined in the following paragraph.

To achieve the global convergence of a hybrid CG technique, one must choose the step length ${\alpha }_{{}_k}$ carefully. Additionally, the selection of a suitable search direction $d_{{}_k}$ must be considered.

The purpose of this procedure is to guarantee that the following sufficient descent condition is satisfied:

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2,$$

(19)

for $C\ge 0$.

Consequently, the following fundamental property is called an angle property. This property and other proposed parameters are critical in developing the proposed method:

$$cos{\theta }_{{}_k}=\frac{g_{{}_{k+1}}^T{d}_{{}_k}}{\parallel g_{{}_{k+1}}\parallel \parallel d_{{}_k}\parallel },$$

(20)

where the angle ${\theta }_{{}_k}$ is between $d_{{}_k}$ and $g_{{}_{k+1}}$.

Hence, we benefit by obtaining the value of ${\theta }_{{}_k}$, which will be used to design a new mixed CG method containing four terms. The three terms that represent the core of our proposed method are inspired by the classical conjugate gradient parameters, which are defined in Formulas (4)–(8).

Therefore, we connect the three terms using the following parameter:

$${\lambda }_{{}_k}=\frac{|g_{{}_{k+1}}^T{d}_{{}_k}|}{\parallel g_{{}_{k+1}}\parallel \parallel d_{{}_k}\parallel }$$

(21)

In addition, we define the following parameter to represent an option for the CG parameter.

$${\chi }_{{}_k}=\frac{\sqrt{|\widehat{\tau }{\lambda }_{{}_k}-\tau |}}{2\overline{n}},$$

(22)

where $\tau$, $\widehat{\tau }$ are fixed real numbers such that $0<\widehat{\tau }<\tau <1$ and $\overline{n}\ge 2$ is an integer.

Then, the three CG parameters are defined as follows:

$${\beta }_{{}_k}^{N1}=max\{0,L{a}_{{}_k}\},$$

(23)

where

$$L{a}_{{}_k}=\frac{\left(y_{{}_k}^T{g}_{{}_{k+1}}\right)\left(d_{{}_k}^T{y}_{{}_k}\right)-2\left|\right|y_{{}_k}{\left|\right|}^2\left(d_{{}_k}^T{g}_{{}_{k+1}}\right)}{{\left(d_{{}_k}^T{y}_{{}_k}\right)}^2},$$

(24)

$${\beta }_{{}_k}^{N2}=max\left\{\frac{{\chi }_{{}_k}{\parallel g_{{}_{k+1}}\parallel }^2}{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)},L{b}_{{}_k}\right\},$$

(25)

where

$$L{b}_{{}_k}=\frac{\left(y_{{}_k}^T{g}_{{}_{k+1}}\right)\left(d_{{}_k}^T{y}_{{}_k}\right)-\left|\right|y_{{}_k}{\left|\right|}^2\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}{r{\parallel d_{{}_k}\parallel }^2{\parallel y_{{}_k}\parallel }^2},$$

(26)

where $0.5\le r<\infty$ is a fixed number,

$${\beta }_{{}_k}^{N3}=max\{▵{\chi }_{{}_k},L{c}_{{}_k}\},$$

(27)

where

$$▵{\chi }_{{}_k}=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{g}_{{}_{k+1}}^T{d}_{{}_k}<0,\hfill \\ {\chi }_{{}_k}\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(28)

$$L{c}_{{}_k}=\frac{2\parallel y_{{}_k}{\parallel }^2\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)\sqrt{s_{{}_{k+1}}}}{n(▵f+\epsilon ){\left(d_{{}_k}^T{y}_{{}_k}\right)}^2},$$

(29)

where $▵f=|f_{{}_{k+1}}-f_{{}_k}|$, $0<\epsilon <1$, $s_{{}_{k+1}}=\parallel x_{{}_{k+1}}-x_{{}_k}\parallel$ at the iteration k and n indicates the number of variables of a test problem.

According to the above relations and formulas, the new search direction is defined as follows:

$$d_{{}_{k+1}}=-g_{{}_{k+1}}+{\beta }_{{}_k}^{N1}{d}_{{}_k}+(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{N2}{d}_{{}_k}-{\lambda }_{{}_k}{\beta }_{{}_k}^{N3}{d}_{{}_k},d_{{}_0}=-g_{{}_0},$$

(30)

Thus, the newly proposed CG approach includes four terms; we name this suggested method “Convex Group Conjugated Gradient”, which is abbreviated as (CGCG).

To render the CGCG method globally convergent, we use the following line search technique:

$$f(x_{{}_k}+{\alpha }_{{}_k}{d}_{{}_k})\le f\left(x_{{}_k}\right)+\delta {\alpha }_{{}_k}{g}_{{}_k}^T{d}_{{}_k},$$

(31)

and

$$g{(x_{{}_k}+{\alpha }_{{}_k}{d}_{{}_k})}^T{d}_{{}_k}\ge \sigma g_{{}_k}^T{d}_{{}_k},$$

(32)

where $\delta \in (0,0.5)$ and $\sigma \in (\delta ,1)$ are constants.

According to the search direction (30) and the Wolfe conditions (31) and (32), we present the explicit steps of the (CGCG) technique, as follows.

Now, we present some numerical facts relating to the above parameters, allowing us to discuss the global convergence and descent properties of the CGCG method.

The purpose of the following remark is to facilitate the convergence analysis proof of the CGCG method.

Remark 1. 

When both sides of (30) are multiplied by the $g_{{}_{k+1}}^T$, we obtain the following:

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}=-{\parallel g_{{}_{k+1}}\parallel }^2+{\beta }_{{}_k}^{N1}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)+(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{N2}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)-{\lambda }_{{}_k}{\beta }_{{}_k}^{N3}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right).$$

(33)

Then, the fourth term of (33) satisfies the following inequality for each iteration k:

$${\lambda }_{{}_k}{\beta }_{{}_k}^{N3}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)\ge 0.$$

(34)

Since $0\le {\lambda }_{{}_k}\le 1$, and if the first branch of (28) is satisfied, then ${\beta }_{{}_k}^{N3}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)=0$; otherwise, ${\beta }_{{}_k}^{N3}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)>0$ since $L{c}_{{}_k}\ge {\chi }_{{}_k}$ and it is clear that $0<{\chi }_{{}_k}<\frac{1}{4}$.

The third term of (33) can be reformulated as follows:

$$(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{N2}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)=\left\{\begin{array}{cc}(1-{\lambda }_{{}_k}){\chi }_{{}_k}{\parallel g_{{}_{k+1}}\parallel }^2\hfill & \mathit{if}\frac{{\chi }_{{}_k}{\parallel g_{{}_{k+1}}\parallel }^2}{g_{{}_{k+1}}^T{d}_{{}_k}}\ge L{b}_{{}_k},\hfill \\ \\ \frac{(1-{\lambda }_{{}_k})\left(y_{{}_k}^T{g}_{{}_{k+1}}\right)\left(d_{{}_k}^T{y}_{{}_k}\right)\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)-\left|\right|y_{{}_k}{\left|\right|}^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2}{r\parallel d_{{}_k}{\parallel }^2{\parallel y_{{}_k}\parallel }^2}\hfill & \mathit{otherwise},\hfill \end{array}\right.$$

(35)

The second term of (33) can be reformulated as follows:

$${\beta }_{{}_k}^{N1}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)=\left\{\begin{array}{cc}0\hfill & \mathit{if}L{a}_{{}_k}\le 0,\hfill \\ \\ \frac{\left(y_{{}_k}^T{g}_{{}_{k+1}}\right)\left(d_{{}_k}^T{y}_{{}_k}\right)\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)-2\left|\right|y_{{}_k}{\left|\right|}^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2}{{\left(d_{{}_k}^T{y}_{{}_k}\right)}^2}\hfill & \mathit{otherwise},\hfill \end{array}\right.$$

(36)

We set the second term of Formula (35) as follows:

$$Mu=\frac{(1-{\lambda }_{{}_k})\left(y_{{}_k}^T{g}_{{}_{k+1}}\right)\left(d_{{}_k}^T{y}_{{}_k}\right)\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)-\left|\right|y_{{}_k}{\left|\right|}^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2}{r\parallel d_{{}_k}{\parallel }^2{\parallel y_{{}_k}\parallel }^2}.$$

The following inequality $u^{T}v\le \frac{1}{2}{\left(\right|\left|u\right||}^2+{\parallel v\parallel }^2)$ is applied to the first expression of the $Mu$ numerator, with $u=d_{{}_k}{g}_{k+1}^T{y}_{{}_k}$, $v=y_{{}_k}{g}_{{}_{k+1}}^T{d}_{{}_k}$.

Then,

$$Mu\le \frac{(1-{\lambda }_{{}_k})\frac{1}{2}\left[\parallel d_{{}_k}{\parallel }^2\parallel g_{k+1}{\parallel }^2\parallel y_{{}_k}{\parallel }^2+\parallel y_{{}_k}{\parallel }^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2-2{\parallel y_{{}_k}\parallel }^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2\right]}{r\parallel d_{{}_k}{\parallel }^2{\parallel y_{{}_k}\parallel }^2}\le \frac{(1-{\lambda }_{{}_k}){\parallel g_{k+1}\parallel }^2}{2r}$$

then,

$$Mu\le \frac{(1-{\lambda }_{{}_k}){\parallel g_{k+1}\parallel }^2}{2r}.$$

(37)

Similarly, the second branch of Formula (36) is rewritten as follows:

$$\widehat{Mu}=\frac{\left(y_{{}_k}^T{g}_{{}_{k+1}}\right)\left(d_{{}_k}^T{y}_{{}_k}\right)\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)-2{\parallel y_{{}_k}\parallel }^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2}{{\left(d_{{}_k}{y}_{{}_k}\right)}^2}\le \frac{\frac{1}{2}\left[\parallel g_{k+1}{\parallel }^2{\left(d_{{}_k}^T{y}_{{}_k}\right)}^2-3{\parallel y_{{}_k}\parallel }^2{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}^2\right]}{{\left(d_{{}_k}^T{y}_{{}_k}\right)}^2}\le \frac{1}{2}{\parallel g_{k+1}\parallel }^2.$$

Therefore,

$$\widehat{Mu}\le \frac{1}{2}{\parallel g_{k+1}\parallel }^2.$$

(38)

Note: As we have mentioned above, the four results that are listed in Formulas (34)–(38) of Remark 1 are applied in the convergence analysis of the suggested method described in Section 2.

The convergence analysis and descent property of the CGCG technique are presented in the following section.

2.1. Convergence Analysis

The convergence analysis of Algorithm 1 is presented as follows. First, two useful hypotheses are listed as follows.

Hypothesis 1. 

Let a function $f\left(x\right)$ be continuously differentiable.

Hypothesis 2. 

In a few neighborhoods N of the level group, $𝓁=\{x\in {\Re }^n:f\left(x\right)\le f\left(x_{{}_0}\right)\}$, the gradient vector $g\left(x\right)$ of the function $f\left(x\right)$ is Lipschitz-continuous.

Therefore, there is a fixed $L<\infty$ that satisfies the following inequality $\parallel g\left(x\right)-g\left(y\right)\parallel \le L\parallel x-y\parallel$, $\forall x,y\in N$.

The following lemma shows that the search direction $d_{{}_{k+1}}$ defined by Formula (30) is a descent direction.

Lemma 1. 

Let $\left\{x_{{}_k}\right\}$ be the sequence obtained using Algorithm 1. If $d_{{}_k}^T{y}_{{}_k}\ne 0$; then,

$$g_{k+1}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2,$$

(39)

where $0\le C<1$.

Proof. 

If $k=0$, it follows from (30) that $g_{{}_1}^T{d}_{{}_1}=-{\parallel g_{{}_1}\parallel }^2<0$, for $k\ge 1$, proving that (39) is true.

According to (34) and Remark 1, the following cases exist that can be used to prove that Formula (39) is true.

Case I: When ${\beta }_{{}_k}^{N1}=0$ and ${\beta }_{{}_k}^{N2}=L{b}_{{}_k}$ and based on (37), then Formula (33) is as follows

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -\parallel g_{{}_{k+1}}{\parallel }^2+(1-{\lambda }_{{}_k})L{b}_{{}_k}\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)\le -\parallel g_{{}_{k+1}}{\parallel }^2+\frac{(1-{\lambda }_{{}_k}){\parallel g_{{}_{k+1}}\parallel }^2}{2r}\le {\parallel g_{{}_{k+1}}\parallel }^2(-1+\frac{1}{2r}),$$

we set $C=-1+\frac{1}{2r}$.

Then,

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2,$$

(40)

where $0\le C<1$.

Therefore, (39) is met.

Case II: When ${\beta }_{{}_k}^{N1}=0$ and ${\beta }_{{}_k}^{N2}={\chi }_{{}_k}\frac{\parallel g_{{}_{k+1}}{\parallel }^2}{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}$, then Formula (33) is as follows.

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -\parallel g_{{}_{k+1}}{\parallel }^2+(1-{\lambda }_{{}_k}){\chi }_{{}_k}\parallel g_{{}_{k+1}}{\parallel }^2\le {\parallel g_{{}_{k+1}}\parallel }^2(-1+{\chi }_{{}_k}),$$

we set $C=-1+{\chi }_{{}_k}$, with $0<{\chi }_{{}_k}<\frac{1}{4}$. Then,

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2,$$

(41)

where $0<C<1$. Therefore, (39) is met.

Case III: When ${\beta }_{{}_k}^{N1}=L{a}_{{}_k}$ and ${\beta }_{{}_k}^{N2}={\chi }_{{}_k}\frac{\parallel g_{{}_{k+1}}{\parallel }^2}{\left(g_{{}_{k+1}}^T{d}_{{}_k}\right)}$, and based on (38), then Formula (33) is as follows:

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -\parallel g_{{}_{k+1}}{\parallel }^2+\frac{1}{2}\parallel g_{k+1}{\parallel }^2+(1-{\lambda }_{{}_k}){\chi }_{{}_k}\parallel g_{{}_{k+1}}{\parallel }^2\le {\parallel g_{{}_{k+1}}\parallel }^2(-\frac{1}{2}+{\chi }_{{}_k}),$$

we set $C=-\frac{1}{2}+{\chi }_{{}_k}$ where $0<{\chi }_{{}_k}<\frac{1}{4}$.

Then,

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2,$$

(42)

where $0<C<1$; thus, (39) is met.

Case IV: When ${\beta }_{{}_k}^{N1}=L{a}_{{}_k}$ and ${\beta }_{{}_k}^{N2}=L{b}_{{}_k}$, and based on Formulas (37) and (38), then Formula (33) is as follows:

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -\parallel g_{{}_{k+1}}{\parallel }^2+\frac{1}{2}\parallel g_{{}_{k+1}}{\parallel }^2+\frac{1}{2r}(1-{\lambda }_{{}_k})\parallel g_{{}_{k+1}}{\parallel }^2\le {\parallel g_{{}_{k+1}}\parallel }^2(-\frac{1}{2}+\frac{1}{2r}),$$

we set $C=-\frac{1}{2}+\frac{1}{2r}$.

Then,

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2$$

(43)

where $0\le C<1$; thus, (39) is met.

Hence, the above four cases show that (39) is true. □

Below, some hypotheses and a useful lemma are presented, where the obtained result was essentially proven by the author of [60] and the author of [61,62].

Lemma 2. 

Let $x_{{}_0}$ be a starting point that satisfies Hypothesis 1. Regard any algorithm to be of Form (18), such that the $d_{{}_k}$ satisfies (19) and the ${\alpha }_{{}_k}$ satisfies conditions (31) and (32).

Hence, the following inequality is met.

$${\displaystyle \sum _{k=0}^{\infty }\frac{{\left(g_{{}_k}^T{d}_{{}_k}\right)}^2}{\parallel d_{{}_k}{\parallel }^2}<\infty }$$

(44)

Theorem 1. 

Suppose that Hypotheses 1 and 2 are met and that the following is satisfied: ${\displaystyle \sum _{k=0}^{\infty }}\frac{1}{\parallel d_{{}_k}{\parallel }^2}=+\infty$. Then, the sequence $\left\{g_{{}_{k+1}}\right\}$ generated using the CGCGC method satisfies the next result.

$$\underset{k\to \infty }{lim}inf\parallel g_{{}_{k+1}}\parallel =0,$$

(45)

Proof. 

Proof by contradiction: assume that (45) is incorrect; hence, for some $\xi >0$.

Then, the next result is correct.

$$\parallel g_{{}_{k+1}}\parallel \ge \xi .$$

(46)

According to (46) and (19), the following result is obtained

$$g_{{}_{k+1}}^T{d}_{{}_{k+1}}\le -C{\parallel g_{{}_{k+1}}\parallel }^2\le -{\xi }^2,$$

(47)

and then

$$\frac{\left(g_{{}_{k+1}}^T{d}_{{}_{k+1}}\right)}{\parallel d_{{}_{k+1}}\parallel }\le \frac{-{\xi }^2}{\parallel d_{{}_{k+1}}\parallel };$$

$$\frac{{\left(g_{{}_{k+1}}^T{d}_{{}_{k+1}}\right)}^2}{\parallel d_{{}_{k+1}}{\parallel }^2}\ge \frac{{\xi }^4}{\parallel d_{{}_{k+1}}{\parallel }^2},$$

By summing the final expression, we obtain

$${\displaystyle \sum _{k=0}^{\infty }\frac{{\left(g_{{}_{k+1}}^T{d}_{{}_{k+1}}\right)}^2}{\parallel d_{{}_{k+1}}{\parallel }^2}\ge \sum _{k=0}^{\infty }\frac{{\xi }^4}{\parallel d_{{}_{k+1}}{\parallel }^2}=\infty .}$$

(48)

Since (48) is in contradiction with (44), (45) is true as long as $k\to \infty$. □

Computational Cost Analysis of CGCG Method

In general, any modified version of the conjugate gradient method has low memory requirements and convergence speed [1] but converges much faster than the steepest descent method [19,63].

Since the quasi-Newton methods engender a need to keep a $n\times n$ matrix $H_{{}_k}$ (the inverse of the approximate Hessian $B_{{}_k}$) or $L_{{}_k}$ (the Cholesky factor of $B_{{}_k}$) in a computer’s memory, a Quasi-Newton method needs $O\left(n^2\right)$ data units [64].

On the other hand, by using the CGCG method, we can compute the values of the gradient vector only; therefore, the CGCG method has an $O\left(n\right)$ memory requirement, i.e., $O\left(n\right)$ complexity for each iteration.

3. Numerical Investigations

Numerical investigations are applied to test the effectiveness and robustness of the CGCG approach by solving a set of unconstrained optimization problems.

Therefore, the performance of Algorithm 1 (CGCG) is tested by solving the 76 benchmark test problems with variable dimensions between 100 to 800,000. These test problems were taken from [65,66].

These test problems have recently been used by many authors to test the efficiency and effectiveness of their proposed algorithms (see, for example, [25,58]).

These test problems are processed on a personal computer (a laptop) with an Intel(R) Core(TM) i5-3230M [email protected] and 2.60 GHz with 4.00 GB of RAM using the Windows 10 operating system.

This CGCG method, which is listed in Algorithm 1, is proposed to obtain a new set of CG methods, as the CGCG method contains four terms.

The performance of the CGCG method is examined in this section. Therefore, 76 test problems are used to complete this task. In addition, five classical CG methods are used to solve these 76 test problems. These five classical CG methods are the HS-CG method defined by (4), the FR-CG method defined by (5), the LS-CG method defined by (7), the DY-CG method defined by (8), and the HZ-CG method defined by (9).

The numerical results of all six CG methods are documented in

Table 1. The Number of Iterations (Itr).

Pr	n	CGCG	HZ	DY	LS	FR	HS	Pr	n	CGCG	HZ	DY	LS	FR	HS
1	6000	32	40	270	238	33	F	39	70,000	2	2	2	2	2	2
2	100,000	39	79	F	F	F	F	40	240,000	2	2	2	2	2	2
3	800,000	26	21	F	101	F	101	41	6000	83	140	52	45	48	30
4	6000	22	21	13	27	22	28	42	30,000	197	200	166	89	147	45
5	90,000	24	19	25	49	26	38	43	4000	35	93	52	1462	F	1501
6	24,000	18	13	18	39	79	28	44	10,000	17	17	20	563	17	57
7	48,000	22	18	18	52	16	29	45	4000	57	58	65	75	54	57
8	2700	18	18	30	43	37	24	46	80,000	86	134	133	123	172	111
9	27,000	19	17	22	65	23	23	47	500,000	144	151	203	190	268	176
10	12,000	21	21	54	28	47	42	48	300	635	823	1178	977	1712	623
11	90,000	27	59	20	60	19	36	49	2000	2108	1450	F	2762	F	2374
12	2400	449	409	562	429	789	264	50	150,000	356	587	F	219	F	191
13	48,000	1178	1111	1831	1977	F	1217	51	5000	2	2	2	2	2	F
14	15,000	770	709	F	1038	F	534	52	50,000	2	2	2	2	2	F
15	60,000	647	1089	F	1415	F	1354	53	500,000	2	2	2	2	2	F
16	12,000	576	2007	F	615	1161	499	54	300	1663	1878	1132	2159	F	1845
17	90,000	856	591	F	1627	F	1226	55	100,000	35	55	F	2629	F	F
18	6000	502	342	F	636	F	378	56	250,000	56	97	F	2030	F	F
19	150,000	442	812	F	2058	2810	1501	57	500,000	120	149	F	523	F	F
20	360	1616	1079	1512	2486	F	1501	58	100	2047	F	F	1326	F	2566
21	3000	407	412	F	553	F	418	59	5000	73	193	F	94	2312	401
22	15,000	529	556	442	515	F	562	60	80,000	53	80	F	264	233	F
23	12,000	467	762	F	533	F	445	61	150	2202	2704	1745	2907	F	2345
24	120,000	620	721	1515	921	F	589	62	500	243	241	400	235	573	168
25	2400	296	349	F	395	F	306	63	5000	960	1169	F	F	F	F
26	24,000	459	420	F	552	F	390	64	2000	12	12	12	663	9	F
27	150	2002	2455	1317	2848	F	1132	65	20,000	10	10	11	481	25	23
28	9000	87	81	90	118	158	72	66	500,000	17	12	32	548	108	F
29	90,000	108	63	71	124	87	91	67	800	637	719	2037	1916	F	F
30	5000	61	60	55	59	54	67	68	2000	1061	749	F	F	F	F
31	150,000	114	115	125	152	166	124	69	8000	686	1025	651	709	F	F
32	7000	40	51	54	58	124	68	70	50,000	1731	1983	F	F	F	F
33	40,000	43	43	53	69	100	172	71	500	583	291	1481	641	F	2214
34	500,000	44	43	194	167	201	299	72	2000	1687	857	F	F	F	F
35	100	862	1574	133	199	170	103	73	500	162	178	297	67	146	48
36	1000	75	50	113	124	F	47	74	1000	112	123	103	61	89	43
37	50,000	99	138	164	246	F	F	75	500	64	55	316	102	387	83
38	200,000	239	307	350	296	F	F	76	8000	95	96	401	111	738	77

,

Table 2. The Number of Function Evaluations (FEs).

Pr	n	CGCG	HZ	DY	LS	FR	HS	Pr	n	CGCG	HZ	DY	LS	FR	HS
1	6000	92	100	878	1050	103	F	39	70,000	15	15	15	15	15	15
2	100,000	113	201	F	F	F	F	40	240,000	13	13	13	13	13	13
3	800,000	99	85	F	536	F	408	41	6000	256	364	203	272	193	177
4	6000	83	90	70	163	81	172	42	30,000	751	670	552	647	603	237
5	90,000	87	82	87	367	91	179	43	4000	267	721	347	15,993	F	1528
6	24000	78	76	81	255	216	168	44	10,000	83	83	85	6096	84	451
7	48,000	85	81	81	312	80	152	45	4000	188	174	170	331	223	170
8	2700	77	75	92	292	108	129	46	80,000	308	376	311	557	624	342
9	27,000	80	87	86	467	86	138	47	500,000	498	475	423	815	823	468
10	12,000	81	81	132	147	135	217	48	300	1406	1770	1246	3472	3811	990
11	90,000	95	183	86	394	95	174	49	2000	4677	3108	F	9941	F	3775
12	2400	969	878	621	1509	1725	444	50	150,000	959	1427	F	1075	F	572
13	48,000	2547	2365	1896	6952	F	1865	51	5000	11	11	11	11	11	F
14	15,000	1741	1483	F	3753	F	876	52	50,000	16	16	16	16	16	F
15	60,000	1428	2321	F	5006	F	2098	53	500,000	12	12	12	12	12	F
16	12,000	1252	4374	F	2261	2581	784	54	300	3730	3963	1181	7450	F	2904
17	90,000	1866	1273	F	5728	F	1926	55	100,000	134	184	F	11,605	F	F
18	6000	1157	722	F	2305	F	635	56	250,000	164	268	F	9226	F	F
19	150,000	1010	1732	F	7527	6240	2445	57	500,000	314	399	F	2562	F	F
20	360	3603	2302	1566	8988	F	2360	58	100	4714	F	F	3219	F	4587
21	3000	955	914	F	2053	F	697	59	5000	173	409	F	462	5308	1259
22	15,000	1143	1183	516	1853	F	891	60	80,000	154	190	F	1156	550	F
23	12,000	1072	1651	F	1852	F	699	61	150	4790	5752	1826	10,182	F	3578
24	120,000	1358	1564	1584	3292	F	932	62	500	505	487	467	846	1772	299
25	2400	659	751	F	1550	F	491	63	5000	2146	2550	F	F	F	F
26	24,000	979	942	F	1855	F	636	64	2000	63	63	63	7281	65	F
27	150	4519	5309	1358	9937	F	1759	65	20,000	72	72	78	5272	93	155
28	9000	250	245	266	718	429	293	66	500,000	107	77	225	6011	614	F
29	90,000	302	208	203	688	222	298	67	800	4542	5666	18,079	18,920	F	F
30	5000	187	188	164	268	199	195	68	2000	7534	4414	F	F	F	F
31	150,000	338	333	281	651	569	371	69	8000	1478	2190	881	2701	F	F
32	7000	121	256	149	295	360	377	70	50,000	3967	4354	F	F	F	F
33	40,000	112	150	209	425	642	894	71	500	4126	1067	12,906	5205	F	18,987
34	500,000	179	160	1012	1259	1040	2107	72	2000	13,364	4084	F	F	F	F
35	100	2514	4234	449	1164	565	394	73	500	547	495	811	460	509	258
36	1000	367	185	793	944	F	175	74	1000	378	386	366	444	296	209
37	50,000	723	1156	1371	2360	F	F	75	500	134	125	372	404	813	167
38	200,000	2238	2895	3007	2885	F	F	76	8000	228	205	463	404	1622	159

and

Table 3. The CPUT.

Pr	n	CGCG	HZ	DY	LS	FR	HS	Pr	n	CGCG	HZ	DY	LS	FR	HS
1	6000	0.13	0.10	0.991	0.93	0.13	F	39	70,000	0.21	0.24	0.29	0.19	0.30	0.21
2	100,000	2.22	2.76	F	F	F	F	40	240,000	0.62	0.68	0.61	0.53	0.69	0.62
3	800,000	19.65	12.15	F	80.67	F	60.77	41	6000	0.03	0.09	0.11	0.06	0.19	0.02
4	6000	0.41	0.33	0.42	0.85	0.49	0.70	42	30,000	2.41	2.71	1.74	1.70	2.71	0.72
5	90,000	4.64	4.31	4.51	17.83	6.16	9.63	43	4000	17.49	46.16	18.48	812.32	F	82.43
6	24,000	1.30	1.34	1.24	3.94	4.25	2.57	44	10,000	27.26	24.97	22.13	1627.78	26.26	115.85
7	48,000	2.77	2.45	2.41	9.22	3.00	4.47	45	4000	0.39	0.35	0.39	0.52	0.47	0.31
8	2700	0.18	0.18	0.23	0.72	0.37	0.29	46	80,000	11.58	13.59	10.06	15.68	22.40	11.88
9	27,000	1.52	1.47	1.43	7.99	1.79	2.37	47	500,000	97.81	104.77	96.01	163.74	183.61	95.10
10	12,000	0.65	0.66	1.04	1.18	1.35	1.62	48	300	0.17	0.19	0.17	0.55	0.46	0.08
11	90,000	5.37	10.18	4.49	20.07	6.52	9.25	49	2000	1.43	0.96	F	2.02	F	1.00
12	2400	1.67	1.59	1.16	2.81	3.86	0.68	50	150,000	21.70	29.56	F	15.78	F	9.46
13	48,000	65.39	68.58	53.76	204.81	F	52.20	51	5000	0.03	0.03	0.10	0.07	0.27	F
14	15,000	18.09	13.91	F	37.72	F	8.69	52	50,000	0.34	0.37	0.39	0.28	0.34	F
15	60,000	55.80	80.29	F	182.95	F	74.44	53	500,000	2.50	3.17	3.56	2.51	2.86	F
16	12,000	10.79	34.13	F	18.51	23.95	6.03	54	300	0.42	0.38	0.21	0.44	0.60	0.24
17	90,000	103.80	69.11	F	309.50	359.16	101.07	55	100,000	2.20	3.19	F	151.62	272.38	F
18	6000	5.01	2.99	F	10.05	25.60	2.48	56	250,000	10.05	16.87	F	484.03	983.93	F
19	150,000	87.50	164.52	F	694.51	633.25	228.25	57	500,000	40.05	49.90	F	266.80	2010.13	F
20	360	1.54	1.76	0.821	4.20	F	1.12	58	100	0.82	F	F	0.43	1.54	0.63
21	3000	2.64	2.62	F	4.28	F	1.48	59	5000	0.13	0.29	F	0.30	3.73	0.73
22	15,000	11.71	12.91	4.44	14.98	F	7.04	60	80,000	1.54	1.66	F	8.66	5.27	F
23	12,000	9.52	12.55	F	13.64	F	4.42	61	150	0.78	0.51	0.24	0.51	F	0.29
24	120,000	105.00	115.81	106.79	208.11	F	60.85	62	500	0.12	0.06	0.11	0.11	0.27	0.07
25	2400	1.27	1.55	F	2.17	F	0.71	63	5000	1.66	1.27	F	F	F	F
26	24,000	16.33	14.79	F	23.12	F	8.78	64	2000	0.02	0.03	0.03	1.59	0.09	F
27	150	0.61	0.55	0.17	0.45	F	0.13	65	20,000	0.19	0.18	0.19	10.16	0.24	0.31
28	9000	0.30	0.22	0.22	0.48	0.34	0.23	66	500,000	7.26	4.92	13.53	336.84	36.76	F
29	90,000	3.40	1.75	1.33	4.29	1.56	2.09	67	800	0.92	0.99	3.19	3.17	F	F
30	5000	0.42	0.46	0.37	0.52	0.44	0.44	68	2000	3.05	1.60	F	F	F	F
31	150,000	21.53	22.64	15.09	37.37	34.91	23.84	69	8000	2.74	3.28	1.45	3.68	F	F
32	7000	0.46	1.02	0.51	1.26	1.11	1.43	70	50,000	41.47	40.49	27.65	F	F	F
33	40,000	2.40	3.23	3.51	9.20	11.58	16.90	71	500	0.61	0.20	2.04	0.70	F	2.53
34	500,000	44.34	42.78	241.73	257.40	207.06	462.33	72	2000	6.51	1.95	F	F	F	F
35	100	0.30	0.41	0.05	0.12	0.11	0.03	73	500	1.16	1.10	1.92	0.90	1.47	0.54
36	1000	0.08	0.08	0.17	0.13	F	0.03	74	1000	3.33	3.51	3.25	3.37	2.89	1.63
37	50,000	6.17	8.02	9.47	13.19	F	F	75	500	0.28	0.30	0.85	0.71	2.14	0.42
38	200,000	69.75	80.96	80.44	70.40	F	F	76	8000	67.61	63.67	107.70	105.84	560.69	40.32

.

Next, the performance of all six algorithms is compared. The six methods are programmed in the MATLAB language (version 8.5.0.197613 (R2015a)).

The numerical comparisons present the advantages and disadvantages of each of the six algorithms. Therefore, three criteria are used to evaluate the performance of all six algorithms. These three criteria are “Itr”, denoting the number of iterations; “EFs”, denoting the functional evaluations; and “CPUT”, denoting the time.

We use the following termination form when running the six algorithms

$$\epsilon =10.00\times {10}^{-6},$$

(49)

or Itr $>3000$, i.e., if the number of iterations exceeds 3000.

Accordingly, when $\parallel g\left(x_{{}_k}\right){\parallel }^2\le \epsilon$ or Itr $\ge 3000$, the algorithm stops. The standard for determining the success of the algorithm in solving a test problem is as follows. If Itr $\le 3000$ and $\parallel g\left(x_{{}_k}\right){\parallel }^2\le \epsilon$, the algorithm has succeeded in solving the problem; otherwise, it has failed to solve the problem, which is denoted by “F” as shown in Table 1, Table 2 and Table 3.

Moreover, to display the test results clearly, we use a performance profile tool developed by Dolan and Moré [67]. More details about the performance profile and how it is used can be found in [67,68,69,70].

The significant feature of the performance profile constitutes its presentation of all results that are listed on one table in one figure by plotting an accumulative distribution function ${\rho }_{{}_s}\left(\tau \right)$ for the many algorithms.

The numerical outcomes recorded in Table 1, Table 2 and Table 3 and all three graphs, which are depicted in Figure 1 and Figure 2, display the performance of all six algorithms.

The results in Table 1 present the number of iterations (Itr) for all six CG parameters. It is clear that the CGCG algorithm (Algorithm 1) was capable of solving all the test problems (76 benchmark test problems).

Hence, the CGCG method satisfies the following condition (success standard): Itr $<3000$ and $\parallel g\left(x_{{}_k}\right){\parallel }^2\le \epsilon$, for all test problems. The second-best performance was exhibited by the HZ method.

According to the graph on the left of Figure 1 with $\tau =10$, the success rates of the six algorithms are ordered as follows: $100\%$, $97\%$, $68\%$, $89\%$, $54\%$, and $74\%$, for CGCG, HZ, DY, LS, FR, and HS, respectively.

The seam notice for the remaining Table 2 and Table 3 for the FEs and the CPU time, respectively.

The performance of the six algorithms examined via performance profiles is shown in Figure 1 and Figure 2. By utilizing the performance profile theory, we generated three performance levels, which are shown Figure 1 and Figure 2.

These two figures are based on the results listed in Table 1, Table 2 and Table 3 for the Itr, EFs, and CPU time for all 76 test problems, respectively.

It is clear from the two figures that the CGCG algorithm has the characteristics of efficiency, reliability, and effectiveness in solving the 76 test problems compared to the other five methods.

The performance of the six algorithms can be summarized by referencing Figure 1 and Figure 2, as follows. The performance of the six methods with respect to the Itr criterion is shown as follows.

At $\tau =1$, the success rates of the 6 algorithms are arranged as follows: $75\%$, $41\%$, $34\%$, $26\%$, $14\%$, and $13\%$ for the LS, CGCG, HZ, HS, FR, and DY methods, respectively, corresponding to the left graph in Figure 1.

At $\tau =2$, the success rates of the six algorithms are ordered as follows: $92\%$, $83\%$, $83\%$, $58\%$, $49\%$, and $28\%$ for the CGCG, HZ, LS, HS, DY, and FR, respectively, as shown in the left graph in Figure 1.

At $\tau =3$, the success rates of the six algorithms are ordered as follows: $96\%$, $89\%$, $84\%$, $64\%$, $58\%$ and $39\%$ for the CGCG, HZ, LS, HS, DY, and FR, respectively, as shown in the left graph in Figure 1.

At $\tau =4$, the success rates of the six algorithms are ordered as follows: $97\%$, $93\%$, $86\%$, $68\%$, $61\%$, and $45\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, shown in the left graph in Figure 1.

At $\tau =5$, the success rates of the six algorithms are ordered as follows: $99\%$, $97\%$, $87\%$, $70\%$, $62\%$, and $47\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, as shown in the left graph of Figure 1.

At $\tau =6$, the success rates of the six algorithms are ordered as follows: $99\%$, $97\%$, $88\%$, $71\%$, $66\%$ and $47\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, as shown in the left graph in Figure 1.

At $\tau =7$, the success rates of the six algorithms are ordered as follows: $99\%$, $97\%$, $88\%$, $72\%$, $67\%$, and $50\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, as shown in the left graph in Figure 1.

At $\tau =8$, the success rates of the six algorithms are ordered as follows: $99\%$, $97\%$, $88\%$, $74\%$, $67\%$, and $51\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, as shown in the left graph in Figure 1.

At $\tau =9$, the success rates of the six algorithms are ordered as follows: $100\%$, $97\%$, $89\%$, $74\%$, $68\%$, and $53\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, as shown in the left graph in Figure 1.

At $\tau =10$, the success rates of the six algorithms are ordered as follows: $100\%$, $97\%$, $89\%$, $74\%$, $68\%$, and $54\%$ for the CGCG, HZ, LS, HS, DY, and FR methods, respectively, as shown in the left graph in Figure 1.

The performance of the six methods with respect to the FEs criterion is shown as follows.

At $\tau =1$, the success rates of the six algorithms are ordered as follows: $36\%$, $30\%$, $28\%$, $25\%$, $8\%$, and $8\%$, for the CGCG, HS, HZ, DY, LS, and FR, methods, respectively, as shown in the right graph of Figure 1.

At $\tau =2$, the success rates of the six algorithms are ordered as follows: $84\%$, $79\%$, $55\%$, $54\%$, $33\%$, and $20\%$, for the CGCG, HZ, DY, HS, FR, and LS methods, respectively, in the right-hand graph in Figure 1.

At $\tau =3$, the success rates of the six algorithms are ordered as follows: $92\%$, $92\%$, $64\%$, $61\%$, $39\%$ and $38\%$, for the CGCG, HZ, HS, DY, FR and LS, for the right graph of Figure 1.

At $\tau =4$, the success rates of the six algorithms are ordered as follows: $99\%$, $96\%$, $66\%$, $63\%$, $61\%$ and $45\%$, for the CGCG, HZ, HS, DY, LS and FR for the right graph of Figure 1.

At $\tau =5$, the success rates of the six algorithms are ordered as follows: $99\%$, $96\%$, $68\%$, $67\%$, $64\%$ and $45\%$, for the CGCG, HZ, HS, LS, DY and FR for the right graph of Figure 1.

At $\tau =6$, the success rates of the six algorithms are ordered as follows: $99\%$, $97\%$, $74\%$, $70\%$, $64\%$ and $47\%$, for the CGCG, HZ, LS, HS, DY and FR for the right graph of Figure 1.

At $\tau =7$, the success rates of the six algorithms are ordered as follows: $99\%$, $97\%$, $76\%$, $70\%$, $66\%$ and $51\%$, for the CGCG, HZ, LS, HS, DY and FR for the right graph of Figure 1.

At $\tau =8$, the success rates of the six algorithms are ordered as follows: $100\%$, $97\%$, $82\%$, $72\%$, $66\%$ and $53\%$, for the CGCG, HZ, LS, HS, DY and FR for the right graph of Figure 1.

At $\tau =9$, the success rates of the six algorithms are ordered as follows: $100\%$, $97\%$, $83\%$, $72\%$, $66\%$ and $53\%$, for the CGCG, HZ, LS, HS, DY and FR for the right graph of Figure 1.

At $\tau =10$, the success rates of the six algorithms are ordered as follows: $100\%$, $97\%$, $83\%$, $72\%$, $67\%$, $53\%$, for the CGCG, HZ, LS, HS, DY, and FR for the right graph of Figure 1.

In addition, for Figure 2 and at $\tau =10$, the success rates of the six algorithms are ordered as follows $100\%$, $97\%$, $68\%$, $84\%$, $55\%$ and $72\%$ for the CGCG, HZ, DY, LS, FR, and HS methods, respectively.

In general, all the graphs show that the curve of the function ${\rho }_{{}_s}\left(\tau \right)$ is almost maximal for the CGCG method. Therefore, the comparison results indicate that the CGCG approach is competitive with, and in all cases superior to, the five other CG methods in terms of efficiency, reliability, and effectiveness with regard to solving the set of test problems.

4. Image Processing

This section constitutes the second part of our numerical experiments. This section aims to render the CGCG algorithm capable of dealing with image-processing problems.

Images are often corrupted by various sources (factors) that may be responsible for the introduction of an artifact in a photo.

Therefore, the number of degraded pixels in a photo corresponds to the amount of noise that has been introduced into the image.

The main sources of noise in a digital image are as follows: [71,72].

• Some environmental conditions may impact the efficiency of an imaging sensor during image acquisition.
• Noise may be introduced into an image through an inappropriate sensor temperature and low light levels.
• In addition, an image can deteriorated (corrupted) due to interference in the transmission channel.
• Similarly, the image may be deteriorated (corrupted) due to dust particles that may exist on the scanner screen.

Images are often deteriorated by batch fuzz due to noisy sensors or transmission channels that lead to the corruption of a number of pixels in the picture.

A batch fuzz is one of the numerous popular noise samples in which only a portion of the pixels is degraded, i.e., the information of the pixels is entirely lost.

Generally, different photos with several implementations require treatment using a fine noise funnel technique to obtain credible effects and thus restore the original picture.

To restore an original photo that has been deteriorated by batch noise, a two-phase scheme is used, in which the first phase entails determining the fuzz-affected pixels in the corrupted photo, which is executed using the adaptive median filter algorithm [73,74,75,76,77].

In the first stage, the adaptive median filter algorithm determines the noise in the corrupted image, as follows. The filter compares each pixel in the distorted image to the surrounding pixels. If one of the pixel values varies significantly from the majority of the surrounding pixels, the pixel is treated as noise. More details about the adaptive median filter algorithm and salt-and-pepper noise removal by median type are available in [73,74,75,76,78,79].

The second phase of the scheme involves restoring the original image by using any algorithm that solves optimization problems.

Chan et al. [73] have applied the two-phase scheme to restore a corrupted image.

Many authors have used these two phases to render CG algorithms cable of restoring images corrupted by impulse noise (see, for example [25,42,43]).

The two-phase scheme can be briefly described as follows.

By applying the first stage, we use the adaptive median filter to select the corrupted pixels [73].

In the second phase, assume that the corrupted photo, denoted by $\delta$, has a size of $\tau -$ by $-\varrho$ and $\delta I=\{1,2,\dots ,\tau \}\times \{1,2,\dots ,\varrho \}$, constituting the index group of the photo $\delta$.

The set $\aleph \subset \delta I$ denotes the set of indices of the noise pixels detected from the first phase, and $|\aleph |$ is the number of elements of ℵ.

Let $V_{i,j}$ be the set of the four closest neighbors of the pixel at pixel location $(i,j)\in \delta I$, i.e., let $V_{i,j}=\{(i,j-1),(i,j+1),(i-1,j),(i+1,j)\}$, and $y_{i,j}$ be the observed pixel value (gray level) of the photo at pixel location $(i,j)$.

Therefore, in the second stage, the noise from the corrupted pixels is removed by solving the following non-smooth problem:

$$\underset{u}{min}\sum _{(i,j)\in \aleph }\left(|u_{i,j}-y_{i,j}|+\frac{\beta }{2}\left[S_{i,j}+{\tilde{S}}_{i,j}\right]\right),$$

(50)

where ${\displaystyle S_{i,j}=\sum _{(n,m)\in V_{i,j}\setminus N}{\phi }_{\alpha }(u_{i,j}-y_{n,m})}$, ${\displaystyle \tilde{S}=\sum _{(n,m)\in V_{i,j}\cap \aleph }{\phi }_{\alpha }(u_{i,j}-y_{n,m})}$, and $\phi$ is a potential edge-preserving function.

Some of these functions were defined in [73], as follows.

$${\phi }_{{}_{\alpha }}\left(t\right)=\left\{\begin{array}{cc}{\left|t\right|}^{\alpha }\hfill & \mathrm{for}1<\alpha \le 2,\hfill \\ \\ \sqrt{\alpha +t^2}\hfill & \mathrm{for}\alpha >0.\hfill \end{array}\right.$$

(51)

In this paper, we use the second branch of (51), employing different values of the $\alpha$, and $u={\left[u_{i,j}\right]}_{i,j}\in \aleph$ is a column vector of length $|\aleph |$ ordered lexicographically.

However, it is time consuming and costly to determine the minimizer point of a non-smooth problem (50) exactly.

The authors of [80] canceled the non-smooth term and presented the subsequent smooth unconstrained optimization problem:

$$\underset{u}{min}{F}_{\alpha }\left(u\right):=\sum _{(i,j)\in \aleph }\left(2\sum _{(i,j)\in V_{i,j}\setminus \aleph }{\phi }_{{}_{\alpha }}(u_{i,j}-y_{i,j})+\sum _{(m,n)\in V_{i,j}\cap \aleph }{\phi }_{{}_{\alpha }}(u_{i,j}-u_{m,n})\right).$$

(52)

Clearly, the greater the fuzz ratio, the greater the size of (52).

Cai et al. [80] revealed that degraded pictures may be repaired efficiently by utilizing the CG parameters to find the minimizer of Problem (52); however, in reality, the fuzz ratio is heightened or actually corresponds to $90\%$.

Newly proposed algorithms of CG parameters for solving this problem (image restoration) can be found in [25,42,81].

Now, we concentrate on using the two-stage procedure to clear salt-and-pepper fuzz that represents a particular state of batch noise.

The adaptive median filter approach that is described by the authors of [76] will be used as the first stage to detect noisy pixels.

Therefore, the CGCG method is modified to obtain three CGCG algorithms, which are adapted to solve Problem (52); hence, this case represents the second stage.

The following iterative form is used to generate the candidate solutions of Problem (52).

$$u_{k+1}=u_{{}_k}+{\alpha }_{{}_k}{d}_{{}_d},$$

(53)

where ${\alpha }_{{}_k}$ represents the step size computed by a line search method, and $d_{{}_k}$ is the search direction, which is defined as follows.

$$d_{{}_{k+1}}=-g_{{}_{k+1}}+{\beta }_{{}_k}{d}_{{}_k}+(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{N_{{}_2}}{d}_{{}_k}-{\lambda }_{{}_k}{\beta }^{N_{{}_3}}{d}_{{}_k},$$

(54)

where ${\beta }_{{}_k}\in \{{\beta }^{HS},{\beta }^{H{Z}^{*}}\}$.

This combination allows for the acquisition of integrated algorithms that inherit the features of the parameters defined in Formulas (4), (9), (25), and (27).

Therefore, the two Formulas (53) and (54) can be run iteratively through one of the following cases.

Case 1: If ${\beta }_{{}_k}={\beta }^{HS}$, then the CGCG-SH algorithm is used.

Case 2: If ${\beta }_{{}_k}={\beta }^{H{Z}^{*}}$, then the CGCG-HZ algorithm is used.

The selected size of the filter window for the adaptive median filter method has a key role in the phases uncovering the noisy pixels in the corrupted image.

Chan et al. [73] presented several volumes of the filter window (W) as noted by the fuzz level as follows:

When the noise level is less than $25\%$, then the maximum of the window is $W_{max}\times W_{max}=5\times 5$. When the noise level is between $25\%$ and $40\%$, $W_{max}\times W_{max}=7\times 7$; if the noise level is between $40\%$ and $60\%$, $W_{max}\times W_{max}=9\times 9$; if the noise level is between $60\%$ and $70\%$, $W_{max}\times W_{max}=13\times 13$; if the noise level is between $70\%$ and $80\%$, $W_{max}\times W_{max}=17\times 17$; if the noise level is between $80\%$ and $85\%$, $W_{max}\times W_{max}=25\times 25$; and if the noise level is between $85\%$ and $90\%$, $W_{max}\times W_{max}=39\times 39$.

In addition, the value of $\alpha$ in Problem (52) may be significant for finding the minimizer of Problem (52). Cai et al. [80] set $\alpha =100$.

Accordingly, we suggest two diffident values of the filter window (W) and the parameter $\alpha$, as follows.

Procedure: A We set $W_{max}\times W_{max}=5\times 5$, and if the noise ratio is less than or equal to $60\%$, then we set $\alpha =500$. Otherwise we set $\alpha =350$.

Consequently, by using Procedure: A and both Case 1 and Case 2, we design two algorithms for solving Problem (52). These tow algorithms are the CGCG-HS1 algorithm and CGCG-HZ algorithm; the research directions of the two algorithms are defined by

$$d_{{}_{k+1}}=-g_{{}_{k+1}}+{\beta }^{HS}{d}_{{}_k}+(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{N_{{}_2}}{d}_{{}_k}-{\lambda }_{{}_k}{\beta }^{N_{{}_3}}{d}_{{}_k},$$

(55)

and

$$d_{{}_{k+1}}=-g_{{}_{k+1}}+{\beta }^{H{Z}^{*}}{d}_{{}_k}+(1-{\lambda }_{{}_k}){\beta }_{{}_k}^{N_{{}_2}}{d}_{{}_k}-{\lambda }_{{}_k}{\beta }^{N_{{}_3}}{d}_{{}_k},$$

(56)

respectively.

Procedure: B If the noise ratio is equal to $30\%$, we set $W_{max}\times W_{max}=3\times 3$; if the noise ratio equals $50\%$, we set $W_{max}\times W_{max}=5\times 5$; if the noise ratio is equal to $70\%$, we set $W_{max}\times W_{max}=7\times 7$; and if the noise ratio is equal to $90\%$, we set $W_{max}\times W_{max}=9\times 9$. Additionally, we set $\alpha =100$ for any noise level.

According to Procedure: B, we used Formula (55) to solve all images problems; this algorithm is abbreviated as CGCG-HS2.

We assess the performance of these three incorporated algorithms in solving image restoration problems by utilizing the peak signal-to-noise ratio [82].

The peak signal-to-fuzz proportion (PSNR) is defined as follows:

$$PSNR=10lo{g}_{10}\frac{{255}^2}{\frac{1}{NM}{\sum }_{(i,j)}{\left(x_{(i,j)}^{res}-x_{(i,j)}^{ori}\right)}^2},$$

(57)

where $x_{(i,j)}^{res}$ and $x_{(i,j)}^{ori}$ are the pixel values of the fixed picture and the original one, respectively.

The examined pictures are Lena ($512\times 512$), Hill ($512\times 512$), Man ($512\times 512$), and Boat ($512\times 512$).

The stopping standard used is defined by the following conditions.

If one of the two conditions is met, $Itr>200$ or

$$\frac{F_{{}_{\alpha }}(U_{{}_k}-F_{{}_{\alpha }}\left(U_{{}_{k-1}}\right))}{F_{{}_{\alpha }}{U}_{{}_k}}\le {10}^{-4}.$$

(58)

The above procedures can be summarized as the following algorithms.

For further clarification of the working mechanism of the three proposed algorithms listed in Algorithms 2–4, the steps applied are shown in Figure 3.

In addition, Figure 4 depicts a graphical abstract of the operational scheme of the three proposed algorithms with real examples.

5. Numerical Results

In this section, we use the same operating environment that is used in Section 3.

Therefore, three criteria are used in this section; these standards are the Itr CPU time (Tcpu) and the PSNR values for the restored images listed in

Table 4. Results regarding the performance of CGCG-HS1, CGCG-HS2, and CGCG-HZ Algorithms for repairing corrupted pictures.

Image	Noise Ratio	CGCG-HS1			CGCG-HZ			CGCG-HS2
		Tcpu	Itr	PSNR	Tcpu	Itr	PSNR	Tcpu	Itr	PSNR
lena	30%	16.13	14	37.22	14.42	12	37.21	16.4649116	19	36.9968113
lena	50%	27.90	17	34.58	19.64	13	34.60	28.99923381	17	34.58358745
lena	70%	39.52	23	31.37	28.87	20	31.37	33.2962394	20	31.4504158
lena	90%	47.87	26	26.33	41.13	26	26.22	54.66880879	26	26.11239207
hill	30%	16.76	19	34.92	15.67	15	34.87	16.72475127	16	34.89737541
hill	50%	28.86	21	32.67	23.17	14	32.57	30.66004241	21	32.66618695
hill	70%	33.09	18	29.69	29.01	17	29.68	35.82486906	23	29.8987122
hill	90%	46.69	25	25.64	40.18	23	25.55	49.82943146	27	25.65882636
man	30%	21.91	15	31.69	16.64	12	31.63	14.54189568	16	31.5757247
man	50%	29.92	20	29.30	21.72	12	29.23	26.76981277	20	29.30118531
man	70%	38.22	24	26.45	33.67	20	26.38	35.66918655	22	26.46714801
man	90%	48.01	26	22.52	40.87	24	22.52	51.93510837	28	22.52534104
boat	30%	17.32	16	33.66	17.25	14	33.62	18.76662682	18	33.65937063
boat	50%	26.25	14	31.15	24.19	16	31.12	27.75435595	14	31.14532822
boat	70%	39.57	21	28.30	29.92	19	28.26	35.84634329	20	28.30099363
boat	90%	47.59	25	24.09	44.98	31	24.20	47.62695771	25	24.11558254
	Total	525.62	324	479.58	441.33	288	479.02	525.38	332	479.35

.

In addition, the graphs of the original, noisy, and repaired pictures are shown in Figure 5, Figure 6, Figure 7 and Figure 8.

The noise levels of the salt-and-pepper noise are as follows: $30\%$, $50\%$, $70\%$, and $90\%$.

The fifth row of Table 4 offers the totals of the three standards, i.e., Itr, Tcpu, and PSNR, for the repaired pictures. These pictures were processed by three algorithms: CGCG-HS1, CGCG-HZ, and CGCG-HS2.

These three criteria show that the CGCG-HS1 algorithm has achieved the best results for the PSNR compared to the CGCG-HS2 and CGCG-HZ algorithms.

With respect to time (Tcpu) and the number of iterations (Itr), the CGCG-HZ algorithm is the best, as it restores all corrupted images of all noises levels within 441.33 s with 288 iterations versus 525.62 s and 324 iterations achieved by applying the CGCG-HS1 algorithm and 525.738 s and 332 iterations achieved using CGCG-HS2 method. However, the CGCG-HS1 algorithm is the best for PSNR, as it its time span is 479.58 against 479.02 and 479.35 for CGCG-HZ and CGCG-HS2, respectively.

In general, the performance of the CGCG-HS1 and CGCG-HZ algorithms is better than the performance of the CGCG-HS2 algorithm. This means that the value of the parameter $\alpha$ plays a key role in minimizing the objective function $F_{{}_{\alpha }}\left(u\right)$ in (52); additionally, the size of the filter window in the adaptive median filter method is very significant when scanning a deteriorated image. The outstanding clarity of the restored images proves the new family of modified hybrid CG methods can be used to solve many problems in different applications.

6. Conclusions and Future Work

A novel four-term CG parameter has been designed and tested in this study. The newly proposed algorithm is abbreviated as CGCG, which stands for “Convex Group Conjugated Gradient”.

The CGCG algorithm solved an unconstrained optimization problem.

The convergence analysis proof of the CGCG approach has been shown. Through solving a set 76 benchmark test problems using the six algorithms, the numerical results indicate that the CGCG algorithm outperforms the five classical CG algorithms. The CGCG algorithm has been adapted into a new family of modified CG methods used to deal with image restoration problems. The median filter method was first applied to detect the noise in a corrupted image. According to the noise level in the corrupted image, we changed the size of the filter window to improve the performance of this proposed family of CG methods. The salt-and-pepper technique was used to corrupt an image as a test problem, where levels of noise are 30–90%. Therefore, four well-known images (test problems) were corrupted with salt-and-pepper noise (at levels from 30 to 90%) to examine the performance of the new family of modified hybrid CG methods.

The superior clarity of the restored images indicates that the new family of modified hybrid CG methods has efficiency, reliability, and effectiveness in dealing with image restoration problems.

Therefore, this new family of modified hybrid CG methods can be adapted to deal with other problems in different applications. In future research, it will be valuable to combine the CGCG method with a meta-heuristic technique in order to possess the features of both techniques. Through this hybridization, we will obtain a hybrid CG meta-heuristic algorithm for dealing with global optimization problems, including unconstrained, constrained, and multi-objective problems.

It is also useful to develop the CGCG approach to deal with a nonlinear system of monotone equations.

Furthermore, the CGCG approach could be modified to deal with the symmetric system of nonlinear equations, image restoration, and deep neural networks used for trains.

7. List of Test Problem

These test problems were taken from [65,66].

COSINE function (CUTE): Pr = 1–3

$$f\left(x\right)=\sum _{i=1}^{n-1}cos(-0.5x_{i+1}+x_{{}_i}^2),x_{{}_0}=[1,\dots ,1]$$

DIXMAAN function (CUTE): Pr = 4–26

$$f\left(x\right)=1+\sum _{i=1}^n\alpha x_{{}_i}^2{\left(\frac{i}{n}\right)}^{k1}+\sum _{i=1}^{n-1}\beta x_{{}_i}^2(x_{i+1}+x_{i+1}^2){\left(\frac{i}{n}\right)}^{k2}+\sum _{i=1}^{2m}\gamma x_{{}_i}^2{x}_{{}_{i+m}}^4{\left(\frac{i}{n}\right)}^{k3}+\sum _{i=1}^m\delta x_{{}_i}{x}_{{}_{i+2m}}{\left(\frac{i}{n}\right)}^{k4},m=n/3,$$

	$\alpha$	$\beta$	$\gamma$	$\delta$	$k1$	$k2$	$k3$	$k4$
A	1	0	0.125	0.125	0	0	0	0
B	1	0.0625	0.0625	0.0625	0	0	0	1
C	1	0.125	0.125	0.125	0	0	0	0
D	1	0.26	0.26	0.26	0	0	0	0
E	1	0	0.125	0.125	1	0	0	1
F	1	0.0625	0.0625	0.0625	1	0	0	1
G	1	0.125	0.125	0.125	1	0	0	1
H	1	0.26	0.26	0.26	1	0	0	1
I	1	0	0.125	0.125	2	0	0	2
J	1	0.0625	0.0625	0.0625	2	0	0	2
K	1	0.125	0.125	0.125	2	0	0	2
L	1	0.26	0.26	0.26	2	0	0	2

where $x_{{}_0}=[2,\dots ,2]$.

DIXON3DQ function (CUTE): Pr = 27

$$f\left(x\right)={(x_{{}_1}-1)}^2+\sum _{i=1}^{n-1}{(x_{{}_i}-x_{{}_{i+1}})}^2+{(x_{{}_n}-1)}^2,x_{{}_0}=[-1,\dots ,-1]$$

DQDRTIC function (CUTE): Pr = 28–31

$$f\left(x\right)=\sum _{i=1}^{n-2}(x_{{}_i}^2+c{x}_{{}_{i+1}}^2+d{x}_{{}_{i+2}}^2),c=100,d=100,x_{{}_0}=[3,\dots ,3].$$

EDENSCH function (CUTE): Pr = 32–34

$$f\left(x\right)=16+\sum _{i=1}^{n-1}\left[{(x_{{}_i}-2)}^4+{(x_{{}_i}{x}_{{}_{i+1}}-2x_{{}_{i+1}})}^2+{(x_{{}_{i+1}}+1)}^2\right],x_{{}_0}=[0,\dots ,0].$$

EG2 function (CUTE): Pr = 35

$$f\left(x\right)=\sum _{i=1}^{n-1}sin(x_{{}_1}+x_{{}_i}^2-1)+0.5sin\left(x_{{}_n}^2\right),x_{{}_0}=[1,\dots ,1].$$

FLETCHCR function (CUTE): Pr = 36–38

$$f\left(x\right)=\sum _{i=1}^{n-1}c{(x_{{}_{i+1}}-x_{{}_i}+1-x_{{}_i}^2)}^2,c=100,x_{{}_0}=[0,\dots ,0].$$

HIMMELBG function (CUTE): Pr = 39–40

$$f\left(x\right)=\sum _{i=1}^{n/2}(2x_{2i-1}^2+3x_{2i}^2)exp(-x_{2i-1}-x_{2i}),x_{{}_0}=[1.5,\dots ,1.5].$$

LIARWHD function (CUTE): Pr = 41–42

$$f\left(x\right)=\sum _{i=1}^{n}4{(-x_{{}_1}+x_{{}_i}^2)}^2+\sum _{i=1}^n{(x_{{}_1}-1)}^2,x_{{}_0}=[4,\dots ,4].$$

Extended Penalty function: Pr = 43–44

$$f\left(x\right)=\sum _{i=1}^{n-1}{(x_{{}_i}-1)}^2+{\left(\sum _{i=1}^n{x}_{{}_i}^2-0.25\right)}^2,x_{{}_0}=[1,2,\dots ,n].$$

QUARTC function (CUTE): Pr = 45–47

$$f\left(x\right)=\sum _{i=1}^n{(x_{{}_i}-1)}^4,x_{{}_0}=[2,\dots ,2].$$

TRIDIA function (CUTE): Pr = 48–49

$$f\left(x\right)={(x_{{}_1}-1)}^2+\sum _{i=1}^{n}i{(2x_{{}_i}-x_{{}_{i-1}})}^2,x_{{}_0}=[1,\dots ,1].$$

Extended Wood function: Pr = 50

$$f\left(x\right)=\sum _{i=1}^n(100{(x_{4i-2}-x_{4i-3}^2)}^2+{(1-x_{4i-3})}^2+90{(x_{4i}-x_{4i-1}^2)}^2+{(1-x_{4i-1})}^2+10{(x_{4i-2}+x_{4i}-2)}^2+0.1{(x_{4i-2}-x_{4i})}^2),$$

$x_{{}_0}=[-3,-1,-3,-1,-3,-1,\dots ,-3,-1]$.

BDEXP function (CUTE): Pr = 51–53

$$f\left(x\right)=\sum _{i=1}^n(x_{{}_i}+x_{i+1})exp(-x_{{}_{i+2}}(x_{{}_i}+x_{i+1})),x_{{}_0}=[1,\dots ,1].$$

BIGGSB 1 function (CUTE): Pr = 54

$$f\left(x\right)={(x_{{}_1}-1)}^2+\sum _{i=1}^{n-1}{(x_{i+1}-x_{{}_i})}^2+{(1-x_{{}_n})}^2,x_{{}_0}=[0,\dots ,0].$$

SINE function: Pr = 55–57

$$f\left(x\right)=\sum _{i=1}^{n-1}sin(-0.5x_{i+1}+x_{{}_i}^2),x_{{}_0}=[1,\dots ,1].$$

FLETCBV3 function (CUTE): Pr = 58

$$f\left(x\right)=\frac{p}{2}{(x_{{}_1}+x_{{}_n})}^2\sum _{i=1}^{n-1}\frac{p}{2}{(x_{{}_i}-x_{{}_{i+1}})}^2-\sum _{i=1}^n\left(\frac{p(h^2+2)}{h^2}{x}_{{}_i}+\frac{cp}{h^2}cos\left(x_{{}_i}\right)\right),$$

where $p=1/{10}^8$, $h=1/(1+n)$, $c=1$ and $x_{{}_0}=[h,2h\dots ,nh]$.

NONSCOMP function (CUTE): Pr = 59–60

$$f\left(x\right)={(x_{{}_1}-1)}^2+\sum _{i=1}^{n}4{(x_{{}_i}-x_{{}_{i-1}}^2)}^2,x_{{}_0}=[3,3,\dots ,3].$$

POWER function (CUTE): Pr = 61

$$f\left(x\right)=\sum _{i=1}^n{\left(i{x}_{{}_i}\right)}^2,x_{{}_0}=[1,1,\dots ,1].$$

Raydan 1 function: Pr = 62–63

$$f\left(x\right)=\sum _{i=1}^n\frac{i}{10}(exp\left(x_{{}_i}\right)-x_{{}_i}),x_{{}_0}=[1,1,\dots ,1].$$

Raydan 2 function: Pr = 64–66

$$f\left(x\right)=\sum _{i=1}^n(exp\left(x_{{}_i}\right)-x_{{}_i}),x_{{}_0}=[1,1,\dots ,1].$$

Diagonal 1 function: Pr = 67–68

$$f\left(x\right)=\sum _{i=1}^n(exp\left(x_{{}_i}\right)-i{x}_{{}_i}),x_{{}_0}=[1/n,1/n,\dots ,1/n].$$

Diagonal 2 function: Pr = 69–70

$$f\left(x\right)=\sum _{i=1}^n(exp\left(x_{{}_i}\right)-\frac{x_{{}_i}}{i}),x_{{}_0}=[1/1,1/2,\dots ,1/n].$$

Diagonal 3 function: Pr = 71–72

$$f\left(x\right)=\sum _{i=1}^n\left(exp\left(x_{{}_i}\right)-isin\left(x_{{}_i}\right)\right),x_{{}_0}=[1,\dots ,1].$$

Extended Rosenbrock function: Pr = 73–74

$$f\left(x\right)=\sum _{i=1}^{n/2}100{\left(x_{{}_{2i}}-x_{{}_{2i-1}}^2\right)}^2+{(1-x_{{}_{2i-1}})}^2,x_{{}_0}=[-1.2,1,\dots ,-1.2,1].$$

TRIDIA function (CUTE): Pr = 75–76

$$f\left(x\right)={(x_{{}_1}-1)}^2+\sum _{i=2}^{n}i{(2x_{{}_i}-x_{{}_{i-1}})}^2,x_{{}_0}=[1,\dots ,1].$$