Abstract
The conjugate gradient (CG) directions are among the important components of the CG algorithms. These directions have proven their effectiveness in many applications—more specifically, in image processing due to their low memory requirements. In this study, we derived a new conjugate gradient coefficient based on the famous quadratic model. The derived algorithm is distinguished by its global convergence and essential descent properties, ensuring robust performance across diverse scenarios. Extensive numerical testing on image restoration and unconstrained optimization problems have demonstrated that the new formulas significantly outperform existing methods. Specifically, the proposed conjugate gradient scheme has shown superior performance compared to the traditional Fletcher–Reeves (FR) conjugate gradient method. This advancement not only enhances computational efficiency on unconstrained optimization problems, but also improves the accuracy and quality of image restoration, making it a highly valuable tool in the field of computational imaging and optimization.
Keywords:
image processing; impulse noise reduction; unconstrained optimization; conjugate gradient methods; line search methods MSC:
90C30; 90C26; 90C06; 90C90; 90C47
1. Introduction
Many real-world applications involve nonlinear optimization problems, particularly, problems of extremely enormous dimensions, and thus require the use of first-order schemes to obtain their solutions. Conjugate gradient (CG) methods, the most popular first-order approaches, have been widely demonstrated to be useful in handling challenging unconstrained and constrained problems, such as problems arising from imaging processing. This is due to the efficiency of the CG algorithm in handling sparse and large-scale systems, which are common in such tasks. These methods are iterative and thus allow for early termination criteria to save time, and can be adapted for nonlinear functions, making their algorithms versatile. In addition, CG schemes efficiently handle minimization problems with regularization, scale well with the size of the problem, and can control parallel processing capabilities, improving their efficacy and speed for high-resolution image processing tasks.
In [1], the authors propose a two-phase strategy that combines the advantages of the adaptive median filter with the variational method in a single approach. The adaptive median filter [2] is used in the initial phase of the processing for salt-and-pepper noise. In such methods, represents the real image and represents the index set of and denotes the set of indices of the noise pixels that were detected during the first phase of the analysis process. Thus, the issue is determining an effective method for minimizing the function as follows:
where is the regularization parameter, are the edge-preserving potential functions, and is the edge-preserving potential function. denotes the set of the four closest neighbors of the pixel at the location , and denotes the observed pixel value of the image at the position with in each row denoting a column vector of one length sorted lexicographically after the other.
It has been demonstrated in [1,2] that the term in (1) allows for the detection of noisy pixels, but it also adds a tiny bias in the restoration of corrupted pixels when used in conjunction with other techniques. Our method begins by detecting the set of all noisy pixels, which occurs in its initial phase. As a result, this phrase is no longer necessary during the restoration phase. This recommends that we remove it from consideration (1). Consequently, we can only analyze the functional aspects of the following:
In this study, we are more interested in investigating the performance of a new Conjugate gradient method on image restoration and unconstrained optimization problem of the form:
where is continuously differentiable (see [3,4,5]). The CG algorithm generates a sequence of iterative points via [6]:
where is a step length and the search direction is generated as:
For further information on the possible choices of the conjugate coefficient , see [7,8]. In general, the global convergence characteristics of CG techniques are widely studied. According to [9], the Fletcher–Reeves (FR) method has been identified as having the best convergence results, while the Hestenes–Stiefel (HS) method has been recognized as one of the most efficient CG methods with good numerical performance but failing to satisfy the global convergence properties under classical line search conditions [10]. The particular choices for the FR and HS methods are:
with . The presentations in [11,12,13,14] provide excellent references to contemporary CG approaches that have yielded significant outcomes, in comparison to those in (6). Because the Hestenes–Stiefel formula meets the conjugacy criterion, it is particularly appealing to require that (see [15])
be satisfied when new methods are developed. The derivations in [16,17] represent an outstanding summary of the evolution of multiple variants of nonlinear conjugate gradient methods with a specific focus on global convergence qualities, as described in [17]. In theory, based on Perry’s conjugacy criterion (7), it may be rewritten as:
In [14], the Hestenes–Stiefel (HS) method has been modified to produce:
As revealed by the numerical findings, the Wu and Chen [17] approach numerically supersedes the HS method [10].
In [18], it is shown that for quadratic functions, the step size is determined exactly as:
For non-quadratic problems, classical line searches, such as cubic interpolation, are employed to find a step size along a generated search direction. For convergence purposes, is usually required to satisfy the strong Wolfe–Powell (SWP) [19] line search conditions:
where [20,21]. Such conditions are particularly beneficial in examining the convergence properties of CG methods. For a more recent study on the CG method, see [22,23,24,25,26,27].
Next, a new CG conjugacy parameter is developed by employing a quadratic model. This is followed by an analysis of the new method’s convergence properties. The new derivation aims to further the numerical behavior of CG methods.
2. Deriving the New Parameter Based on the Quadratic Model
In this section, we present the derivation process of the new conjugate gradient formulas. The algorithm used for the computational experiment is further presented at the end of this section. The motivation for constructing novel conjugate gradient parameters via the quadratic model is to enhance the convergence rate and accuracy of the CG algorithm. By leveraging the second-order curvature and presenting modified formulas, the updated algorithm can provide improved direction and step length adjustments, and this would lead to more effective minimization, particularly in large-scale and complex problems in image restoration and unconstrained optimization, which need robust optimization procedures. The new CG parameter is derived using the quadratic model:
where the corresponding gradient is given as:
The second-order curvature is derived from (12) and (13) to obtain:
Using the above equation to obtain:
where
Using (8) and (15) in the above equation, we obtain:
Since , this implies that:
which yields:
Additionally, for exact line search, (16) leads to two suggested expressions as follows:
and
We refer to the three alternatives of as BT1, BT2, and BT3, respectively, as indicated in (16), (20), and (21). Introducing multiple forms of , including BT1, BT2, and BT3, offers flexibility in selecting the most suitable parameter for different problem contexts, improving the approach’s performance and adaptability.
The algorithmic steps (Algorithm 1) for the derived method are summarized as:
| Algorithm 1. The new conjugate gradient algorithm for minimizing. |
| . |
| then stop. |
| by (9) and (10). |
| by (19). |
| . |
| and go to stage 1. |
3. Convergence Analysis of the Uniformly Convex Function
The global convergence analysis for the derived methods is considered in this section. The following assumptions are needed:
Assumption 1.
is bounded below on and bounded on the set .
Assumption 2.
The gradient is Lipschitz continuous, i.e., there exists a non-negative steady L such that
Under these assumptions on the objective function, there exists a steady , such that . More details can be found in [28,29].
We start by proving the descent property for the new algorithm in the following theorem.
Theorem 1.
Let and be generated by (5) with the choices (19), (20), and (21) for the conjugacy parameter, then is a downhill direction.
Proof of Theorem 1.
Since we obtained . Multiplying in (5) by and using Equation (13), we have:
By Lipschitz condition:
Combining (23) with (24), we obtain:
Hence, (25) yields:
Next, from (26), we have:
Hence, the generated directions are downhill. □
To show that the new method converges globally, we employ the following lemma, which was proven in [19,20]
Lemma 1.
If positive constants, and , exist, such that for
and
for any positive integer , the inequality, (44) holds for at least values of .
Using the condition of Lemma 1, we can prove the following result.
Theorem 2.
Assume that is a uniformly convex function on , i.e., there exists a stable , that satisfies:
If the conditions in Lemma 1 hold, then:
Proof of Theorem 2:
The proof is similar to the ones in [20,21]. □
4. Numerical Results
In this section, numerical data is presented to demonstrate the efficiency of the BT1, BT2, and BT3 algorithms in reducing salt-and-pepper impulse noise by lowering the threshold in (3) and further solving unconstrained optimization problems. The parameters chosen for the line search in (11) for the BT1, BT2, and BT3 procedures are and . All simulations are conducted on a PC with MATLAB 2015a. The BT1, BT2, and BT3 techniques are compared to the FR method in terms of performance efficiency. It is vital to emphasize that the speed at which the obtained decrease in (3) is the primary focus of this research. The Signal-to-Noise Ratio is used to evaluate the quality of the recovered image:
where and denote the pixel values of the restored image and the original image, respectively. The stopping criteria for both techniques are as follows:
Table 1 reports the computed PSNR (peak signal-to-noise ratio), in addition to the iterations count (NI) and function evaluations (NF) for each of the tested methods, as opposed to the standard FR method.
Table 1.
Numerical results of FR, BT1, BT2, and BT3 algorithms.
As demonstrated in Table 1, the BT1, BT2, and BT3 are more efficient as they require the fewest iterations and function evaluations compared to the FR method. Furthermore, the PSNR values generated by all three new approaches are superior. The restoration results achieved utilizing the FR, BT1, BT2, and BT3 algorithms are shown in Figure 1, Figure 2, Figure 3 and Figure 4. These findings demonstrate that the recommended image-correcting procedures BT1, BT2, and BT3 are reliable and effective.
Figure 1.
Demonstrates the results of algorithms FR, BT1, BT2, and BT3 of 256 × 256 Lena image.
Figure 2.
Demonstrates the results of algorithms FR, BT1, BT2, and BT3 of 256 × 256 House image.
Figure 3.
Demonstrates the results of algorithms FR, BT1, BT2, and BT3 of 256 × 256 Elaine image.
Figure 4.
Demonstrates the results of algorithms FR, BT1, BT2, and BT3 of 256 × 256 Cameraman image.
Next, the numerical performance of the proposed method was evaluated on a total of twenty-three unconstrained optimization problems under the strong Wolfe line search. Three metrics including the number of iterations (NOI), the number of function evaluation (NOF), and computational time (CPUT) are used to measure the efficiency of all algorithms. The termination condition was also set as:
- If was not satisfied.
- If iterations exceed 2000.
Table 2 presents a detailed performance of all the algorithms. The symbol (***) is used to denote the point where an algorithm fails to satisfy the above conditions. The symbol (##) means a failure to converge to a solution.
Table 2.
Performance comparison based on NOI, NOF, and CPU time.
The above results are further analyzed using the performance profile tool introduced by Dolan and More [30]. The following figures present the visual illustration of the results from Table 2.
As seen in Figure 5 and Figure 6, a percent P(τ) of the problems for which the technique is within a factor τ of the optimal time is displayed for each method. A method’s quickest percentage of test problems is indicated by the vertical axis on the left side of the curves.
Figure 5.
Performance profile based on NOI.
Figure 6.
Performance profile based on NOF.
The figures indicate that the proposed algorithms outperformed the classical FR algorithm based on the number of iterations and function evaluation. This shows that the new methods are ranked out of the FR method, thus competing favorably with the existing method.
5. Conclusions
In conclusion, we have developed a new modified conjugate gradient formula and introduced three new conjugate gradient methods, BT1, BT2, and BT3, which propose different options for the conjugacy parameter. The new choices are designed to enhance image processing tasks, particularly in image restoration applications. By employing Wolfe line search conditions, we established the global convergence properties of these new methods. Our comprehensive simulation studies demonstrated that the BT1, BT2, and BT3 methods significantly reduce the number of iterations and function evaluations required, thus improving the computational efficiency of the methods on unconstrained optimization problems. Moreover, these methods were further shown to effectively restore image quality, surpassing the performance of the traditional conjugate gradient method.
The results highlight the potential of BT1, BT2, and BT3 methods to advance the field of image processing. Their ability to achieve higher accuracy with less computational effort makes them valuable tools for practitioners and researchers equally. Future work will focus on further optimizing these methods and exploring their application to a broader range of image processing challenges.
Author Contributions
Conceptualization, B.A.H. and I.A.R.M.; methodology I.M.S. and I.A.R.M.; validation I.M.S. and T.A.A.; writing original version, I.A.R.M., R.M.S. and B.A.H.; Investigation I.M.S. and R.M.S. All authors have read and agreed to the published version of the manuscript.
Funding
School of Arts and Sciences, University of Central Asia, Naryn, Kyrg Republic.
Data Availability Statement
The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding authors.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Xue, W.; Ren, J.; Zheng, X.; Liu, Z.; Liang, Y. A new DY conjugate gradient method and applications to image denoising. IEICE Trans. Inf. Syst. 2018, 12, 2984–2990. [Google Scholar] [CrossRef] [Scilit]
- Yu, G.; Huang, J.; Zhou, Y. A descent spectral conjugate gradient method for impulse noise removal. Appl. Math. Lett. 2010, 23, 555–560. [Google Scholar] [CrossRef] [Scilit]
- Huang, T.; Yang, G.; Tang, G. A fast two-dimensional median filtering algorithm. Image Process. Based Partial Differ. Equ. 1979, 27, 13–18. [Google Scholar] [CrossRef] [Scilit]
- Sulaiman, I.M.; Supian, S.; Mamat, M. New class of hybrid conjugate gradient coefficients with guaranteed descent and efficient line search. IOP Conf. Ser. Mater. Sci. Eng. 2019, 621, 012021. [Google Scholar] [CrossRef] [Scilit]
- Awwal, A.M.; Yahaya, M.M.; Pakkaranang, N.; Pholasa, N. A New Variant of the Conjugate Descent Method for Solving Unconstrained Optimization Problems and Applications. Mathematics 2024, 12, 2430. [Google Scholar] [CrossRef] [Scilit]
- Malik, M.; Abas, S.S.; Mamat, M.; Mohammed, I.S. A new hybrid conjugate gradient method with global convergence properties. Int. J. Adv. Sci. Technol. 2020, 29, 199–210. [Google Scholar]
- Hager, W.W.; Zhang, H. A survey of nonlinear conjugate gradient methods. Pac. J. Optim. 2006, 2, 35–58. [Google Scholar]
- Hassan, B.A.; Jabbar, H.N.; Laylani, Y.A.; Moghrabi, I.A.R.; Alissa, A.J. An enhanced fletcher-reeves-like conjugate gradient methods for image restoration. Int. J. Electr. Comput. Eng. 2023, 13, 6268–6276. [Google Scholar] [CrossRef] [Scilit]
- Fletcher, R. Function minimization by conjugate gradients. Comput. J. 1964, 7, 149–154. [Google Scholar] [CrossRef] [Scilit]
- Hestenes, M.R.; Stiefel, E. Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 1952, 49, 409–436. [Google Scholar] [CrossRef] [Scilit]
- Dai, Y.H.; Yuan, Y. A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property. SIAM J. Optim. 1999, 10, 177–182. [Google Scholar] [CrossRef] [Scilit]
- Fletcher, R. Practical Methods of Optimization; Wiley: Hoboken, NJ, USA, 1987. [Google Scholar]
- Liu, Y.; Storey, C. Efficient generalized conjugate gradient algorithms, part 1: Theory. J. Optim. Theory Appl. 1991, 69, 129–137. [Google Scholar] [CrossRef] [Scilit]
- Polak, E.; Ribiere, G. Note sur la Convergence de Directions Conjugate, Revue Francaise Informant. Reserche. Opertionelle 1969, 3, 35–43. [Google Scholar]
- Perry, A. A Modified Conjugate Gradient Algorithm. Oper. Res. 1978, 26, 1073–1078. [Google Scholar] [CrossRef] [Scilit]
- Moghrabi, I.A.R. A new scaled secant-type conjugate gradient algorithm. In Proceedings of the 2017 European Conference on Electrical Engineering and Computer Science, EECS 2017, Bern, Switzerland, 17–19 November 2017. [Google Scholar] [CrossRef] [Scilit]
- Wu, C.; Chen, G. New type of conjugate gradient algorithms for unconstrained optimization problems. J. Syst. Eng. Electron. 2010, 21, 1000–1007. [Google Scholar] [CrossRef]
- Nocedal, J.; Wright, S.J. Numerical Optimization-Springer Series in Operations Research; Springer: Berlin/Heidelberg, Germany, 2006. [Google Scholar]
- Wolfe, P. Convergence conditions for ascent methods. II: Some corrections. SIAM Rev. 1971, 3, 185–188. [Google Scholar] [CrossRef] [Scilit]
- Hassan, B.A.; Moghrabi, I.A.R. A modified secant equation quasi-Newton method for unconstrained optimization. J. Appl. Math. Comput. 2023, 69, 451–464. [Google Scholar] [CrossRef] [Scilit]
- Dai, Y.; Han, J.; Liu, G.; Sun, D.; Yin, H.; Yuan, Y.X. Convergence Properties of Nonlinear Conjugate Gradient Methods. SIAM J. Optim. 2000, 10, 345–358. [Google Scholar] [CrossRef] [Scilit]
- Ibrahim, S.M.; Salihu, N. Two sufficient descent spectral conjugate gradient algorithms for unconstrained optimization with application. Optim. Eng. 2024, 31, 1–26. [Google Scholar] [CrossRef] [Scilit]
- Hassan, B.A.; Taha, M.W.; Kadoo, F.H.; Mohammed, S.I. A new modification into Quasi-Newton equation for solving unconstrained optimization problems. In AIP Conference Proceedings; AIP Publishing LLC: Melville, NY, USA, 2022; Volume 2394. [Google Scholar]
- Salihu, N.; Kumam, P.; Sulaiman, I.M.; Arzuka, I.; Kumam, W. An efficient Newton-like conjugate gradient method with restart strategy and its application. Math. Comput. Simul. 2024, 226, 354–372. [Google Scholar] [CrossRef] [Scilit]
- Malik, M.; Mamat, M.; Abas, S.S.; Sulaiman, I.M. Performance Analysis of New Spectral and Hybrid Conjugate Gradient Methods for Solving Unconstrained Optimization Problems. IAENG Int. J. Comput. Sci. 2021, 48, 66–79. [Google Scholar]
- Salihu, N.; Kumam, P.; Muhammad Yahaya, M.; Seangwattana, T. A revised Liu–Storey Conjugate gradient parameter for unconstrained optimization problems with applications. Eng. Optim. 2024, 1–25. [Google Scholar] [CrossRef] [Scilit]
- Ibrahim, A.H.; Rapajić, S.; Kamandi, A.; Kumam, P.; Papp, Z. Relaxed-inertial derivative-free algorithm for systems of nonlinear pseudo-monotone equations. Comput. Appl. Math. 2024, 43, 239. [Google Scholar] [CrossRef] [Scilit]
- Hager, W.W.; Zhang, H. A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM J. Optim. 2005, 16, 170–192. [Google Scholar] [CrossRef] [Scilit]
- Yuan, G.; Wei, Z.; Lu, X. Global convergence of BFGS and PRP methods under a modified weak Wolfe–Powell line search. Appl. Math. Model. 2018, 47, 811–825. [Google Scholar] [CrossRef] [Scilit]
- Dolan, E.D.; More, J.J. Benchmarking optimization software with performance profiles. Math. Program. 2002, 91, 201–213. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).









