Two Classes of Restart Algorithms for Solving Pseudomonotone Nonlinear Equations

Abstract

In this study, we introduce two efficient derivative-free algorithms enhanced by a restart strategy to solve nonlinear pseudomonotone equations. We demonstrate that the algorithm’s search direction is both descent and bounded, and under the assumptions of pseudomonotonicity and continuity, the algorithm generates globally convergent sequences toward the solutions. Numerical experiments on benchmark test problems highlight the computational efficiency of our proposed algorithm compared to several existing methods. Additionally, we illustrate the algorithm’s applicability to logistic regression problems, showcasing its practical relevance.

1. Introduction

Conjugate Gradient (CG) algorithms are highly efficient for solving nonlinear problems, particularly those of large scale. Their efficiency stems from the fact that they avoid storing $n\times n$ matrices, instead relying on $n\times 1$ column vectors. These algorithms are particularly effective for solving unconstrained minimization problems of the following form:

$$\underset{x\in {\mathcal{R}}^n}{min}f\left(x\right),$$

where $f:{\mathcal{R}}^n\to \mathcal{R}$ is continuously differentiable and bounded below. To solve this problem using the CG method, an initial point $x_0$ is chosen, and the following iterative procedure is applied:

$$x_{k+1}=x_k+{\alpha }_k{d}_k,k=0,1,\dots ,$$

where the sequence ${\left\{x_k\right\}}_{k=1}^{\infty }$ is expected to converge to the solution. The search direction $d_k$ is determined by the following:

$$d_k:=\left\{\begin{array}{cc}-F\left(x_k\right),\hfill & \mathrm{if}k=0,\hfill \\ -F\left(x_k\right)+{\beta }_k{d}_{k-1},\hfill & \mathrm{if}k\ge 1,\hfill \end{array}\right.$$

where $F:=\nabla f$ is the gradient of f, and ${\beta }_k$ is a CG parameter that varies across different CG algorithms. The step size ${\alpha }_k$ is obtained via a line search along the direction $d_k$. A first-order necessary condition for optimality states that if $\tilde{x}$ is a minimizer of f, then

$$F\left(\tilde{x}\right)=0.$$

(1)

Problems of this form arise in numerous high-impact applications, including signal recovery [1], image restoration [2], logistic regression [3], power flow equations [4], and variational inequalities [5].

For a CG algorithm to be both convergent and efficient, the search direction $d_k$ must satisfy certain criteria. These include the boundedness of the search direction and the sufficient descent property:

$$F{\left(x_k\right)}^T{d}_k\le -c{\parallel F\left(x_k\right)\parallel }^2,c>0.$$

(2)

However, some CG algorithms may fail to achieve global convergence if they do not satisfy the sufficient descent condition.

A restart strategy is often used to ensure that the search direction $d_k$ satisfies the sufficient descent condition at each iteration. Various restart algorithms have been proposed in the literature. For example, Powell [6] introduced a restart technique to improve convergence rates and ensure the positive definiteness of the search direction. Other researchers, such as Bolan and Kowalik [7], Shanno [8], Knockaert and Zutter [9], and Dai [10], have also developed restart procedures to guarantee global convergence in their algorithms.

Despite these advancements, many restart strategies based on the Powell criterion can suffer from slow convergence because they discard information from previous search directions. To address this, Jiang et al. [11] proposed efficient restart algorithms that take advantage of previous search directions, ensuring global convergence under Wolfe line search conditions. Their work demonstrated better performance over traditional restart methods.

In recent years, derivative-free methods have gained attention for solving problems where the gradient $\nabla f$ is not available. These methods rely solely on function evaluations, making them suitable for problems where f is not differentiable. Cheng [12] proposed a PRP restart algorithm for solving such problems, demonstrating its efficiency over non-restart algorithms. However, the effectiveness of the restart strategy was not fully validated due to limited numerical experiments. In [13], Li et al. proposed two classes of derivative-free methods with self-adaptive and restart features. The proposed methods, which extend MCG1 and MCG2 [14], generate descent directions and achieve global convergence. Additionally, the linear convergence rate of these methods was established under the local error bound condition and their effectiveness in solving constrained nonlinear equations together with application was demonstrated. However, the methods in [12,13] require F to be Monotone and Lipschitz continuous. For more on derivative-free methods, interested readers are referred to [1,2,15,16,17,18].

Pseudomonotonicity, a weaker condition than monotonicity, has also been explored in the context of CG algorithms. Recent work by Liu et al. [19], Awwal et al. [20], and Liu et al. [21] has focused on solving problems involving pseudomonotone and continuous operators, though their algorithms do not incorporate restart strategies. More details about pseudomonotone operators can also be found in [22].

In this work, we propose two efficient restart algorithms for solving problems involving pseudomonotone and continuous operators. The algorithm is designed to ensure global convergence while maintaining computational efficiency. The manuscript is organized as follows: Section 2 introduces the proposed algorithm and its convergence properties, Section 3 presents numerical experiments demonstrating the algorithm’s performance, Section 4 applies the method to logistic regression models, and Section 5 concludes the work.

2. Derivative-Free Projection Method with Restart Technique

In this section, we propose two classes of restart derivative-free algorithm to solve (1) with $F:{\mathcal{R}}^n\to {\mathcal{R}}^n$ being

1.

pseudomonotone, i.e., for all $x_1,x_2\in {\mathcal{R}}^n$

$$F{\left(x_1\right)}^T(x_2-x_1)\ge 0\mathrm{implies}F{\left(x_2\right)}^T(x_2-x_1)\ge 0,$$

(3)

2.

and continuous.

To begin with, we recall the SPCG2+ method for solving the unconstrained optimization problem proposed by Liu et al. [23]. The search direction is defined as follows:

$$d_k=\left\{\begin{array}{cc}-F_k,\hfill & k=0,\hfill \\ -\left(1+{\beta }_k{\displaystyle \frac{F_k^T{d}_{k-1}}{\parallel F_k{\parallel }^2}}\right)F_k+{\beta }_k{d}_{k-1},\hfill & k\ge 1,\hfill \end{array}\right.$$

where ${\beta }_k$ is defined as

$${\beta }_k={\displaystyle \frac{F_k^T{y}_{k-1}}{d_{k-1}^T{y}_{k-1}}}-{\displaystyle \frac{F_k^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}},y_{k-1}:=F\left(x_k\right)-F\left(x_{k-1}\right),$$

(4)

In order to obtain the global convergence of the algorithm, the search direction generated by the algorithm is required to satisfy the sufficient descent property. As such, we propose the following two classes of restart methods with the following search directions:

$$d_k^I:=\left\{\begin{array}{cc}-F\left(x_k\right),\hfill & \mathrm{if}k=0,\hfill \\ -\left(1+{\beta }_k{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel F\left(x_k\right){\parallel }^2}}\right)F\left(x_k\right)+{\beta }_k{d}_{k-1},(k>0)\hfill & \mathrm{if}\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel \le d_{k-1}^T{y}_{k-1},\hfill \\ -F\left(x_k\right)+\mu {\displaystyle \frac{F{\left(x_k\right)}^T{y}_{k-1}}{\parallel y_{k-1}{\parallel }^2}}{y}_{k-1},(k>0)\hfill & \mathrm{otherwise},\left|\mu \right|<1.\hfill \end{array}\right.$$

(5)

and

$$d_k^{II}:=\left\{\begin{array}{cc}-F\left(x_k\right),\hfill & \mathrm{if}k=0,\hfill \\ -\left(1+{\beta }_k{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel F\left(x_k\right){\parallel }^2}}\right)F\left(x_k\right)+{\beta }_k{d}_{k-1},(k>0)\hfill & \mathrm{if}\parallel y_{k-1}\parallel \parallel d_{k-1}\parallel \le d_{k-1}^T{y}_{k-1},\hfill \\ -F\left(x_k\right)+\mu {\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}{d}_{k-1},(k>0)\hfill & \mathrm{otherwise},\left|\mu \right|<1.\hfill \end{array}\right.$$

(6)

Note that the restart conditions in $d_k^I$ and $d_k^{II}$ are necesssry in order for the search direction to satisfy inequality (2). This inequality is neccessary for the algorithm to converge. However, these restart conditions may not always hold. As such, we define $d_k^I$ and $d_k^{II}$ by $-F\left(x_k\right)+\mu {\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}{d}_{k-1},\left|\mu \right|<1$, which always satisfies (2). In addition, it has been shown to be an efficient search direction when it comes to the practical aspect.

Next, we present two restart algorithms that generate approximate solutions of the pseudomonotone Equation (1) with search direction $d_k$ in (5) and (6) together with their convergence analysis.

Lemma 1.

Let $F\left(x_0\right),\dots ,F\left(x_k\right)$ be nonzero for some $k\ge 0$, where $x_0,\dots ,x_k$ are the first $k+1$ iterates obtained via the Algorithm 1. If $d_k$ is defined by (5) or (6) with $\ell =1-\mu$, then

$$F{\left(x_k\right)}^T{d}_k\le -\ell {\parallel F\left(x_k\right)\parallel }^2.$$

(7)

Algorithm 1 Restart Derivative-free Projection Method (RDFPM)

1: Initialization: Select an initial vector $x_0\in {\mathcal{R}}^n$, $r\in (0,1)$, $\kappa \in (0,2)$, $\sigma \in (0,1)$. Set $k=0.$ 2: Step 1. Evaluate $F\left(x_k\right)$. If $\parallel F\left(x_k\right)\parallel \le tol$ then stop, else go to Step 2. 3: Step 2. Compute the search direction $d_k$ by (5) or (6). 4: Step 3. Set $z_k=x_k+{\alpha }_k{d}_k$ with ${\alpha }_k:=r^{{\iota }_k}$ and $${\iota }_k:=min\left\{i\in \mathbb{N}\cup \left\{0\right\}:-F{(x_k+r^i{d}_k)}^T{d}_k\ge \sigma r^i\parallel F(x_k+r^i{d}_k)\parallel \parallel d_k{\parallel }^2\right\}.$$ 5: Step 4. If $\parallel F\left(z_k\right)\parallel =0$, stop, else go to Step 5. 6: Step 5. Set $$x_{k+1}=x_k-\gamma {\displaystyle \frac{{F\left(z_k\right)}^T(x_k-z_k)}{\parallel F\left(z_k\right){\parallel }^2}}F\left(z_k\right).$$ (8) 7: Step 6. Let $k=k+1$ and repeat the process from Step 1.

Proof. 

Consider $d_k$ defined by (5), then

Case 1: When $k=0$, $F{\left(x_0\right)}^T{d}_0=-\parallel F\left(x_0\right){\parallel }^2\le -(1-\mu ){\parallel F\left(x_0\right)\parallel }^2$.

Case 2: When $k>0$ and $\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel \le d_{k-1}^T{y}_{k-1}$, we get

$$\begin{array}{cc}\hfill F{\left(x_k\right)}^T{d}_k& =-\left(1+{\beta }_k{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel F\left(x_k\right){\parallel }^2}}\right)F{\left(x_k\right)}^{T}F\left(x_k\right)+{\beta }_{k}F{\left(x_k\right)}^T{d}_{k-1}\hfill \\ \hfill & =-\parallel F\left(x_k\right){\parallel }^2-{\beta }_{k}F{\left(x_k\right)}^T{d}_{k-1}+{\beta }_{k}F{\left(x_k\right)}^T{d}_{k-1}\hfill \\ \hfill & =-\parallel F\left(x_k\right){\parallel }^2\hfill \\ \hfill & \le -(1-\mu )\parallel F\left(x_k\right){\parallel }^2.\hfill \end{array}$$

Case 3: If $k>0$ and $\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel >d_{k-1}^T{y}_{k-1}$, we get

$$\begin{array}{cc}\hfill F{\left(x_k\right)}^T{d}_k& =-\parallel F\left(x_k\right){\parallel }^2+\mu {\displaystyle \frac{{\left(F{\left(x_k\right)}^T{y}_{k-1}\right)}^2}{\parallel y_{k-1}{\parallel }^2}}\hfill \\ \hfill & \le -\parallel F\left(x_k\right){\parallel }^2+\mu {\displaystyle \frac{\parallel F\left(x_k\right){{\parallel }^2{y}_{k-1}\parallel }^2}{\parallel y_{k-1}{\parallel }^2}}\hfill \\ \hfill & \le -\parallel F\left(x_k\right){\parallel }^2+\mu {\parallel F\left(x_k\right)\parallel }^2\hfill \\ \hfill & =-{(1-\mu )\parallel F\left(xk\right)\parallel }^2.\hfill \end{array}$$

Therefore, for all three cases above, we end up with (7).

The proof for the case when $d_k$ is defined by (6) is similar because it is independent of the restart condition, so we omit it.    □

Remark 1.

As the Algorithm 1 generates $k+1$ iterates $x_0,\dots x_k$ for $k\ge 0$ with $F\left(x_j\right)\ne 0$ for $j=0,1,\dots ,k$, we can deduce from Lemma 1 that for $j=0,1,\dots ,k$

$$-\parallel F\left(x_j\right)\parallel \parallel d_j\parallel \le F{\left(x_j\right)}^T{d}_j\le -\ell {\parallel F\left(x_j\right)\parallel }^2<0$$

implies

$$0<\ell \parallel F\left(x_j\right)\parallel \le \parallel d_j\parallel .$$

(9)

This means that the Algorithm 1 runs as long as $x_k$ is not a solution, i.e., $F\left(x_k\right)\ne 0$.

Before stating the next Lemma, we would like to assume that the solution set to problem (1) is nonempty.

Lemma 2.

Let $\tilde{x}$ be a solution of problem (1) for a pseudomonotone and continuous operator F. If a sequence ${\left\{x_k\right\}}_{k=1}^{\infty }$ is generated by the Algorithm 1, then

$$\parallel x_{k+1}-\tilde{x}{\parallel }^2\le \parallel x_k-\tilde{x}{\parallel }^2-\gamma (2-\gamma ){\sigma }^2{\parallel x_k-z_k\parallel }^4.$$

Precisely, ${\displaystyle \underset{k\to \infty }{lim}\parallel x_k-\tilde{x}\parallel }$ exists.

Proof. 

Since $\tilde{x}$ is a solution of (1) and F is pseudomonotone, then

$$F{\left(\tilde{x}\right)}^T(z_k-\tilde{x})\ge 0⟹F{\left(z_k\right)}^T(z_k-\tilde{x})\ge 0.$$

(10)

In addition, by the definition of $z_k$ in the Algorithm 1, we have

$$-F{\left(z_k\right)}^T{d}_k\ge \sigma {\alpha }_k\parallel F\left(z_k\right)\parallel \parallel d_k{\parallel }^2,$$

which implies

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill F{\left(z_k\right)}^T(x_k-z_k)=& F{\left(z_k\right)}^T(-{\alpha }_k{d}_k)=-{\alpha }_{k}F{\left(z_k\right)}^T{d}_k\hfill \\ \hfill \ge & \sigma {\alpha }_k^2\parallel F\left(z_k\right)\parallel \parallel d_k{\parallel }^2=\sigma \parallel F\left(z_k\right)\parallel \parallel x_k-z_k{\parallel }^2\ge 0.\hfill \end{array}\end{array}$$

(11)

Next, we estimate $\parallel x_{k+1}-\tilde{x}{\parallel }^2$ using (8), (10), (11) and $\gamma \in (0,2)$:

$$\begin{array}{cc}\hfill \parallel x_{k+1}-\tilde{x}{\parallel }^2& ={∥x_k-\gamma {\displaystyle \frac{F{\left(z_k\right)}^T(x_k-z_k)}{\parallel F\left(z_k\right){\parallel }^2}}\mathsf{\Phi }\left(z_k\right)-\tilde{x}∥}^2\hfill \\ \hfill & =\parallel x_k-\tilde{x}{\parallel }^2-2\gamma {\displaystyle \frac{F{\left(z_k\right)}^T(x_k-z_k)}{\parallel F\left(z_k\right){\parallel }^2}}F{\left(z_k\right)}^T(x_k-\tilde{x})+{\gamma }^2{\displaystyle \frac{{\left(F{\left(z_k\right)}^T(x_k-z_k)\right)}^2}{\parallel F\left(z_k\right){\parallel }^2}}\hfill \\ \hfill & =\parallel x_k-\tilde{x}{\parallel }^2-2\gamma {\displaystyle \frac{{\left(F{\left(z_k\right)}^T(x_k-z_k)\right)}^2}{\parallel F\left(z_k\right){\parallel }^2}}-2\gamma {\displaystyle \frac{\left(F{\left(z_k\right)}^T(x_k-z_k)\right)\left(F{\left(z_k\right)}^T(z_k-\tilde{x})\right)}{\parallel F\left(z_k\right){\parallel }^2}}\hfill \\ \hfill & +{\gamma }^2{\displaystyle \frac{{\left(F{\left(z_k\right)}^T(x_k-z_k)\right)}^2}{\parallel F\left(z_k\right){\parallel }^2}}\hfill \\ \hfill & \le \parallel x_k-\tilde{x}{\parallel }^2-\gamma (2-\gamma ){\displaystyle \frac{{\left(F{\left(z_k\right)}^T(x_k-z_k)\right)}^2}{\parallel \mathsf{\Phi }\left(z_k\right){\parallel }^2}}\hfill \\ \hfill & \le \parallel x_k-\tilde{x}{\parallel }^2-\gamma (2-\gamma ){\sigma }^2{\parallel x_k-z_k\parallel }^4.\hfill \end{array}$$

$\{\parallel x_k-\tilde{x}\parallel \}$ is a decreasing sequence from the last inequality, and, hence, the limit exists.    □

Remark 2.

It is interesting to note that Lemma 2 confirms that for any zero $\tilde{x}$ of F, $\{\parallel x_k-\tilde{x}\parallel \}$ is a decreasing sequence.

Lemma 3.

Suppose ${\left\{x_k\right\}}_{k=1}^{\infty }$ is generated by the Algorithm 1 and F is continuous, then for all k, there exists $M>0$, such that

$$\parallel F\left(x_k\right)\parallel \le M.$$

(12)

In other words, the sequence $\left\{F\left(x_k\right)\right\}$ is bounded.

Proof. 

As the sequence $\{\parallel x_k-\tilde{x}\parallel \}$ converges by Lemma 2, then there exists $M_1>0$, such that for all k,

$$\parallel x_k-\tilde{x}\parallel \le M_1.$$

By the triangle inequality, we end up with

$$\parallel x_k\parallel \le M_1+\parallel \tilde{x}\parallel :=M_2.$$

Now, since ${\left\{x_k\right\}}_{k=1}^{\infty }$ is bounded and F is continuous, $\left\{F\left(x_k\right)\right\}$ is also bounded.    □

Lemma 4.

Let ${\alpha }_k$ and $d_k$ be generated via the Algorithm 1. Then

$$\underset{k\to \infty }{lim}{\alpha }_k\parallel d_k\parallel =0.$$

Proof. 

By Lemma 2, we have that $\gamma (2-\gamma ){\sigma }^2\parallel x_k-z_k{\parallel }^4\le \parallel x_k-\tilde{x}{\parallel }^2-{\parallel x_{k+1}-\tilde{x}\parallel }^2$; we get

$$\begin{array}{cc}\hfill \sum _{k=0}^{\infty }({\alpha }_k\parallel d_k{\parallel )}^4=\sum _{k=0}^{\infty }{\parallel x_k-z_k\parallel }^4\le & {\displaystyle \frac{1}{\gamma (2-\gamma ){\sigma }^2}}\sum _{k=0}^{\infty }(\parallel x_k-\tilde{x}{\parallel }^2-\parallel x_{k+1}-\tilde{x}{\parallel }^2)\hfill \\ \hfill \le & {\displaystyle \frac{\parallel x_0-\tilde{x}{\parallel }^2}{\gamma (2-\gamma ){\sigma }^2}}-\underset{k\to \infty }{lim}{\parallel x_k-\tilde{x}\parallel }^2<\infty .\hfill \end{array}$$

Hence,

$$\underset{k\to \infty }{lim}{\alpha }_k\parallel d_k\parallel =0.$$

   □

Lemma 5.

For all k, $d_k$ defined by Equation (5) or (6) is bounded. That is

$$\parallel d_k\parallel \le C,C>0.$$

(13)

Proof. 

Let $d_k$ be defined by (5), then

• For $k=0$,  $\parallel d_k\parallel =\parallel -F\left(x_k\right)\parallel =\parallel -F\left(x_k\right)\parallel$.
• For $k>0$ and $\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel \le d_{k-1}^T{y}_{k-1}$, we have

  $$\begin{array}{cc}\hfill \parallel d_k^I\parallel & =∥-F\left(x_k\right)+{\beta }_k{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel F\left(x_k\right){\parallel }^2}}F\left(x_k\right)+{\beta }_k{d}_{k-1}∥\hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +{\displaystyle \frac{\left|\beta \right|\parallel F\left(x_k\right){\parallel }^2\parallel d_{k-1}\parallel }{\parallel F\left(x_k\right){\parallel }^2}}+\left|\beta \right|\parallel d_{k-1}\parallel \hfill \\ \hfill & =\parallel F\left(x_k\right)\parallel +2\left|{\displaystyle \frac{F{\left(x_k\right)}^T{y}_{k-1}}{d_{k-1}^T{y}_{k-1}}}-{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}\right|\parallel d_{k-1}\parallel \hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +2{\displaystyle \frac{|F{\left(x_k\right)}^T{y}_{k-1}|}{|d_{k-1}^T{y}_{k-1}|}}\parallel d_{k-1}\parallel +2{\displaystyle \frac{|F{\left(x_k\right)}^T{d}_{k-1}|}{\parallel d_{k-1}{\parallel }^2}}\parallel d_{k-1}\parallel \hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +2{\displaystyle \frac{\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel }{|d_{k-1}^T{y}_{k-1}|}}+2{\displaystyle \frac{\parallel F\left(x_k\right)\parallel \parallel d_{k-1}{\parallel }^2}{\parallel d_{k-1}{\parallel }^2}}\hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +2+2\parallel F\left(x_k\right)\parallel \hfill \\ \hfill & =3\parallel F\left(x_k\right)\parallel +2\hfill \\ \hfill & \le 3M+2\hfill \end{array}$$
• For $k>0$ and $\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel >d_{k-1}^T{y}_{k-1}$, we have

  $$\begin{array}{cc}\hfill \parallel d_k^I\parallel & =∥-F\left(x_k\right)+\mu {\displaystyle \frac{F{\left(x_k\right)}^T{y}_{k-1}}{\parallel y_{k-1}{\parallel }^2}}{y}_{k-1}∥\hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +\mu {\displaystyle \frac{\parallel F\left(x_k\right)\parallel \parallel y_{k-1}{\parallel }^2}{\parallel y_{k-1}{\parallel }^2}}\hfill \\ \hfill & =\parallel F\left(x_k\right)\parallel +\mu \parallel F\left(x_k\right)\parallel =(1+\mu )\parallel F\left(x_k\right)\parallel .\hfill \end{array}$$

On the other hand, if $d_k$ be defined by (6), then

• For $k=0$,  $\parallel d_k\parallel =\parallel -F\left(x_k\right)\parallel =\parallel -F\left(x_k\right)\parallel$.
• For $k>0$ and $\parallel y_{k-1}\parallel \parallel d_{k-1}\parallel \le d_{k-1}^T{y}_{k-1}$, we have

  $$\begin{array}{cc}\hfill \parallel d_k^{II}\parallel & =∥-F\left(x_k\right)+{\beta }_k{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel F\left(x_k\right){\parallel }^2}}F\left(x_k\right)+{\beta }_k{d}_{k-1}∥\hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +{\displaystyle \frac{\left|\beta \right|\parallel F\left(x_k\right){\parallel }^2\parallel d_{k-1}\parallel }{\parallel F\left(x_k\right){\parallel }^2}}+\left|\beta \right|\parallel d_{k-1}\parallel \hfill \\ \hfill & =\parallel F\left(x_k\right)\parallel +2\left|{\displaystyle \frac{F{\left(x_k\right)}^T{y}_{k-1}}{d_{k-1}^T{y}_{k-1}}}-{\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}\right|\parallel d_{k-1}\parallel \hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +2{\displaystyle \frac{|F{\left(x_k\right)}^T{y}_{k-1}|}{|d_{k-1}^T{y}_{k-1}|}}\parallel d_{k-1}\parallel +2{\displaystyle \frac{|F{\left(x_k\right)}^T{d}_{k-1}|}{\parallel d_{k-1}{\parallel }^2}}\parallel d_{k-1}\parallel \hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +2{\displaystyle \frac{\parallel F\left(x_k\right)\parallel \parallel y_{k-1}\parallel \parallel d_{k-1}\parallel }{|d_{k-1}^T{y}_{k-1}|}}+2{\displaystyle \frac{\parallel F\left(x_k\right)\parallel \parallel d_{k-1}{\parallel }^2}{\parallel d_{k-1}{\parallel }^2}}\hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +2\parallel F\left(x_k\right)\parallel +2\parallel F\left(x_k\right)\parallel \hfill \\ \hfill & =5\parallel F\left(x_k\right)\parallel .\hfill \end{array}$$
• For $k>0$ and $\parallel y_{k-1}\parallel \parallel d_{k-1}\parallel >d_{k-1}^T{y}_{k-1}$, we have

  $$\begin{array}{cc}\hfill \parallel d_k^{II}\parallel & =∥-F\left(x_k\right)+\mu {\displaystyle \frac{F{\left(x_k\right)}^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}{d}_{k-1}∥\hfill \\ \hfill & \le \parallel F\left(x_k\right)\parallel +\mu {\displaystyle \frac{\parallel F\left(x_k\right)\parallel \parallel d_{k-1}{\parallel }^2}{\parallel d_{k-1}{\parallel }^2}}\hfill \\ \hfill & =\parallel F\left(x_k\right)\parallel +\mu \parallel F\left(x_k\right)\parallel =(1+\mu )\parallel F\left(x_k\right)\parallel .\hfill \end{array}$$

That is, with $C=max\{5M,3M+2,1+\mu \}$, we have

$$\parallel d_k\parallel \le C,k>0.$$

□

Proposition 1.

If ${\left\{x_k\right\}}_{k=1}^{\infty }$ is generated by the Algorithm 1 with a pseudomonotone and continuous operator Φ, then

$$\underset{k\to \infty }{lim\; inf}\parallel F\left(x_k\right)\parallel =0.$$

(14)

Proof. 

Suppose by contradiction that ${\displaystyle \underset{k\to \infty }{lim\; inf}\parallel F\left(x_k\right)\parallel >0}$. This implies that there exists $\kappa >0$ with

$$\underset{k}{inf}\parallel F\left(x_k\right)\parallel \ge \kappa .$$

(15)

Using (15) together with (9) and (13), we have

$$\ell \kappa \le \ell \parallel F\left(x_k\right)\parallel \le \parallel d_k\parallel \le C,k>0.$$

So ${lim}_{k\to \infty }{\alpha }_k\parallel d_k\parallel =0$ obtained in Lemma 4 implies

$$\underset{k\to \infty }{lim}{\alpha }_k=0.$$

(16)

In addition, as $\left\{x_k\right\}$ and $\left\{d_k\right\}$ are bounded, there exist convergent subsequences $\left\{x_{k_j}\right\}$ and $\left\{d_{k_j}\right\}$, such that

$$\underset{j\to \infty }{lim}{x}_{k_j}=\tilde{x}\mathrm{and}\underset{j\to \infty }{lim}{d}_{k_j}=\tilde{d}.$$

Also, from (7),

$$-F{\left(x_{k_j}\right)}^T{d}_{k_j}\ge \ell {\parallel F\left(x_{k_j}\right)\parallel }^2$$

By the continuity of F, (15), and allowing $j\to \infty$, we have

$$-F{\left(\tilde{x}\right)}^T\tilde{d}\ge \ell {\parallel F\left(\tilde{x}\right)\parallel }^2>\ell {\kappa }^2>0.$$

(17)

On the other hand, from Step 4 of the Algorithm 1, we have

$${-F(x_{k_j}+r^{-1}{\alpha }_{k_j}{d}_{k_j})}^T{d}_{k_j}<\sigma r^{-1}{\alpha }_{k_j}\parallel F(x_{k_j}+r^{-1}{\alpha }_{k_j}{d}_{k_j})\parallel \parallel d_{k_j}{\parallel }^2.$$

Combining the above argument together with (16), we have

$$-F{\left(\tilde{x}\right)}^T\tilde{d}\le 0,$$

which contradicts (17). Hence, (14) is true.   □

The next Theorem confirms the convergence of a sequence generated by the Algorithm 1 to a zero of the pseudomonotone and continuous operator F.

Theorem 1.

Suppose F is pseudomonotone and continuous; then, a sequence ${\left\{x_k\right\}}_{k=1}^{\infty }$ generated via the Algorithm 1 converges to a zero $\overline{x}$ of F, that is, $F\left(\overline{x}\right)=0.$

Proof. 

From Proposition 1, one can choose a subsequence $\left\{x_{k_j}\right\}$ such that $\left\{F\left(x_{k_j}\right)\right\}$ converges to 0. Because $\left\{x_k\right\}$ is bounded, we may further extract a convergent subsequence from $\left\{x_{k_j}\right\}$, which we also denote by $\left\{x_{k_j}\right\}$ for simplicity. Let $\overline{x}={lim}_{j\to \infty }{x}_{k_j}$. Since F is continuous, we have

$$F\left(\overline{x}\right)=\underset{j\to \infty }{lim}F\left(x_{k_j}\right)=0.$$

Furthermore, since ${\displaystyle \underset{k\to \infty }{lim}\parallel x_k-\overline{x}\parallel }$ exists by Lemma 2 and ${lim}_{j\to \infty }\parallel x_{k_j}-\overline{x}\parallel =0$, then the sequence $\left\{x_k\right\}$ must converge to $\overline{x}$.   □

3. Numerical Experiments

In this section, we evaluate the performance of the proposed methods; that is, Algorithm 1 implemented with search direction (5) and search direction (6). We refer to these methods as RDFPM I and RDFPM II, respectively. The performance of these methods is compared against recently proposed methods in the literature; RDFIA [24], a relaxed-inertial derivative-free algorithm for pseudomonotone systems, and HDFPM [25], a memoryless BFGS hyperplane projection method for monotone equations. All experiments were executed in MATLAB R2025a Prerelease Update 2 (25.1.0.2833191) on an Apple Mac OS system with an M3 chip, 8GB using unified parameters for both proposed methods: $r=0.8$, $\sigma =0.001$, $\gamma =1.2$, and $\mu =0.6$.

Seven benchmark nonlinear equations from the literature were selected as test problems. These problems include Problem 1 (Problem 2 in [26]), Problem 2 (Logarithmic function in [27]), Problem 3 (Example 4.1 in [28]), Problem 4 (Modified problem 10 in [29]), Problem 5 (Strictly convex function II in [27], Problem 6 (Nonsmooth function Problem 2 in [30]), and Problem 7 (Problem 4.6 in [31]). To assess the performance of each method, we evaluated four problem dimensions: $n=2000$, $15,000$, $75,000$, and $150,000$. For each dimension, six initial points were tested: five deterministic vectors ${\mathbf{x}}_0=\alpha ·\mathrm{ones}(n,1)$ with $\alpha \in \{0.2,0.5,1.0,1.2,1.5\}$, and one randomized initial point ${\mathbf{x}}_0=\mathrm{rand}(n,1)$. Each algorithm was implemented under identical experimental conditions (that is, all methods were tested on the same problem instances, dimensions, initial points and stopping criteria) to ensure fairness. The algorithms terminated when the residual satisfied $\parallel F\left(z_k\right)\parallel \le {10}^{-6}$ or $\parallel F\left(x_k\right)\parallel \le {10}^{-6}$, or after a maximum of 2000 iterations.

Detailed results of the numerical experiments are presented in Table A1, Table A2, Table A3, Table A4, Table A5, Table A6 and Table A7 in the following link: https://github.com/ibrahimkarym/Two-classes-of-restart.git (accessed on 1 November 2025) or at the Appendix A. To assess the performance of the methods, we used the Dolan and Moré [32] performance profile, which allows us to plot the results. This profile provides an insightful comparison by summarizing the efficiency of each algorithm in terms of its ability to minimize iterations, function evaluations, and computation time. The best solver is the curve that tops the plot. Across all problems and initializations, RDFPM I and II consistently outperformed RDFIA and HDFPM. These results in Figure 1, Figure 2 and Figure 3 confirm that RDFPM I and II are robust and efficient for solving nonlinear equations.

Figure 1. Performance profile for iteration.

Figure 2. Performance profile for fval.

Figure 3. Performance profile for time.

4. Application to Logistic Regression

In this section, we consider applying the two proposed methods (RDFPM I and RDFPM II) to solve the regularized centralized logistic regression problem. This problem is formulated as follows

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log(1+exp(-b_i{a}_i^{\top }x))+{\displaystyle \frac{\varrho }{2}}{\parallel x\parallel }^2,$$

(18)

where $\varrho >0$ is a regularization parameter that promotes stability and prevents overfitting, and the logistic loss function ${\displaystyle \frac{1}{N}}{\sum }_{i=1}^{N}log(1+exp(-b_i{a}_i^{\top }x))$ models the classification error based on the dataset consisting of pairs $(a_i,b_i)\in {\mathbb{R}}^n\times \{-1,1\}$. Given the strong convexity of f, the optimal solution $x^{*}\in {\mathbb{R}}^n$ is uniquely characterized as the root of the corresponding system of nonlinear monotone equations [33]:

$$F\left(x\right):=\nabla f\left(x\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{-b_{i}exp(-b_i{a}_i^{\top }x)a_i}{1+exp(-b_i{a}_i^{\top }x)}}+\varrho x=0.$$

(19)

To solve (19), we employ our newly developed algorithm, leveraging its efficiency and robustness in handling large-scale nonlinear systems. We compare its performance with the HDFPM method [25]. For fairness, the parameter settings for RDFPM I, RDFPM II, and HDFPM follow those used in previous experiments. For benchmarking, we utilized real-world datasets from the LIBSVM repository [34], available at https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ (accessed on 1 November 2025). Specifically, the benchmark instances a1a–a9a and colon-cancer were loaded using the libsvmread function. No additional preprocessing was performed. There was no imputation, categorical encoding, or feature scaling—and the logistic objective was evaluated directly on the full dataset without a train/validation/test split to ensure a consistent optimization setting across all solvers. Details on the selected datasets are provided in Table 1.

Table 1. LIBSVM instance statistics and performance metrics.

			RDFPM I				RDFPM II				HDFPM
Dataset	Data Points N	Variables n	NITER	NFVAL	CPUT	NormF	NITER	NFVAL	CPUT	NormF	NITER	NFVAL	CPUT	NormF
a1a.t	30,956	123	121.6	364.8	0.7438	9.0798 × 10−7	134.2	458.6	0.9197	9.6339 × 10−7	213.8	899.4	1.7939	9.7467 × 10−7
a2a.t	30,296	123	121.8	365.8	0.7349	9.1922 × 10−7	143.6	502.6	1.0157	9.4277 × 10−7	220.2	929.4	1.8716	9.1506 × 10−7
a3a.t	29,376	123	120.8	364.0	0.7181	9.2259 × 10−7	156.0	551.0	1.0830	9.1583 × 10−7	216.8	913.6	1.7915	9.3153 × 10−7
a4a.t	27,780	123	122.0	366.8	0.6880	9.2408 × 10−7	128.2	433.6	0.8135	9.0834 × 10−7	216.4	912.0	1.7091	9.4888 × 10−7
a5a.t	26,147	123	121.2	363.6	0.6546	9.0098 × 10−7	167.2	596.6	1.0580	7.8891 × 10−7	218.0	919.6	1.6275	9.3824 × 10−7
a6a.t	21,341	123	120.4	361.2	0.5008	9.3515 × 10−7	139.8	486.6	0.6736	8.7210 × 10−7	216.8	914.0	1.2638	9.6296 × 10−7
a7a.t	16,461	123	121.8	365.8	0.3813	9.3367 × 10−7	139.8	474.0	0.4980	8.7366 × 10−7	212.0	891.6	0.9357	9.7320 × 10−7
a8a.t	9865	122	110.8	339.6	0.1987	9.3058 × 10−7	146.8	501.4	0.2925	9.0627 × 10−7	208.2	875.2	0.5106	9.7813 × 10−7
a9a.t	16,281	122	120.2	361.2	0.3714	9.1402 × 10−7	148.8	517.8	0.5377	8.8241 × 10−7	210.6	886.4	0.9208	9.2898 × 10−7

Experiments were conducted in MATLAB (details mentioned earlier). The global random number generator was fixed to mt19937ar with seed 97006855 using RandStream.setGlobalStream. To ensure deterministic behavior and comparable wall-clock timing, we restricted BLAS operations to a single thread by setting maxNumCompThreads(1) and the environment variables OMP_NUM_THREADS=1 and MKL_NUM_THREADS=1; no parallel pools were used.

The numerical experiments are conducted by initializing x using the MATLAB script “$4\ast \left(\mathrm{rand}\right(n,1)-0.5)$” and setting the regularization parameter to $\varrho =0.1$. Each test instance is evaluated across five independent runs, and we report the average performance metrics. As demonstrated in Table 1, our proposed algorithms achieves superior efficiency, outperforming HDFPM in terms of computational time, iteration count, and function evaluations. This highlights the effectiveness of our algorithm in addressing regularized centralized logistic regression problems.

5. Conclusions

In this paper, we propose two classes of conjugate gradient algorithm that integrate restart conditions to solve nonlinear pseudomonotone equations. We have analyzed its global convergence, particularly in cases where the pseudomonotone operator is continuous. To evaluate the efficiency of the algorithm, we conducted numerical experiments on some benchmark test problems. In addition, we evaluated its performance by applying it to a logistic regression model, demonstrating its competitiveness against existing methods. The results highlight the superior efficiency of our algorithms. It successfully solved a larger number of problems while requiring fewer iterations, function evaluations, and CPU time compared to other approaches. Furthermore, we aim to extend our algorithms to multivalued pseudomonotone operators, enhancing their applicability and efficiency for a broader range of problems.