Method of Third-Order Convergence with Approximation of Inverse Operator for Large Scale Systems

Abstract

A vital role in the dynamics of physical systems is played by symmetries. In fact, these studies require the solution for systems of equations on abstract spaces including on the finite-dimensional Euclidean, Hilbert, or Banach spaces. Methods of iterative nature are commonly used to determinate the solution. In this article, such methods of higher convergence order are studied. In particular, we develop a two-step iterative method to solve large scale systems that does not require finding an inverse operator. Instead of the operator’s inverting, it uses a two-step Schultz approximation. The convergence is investigated using Lipschitz condition on the first-order derivatives. The cubic order of convergence is established and the results of the numerical experiment are given to determine the real benefits of the proposed method.

1. Introduction

In this article, F denotes an operator acting on a Banach space X with values inside a Banach space Y and $D\subseteq X$ is convex. To find the approximate solution p of

$$F\left(x\right)=0$$

(1)

we often utilize the method [1,2,3,4,5]

$$\begin{array}{c}{x}_{k+1}=x_k-F^{\prime }{\left(x_k\right)}^{-1}F\left(x_k\right),k\ge 0,\hfill \end{array}$$

(2)

due to Newton or its differential or difference modifications. Iterative methods are used since p can be found in closed form only in special cases. That is the first benefit of using iterative methods. However, at each iteration, it is necessary to find one or more inverse operators. This is one setback for using Newton-type methods [1,2,3,4,5]. Since it is not always easy to do, we bypass this obstacle by finding approximations for inversion using only multiplications of linear operators. This is one major benefit. There is the novelty of our paper, since we contribute in this direction.

Methods with approximation of inverse operator are based on different ideas. Some ideas intend to construct approximations to the solution of a nonlinear Equation (1), and others to construct approximations to the inverse operator. There are two approaches to approximate an inverse operator: successive approximation (SA) and parallel (synchronous and asynchronous) approximations. In methods with the successive approximation, the calculations in separate branches are performed alternately. In methods with the parallel approximation requiring the computation of an inverse of a linear operator, the computations in separate branches of the method are performed in parallel. Such methods are effective for numerically solving Equation (1) in the parallel processor system with common memory [6,7,8].

Many authors have investigated methods with SA of inverse operator. For example, the local convergence of modifications of Newton and Steffensen methods are studied in [9]. In [10,11,12], the authors studied the semilocal convergence (SLC) for Ulm method [9]

$$\begin{array}{c}{A}_{k+1}=A_k+(E-A_k{F}^{\prime }\left(x_k\right))A_k,\hfill \\ x_{k+1}=x_k-A_{k+1}F\left(x_k\right),k\ge 0\hfill \end{array}$$

(3)

and its difference analog

$$\begin{array}{c}{A}_{k+1}=A_k+(E-A_{k}F(x_{k-1},x_k))A_k,\hfill \\ x_{k+1}=x_k-A_{k+1}F\left(x_k\right),k\ge 0.\hfill \end{array}$$

(4)

Here, E is the identity operator in X and $F(u,v)$ stands for the first-order divided difference for operator $F\left(x\right)$ [13]. Moreover, $x_0$ and $A_0$ are initial approximations for a solution $x_{*}$ and an inverse operator $A_{*}={\left[F^{\prime }\left(x_{*}\right)\right]}^{-1}$, respectively. Furthermore, the closer $x_0$ and $A_0$ are to p, $A_{*}$ the better Equation (4) performs. However, there are no preconditioners. An investigation of the accelerated Newton method and a two-parametric secant-type method with SA of the inverse operator is performed in [14,15] (see also [16,17,18]). It is worth noting that Ulm type methods provide the same order of convergence as Newton’s or Steffensen’s but without using inverses which are very expensive to compute in general. In certain cases, even the ratio of convergence may be smaller (see the numerical section).

A method with synchronous inverse approximation

$$\begin{array}{c}{A}_{k+1}=A_k+(E-A_k{F}^{\prime }\left(x_k\right))A_k,\hfill \\ x_{k+1}=x_k-A_{k}F\left(x_k\right),k\ge 0\hfill \end{array}$$

(5)

and a method with the asynchronous approximation were considered by A. Rooze [6,7]. Some modifications of a method by Gauss–Newton with SA of the inverse operator to solve a nonlinear least squares problems were proposed by Iakymchuk, Shakhno [8].

In 1983, the authors built third-, fifth-, and sixth-order methods with approximation of inverse operator for solving operator Equations [19]. In particular, the third-order method has the form

$$\begin{array}{c}{y}_k=x_k-A_{k}F\left(x_k\right),\hfill \\ x_{k+1}=y_k-A_{k}F\left(y_k\right),\hfill \\ B_k=A_k(2E-F(x_{k+1},u_{k+1})A_k),\hfill \\ A_{k+1}=B_k(2E-F(x_{k+1},u_{k+1})B_k),k\ge 0,\hfill \end{array}$$

(6)

where $u_{k+1}=x_{k+1}-{\beta }_{k+1}F\left(x_{k+1}\right),{\beta }_{k+1}$ is a real parameter.

To increase the order of convergence, efficiency and applicability of the aforementioned Ulm-type method, by replacing the divided differences by Fréchet derivatives in Equation (6), we develop the method

$$\begin{array}{c}{y}_k=x_k-A_{k}F\left(x_k\right),\hfill \\ x_{k+1}=y_k-A_{k}F\left(y_k\right),\hfill \\ B_k=A_k(2E-F^{\prime }\left(x_{k+1}\right)A_k),\hfill \\ A_{k+1}=B_k(2E-F^{\prime }\left(x_{k+1}\right)B_k),k\ge 0.\hfill \end{array}$$

(7)

This method is a two-step modification of Newton method [2,20]:

$$\begin{array}{c}{y}_k=x_k-F^{\prime }{\left(x_k\right)}^{-1}F\left(x_k\right),\hfill \\ x_{k+1}=y_k-F^{\prime }{\left(x_k\right)}^{-1}F\left(y_k\right),k\ge 0.\hfill \end{array}$$

(8)

It is easy to see that the method in Equation (7) coincides with the third-order iterative process

$$\begin{array}{c}{y}_k=x_k-B_{k}F\left(x_k\right),\hfill \\ x_{k+1}=y_k-B_{k}F\left(y_k\right),\hfill \\ B_{k+1}=B_k+B_k(2E-F^{\prime }\left(x_{k+1}\right)B_k)(E-F^{\prime }\left(x_{k+1}\right)B_k),k\ge 0,\hfill \end{array}$$

(9)

which was proposed by Esquerro, Hernández [21]. They only investigated the semilocal convergence (SLC) of the method in Equation (9). Obviously, the methods in Equations (7) and (9) can be considered as methods SA for inverse operator.

We examine conditions for local convergence (LC) of the method in Equation (7) including its convergence order. We note that LC results are very important, since they demonstrate the degree of difficulty in selecting starters $x_0$ that guarantee convergence to p.

Another novelty of the developed method in Equation (7) lies in the fact that the inverse at each step is not calculated compared to other methods; the method is suitable to solve large scale systems; and it has better rate and convergence order than other methods using related information (see also benefits reported in Section 3. Indeed, there are significant developments in the study of Ulm-type methods. Moreover, these ideas can be used to expand the applicability of other methods along the same lines.

Section 2 contains the LC study of method in Equation (7). Large scale systems are solved in Section 3, where X and Y are specialized to be finite-dimensional Euclidean spaces. The study of solving large scale systems is of extreme importance, since most problems from diverse disciplines reduce to solving such systems [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22].

2. LC study for the Method in Equation (7)

The following theorem establishes the conditions under which the iterative process in Equation (7) is convergent.

Theorem 1.

Assume that:

(1) 

Operator $A_{*}={\left[F^{\prime }\left(p\right)\right]}^{-1}$ exists;

$$\parallel A_{*}\parallel \le B$$

(10)

and

$$\parallel F^{\prime }\left(p\right)\parallel \le C,$$

(11)

where p is the solution of Equation (1).

(2) 

In ball $U=\{x\in D:\parallel x-p\parallel \le R_0\}$, the following estimate is satisfied

$$\parallel F^{″}\left(x\right)\parallel \le L,$$

(12)

where $R_0=max\{r_0,a_1{r}_0^2,a_2{r}_0^3\},r_0=max\{\parallel x_0-p\parallel ,\parallel A_0-A_{*}\parallel \},$

${\displaystyle a_1=C+\frac{3}{2}BL+\frac{3}{2}L{r}_0}$;

${\displaystyle a_2=(C+\frac{3}{2}BL{a}_1{r}_0+\frac{3}{2}L{a}_1{r}_0^2)a_1}$.

(3) 

Initial approximations $x_0$, $A_0$ are such that

$$q{r}_0<1,$$

(13)

where $q={(max\{a_2,a_3\})}^{\frac{1}{2}}$, $a_3=C{\gamma }^2{r}_0+L{a}_2{(C+\gamma r_0^2)}^2$, $\gamma =C+L{(B+r_0)}^2{a}_2{r}_0$.

Then, sequences $\left\{x_k\right\},\left\{A_k\right\},k\ge 0$, generated by Equation (7), converge to $p,A_{*}$, respectively, and

$$r_k=max\{\parallel A_k-A_{*}\parallel ,\parallel x_k-p\parallel \}\le {\left(q{r}_0\right)}^{3^k-1}{r}_0,k\ge 0.$$

(14)

Proof. 

The proof is performed by mathematical induction.

It follows from

$$\parallel x_0-p\parallel \le r_0={\left(q{r}_0\right)}^{3^0-1}{r}_0,$$

that $x_0\in U$, and the estimate in Equation (14) is true for $k=0$. Suppose that $x_k\in U$ and the estimate in Equation (14) is satisfied for $k\ge 0$. It follows that $r_k\le r_0$, since $q{r}_0<1$ is provided by Equation (13). Taking into account Equation (10) and the definition $r_0$, we get

$$\parallel A_k\parallel \le \parallel A_{*}\parallel +\parallel A_k-A_{*}\parallel \le B+r_0.$$

(15)

We obtain from the first equality of Equation (7) and the Taylor’s formula

$$\begin{array}{ccc}\hfill p-y_k& =& p-x_k+A_k(F\left(x_k\right)-F\left(p\right))=p-x_k+A_k{F}^{\prime }\left(x_k\right)(x_k-p)\hfill \\ & \hfill & {\displaystyle -A_k{\int }_0^1{F}^{\prime \prime }(x_k+t(p-x_k)){(p-x_k)}^2(1-t)dt}\hfill \\ \hfill & =& {\displaystyle -A_k{\int }_0^1{F}^{\prime \prime }(x_k+t(p-x_k)){(p-x_k)}^2(1-t)dt}\hfill \\ & \hfill & +(E-A_k{F}^{\prime }\left(x_k\right))(p-x_k).\hfill \end{array}$$

(16)

By the condition in Equation (11) and the estimate in Equation (15), we have since

$$\begin{array}{ccc}\hfill E-A_k{F}^{\prime }\left(x_k\right)& =& (A_{*}-A_k)F^{\prime }\left(p\right)+A_k(F^{\prime }\left(p\right)-F^{\prime }\left(x_k\right)),\hfill \end{array}$$

that

$$\begin{array}{ccc}\hfill \parallel E-A_k{F}^{\prime }\left(x_k\right)\parallel & \le & \parallel F^{\prime }\left(p\right)\parallel \parallel A_{*}-A_k\parallel +\parallel A_k\parallel \parallel F^{\prime }\left(p\right)-F^{\prime }\left(x_k\right)\parallel \hfill \\ \hfill & \le & C\parallel A_{*}-A_k\parallel +(B+r_0)L\parallel p-x_k\parallel \hfill \\ \hfill & \le & r_k(C+L(B+r_0))=a{r}_k,\hfill \end{array}$$

(17)

where $a=C+L(B+r_0)$.

It follows from Equations (16) and (17) that

$$\begin{array}{ccc}\hfill \parallel p-y_k\parallel & \le & {\displaystyle \parallel A_k\parallel \parallel {\int }_0^{1}F″(x_k+t(p-x_k)){(p-x_k)}^2(1-t)dt\parallel }\hfill \\ & \hfill & +\parallel E-A_k{F}^{\prime }\left(x_k\right)\parallel \parallel p-x_k\parallel \hfill \\ \hfill & \le & {\displaystyle (B+r_0)\frac{L}{2}\parallel p-x_k{\parallel }^2+(C+L(B+r_0))r_k\parallel p-x_k\parallel }\hfill \\ \hfill & \le & {\displaystyle (B+r_0)\frac{L}{2}{r}_k^2+(C+L(B+r_0))r_k^2=(C+\frac{3}{2}BL+\frac{3}{2}L{r}_0)r_k^2,}\hfill \end{array}$$

thus

$$\parallel p-y_k\parallel \le a_1{r}_k^2.$$

(18)

In addition, $\parallel p-y_k\parallel \le a_1{r}_0^2\le R_0$, and whence it follows $y_k\in U$.

We obtain from the second equality of Equation (7) and the Taylor’s formula

$$\begin{array}{ccc}\hfill x_{k+1}-p& =& y_k-p-A_k(F\left(y_k\right)-F\left(p\right))\hfill \\ \hfill & =& {\displaystyle A_k{\int }_0^{1}F″(y_k+t(p-y_k)){(p-y_k)}^2(1-t)dt}\hfill \\ & \hfill & +(E-A_k{F}^{\prime }\left(y_k\right))(y_k-p).\hfill \end{array}$$

(19)

Hence, we have, given the conditions in Equations (10)–(12) and the estimates in Equations (15), (17), and (18),

$$\begin{array}{ccc}\hfill \parallel x_{k+1}-p\parallel & \le & (B+\parallel A_{*}-A_k\parallel )\frac{L}{2}{\parallel p-y_k\parallel }^2\hfill \\ & \hfill & +\left(C\parallel A_{*}-A_k\parallel +(B+\parallel A_{*}-A_k\parallel )L\parallel p-y_k\parallel \right)\parallel p-y_k\parallel \hfill \\ \hfill & \le & {\displaystyle C\parallel A_{*}-A_k\parallel \parallel p-y_k\parallel +\frac{3}{2}BL\parallel p-y_k{\parallel }^2+\frac{3}{2}L\parallel p-y_k{\parallel }^2\parallel A_{*}-A_k\parallel }\hfill \\ \hfill & \le & {\displaystyle \parallel p-y_k\parallel (C{r}_k+\frac{3}{2}BL\parallel p-y_k\parallel +\frac{3}{2}L{r}_k\parallel p-y_k\parallel )}\hfill \\ \hfill & \le & {\displaystyle (C+\frac{3}{2}BL{a}_1{r}_0+\frac{3}{2}L{a}_1{r}_0^2)a_1{r}_k^3=a_2{r}_k^3.}\hfill \end{array}$$

(20)

Thus, $x_{k+1}\in U$, since $\parallel x_{k+1}-p\parallel \le a_2{r}_0^3\le R_0$.

On the other hand, based on the third formula of Equation (7)

$$A_{*}-B_k=(A_{*}-A_k)F^{\prime }\left(p\right)(A_{*}-A_k)-A_k(F^{\prime }\left(p\right)-F^{\prime }\left(x_{k+1}\right))A_k.$$

Hence, we have, given Equations (11) and (20),

$$\begin{array}{ccc}\hfill \parallel A_{*}-B_k\parallel & \le & \parallel F^{\prime }\left(p\right)\parallel \parallel A_{*}-A_k{\parallel }^2+\parallel A_k{\parallel }^2\parallel F^{\prime }\left(p\right)-F^{\prime }\left(x_{k+1}\right)\parallel \hfill \\ \hfill & \le & C{r}_k^2+\parallel A_k{\parallel }^{2}L\parallel x_{k+1}-p\parallel \hfill \\ \hfill & \le & (C+L{(B+r_0)}^2{a}_2{r}_0)r_k^2=\gamma r_k^2.\hfill \end{array}$$

(21)

In accordance with the fourth equality of Equation (7),

$$A_{*}-A_{k+1}=-B_k(F^{\prime }\left(p\right)-F^{\prime }\left(x_{k+1}\right))B_k+(A_{*}-B_k)F^{\prime }\left(p\right)(A_{*}-B_k).$$

From this relationships, based on the conditions in Equations (11) and (12) and estimates in Equations (20) and (21), we get in turn

$$\begin{array}{ccc}\hfill \parallel A_{*}-A_{k+1}\parallel & \le & \parallel F^{\prime }\left(p\right)\parallel \parallel A_{*}-B_k{\parallel }^2+\parallel B_k{\parallel }^2\parallel F^{\prime }\left(p\right)-F^{\prime }\left(x_{k+1}\right)\parallel \hfill \\ \hfill & \le & C{\gamma }^2{r}_k^4+\parallel B_k{\parallel }^{2}L\parallel x_{k+1}-p\parallel \hfill \\ \hfill & \le & (C{\gamma }^2{r}_0+L{a}_2{(C+\gamma r_0^2)}^2)r_k^3=a_3{r}_k^3.\hfill \end{array}$$

(22)

Given induction assumptions we obtain, based on the estimates in Equations (20) and (22),

$$\begin{array}{ccc}\hfill r_{k+1}& =& max\{\parallel x_{k+1}-p\parallel ,\parallel A_{k+1}-A_{*}\parallel \}=max\{a_2,a_3\}{r}_k^3\hfill \\ & =& q^2{r}_k^3=q^2{\left({\left(q{r}_0\right)}^{3^k-1}{r}_0\right)}^3={\left(q{r}_0\right)}^{3^{k+1}-1}{r}_0.\hfill \end{array}$$

That is, Equation (14) is fulfilled for an iteration $k+1$. The induction is complete.

Moreover, it follows from the estimate in Equation (14) for $k\to \infty$ the convergence of sequences $\left\{x_k\right\}$ and $\left\{A_k\right\}$. ☐

Next, a uniqueness result follows.

Proposition 1.

Suppose: Equation (1) has a solution $p\in D_0$, where $D_0\subseteq D$ is convex and

$$\parallel A_{*}(F^{\prime }\left(p\right)-F^{\prime }\left(x\right))\parallel \le L_0\parallel p-x\parallel \mathit{for}x\in D_0\mathit{and}{L}_0>0.$$

(23)

Set ${\displaystyle U_0=\{x\in D:\parallel x-p\parallel <\frac{1}{L_0}\}}$ and $S=D_0\cap U_0$.

Then, p uniquely solves (1) on S.

Proof. 

Consider for $v\in S$ with $F\left(v\right)=0$, operator ${\displaystyle T={\int }_0^1{F}^{\prime }(v+a(p-v))da}$. Using Equation (23), we obtain

$$\begin{array}{ccc}\hfill \parallel A_{*}(T-F^{\prime }\left(p\right))\parallel & \le & {\displaystyle L_0{\int }_0^1\parallel v+a(p-v)-p\parallel da}\hfill \\ \hfill & \le & {\displaystyle L_0{\int }_0^1(1-a)\parallel v-p\parallel da\le \frac{L_0}{2}\parallel v-p\parallel <1,}\hfill \end{array}$$

(24)

thus T is invertible. Hence, $p=v$ follows from

$$T(p-v)=F\left(p\right)-F\left(v\right)=0.$$

 ☐

The method in Equation (7) has a higher convergence rate than the Newton and Steffensen method. Moreover, in contrast, it does not require an inverse operator. Furthermore, the convergence order of method in Equation (7) is larger than for methods with the successive approximation in Equations (3) and (4).

3. Numerical Experiments

We used large scale test problems from Luksan [22] for the numerical study of the methods. Calculations were performed in Octave 5.1.0. Stopping iterative processes occurred under the condition

$$\parallel x_{k+1}-x_k\parallel \le {10}^{-10}.$$

The initial approximations were calculated by the rules $x_0=\overline{w}$, $y_0=x_0+{10}^{-4}$. We compared methods by the number of iterations required to obtain an approximate solution.

Table 1. The number of iterations for different values of s.

Method	Example 1			Example 2			Example 3
	0.4	1	5	1	5	100	0.01	0.1	1
Equation (2)	5	7	11	5	7	11	15	18	21
Equation (3)	7	8	15	7	9	15	22	27	33
Equation (7)	4	5	9	4	5	9	13	15	18
Equation (8)	4	5	8	4	5	8	11	13	16

shows the results for $m=100$.

Example 1.

Trigonometric-exponential function.

$$\begin{array}{ccc}\hfill h_i& =& 3w_i^3+2w_{i+1}-5+sin\left(w_i-w_{i+1}\right)sin\left(w_i+w_{i+1}\right),i=1,\hfill \\ \hfill h_i& =& 3w_i^3+2w_{i+1}-5+sin\left(w_i-w_{i+1}\right)sin\left(w_i+w_{i+1}\right)+\hfill \\ & & +4w_i-w_{i-1}exp\left(w_{i-1}-w_i\right)-3,1<i<m,\hfill \\ \hfill h_i& =& 4w_i-w_{i-1}exp\left(w_{i-1}-w_i\right)-3,i=m,\hfill \\ \hfill {\overline{w}}_i& =& 2s,s\in \mathrm{I}\mathrm{R},i=\overline{1,m}.\hfill \end{array}$$

Example 2.

Consider tridiagonal function due to Broyden.

$$\begin{array}{ccc}\hfill h_i& =& w_i(0.5w_i-3)+2w_{i+1}-1,i=1,\hfill \\ \hfill h_i& =& w_i(0.5w_i-3)+w_{i-1}+2w_{i+1}-1,1<i<m,\hfill \\ \hfill h_i& =& w_i(0.5w_i-3)+w_{i-1}-1,i=m,\hfill \\ \hfill {\overline{w}}_i& =& -s,s\in \mathrm{I}\mathrm{R},i=\overline{1,m}.\hfill \end{array}$$

Example 3.

Counter current reactor problem.

$$\begin{array}{ccc}\hfill \alpha & =& 1/2,\hfill \\ \hfill h_i& =& \alpha -(1-\alpha )w_{i+2}-w_i(1+4w_{i+1}),i=1,\hfill \\ \hfill h_i& =& -(2-\alpha )w_{i+2}-w_i(1+4w_{i-1}),i=2,\hfill \\ \hfill h_i& =& \alpha w_{i-2}-(1-\alpha )w_{i+2}-w_i(1+4w_{i+1}),mod(i,2)=0,2<i<m-1,\hfill \\ \hfill h_i& =& \alpha w_{i-2}-(2-\alpha )w_{i+2}-w_i(1+4w_{i-1}),mod(i,2)=1,2<i<m-1,\hfill \\ \hfill h_i& =& \alpha w_{i-2}-w_i(1+4w_{i+1}),i=m-1,\hfill \\ \hfill h_i& =& \alpha w_{i-2}-(2-\alpha )-w_i(1+4w_{i-1}),i=m,\hfill \\ \hfill {\overline{w}}_i& =& 1000s,s\in \mathrm{I}\mathrm{R},i=\overline{1,m}.\hfill \end{array}$$

The comparison by the number of iterations was performed to confirm the theoretical results about the higher convergence order of the studied method in Equation (7) than for the Ulm method in Equation (3). Obtained results also show that the methods with the approximation of inverse operator are somewhat inferior to the corresponding methods with the calculation of the inverse operator. However, the benefit of these methods is that they can be used when finding the inverse operator is impossible or difficult.

The graphs show the values of error $\parallel x_k-p\parallel$ at each iteration (see Figure 1). These are results for $s=5$ for Example 1, $s=100$ for Example 2, and $s=0.1$ for Example 3. We draw graphs using a logarithmic scale for both of the axes.

We see from the results on graphs that the errors of the methods with the approximation of inverse operator in Equations (3) and (7) decrease more slowly than for the basic methods in Equations (2) and (8), respectively. Moreover, obtained numerical results confirm that the methods in Equations (7) and (8) have a higher convergence rate than those in Equations (2) and (3).

4. Conclusions

The LC of method in Equation (7) with approximation of inverse operator for solving large scaled systems of equations is studied. The cubic order and radius of convergence of this method are determined. Numerical results are presented which demonstrate the computational benefits of the method.