Optimal Combination of the Splitting–Linearizing Method to SSOR and SAOR for Solving the System of Nonlinear Equations

Abstract

The symmetric successive overrelaxation (SSOR) and symmetric accelerated overrelaxation (SAOR) are conventional iterative methods for solving linear equations. In this paper, novel approaches are presented by combining a splitting–linearizing method with SSOR and SAOR for solving a system of nonlinear equations. The nonlinear terms are decomposed at two sides through a splitting parameter, which are linearized around the values at the previous step, obtaining a linear equation system at each iteration step. The optimal values of parameters are determined to minimize the reciprocal of the maximal projection, which are sought in preferred ranges using the golden section search algorithm. Numerical tests assess the performance of the developed methods, namely, the optimal splitting symmetric successive over-relaxation (OSSSOR), and the optimal splitting symmetric accelerated over-relaxation (OSSAOR). The chief advantages of the proposed methods are that they do not need to compute the inverse matrix at each iteration step, and the computed orders of convergence by OSSSOR and OSSAOR are between 1.5 and 5.61; they, without needing the inner iterations loop, converge very fast with saving CPU time to find the true solution with a high accuracy.

1. Introduction

The paper is focused on novel iterative methods for solving

$$\mathbf{F}\left(\mathbf{x}\right)=\mathbf{0},\mathbf{x}:={(x_1,\dots ,x_n)}^T\in {\mathbb{R}}^n,\mathbf{F}:={(F_1,\dots ,F_n)}^T\in {\mathbb{R}}^n.$$

(1)

Nonlinear equations are ubiquitous as the main mathematical models for depicting physical phenomena. Many nonlinear ordinary differential equations and partial differential equations after discretizations become finite dimensional systems of nonlinear equations. In practice, some decomposition techniques in linear algebra such as the Cholesky decomposition $\mathbf{L}{\mathbf{L}}^{\mathrm{T}}$, $\mathbf{L}\mathbf{U}$ decomposition as well as the $\mathbf{Q}\mathbf{R}$ decomposition of a given matrix, can also be viewed as nonlinear equations for finding $\mathbf{L}$, $\mathbf{U}$, $\mathbf{Q}$ and $\mathbf{R}$. In the $\mathbf{L}\mathbf{U}$ decomposition of a matrix $\mathbf{A}=\mathbf{L}\mathbf{U}$, $\mathbf{L}$ is a lower triangular matrix, and $\mathbf{U}$ is an upper triangular matrix. In the $\mathbf{Q}\mathbf{R}$ decomposition of a matrix $\mathbf{A}=\mathbf{Q}\mathbf{R}$, $\mathbf{Q}$ is an orthonormal matrix, and $\mathbf{R}$ is an upper triangular matrix.

In the study of evolutionary type algorithm for solving nonlinear equations, Liu [1] has constructed an invariant manifold by

$$h(\mathbf{x},t)=\frac{Q\left(t\right)}{2}{\parallel \mathbf{F}\left(\mathbf{x}\right)\parallel }^2-\frac{1}{2}{\parallel \mathbf{F}\left({\mathbf{x}}_0\right)\parallel }^2=0,$$

(2)

where $\parallel \mathbf{F}\parallel$ is the residual of Equation (1) defined in terms of the Euclidean norm; $Q\left(0\right)=1$ and $Q\left(t\right)>0$ is a monotonically increasing function of t, and ${\mathbf{x}}_0$ is an initial guess of solution.

Let $\mathbf{B}=\left[b_{ij}\right]$ be the Jacobian matrix of Equation (1) with $b_{ij}=\partial F_i/\partial x_j$. Liu [1] found an important function to qualify the convergence behavior of an iterative scheme:

$$a_0\left(\mathbf{x}\right):=\frac{{\parallel \mathbf{F}\parallel }^2{\parallel \mathbf{E}\mathbf{F}\parallel }^2}{{[\mathbf{F}·\left(\mathbf{E}\mathbf{F}\right)]}^2}\ge 1,$$

(3)

where $\mathbf{E}=\mathbf{B}{\mathbf{B}}^T$. When $a_0\left(\mathbf{x}\right)$ tends to one, the solution $\mathbf{x}$ is obtained approximately. Liu [1] proposed a method to find a better descent direction $\mathbf{u}$ by minimizing

$$\underset{\mathbf{u},\mathbf{v}=\mathbf{B}\mathbf{u}}{min}\left\{a_0\left(\mathbf{x}\right):=\frac{{\parallel \mathbf{F}\parallel }^2{\parallel \mathbf{v}\parallel }^2}{{(\mathbf{F}·\mathbf{v})}^2}\right\}\ge 1.$$

(4)

It is interesting that the Newtonian descent direction $\mathbf{u}$ is given by

$$\mathbf{B}\mathbf{u}=\mathbf{F},$$

(5)

which renders $\mathbf{v}=\mathbf{B}\mathbf{u}=\mathbf{F}$, and hence, $a_0\left(\mathbf{x}\right)=1$ in Equation (4). In the sense of the optimality of Equation (4), the Newton iteration method is the optimal one, which is convergent quadratically. However the Newton iteration method is sensitive to the initial guess of the solution and the computation of ${\mathbf{B}}_k^{-1}$ at each iteration step is expensive. Indeed, there are many effective iteration methods by extending and modifying the Newton iteration method to solve the nonlinear Equations [2,3,4,5,6]. Recently, Al-Obaidi and Darvish [7] gave a comparative study on the qualification criteria for three categories nonlinear solvers for solving nonlinear equations. The multi-point and higher-order iterative algorithms based on the Newton technique for solving nonlinear equations can be seen in [8,9].

The methods of symmetric successive over-relaxation and symmetric accelerated over-relaxation [10,11,12,13] iteratively solve $\mathbf{x}\in {\mathbb{R}}^n$ from

$$\mathbf{A}\mathbf{x}=\mathbf{b},$$

(6)

where the coefficient matrix $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ has nonzero diagonal elements, and $\mathbf{b}\in {\mathbb{R}}^n$ is given.

Let $\mathbf{y}:=\mathbf{A}\mathbf{x}$. We attempt to find the optimal approximation of $\mathbf{b}$ to $\mathbf{y}$ by an optimization method. The orthogonal projection is regarded as the approximation to $\mathbf{b}$ by $\mathbf{y}$, whose error vector is written as

$$\mathbf{e}:=\mathbf{b}-\mathbf{b}·\frac{\mathbf{y}}{\parallel \mathbf{y}\parallel }\frac{\mathbf{y}}{\parallel \mathbf{y}\parallel }.$$

(7)

The best approximation can be found by minimizing

$$\underset{\mathbf{x}}{min}{\parallel \mathbf{e}\parallel }^2=\underset{\mathbf{x}}{min}\left\{{\parallel \mathbf{b}\parallel }^2-\frac{{(\mathbf{b}·\mathbf{y})}^2}{{\parallel \mathbf{y}\parallel }^2}\right\},$$

(8)

or equivalently maximizing the square of the orthogonal projection of $\mathbf{b}$ to $\mathbf{y}$, i.e.,

$$\underset{\mathbf{x}}{max}\left\{\frac{{(\mathbf{b}·\mathbf{y})}^2}{{\parallel \mathbf{y}\parallel }^2}\right\}.$$

(9)

Due to this reason, the solution of the above equation will be named the maximal projection solution.

Since $\mathbf{b}$ is a given nonzero constant vector, we can recast Equation (9) to

$$\underset{\mathbf{x}}{max}\left\{\frac{{(\mathbf{b}·\mathbf{y})}^2}{{\parallel \mathbf{b}\parallel }^2{\parallel \mathbf{y}\parallel }^2}\right\},$$

(10)

which does not influence the solution of $\mathbf{x}$. The following merit function

$$\underset{\mathbf{x}}{min}\left\{f=\frac{{\parallel \mathbf{b}\parallel }^2{\parallel \mathbf{y}\parallel }^2}{{(\mathbf{b}·\mathbf{y})}^2}\right\}\ge 1,$$

(11)

which is the reciprocal of Equation (10), has the same form to Equation (4); it is used to seek the optimal searching direction in the numerical solution of nonlinear equations.

Based on Equation (11), Liu [14] developed a double optimal iterative algorithm (DOIA) to solve the nonlinear equations by solving

$${\mathbf{B}}_k{\mathbf{u}}_k={\mathbf{F}}_k,$$

(12)

at each iteration step to find the best descent direction ${\mathbf{u}}_k$ in an affine Krylov subspace.

In the methods of symmetric successive over-relaxation (SSOR), accelerated over-relaxation (AOR) and symmetric accelerated over-relaxation (SAOR) the optimal values of the parameters are usually difficult to achieve [11,12,15]. Here, we are going to search the optimal values of the parameters used in the combination of a splitting technique to the SSOR and SAOR and optimize the related parameters by the minimization technique in Equation (11) to solve the nonlinear equations.

Basically, many algorithms, like the DOIA in [14] and the double iterations method in [16], required us to solve a linear system (12) in each iteration step to determine the precise descent vector ${\mathbf{u}}_k$ in the current step. This is carried out in general through an inner iterations loop. In the proposed iterative algorithms, the SSOR and SAOR are automatically merged into the outer iterations, which do not need to carry out the loop of inner iterations at each iteration step; refer to the pseudo-code to be presented in Section 4. In doing so, a lot of computational time can be saved. Therefore, the main superiority of the new method are its low computational complexity, saving computational time and not sensitive to the initial guess of ${\mathbf{x}}_0$. However, a major drawback of the SSOR and SAOR is that some values of the parameters must be properly selected. We are going to overcome this difficulty in this paper, and we propose a novel linearization method which is different from the Newton iteration method; the computation of ${\mathbf{B}}_k$ and its inversion ${\mathbf{B}}_k^{-1}$ at each iteration step are no longer needed.

Other sections of the paper are arranged as follows. Section 2 briefly sketches some conventional iterative algorithms for solving linear equations, including the successive over-relaxation (SOR) method, its symmetrization as the symmetry successive over-relaxation (SSOR) method, and the symmetry accelerated over-relaxation (SAOR) method. In Section 3, a matrix–vector form of nonlinear equations is introduced, which is linearized to a step-by-step varying linear system through a splitting parameter. By applying SSOR and minimizing a merit function based on the maximal projection method, we develop an optimal splitting symmetry successive over-relaxation (OSSSOR); combining the splitting–linearizing technique with SAOR, we develop an optimal splitting symmetry accelerated over-relaxation (OSSAOR) method for solving nonlinear equations. In Section 4, a pseudo-code for OSSSOR is sketched. In Section 5, a local convergence analysis is initiated. Some examples of nonlinear equations are examined in Section 6 by OSSSOR and OSSAOR. Finally, the conclusions in Section 7 summarize and highlight the main achievements.

2. Reviews of Two Symmetric Iterative Methods for Linear Equations

In Equation (6), the coefficient matrix is decomposed to

$$\mathbf{A}=\mathbf{D}-\mathbf{U}-\mathbf{L},$$

(13)

where $\mathbf{D}$, $\mathbf{U}$ and $\mathbf{L}$ are, respectively, the diagonal, strictly upper triangular and strictly lower triangular parts of $\mathbf{A}$.

It follows from Equations (6) and (13) the SOR iterative method [13]:

$$(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_k+w\mathbf{U}{\mathbf{x}}_k.$$

(14)

For the purpose of convergence, the relaxation parameter w is constrained by $0<w<2$. In Equation (14), owing to the lower triangular matrix of $\mathbf{D}-w\mathbf{L}$, ${\mathbf{x}}_{k+1}$ is easily computed from the previous step value ${\mathbf{x}}_k$ by using the forward substitution method.

On the other hand, the SSOR is [17]

$$\begin{array}{ccc}\hfill & & (\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1/2}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_k+w\mathbf{U}{\mathbf{x}}_k,\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill & & (\mathbf{D}-w\mathbf{U}){\mathbf{x}}_{k+1}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_{k+1/2}+w\mathbf{L}{\mathbf{x}}_{k+1/2},\hfill \end{array}$$

(16)

which is convergent faster than SOR.

While the AOR is [11]

$$(\mathbf{D}-\sigma \mathbf{L}){\mathbf{x}}_{k+1}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_k+w\mathbf{U}{\mathbf{x}}_k+(w-\sigma )\mathbf{L}{\mathbf{x}}_k,$$

(17)

a symmetric AOR (SAOR) is [12]

$$\begin{array}{ccc}\hfill & & (\mathbf{D}-\sigma \mathbf{L}){\mathbf{x}}_{k+1/2}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_k+w\mathbf{U}{\mathbf{x}}_k+(w-\sigma )\mathbf{L}{\mathbf{x}}_k,\hfill \\ & & (\mathbf{D}-\sigma \mathbf{U}){\mathbf{x}}_{k+1}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_{k+1/2}+w\mathbf{L}{\mathbf{x}}_{k+1/2}+(w-\sigma )\mathbf{U}{\mathbf{x}}_{k+1/2}.\hfill \end{array}$$

(18)

3. Nonlinear Equations

Recently, Liu et al. [18] applied the dynamical optimal techniques in the SSOR, AOR, and SAOR for solving linear equations. We extend these results to the system of nonlinear equations in Equation (1), which can be recast to a matrix–vector form:

$$\tilde{\mathbf{A}}\mathbf{x}+\mathbf{B}\left(\mathbf{x}\right)\mathbf{x}=\tilde{\mathbf{b}},$$

(19)

where $\tilde{\mathbf{A}}\in {\mathbb{R}}^{n\times n}$ is a constant matrix, $\mathbf{B}\in {\mathbb{R}}^{n\times n}$ is a matrix function of $\mathbf{x}$, and $\tilde{\mathbf{b}}\in {\mathbb{R}}^n$ is a constant vector.

To Equation (19), we add a splitting parameter s:

$$\tilde{\mathbf{A}}\mathbf{x}+s\mathbf{B}\left(\mathbf{x}\right)\mathbf{x}=\tilde{\mathbf{b}}+(s-1)\mathbf{B}\left(\mathbf{x}\right)\mathbf{x},$$

(20)

which can be linearized around ${\mathbf{x}}_k$ by

$$\tilde{\mathbf{A}}\mathbf{x}+s\mathbf{B}\left({\mathbf{x}}_k\right)\mathbf{x}=\tilde{\mathbf{b}}+(s-1)\mathbf{B}\left({\mathbf{x}}_{\mathbf{k}}\right){\mathbf{x}}_k.$$

(21)

Upon letting

$$\mathbf{A}\left(s\right):=\tilde{\mathbf{A}}+s\mathbf{B}\left({\mathbf{x}}_k\right),\mathbf{b}\left(s\right):=\tilde{\mathbf{b}}+(s-1)\mathbf{B}\left({\mathbf{x}}_{\mathbf{k}}\right){\mathbf{x}}_k,$$

(22)

we can derive a step-by-step varying linear system:

$$\mathbf{A}\left(s\right)\mathbf{x}=\mathbf{b}\left(s\right),$$

(23)

where both $\mathbf{A}$ and $\mathbf{b}$ are functions of s and ${\mathbf{x}}_k$. However, for saving notations, the dependence on ${\mathbf{x}}_k$ is not written out explicitly.

3.1. Optimal Splitting Symmetric Successive Over-Relaxation Method

According to the SSOR in Section 2, we can derive an iterative scheme for Equation (23):

$$\begin{array}{ccc}\hfill & & [\mathbf{D}\left(s\right)-w\mathbf{L}\left(s\right)]{\mathbf{x}}_{k+1/2}=w\mathbf{b}\left(s\right)+(1-w)\mathbf{D}\left(s\right){\mathbf{x}}_k+w\mathbf{U}\left(s\right){\mathbf{x}}_k,\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill & & [\mathbf{D}\left(s\right)-w\mathbf{U}\left(s\right)]{\mathbf{x}}_{k+1}=w\mathbf{b}\left(s\right)+(1-w)\mathbf{D}\left(s\right){\mathbf{x}}_{k+1/2}+w\mathbf{L}\left(s\right){\mathbf{x}}_{k+1/2}.\hfill \end{array}$$

(25)

Now, $\mathbf{D}$, $\mathbf{U}$ and $\mathbf{L}$ are also functions of s and ${\mathbf{x}}_k$, such that it is a nonlinear iterative scheme. Equations (24) and (25) involve two parameters $(s,w)$ to be determined as follows. Let

$${\mathbf{H}}_1=[\mathbf{D}\left(s\right)-w\mathbf{U}\left(s\right)]{\mathbf{x}}_{k+1/2},{\mathbf{H}}_2=w\mathbf{b}\left(s\right)+(1-w)\mathbf{D}\left(s\right){\mathbf{x}}_{k+1/2}+w\mathbf{L}\left(s\right){\mathbf{x}}_{k+1/2},$$

(26)

where ${\mathbf{x}}_{k+1/2}$ is calculated from Equation (24). Insert them into Equation (11), we can seek the optimal values of $(s_k,w_k)$ by minimizing

$$\underset{(s_k,w_k)\in (a,b)\times (c,d)}{min}\left\{f_{k+1/2}=\frac{\parallel {\mathbf{H}}_1{\parallel }^2{\parallel {\mathbf{H}}_2\parallel }^2}{{({\mathbf{H}}_1·{\mathbf{H}}_2)}^2}\right\}.$$

(27)

The algorithm in Equations (24), (25) and (27) is a new iterative scheme for solving nonlinear equations system (1), which is shortened as an optimal splitting symmetric successive over-relaxation (OSSSOR) method. A two-dimensional golden section search algorithm as listed in the Appendix A is applied for determining $(s_k,w_k)$ in Equation (27), of which only a few steps are spent at each iteration, when a loose convergence criterion is taken.

3.2. Optimal Splitting Symmetric Accelerated Over-Relaxation Method

In general, the symmetric accelerated over-relaxation method (SAOR) is convergent faster than the symmetric successive over-relaxation method (SSOR) for solving linear equations. According to the SAOR in Section 2, we can derive a nonlinear iterative scheme for Equation (23):

$$\begin{array}{ccc}\hfill & & [\mathbf{D}\left(s\right)-\sigma \mathbf{L}\left(s\right)]{\mathbf{x}}_{k+1/2}=w\mathbf{b}\left(s\right)+(1-w)\mathbf{D}\left(s\right){\mathbf{x}}_k+w\mathbf{U}\left(s\right){\mathbf{x}}_k+(w-\sigma )\mathbf{L}\left(s\right){\mathbf{x}}_k,\hfill \end{array}$$

(28)

$$\begin{array}{ccc}\hfill & & [\mathbf{D}\left(s\right)-\sigma \mathbf{U}\left(s\right)]{\mathbf{x}}_{k+1}=w\mathbf{b}\left(s\right)+(1-w)\mathbf{D}\left(s\right){\mathbf{x}}_{k+1/2}+w\mathbf{L}\left(s\right){\mathbf{x}}_{k+1/2}+(w-\sigma )\mathbf{U}\left(s\right){\mathbf{x}}_{k+1/2}.\hfill \end{array}$$

(29)

Equations (28) and (29) involve three parameters $(s,w,\sigma )$ to be determined as follows. Let

$$\begin{array}{ccc}\hfill & & {\mathbf{R}}_1=[\mathbf{D}\left(s\right)-\sigma \mathbf{U}\left(s\right)]{\mathbf{x}}_{k+1/2},\hfill \\ & & {\mathbf{R}}_2=w\mathbf{b}\left(s\right)+(1-w)\mathbf{D}\left(s\right){\mathbf{x}}_{k+1/2}+w\mathbf{L}\left(s\right){\mathbf{x}}_{k+1/2}+(w-\sigma )\mathbf{U}\left(s\right){\mathbf{x}}_{k+1/2},\hfill \end{array}$$

(30)

where ${\mathbf{x}}_{k+1/2}$ is calculated from Equation (28). Insert them into Equation (11), obtaining

$$\underset{(s_k,w_k,{\sigma }_k)\in (A,B)\times (a,b)\times (c,d)}{min}\left\{\frac{\parallel {\mathbf{R}}_1{\parallel }^2{\parallel {\mathbf{R}}_2\parallel }^2}{{({\mathbf{R}}_1·{\mathbf{R}}_2)}^2}\right\};$$

(31)

it is used to determine $(s_k,w_k,{\sigma }_k)$. However, this minimization in three dimensions would spend much time. We take a simpler one by

$$\underset{s_k\in [A,B]}{min}\underset{(w_k,{\sigma }_k)\in (a,b)\times (c,d)}{min}\left\{\frac{\parallel {\mathbf{R}}_1{\parallel }^2{\parallel {\mathbf{R}}_2\parallel }^2}{{({\mathbf{R}}_1·{\mathbf{R}}_2)}^2}\right\},$$

(32)

where we let $s_k$ run in a few points in $(A,B)$ by $s_k^j=A+(j-1)(B-A)/(N_s-1),$ $j=1,\dots ,N_s$ with $N_s=6$. The minimization in the second part is carried out by the golden section search algorithm.

The algorithm in Equations (28), (29) and (32) is a new iterative scheme for solving nonlinear equations system (1), which is shortened as an optimal splitting symmetric accelerated over-relaxation (OSSAOR) method.

4. Numerical Algorithm

In this section, we take OSSSOR as an example to write its pseudo-code, where the two-dimensional golden section algorithm is listed in the Appendix A. Before that, we need two codes to solve Equations (24) and (25) forward and backward, respectively. In the code, n is the dimension of the nonlinear equations, b(I) is the I-component of $\mathbf{b}$, and A(I,J) is the IJ-component of $\mathbf{A}$; $\mathbf{A}$ and $\mathbf{b}$ are inputs, and $\mathbf{c}$ is the output (Algorithms 1–3).

Algorithm 1: FSOL  (n,A,b,c)

Give n, $\mathbf{A}$ and $\mathbf{b}$ DO I = 1,n SUM = 0 DO J = 1, I − 1 SUM = SUM + A(I,J) × c(J) Enddo of J c(I) = (b(I) − SUM)/A(I,I) Enddo of I

Algorithm 2: BSOL (n,A,b,c)

Give n, $\mathbf{A}$ and $\mathbf{b}$ DO K = 1,n I = n − K+1 SUM = 0 DO J = I + 1,n SUM = SUM + A(I,J)×c(J) Enddo of J c(I) = (b(I) − SUM)/A(I,I) Enddo of K

Algorithm 3: OSSSOR

1: Give n, $\mathbf{F}$, initial value ${\mathbf{x}}_0$, and $\epsilon$ 2: Give a, b, c and d 3: Do $k=0,1,\dots$ 4: Call GOLDEN(a, b, c, d, $f_{k+1/2}$, $s_k$, $w_k$) 5: Compute ${\mathbf{A}}_k\left(s_k\right)=\tilde{\mathbf{A}}+s_k\mathbf{B}\left({\mathbf{x}}_k\right)$ 6: Compute ${\mathbf{b}}^o\left(s_k\right)=\tilde{\mathbf{b}}+(s_k-1)\mathbf{B}\left({\mathbf{x}}_{\mathbf{k}}\right){\mathbf{x}}_k$ 7: Compute ${\mathbf{D}}_k\left(s_k\right)$, ${\mathbf{L}}_k\left(s_k\right)$ and ${\mathbf{U}}_k\left(s_k\right)$ 8: Compute ${\mathbf{A}}_k\left(s_k\right)={\mathbf{D}}_k\left(s_k\right)-w_k{\mathbf{L}}_k\left(s_k\right)$ 9: Compute ${\mathbf{b}}_k\left(s_k\right)=w_k{\mathbf{b}}^o\left(s_k\right)+(1-w_k){\mathbf{D}}_k\left(s_k\right){\mathbf{x}}_k+w_k{\mathbf{U}}_k\left(s_k\right){\mathbf{x}}_k$ 10: Call FSOL(n,${\mathbf{A}}_k\left(s_k\right)$,${\mathbf{b}}_k\left(s_k\right)$,${\mathbf{x}}_{k+1/2}$) 11: Compute ${\mathbf{A}}_k\left(s_k\right)={\mathbf{D}}_k\left(s_k\right)-w_k{\mathbf{U}}_k\left(s_k\right)$ 12: Compute ${\mathbf{b}}_k\left(s_k\right)=w_k{\mathbf{b}}^o\left(s_k\right)+(1-w_k){\mathbf{D}}_k\left(s_k\right){\mathbf{x}}_{k+1/2}+w_k{\mathbf{L}}_k\left(s_k\right){\mathbf{x}}_{k+1/2}$ 13: Call BSOL(n,${\mathbf{A}}_k\left(s_k\right)$,${\mathbf{b}}_k\left(s_k\right)$,${\mathbf{x}}_{k+1}$) 14: If $\parallel \mathbf{F}\left({\mathbf{x}}_{k+1}\right)\parallel <\epsilon$, stop 15: Otherwise, go to 3

We notice that ${\mathbf{b}}^o$ denotes $\mathbf{b}\left(s\right)$ in Equation (22); $f_{k+1/2}$ in Equation (27) is the input of GOLDEN, whose outputs are $s_k$ and $w_k$; the output of FSOL is ${\mathbf{x}}_{k+1/2}$; and the output of BSOL is ${\mathbf{x}}_{k+1}$.

5. Local Convergence

We suppose that at the kth iteration the nonlinear equations are linearized to

$${\mathbf{A}}^{\left(k\right)}\mathbf{x}={\mathbf{b}}^{\left(k\right)},$$

(33)

where

$${\mathbf{A}}^{\left(k\right)}={\mathbf{D}}^{\left(k\right)}-{\mathbf{U}}^{\left(k\right)}-{\mathbf{L}}^{\left(k\right)}.$$

(34)

We first discuss the SOR:

$$({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=w{\mathbf{b}}^{\left(k\right)}+(1-w){\mathbf{D}}^{\left(k\right)}{\mathbf{x}}_k+w{\mathbf{U}}^{\left(k\right)}{\mathbf{x}}_k.$$

(35)

Theorem 1.

A new iterative form of SOR in Equation (35) is

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,$$

(36)

where ${\mathbf{u}}_k$ is the kth step descent vector, obtained from a forward solution of

$$\begin{array}{ccc}\hfill & & ({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{u}}_k=w{\mathbf{r}}_k;\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& & {\mathbf{r}}_k={\mathbf{b}}^{\left(k\right)}-{\mathbf{A}}^{\left(k\right)}{\mathbf{x}}_k\hfill \end{array}$$

(38)

is the kth step residual vector.

Proof. 

We rewrite Equation (35) to

$$({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{b}}^{\left(k\right)}+[(1-w){\mathbf{D}}^{\left(k\right)}+w{\mathbf{U}}^{\left(k\right)}-({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)})]{\mathbf{x}}_k,$$

(39)

which can be rearranged to

$$({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{b}}^{\left(k\right)}-w({\mathbf{D}}^{\left(k\right)}-{\mathbf{U}}^{\left(k\right)}-{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k.$$

(40)

In view of Equations (34) and (38), we have

$$({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{b}}^{\left(k\right)}-w{\mathbf{A}}^{\left(k\right)}{\mathbf{x}}_k=({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{r}}_k;$$

(41)

removing $({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k$ to the left-hand side yields

$$({\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)})({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)=w{\mathbf{r}}_k.$$

(42)

In terms of ${\mathbf{u}}_k$ in Equation (36), we can derive Equation (37). □

Motivated by the new form of the SOR in Equations (36)–(38), we can prove the following result.

Theorem 2.

For SOR, if the following condition is satisfied

$$\parallel {\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k{\parallel }^2-2{\mathbf{r}}_k^T{\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k<0,$$

(43)

then the iterative algorithm is local convergence with

$$\parallel {\mathbf{r}}_{k+1}\parallel <\parallel {\mathbf{r}}_k\parallel .$$

(44)

Proof. 

In terms of residual vector, we can rewrite Equation (36) to

$${\mathbf{r}}_{k+1}={\mathbf{r}}_k-{\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k.$$

(45)

Taking the squared norms of both sides yields

$$\parallel {\mathbf{r}}_{k+1}{\parallel }^2=\parallel {\mathbf{r}}_k{\parallel }^2-2{\mathbf{r}}_k^T{\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k+{\parallel {\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k\parallel }^2.$$

(46)

If the condition (43) is satisfied, then

$$\parallel {\mathbf{r}}_{k+1}{\parallel }^2<{\parallel {\mathbf{r}}_k\parallel }^2.$$

(47)

Taking the square roots of both sides, Equation (44) is proven. □

As shown in Equation (11), we choose the optimal values of the parameters s and w to minimize

$$\underset{s,w}{min}\left\{f=\frac{\parallel {\mathbf{r}}_k{\parallel }^2{\parallel {\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k\parallel }^2}{{\left({\mathbf{r}}_k^T{\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k\right)}^2}\right\}\ge 1.$$

(48)

If ${\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k={\mathbf{r}}_k$ is satisfied, $f=1$ can be achieved. Then by means of Equation (46), we can derive

$$\parallel {\mathbf{r}}_{k+1}{\parallel }^2=\parallel {\mathbf{r}}_k{\parallel }^2-{\parallel {\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k\parallel }^2.$$

(49)

It guarantees the local convergence of the proposed iterative algorithm. For the SSOR the similar results can be derived for the backward part with ${\mathbf{D}}^{\left(k\right)}-w{\mathbf{L}}^{\left(k\right)}$ in Equation (37) being replaced by ${\mathbf{D}}^{\left(k\right)}-w{\mathbf{U}}^{\left(k\right)}$. We can conclude that

$$\parallel {\mathbf{r}}_{k+1}\parallel <\parallel {\mathbf{r}}_{k+1/2}\parallel <\parallel {\mathbf{r}}_k\parallel$$

(50)

in the SSOR. Thus, the local convergence of SSOR can be verified.

For the AOR,

$$({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=w{\mathbf{b}}^{\left(k\right)}+(1-w){\mathbf{D}}^{\left(k\right)}{\mathbf{x}}_k+w{\mathbf{U}}^{\left(k\right)}{\mathbf{x}}_k+(w-\sigma ){\mathbf{L}}^{\left(k\right)}{\mathbf{x}}_k,$$

(51)

the proof of local convergence is given as follows, which involves three parameters.

Theorem 3.

A new iterative form of AOR in Equation (51) is

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,$$

(52)

where ${\mathbf{u}}_k$ is the kth step descent vector, obtained from a forward solution of

$$\begin{array}{ccc}\hfill & & ({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{u}}_k=w{\mathbf{r}}_k;\hfill \end{array}$$

(53)

$$\begin{array}{ccc}& & {\mathbf{r}}_k={\mathbf{b}}^{\left(k\right)}-{\mathbf{A}}^{\left(k\right)}{\mathbf{x}}_k\hfill \end{array}$$

(54)

is the kth step residual vector.

Proof. 

We rewrite Equation (51) to

$$\begin{array}{ccc}\hfill & & ({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{b}}^{\left(k\right)}\hfill \\ & & +(1-w){\mathbf{D}}^{\left(k\right)}{\mathbf{x}}_k+w{\mathbf{U}}^{\left(k\right)}{\mathbf{x}}_k+(w-\sigma ){\mathbf{L}}^{\left(k\right)}{\mathbf{x}}_k-({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k.\hfill \end{array}$$

(55)

In view of Equations (34) and (54), which can be rearranged to

$$({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_{k+1}=({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{b}}^{\left(k\right)}-w{\mathbf{A}}^{\left(k\right)}{\mathbf{x}}_k=({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k+w{\mathbf{r}}_k;$$

(56)

removing $({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)}){\mathbf{x}}_k$ to the left-hand side yields

$$({\mathbf{D}}^{\left(k\right)}-\sigma {\mathbf{L}}^{\left(k\right)})({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)=w{\mathbf{r}}_k.$$

(57)

In terms of ${\mathbf{u}}_k$ in Equation (52), we can derive Equation (53). □

Theorem 4.

For AOR, if the following condition is satisfied

$$\parallel {\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k{\parallel }^2-2{\mathbf{r}}_k^T{\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k<0,$$

(58)

then the iterative algorithm is local convergence with

$$\parallel {\mathbf{r}}_{k+1}\parallel <\parallel {\mathbf{r}}_k\parallel .$$

(59)

Proof. 

The proof is same to that given in Theorem 2. □

If ${\mathbf{A}}^{\left(k\right)}{\mathbf{u}}_k={\mathbf{r}}_k$ is satisfied, $f=1$ can be achieved, and the iterative algorithm is local convergence as shown in Equation (49). Numerical examples given below will show that f fast tends to 1.

6. Numerical Tests of Nonlinear Equations

In this section, we test the performance of OSSSOR and OSSAOR for solving nonlinear equations. To further asses the performance of the proposed algorithms, we compute the convergence order. For solving a scalar equation $f\left(x\right)=0$, if the iterative sequence $\left\{x_n\right\}$ converges to $x^{*}$ with $f\left(x^{*}\right)=0$, the convergence order $\zeta$ can be estimated by

$$|x_{n+1}-x^{*}|=\beta |x_n-x^{*}{|}^{\zeta },$$

(60)

where $\beta >0$ is a constant. Taking the operation ln on both sides yields

$$\begin{array}{ccc}\hfill & & ln|x_{n+1}-x^{*}|=ln\beta +\zeta ln|x_n-x^{*}|,\hfill \end{array}$$

(61)

$$\begin{array}{ccc}\hfill & & ln|x_n-x^{*}|=ln\beta +\zeta ln|x_{n-1}-x^{*}|.\hfill \end{array}$$

(62)

Subtracting Equation (61) by Equation (62), we can approximate $\zeta$ by

$$COC:=\frac{ln\left|\frac{x_{n+1}-x^{*}}{x_n-x^{*}}\right|}{ln\left|\frac{x_n-x^{*}}{x_{n-1}-x^{*}}\right|}.$$

(63)

Similarly, for solving the nonlinear equations, we can generate the sequence $\parallel {\mathbf{x}}_k\parallel ,$ $k=1,\dots ,k_0$, upon giving the initial value ${\mathbf{x}}_0$. We define

$$COC:=\frac{ln\frac{\parallel {\mathbf{x}}_{k_0}-{\mathbf{x}}^{*}\parallel }{\parallel {\mathbf{x}}_{k_0-1}-{\mathbf{x}}^{*}\parallel }}{ln\frac{\parallel {\mathbf{x}}_{k_0-1}-{\mathbf{x}}^{*}\parallel }{\parallel {\mathbf{x}}_{k_0-2}-{\mathbf{x}}^{*}\parallel }},$$

(64)

where ${\mathbf{x}}^{*}$ is the exact solution and $k_0$ is the number of iterations.

6.1. Example 1

First, we solve [19]

$$\begin{array}{ccc}\hfill & & F_1(x,y)=x^3-3x{y}^2+a_1(2x^2+xy)+b_1{y}^2+c_{1}x+a_{2}y=0,\hfill \end{array}$$

(65)

$$\begin{array}{ccc}\hfill & & F_2(x,y)=3x^{2}y-y^3-a_1(4xy-y^2)+b_2{x}^2+c_2=0.\hfill \end{array}$$

(66)

These equations have been solved in [14,19,20] by different methods, and they found five solutions.

Hirsch and Smale [19] derived a continuous Newton method governed by the following differential equation:

$$\begin{array}{ccc}\hfill & & \dot{\mathbf{x}}\left(t\right)=-{\mathbf{B}}^{-1}\left(\mathbf{x}\right)\mathbf{F}\left(\mathbf{x}\right),\hfill \end{array}$$

(67)

$$\begin{array}{ccc}\hfill & & \mathbf{x}\left(0\right)={\mathbf{x}}_0.\hfill \end{array}$$

(68)

It can be seen that Equation (67) is difficult to calculate, because it includes an inverse of the Jacobian matrix.

Equations (65) and (66) can be written as

$$\left[\begin{array}{cc}{c}_1& a_2\\ 0& 0\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{cc}{x}^2+2a_{1}x& b_{1}y-3xy+a_{1}x\\ b_{2}x& 3x^2-y^2+a_{1}y-4a_{1}x\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}0\\ -c_2\end{array}\right].$$

(69)

We take $a_1=25$, $b_1=-1$, $c_1=-2$, $a_2=-3$, $b_2=-4$ and $c_2=-5$. Hirsch and Smale [19] used the continuous Newton algorithm to calculate this problem and obtained $(x,y)=(39.0207,38.2417)$. Inserting it into $F_1$ and $F_2$, we find that $(F_1,F_2)=(-0.339,-0.117)$, which indicates that the result obtained in [19] is not accurate.

In [21], with $(x_0,y_0)=(0.1,0.1)$ and the number of iterations (NI) = 101, $(x,y)=(0.134212,0.811128)$ and $\left(\right|F_1|,|F_2\left|\right)=(1.141\times {10}^{-7},8.913\times {10}^{-7})$ were obtained. By using OSSSOR, we take $(a,b)\times (c,d)=(0.9,1.6)\times (0.5,1.9)$ and $(x_0,y_0)=(1,1)$, and NI = 49 under $\epsilon ={10}^{-15}$ as shown in Figure 1a by a solid line. In the whole paper, “Residual” in a figure means $\parallel \mathbf{F}\parallel$, which abides the convergence criterion by $\parallel \mathbf{F}\parallel <\epsilon$.

Here, $(x,y)=(-0.163635,0.230529)$ is obtained by the OSSSOR, and $\left(\right|F_1|,|F_2\left|\right)=(2.22\times {10}^{-16},8.88\times {10}^{-16})$ are much smaller than that in [21]. Moreover, it is much more accurate than that computed in [19]. Figure 1b,c show s and w. For the OSSSOR, COC = 5.61 and the CPU time is 0.32 s.

By using OSSAOR, we take $(A,B)=(0.8,1.3)$, $(a,b)\times (c,d)=(0.5,0.6)\times (0.5,0.9)$ and $(x_0,y_0)=(1,1)$, and NI = 43 under $\epsilon ={10}^{-15}$ is shown in Figure 1a by a dashed line. The solution $(x,y)=(-0.163635,0.230529)$ is obtained, and $\left(\right|F_1|,|F_2\left|\right)=(3.33\times {10}^{-16},0)$ are much smaller than that in [21]. Figure 2a–c reveal s, w and $\sigma$. After ten steps, w and $\sigma$ tend to constant values. For the OSSAOR, COC = 3.67, and the CPU time is 0.29 s.

Starting from an initial value $(x_0,y_0)=(10,10)$ and under a convergence criterion $\epsilon ={10}^{-10}$, the OSSAOR converges with NI = 31 to

$$(x,y)=(-0.163635,0.230529),$$

and the OSSSOR converges with NI = 40 to the same solution. In [14], the double optimal iterative algorithm (DOIA) with NI = 223 tends to

$$(x,y)=(0.134212,0.811128).$$

On the other hand, the Newton iterative algorithm with NI = 410 converges to

$$(x,y)=(39.020711,38.241665).$$

Different methods may lead to different solutions; although, they used the same initial guesses.

In Figure 3, we compare the convergence speeds of those four methods, of which OSSSOR and OSSAOR are convergent faster than DOIA and the Newton method.

6.2. Example 2

Next, we apply the OSSSOR and OSSAOR to solve a nonlinear boundary value problem:

$$u^{″}=\frac{3}{2}{u}^2,u\left(0\right)=4,u\left(1\right)=1.$$

(70)

The exact solution of Equation (70) is

$$u\left(x\right)=\frac{4}{{(1+x)}^2}.$$

(71)

By introducing a finite difference discretization of u at the grid points, we can obtain

$$\begin{array}{ccc}\hfill & & F_i=\frac{1}{{(\Delta x)}^2}(u_{i+1}-2u_i+u_{i-1})-\frac{3}{2}{u}_i^2=0,i=1,\dots ,n,\hfill \\ & & u_0=4,u_{n+1}=1,\hfill \end{array}$$

(72)

where $\Delta x=1/(n+1)$ is the grid length.

By using OSSSOR, we take $n=89$, $(a,b)\times (c,d)=(1,3)\times (1.95,1.97)$, and NI = 157 under $\epsilon ={10}^{-5}$ is shown in Figure 4a by a solid line. At the final step, the residual $9.06\times {10}^{-11}$ is obtained. For the OSSSOR, COC = 1.67, and the CPU time is 1.79 s.

Figure 4b,c show s and w. The maximum error (ME) obtained by the OSSSOR is $7.91\times {10}^{-5}$. It is convergent faster and more accurate than that obtained in [21], where NI $>8000$ under $\epsilon ={10}^{-4}$ and the error is larger than ${10}^{-3}$.

By using OSSAOR, we take $(A,B)=(0.8,1.5)$, $(a,b)\times (c,d)=(1.92,1.98)\times (1.92,1.98)$, and NI = 117 under $\epsilon ={10}^{-5}$ is shown in Figure 4a by a dashed line, which is convergent faster than the OSSAOR. For the OSSAOR, COC = 1.66, and the CPU time is 1.35 s.

ME obtained by the OSSAOR is $4.03\times {10}^{-5}$. In Figure 5a–c, s, w and $\sigma$ tend to constant values after the first few steps.

6.3. Example 3

We test the following nonlinear Poisson equation by OSSSOR and OSSAOR:

$$\begin{array}{ccc}\hfill & & \Delta u(x,y)+p(x,y)u^3(x,y)=0,0<x<1,0<y<1,\hfill \end{array}$$

(73)

$$\begin{array}{ccc}& & p(x,y)=-4(x^2+y^2+9).\hfill \end{array}$$

(74)

The exact solution of Equation (73) is

$$u(x,y)=\frac{-1}{x^2+y^2-9}.$$

(75)

Let $u_{i,j}^k:=u^k(x_i,y_j)$ be a kth step numerical value where $(x_i,y_j)=(i\Delta x,j\Delta y)$ with $\Delta x=1/(n_1+1)$ and $\Delta y=1/(n_2+1)$. The splitting form of Equation (73) is

$$\Delta u(x,y)+sp(x,y)u^k{(x,y)}^{2}u(x,y)=(s-1)p(x,y)u^k{(x,y)}^3.$$

(76)

Because $u^k(x,y)$ is obtained at the previous step, Equation (76) is a linear Poisson equation to find the value of $u(x,y)$ at the next step.

From Equation (76), the linearized equations are given by

$$\begin{array}{ccc}\hfill & & F_{i,j}=\frac{1}{{(\Delta x)}^2}[u_{i+1,j}-2u_{i,j}+u_{i-1,j}]+\frac{1}{{(\Delta y)}^2}[u_{i,j+1}-2u_{i,j}+u_{i,j-1}]\hfill \\ & & +sp(x_i,y_j){\left[u_{i,j}^k\right]}^2{u}_{i,j}-(s-1)p(x_i,y_j){\left[u_{i,j}^k\right]}^3=0,1\le i\le n_1,1\le j\le n_2,\hfill \end{array}$$

(77)

where $u_{0,j}=g_1\left(y_j\right)=1/(9-y_j^2)$, $u_{n_1+1,j}=g_3\left(y_j\right)=1/(8-y_j^2)$, $u_{i,0}=g_2\left(x_i\right)=1/(9-x_i^2)$, and $u_{i,n_2+1}=g_4\left(x_i\right)=1/(8-x_i^2)$.

Letting $K=n_2(i-1)+j$, $x_K$ in Equation (23) and $F_K$ are computed by

$$\begin{array}{c}\mathrm{D}\mathrm{o}i=1,n_1,\hfill \\ \mathrm{D}\mathrm{o}j=1,n_2,\hfill \\ K=n_2(i-1)+j,\hfill \\ x_K=u_{i,j},\hfill \\ F_K=F_{i,j},\hfill \end{array}$$

(78)

while $\mathbf{b}$ and $\mathbf{A}$ are constructed by

$$\begin{array}{c}\mathrm{D}\mathrm{o}i=1,n_1,\hfill \\ \mathrm{D}\mathrm{o}j=1,n_2,\hfill \\ K=n_2(i-1)+j,\hfill \\ b_K=(s-1)p(x_i,y_j){\left[u_{i,j}^k\right]}^3,\hfill \\ L_1=n_2(i-2)+j,L_2=n_2(i-1)+j-1,L_3=n_2(i-1)+j,\hfill \\ L_4=n_2(i-1)+j+1,L_5=n_{2}i+j,\hfill \\ A_{K,L_1}=A_{K,L_5}=\frac{1}{{(\Delta x)}^2},A_{K,L_2}=A_{K,L_4}=\frac{1}{{(\Delta y)}^2},\hfill \\ A_{K,L_3}=-\frac{2}{{(\Delta x)}^2}-\frac{2}{{(\Delta y)}^2}sp(x_i,y_j){\left[u_{i,j}^k\right]}^2.\hfill \end{array}$$

(79)

Here, we have $n=n_1\times n_2$ equations. For the point on the boundary, we take $b_k$ by the given boundary conditions.

By using OSSSOR, we take $n=n_1\times n_2=20\times 20$, $(a,b)\times (c,d)=(1,3)\times (1.5,2)$, and it converges with NI = 79 under $\epsilon ={10}^{-5}$ as shown in Figure 6a by a solid line. Figure 6b,c show s and w. For the OSSSOR, COC = 1.59, and the CPU time is 13.89 s.

ME obtained by the OSSSOR is $8.73\times {10}^{-6}$. It is more accurate than that obtained by Liu [14], whose ME is $5.36\times {10}^{-3}$. It is much better than that in [1], which spent over 1000 steps and with ME = $8.57\times {10}^{-3}$.

By using OSSAOR, we take $(A,B)=(0.8,1.2)$, $(a,b)\times (c,d)=(1.7,1.8)\times (1.7,1.8)$, and it converges with NI = 51 under $\epsilon ={10}^{-5}$ as shown in Figure 6a by a dashed line. ME obtained by the OSSAOR is $6.62\times {10}^{-6}$. Both the convergence speed and accuracy are better than that obtained by the OSSSOR. Figure 7a–c plot s, w and $\sigma$. For the OSSAOR, COC = 1.54, and the CPU time is 16.99 s.

6.4. Example 4

An almost linear equations system was given by Brown [22]:

$$\begin{array}{ccc}\hfill & & F_i=x_i+\sum _{j=1}^n{x}_j-(n+1),i=1,\dots ,n-1,\hfill \end{array}$$

(80)

$$\begin{array}{ccc}\hfill & & F_n=\prod _{j=1}^n{x}_j-1,\hfill \end{array}$$

(81)

which has a closed-form solution of $x_i=1,i=1,\dots ,n$.

As demonstarted by Han and Han [23], Brown [22] solved this problem with $n=5$ by the Newton iterative algorithm and gave an incorrectly converged solution $(-0.579,-0.579$, $-0.579,-0.579,8.90)$. For $n=10$ and 30, Brown [22] found that the Newton iterative algorithm diverges.

By using OSSSOR, we take $n=10$, $x_i^0=1.5,i=1,\dots ,10$, $(a,b)\times (c,d)=(1.3,1.5)\times (0.5,1)$, and NI = 223 under $\epsilon ={10}^{-10}$ is shown in Figure 8a by a solid line. Figure 8b,c plotted s and w. ME = $9.27\times {10}^{-10}$ is obtained by the OSSSOR. For the OSSSOR, COC = 1.8 and the CPU time is 1.32 s.

We fix $\epsilon ={10}^{-5}$ and consider different n in the comparison between the Newton method and OSSSOR in

Table 1. For Brown’s problem with different dimension n, comparing the performance of Newton method and OSSSOR.

n	4	5	6	7	10	15
NI (Newton)	12	15	15	8	>1000	>1000
NI (OSSSOR)	28	41	55	71	134	199
ME (Newton)	$8.15\times {10}^{-9}$	7.9	$5.42\times {10}^{-6}$	11.59	10	15
ME (OSSSOR)	$1.32\times {10}^{-5}$	$1.2\times {10}^{-5}$	$3.66\times {10}^{-5}$	$6.84\times {10}^{-5}$	$2.92\times {10}^{-5}$	$1.14\times {10}^{-4}$

, where the number of iterations (NI) and ME are compared. Table 1 shows that the OSSSOR outperforms the Newton method when $n\ge 10$.

Comparing the OSSAOR to OSSSOR, we take $n=10$, which is convergent with NI = 145 under $\epsilon ={10}^{-10}$ as shown in Figure 8a by a dashed line. ME = $3.65\times {10}^{-10}$ is obtained by the OSSAOR. Both the convergence speed and accuracy are slightly better than that obtained by the OSSSOR.

By using OSSAOR, we take $n=15$, $x_i^0=1.5,i=1,\dots ,15$, $(A,B)=(1.25,1.7)$, $(a,b)\times (c,d)=(0.2,0.5)\times (0.2,0.5)$; it converges with NI = 91 under $\epsilon ={10}^{-5}$ and obtains ME = $5.64\times {10}^{-5}$. The OSSAOR is convergent faster and more accurately than the OSSSOR shown in Table 1. Figure 9a–c plot s, w and $\sigma$. For the OSSAOR, COC = 1.68, and the CPU time is 0.58 s.

6.5. Example 5

We test the optimal splitting technique with the Cholesky decomposition [17]:

$$\mathbf{L}{\mathbf{L}}^T=\mathbf{H},$$

(82)

where $\mathbf{H}$ is a given $m\times m$ positive-definite matrix, and $\mathbf{L}$ is a lower triangular matrix with $n=(m+1)m/2$ unknown values. We split the nonlinear matrix equation to

$$s{\mathbf{L}}_k{\mathbf{L}}^T=\mathbf{H}+(s-1){\mathbf{L}}_k{\mathbf{L}}_k^T.$$

(83)

Starting from an initial guess ${\mathbf{L}}_0$, we apply the iteration

$${\mathbf{L}}_{k+1}^T={\left(s{\mathbf{L}}_k\right)}^{-1}[\mathbf{H}+(s-1){\mathbf{L}}_k{\mathbf{L}}_k^T],$$

(84)

and the optimal splitting technique to determine $s_k$ is given by

$$\underset{s_k\in (a,b)}{min}\left\{f=\frac{\parallel s_k{\mathbf{L}}_k{\mathbf{L}}_k^T{\parallel }^2{\parallel \mathbf{H}+(s_k-1){\mathbf{L}}_k{\mathbf{L}}_k^T\parallel }^2}{{[\left(s_k{\mathbf{L}}_k{\mathbf{L}}_k^T\right)·(\mathbf{H}+(s_k-1){\mathbf{L}}_k{\mathbf{L}}_k^T)]}^2}\right\}.$$

(85)

We take the Hilbert matrix with $h_{ij}=1/(i+j-1)$ as a demonstrative case. By using the optimal splitting technique, we take $m=4$, $n=10$, $(a,b)=(1.45,1.55)$, and it converges with NI = 39 under $\epsilon ={10}^{-15}$ as shown in Figure 10a. The residual error is very small with $\parallel \mathbf{L}{\mathbf{L}}^T-\mathbf{H}\parallel =9.03\times {10}^{-16}$. $\mathbf{L}$ obtained is almost exact. Figure 10b,c show f and w; f tends to 1 very fast.

In

Table 2. For the Cholesky decomposition equation of the Hilbert matrices with different dimension m, comparing NI, ME and RES.

m	3	5	10	15	20	25
NI	13	17	25	32	38	42
ME	$3.59\times {10}^{-6}$	$3.90\times {10}^{-6}$	$3.07\times {10}^{-6}$	$2.09\times {10}^{-6}$	$1.48\times {10}^{-6}$	$1.69\times {10}^{-6}$
RES	$5.45\times {10}^{-6}$	$5.79\times {10}^{-6}$	$8.19\times {10}^{-6}$	$6.77\times {10}^{-6}$	$7.78\times {10}^{-6}$	$7.97\times {10}^{-6}$

, we compare NI, ME and the residual (RES) for different dimension m of the Hilbert matrix, where we fix $\epsilon ={10}^{-5}$. The data in Table 2 show that the optimal splitting technique can perform well. For $m=25$, COC = 2.35 and the CPU time is 0.42 s.

6.6. Example 6

Finally, we test the LU decomposition [17]:

$$\mathbf{L}\mathbf{U}=\mathbf{G},$$

(86)

where $\mathbf{G}=\left[g_{ij}\right]$ is a given $m\times m$ matrix, and $\mathbf{L}$ is a lower triangular matrix with unit diagonal elements, and $\mathbf{U}$ is an upper triangular matrix. Here, we have $n=m^2$ unknown values. We apply the splitting–linearizing technique to the pair of iterations:

$$\begin{array}{ccc}\hfill & & {\mathbf{U}}_{k+1}={\left(s{\mathbf{L}}_k\right)}^{-1}[\mathbf{G}+(s-1){\mathbf{L}}_k{\mathbf{U}}_k],\hfill \end{array}$$

(87)

$$\begin{array}{ccc}\hfill & & {\mathbf{L}}_{k+1}=[\mathbf{G}+(s-1){\mathbf{L}}_k{\mathbf{U}}_k]{\left(s{\mathbf{U}}_k\right)}^{-1}.\hfill \end{array}$$

(88)

We take $g_{ij}=i+2j$ as a demonstrative case. By using the optimal splitting technique, we take $m=6$, $n=36$, $(a,b)=(1.6,2.2)$, and NI = 69 under $\epsilon ={10}^{-12}$ is shown in Figure 11a. The residual error is quite small with $\parallel \mathbf{L}\mathbf{U}-\mathbf{G}\parallel =8.04\times {10}^{-13}$. $\mathbf{L}$ and $\mathbf{U}$ obtained are almost exact. Figure 11b,c show f and w; f tends to 1 very fast. For the optimal splitting technique, COC = 4.47 and the CPU time is 0.34 s.

7. Conclusions

In the paper, the nonlinear terms in the nonlinear equations were decomposed at two sides through a splitting parameter. Then we linearized the nonlinear equations around the values at the previous step and derived a step-by-step linear system to determine the values at the next step. In doing so, we do not need to compute the Jacobian matrix and its inversion at each iteration step. We have combined the splitting technique to the classical methods SSOR and SAOR, whose parameters are optimized by using the maximal projection methods. In summary, the key outcomes of the proposed optimal splitting SSOR (OSSSOR) and optimal splitting SAOR (OSSAOR) are pointed out as follows.

• There are two parameters in the OSSSOR, while that in the OSSAOR, there are three.
• Using the maximal projection technique, we derived optimal values of the parameters in the OSSSOR and OSSAOR to accelerate the convergence speed.
• Searching the minimization in a preferred range is easily performed through a few operations in the golden section search algorithms.
• The new methods could provide a good choice of the optimal values of the parameters at each iteration.
• Numerical tests indicated that the OSSAOR is convergent faster than the OSSSOR; however, the OSSAOR is more expensive than the OSSSOR to compute three parameters.
• The proposed OSSSOR and OSSAOR can provide very accurate solutions through a few iterations, as reflected in the very small value of the residual, and the high values of COC.

It is significant that the SSOR and SAOR are for the first time automatically merged into the new methods of OSSSOR and OSSAOR without needing the inner iterations loop; they are low in computational complexity and save a lot of CPU time. The main difficulties in the SSOR and SAOR are some of the involved parameters; we overcame this difficulty in the paper by choosing the proper values of parameters, speeding up the convergence. In the future, the same idea may be extended to the nonlinear matrix equations, like as the Riccati matrix equation, and many other nonlinear matrix equations.