Two Extrapolation Techniques on Splitting Iterative Schemes to Accelerate the Convergence Speed for Solving Linear Systems

Abstract

For the splitting iterative scheme to solve the system of linear equations, an equivalent form in terms of descent and residual vectors is formulated. We propose an extrapolation technique using the new formulation, such that a new splitting iterative scheme (NSIS) can be simply generated from the original one by inserting an acceleration parameter preceding the descent vector. The spectral radius of the NSIS is proven to be smaller than the original one, and so has a faster convergence speed. The orthogonality of consecutive residual vectors is coined into the second NSIS, from which a stepwise varying orthogonalization factor can be derived explicitly. Multiplying the descent vector by the factor, the second NSIS is proven to be absolutely convergent. The modification is based on the maximal reduction of residual vector norm. Two-parameter and three-parameter NSIS are investigated, wherein the optimal value of one parameter is obtained by using the maximization technique. The splitting iterative schemes are unified to have the same iterative form, but endowed with different governing equations for the descent vector. Some examples are examined to exhibit the performance of the proposed extrapolation techniques used in the NSIS.

1. Introduction

In the paper, we improve some splitting iteration methods for solving a system of linear equations:

$$\mathbf{A}\mathbf{x}=\mathbf{b},\mathbf{b},\mathbf{x}\in {\mathbb{R}}^n,\mathbf{A}\in {\mathbb{R}}^{n\times n}.$$

(1)

For solving Equation (1) a lot of well-developed methods were discussed in the text books [1,2,3,4,5]. In general the Krylov subspace method is employed for solving the large scale linear problem. An m-dimensional Krylov subspace is considered:

$${\mathcal{K}}_m(\mathbf{A},{\mathbf{r}}_0):=\mathrm{Span}\{{\mathbf{r}}_0,\mathbf{A}{\mathbf{r}}_0,\dots ,{\mathbf{A}}^{m-1}{\mathbf{r}}_0\}.$$

(2)

In the generalized minimal residual (GMRES) method [6], $\mathbf{r}={\mathbf{r}}_0-\mathbf{A}\mathbf{u}$ is perpendicular to ${\mathcal{L}}_m=\mathbf{A}{\mathcal{K}}_m$ defined by Equation (2), i.e.,

$$\mathbf{r}\perp {\mathcal{L}}_m.$$

(3)

The best descent vector $\mathbf{u}\in {\mathcal{K}}_m$ is sought to minimize the residual [3]:

$$min\parallel \mathbf{r}\parallel ,$$

where

$$\mathbf{r}=\mathbf{b}-\mathbf{A}\mathbf{x}={\mathbf{r}}_0-\mathbf{A}\mathbf{u}$$

is the residual vector, whose Euclidean norm is denoted by $\parallel \mathbf{r}\parallel$. Equation (3) is one of the Petrov-Galerkin method to search $\mathbf{u}\in {\mathcal{K}}_m$. The perpendicularity is a vital guidance for developing effective iterative algorithms, which was employed in [7,8] to enhance the stability of GMRES and SOR.

Dongarra and Sullivan [9] introduced the Krylov subspace method as the top ten algorithms. Bai [10] demonstrated the motivations and realizations of Krylov subspace methods for large sparse linear systems. Recently, Bouyghf et al. [11] provided a unified approach to the Krylov subspace methods for solving linear systems.

Suppose that $\mathbf{A}$ is decomposed by

$$\mathbf{A}=\mathbf{M}-\mathbf{N},$$

(4)

where $\mathbf{M}$ is a nonsingular matrix. The $(\mathbf{M},\mathbf{N})$ splitting iterative scheme for Equation (1) is [12]

$$\mathbf{M}{\mathbf{x}}_{k+1}=\mathbf{N}{\mathbf{x}}_k+\mathbf{b},$$

(5)

where ${\mathbf{x}}_k$ is the kth step value of $\mathbf{x}$. The convergence of the iteration in Equation (5) is guaranteed if

$$\rho =\rho \left(\mathbf{G}\right)<1,$$

(6)

where $\mathbf{G}={\mathbf{M}}^{-1}\mathbf{N}$ is the iteration matrix, and $\rho$ is the spectral radius of $\mathbf{G}$.

The Gaussian elimination method is a classical exact method, but it is not suitable for use in either scientific or engineering practice. The Gauss–Seidel method is a semi-exact method by finding the exact values of variables at each iteration step by a forward-substitution technique. Without needing extra parameter is its main advantage; however, it is convergent slowly.

Many methods are special cases of the $(\mathbf{M},\mathbf{N})$ splitting iterative scheme, namely the Jacobi method, the Gauss–Seidel method, the successive overrelaxation (SOR) method, and the accelerated overrelaxation (AOR) method. The SOR method was developed in [13]. For multi-parameter splitting methods, the two-parameter AOR method [14]:

$$(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=r\mathbf{b}+[(1-r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k,$$

(7)

is a generalization of SOR, which is equipped with a relaxation parameter w and an acceleration parameter r for controlling the convergence behavior.

The $(\mathbf{M},\mathbf{N})$ splitting for Equation (7) is

$$\mathbf{A}=\mathbf{M}-\mathbf{N}=\mathbf{D}-\mathbf{U}-\mathbf{L},$$

(8)

where

$$\mathbf{M}={\displaystyle \frac{1}{r}}(\mathbf{D}-w\mathbf{L}),\mathbf{N}={\displaystyle \frac{1}{r}}[(1-r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}].$$

If $r=w$, the SOR iterative method

$$(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=w\mathbf{b}+(1-w)\mathbf{D}{\mathbf{x}}_k+w\mathbf{U}{\mathbf{x}}_k$$

(9)

is recovered from Equation (7).

The generalizations of AOR method for different linear problems can refer to [15,16,17]; the convergence behaviors were analyzed in [18,19]. To improve the convergence speed the preconditioned AOR methods were developed. Different kinds of preconditioners have been examined to improve the speed of convergence of the iterative methods. A growing interest is to study the preconditioners to ameliorate the speed of convergence of iterative methods in a lot of literature [20,21,22,23,24,25,26,27].

Liu and his co-workers [28,29] reformulated SOR and AOR in terms of descent and residual vectors. The reaccelerated version of AOR was discussed in [30], and the generalization to the three-parameter method of AOR was carried out in [31].

Given an initial guess ${\mathbf{x}}_0$ for an iterative scheme, it follows from Equation (1) a residual vector ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$. Knowing ${\mathbf{r}}_0$, one attempts to search a descent vector $\mathbf{u}$, which can correct the solution to $\mathbf{x}={\mathbf{x}}_0+\mathbf{u}$ to abide the rule of $\parallel \mathbf{r}\parallel <\parallel {\mathbf{r}}_0\parallel$. The residual and descent vectors are two very fundamental concepts in the formulation of the iterative schemes. We will point out that the purpose of extrapolation techniques [32,33,34,35] is designed a feasible manner to search a better descent vector basically, such that the residuals can be reduced quickly.

In the paper we will unify the splitting iteration methods, and simplify the extrapolation techniques from the formulation in terms of descent and residual vectors. The new ideas in this formulation to solve the linear system are developed by improving the existent splitting iteration methods, of which the maximization technique and the spectral analysis of iteration matrix are concerned with. Absolute convergence of the new splitting iterative scheme (NSIS) is proven.

2. A Reduction of Spectral Radius

Through the descent vector and the kth step residual vector

$$\mathbf{u}=\mathbf{x}-{\mathbf{x}}_k,$$

(10)

$${\mathbf{r}}_k=\mathbf{b}-\mathbf{A}{\mathbf{x}}_k,$$

(11)

Equation (1) can be written as

$$\mathbf{A}\mathbf{u}={\mathbf{r}}_k.$$

(12)

Equation (12) shows that the best descent vector is $\mathbf{u}={\mathbf{A}}^{-1}{\mathbf{r}}_k$. However, if we can find ${\mathbf{A}}^{-1}$, the solution of Equation (1) is already given by $\mathbf{x}={\mathbf{A}}^{-1}\mathbf{b}$ exactly. The main difficulty is that finding ${\mathbf{A}}^{-1}$ is absolutely not an easy task.

Lemma 1. 

The $(\mathbf{M},\mathbf{N})$ splitting iterative scheme (5) for solving Equation (1) can be reformulated as

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,\mathbf{M}{\mathbf{u}}_k={\mathbf{r}}_k.$$

(13)

Proof. 

By means of Equation (5), we have

$$\mathbf{M}{\mathbf{x}}_{k+1}=\mathbf{M}{\mathbf{x}}_k+(\mathbf{N}-\mathbf{M}){\mathbf{x}}_k+\mathbf{b};$$

it becomes, with the aids of Equations (4) and (11),

$$\mathbf{M}({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)={\mathbf{r}}_k.$$

We end the proof of Lemma 1 by setting ${\mathbf{u}}_k={\mathbf{x}}_{k+1}-{\mathbf{x}}_k$ in view of Equation (10).  □

We modify Equation (13) and prove the following result.

Theorem 1. 

Inserting an acceleration parameter α preceding ${\mathbf{u}}_k$ in Equation (13), a new splitting iterative scheme for Equation (1) reads as

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+\alpha {\mathbf{u}}_k,\mathbf{M}{\mathbf{u}}_k={\mathbf{r}}_k,$$

(14)

Under the following condition:

$$1<\alpha <{\displaystyle \frac{1}{1-\rho \left(\mathbf{G}\right)}},$$

(15)

where $\mathbf{G}={\mathbf{M}}^{-1}\mathbf{N}$ is the iteration matrix for Equation (5), the spectral radius for Equation (14), denoted by ${\rho }_{\alpha }$, satisfies.

$${\rho }_{\alpha }<\rho \left(\mathbf{G}\right).$$

(16)

Proof. 

When the first one in Equation (14) is multiplied by $\mathbf{M}$, and Equations (4) and (11), and the second one in Equation (14) are taken into account, it generates

$$\begin{array}{ccc}& & \mathbf{M}{\mathbf{x}}_{k+1}=\mathbf{M}{\mathbf{x}}_k+\alpha \mathbf{M}{\mathbf{u}}_k=\mathbf{M}{\mathbf{x}}_k+\alpha {\mathbf{r}}_k\hfill \\ & & =\mathbf{M}{\mathbf{x}}_k+\alpha (\mathbf{b}-\mathbf{A}{\mathbf{x}}_k)=\mathbf{M}{\mathbf{x}}_k+\alpha \mathbf{b}-\alpha (\mathbf{M}-\mathbf{N}){\mathbf{x}}_k\hfill \\ & & =(1-\alpha )\mathbf{M}{\mathbf{x}}_k+\alpha \mathbf{N}{\mathbf{x}}_k+\alpha \mathbf{b}.\hfill \end{array}$$

The associated iteration matrix is

$${\mathbf{G}}_{\alpha }=(1-\alpha ){\mathbf{I}}_n+\alpha \mathbf{G}.$$

(17)

It follows that

$${\rho }_{\alpha }=1-\alpha +\alpha \rho \left(\mathbf{G}\right).$$

(18)

If $\alpha >1$ and $0<\rho \left(\mathbf{G}\right)<1$ owing to Equation (6), then

$$(\alpha -1)\rho \left(\mathbf{G}\right)<\alpha -1$$

implies

$${\rho }_{\alpha }=1-\alpha +\alpha \rho \left(\mathbf{G}\right)<\rho \left(\mathbf{G}\right).$$

To meet the requirement of

$${\rho }_{\alpha }=1-\alpha +\alpha \rho \left(\mathbf{G}\right)>0,$$

$\alpha <1/[1-\rho (\mathbf{G}\left)\right]$ follows readily.  □

Equation (14) can be also derived from Equation (5) after multiplying $\alpha$ and using a substitution technique:

$$\alpha \mathbf{M}{\mathbf{x}}_{k+1}=\alpha \mathbf{N}{\mathbf{x}}_k+\alpha \mathbf{b},$$

(19)

$${\mathbf{x}}_{k+1}\leftarrow {\displaystyle \frac{1}{\alpha }}{\mathbf{x}}_{k+1}+\left(1-{\displaystyle \frac{1}{\alpha }}\right){\mathbf{x}}_k.$$

(20)

It results in

$$\begin{array}{c}\mathbf{M}{\mathbf{x}}_{k+1}+(\alpha -1)\mathbf{M}{\mathbf{x}}_k=\alpha \mathbf{N}{\mathbf{x}}_k+\alpha \mathbf{b},\hfill \\ \mathbf{M}{\mathbf{x}}_{k+1}-\mathbf{M}{\mathbf{x}}_k=\alpha (\mathbf{N}-\mathbf{M}){\mathbf{x}}_k+\alpha \mathbf{b},\hfill \\ \mathbf{M}{\mathbf{x}}_{k+1}-\mathbf{M}{\mathbf{x}}_k=\alpha (\mathbf{b}-\mathbf{A}{\mathbf{x}}_k),\hfill \\ \mathbf{M}({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)=\alpha {\mathbf{r}}_k,\hfill \\ {\mathbf{x}}_{k+1}-{\mathbf{x}}_k=\alpha {\mathbf{M}}^{-1}{\mathbf{r}}_k=\alpha {\mathbf{u}}_k.\hfill \end{array}$$

(21)

This is the usual extrapolation method used in the iterative scheme with $\alpha$ as an extrapolation parameter [33].

The extrapolation technique consists of a multiplication in Equation (19), and an extrapolation in Equation (20), which will be named an extrapolation technique for the $(\mathbf{M},\mathbf{N})$ splitting iterative scheme (5) with the parameter $\alpha$, or simply, an $\alpha$-extrapolation.

We now realize the extrapolation technique for the SOR method in Equation (9) as follows:

$$\begin{array}{c}\alpha (\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=w\alpha \mathbf{b}+(1-r)\alpha \mathbf{D}{\mathbf{x}}_k+w\alpha \mathbf{U}{\mathbf{x}}_k,\hfill \\ \alpha (\mathbf{D}-w\mathbf{L})\left[{\displaystyle \frac{1}{\alpha }}{\mathbf{x}}_{k+1}+\left(1-{\displaystyle \frac{1}{\alpha }}\right){\mathbf{x}}_k\right]=w\alpha \mathbf{b}+(1-r)\alpha \mathbf{D}{\mathbf{x}}_k+w\alpha \mathbf{U}{\mathbf{x}}_k,\hfill \\ (\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=w\alpha \mathbf{b}+(1-r)\alpha \mathbf{D}{\mathbf{x}}_k+w\alpha \mathbf{U}{\mathbf{x}}_k+(1-\alpha )(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_k,\hfill \\ (\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=w\alpha \mathbf{b}+(1-r\alpha )\mathbf{D}{\mathbf{x}}_k+w\alpha \mathbf{U}{\mathbf{x}}_k+(w\alpha -w)\mathbf{L}{\mathbf{x}}_k.\hfill \end{array}$$

(22)

Setting $w\alpha =r$, we can derive Equation (7). Therefore, the AOR in Equation (7) is an $\alpha$-extrapolation of the SOR in Equation (9).

Suppose that the spectral radius of SOR is ${\rho }_{\mathrm{SOR}}$. According to Equation (15), when $(r,w)$ in the AOR method are taken as

$$1<{\displaystyle \frac{r}{w}}<{\displaystyle \frac{1}{1-{\rho }_{\mathrm{SOR}}}},$$

(23)

AOR is convergent faster than SOR.

Remark 1. 

Comparing to Equation (13), α only appears in the first one before ${\mathbf{u}}_k$ in Equation (14), and the second ones are unchanged. The α-extrapolation in Equation (14) within the formulation in terms of descent and residual vectors is simpler than the extrapolation technique applied to the original iterative equation, like that in Equation (21) for the general $(\mathbf{M},\mathbf{N})$ method, and that in Equation (22) for the method of SOR to obtain the AOR method. AOR is the $\alpha =r/w$-extrapolation of SOR.

By absorbing α into ${\mathbf{u}}_k$ and renaming it to ${\mathbf{u}}_k$ again, Equation (14) can also be written as

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,\mathbf{M}{\mathbf{u}}_k=\alpha {\mathbf{r}}_k.$$

(24)

Comparing to Equation (13), it has the same iterative form, but the governing equations of ${\mathbf{u}}_k$ are different; ${\mathbf{u}}_k$ in Equation (24) is enhanced by an acceleration factor α. Both Equations (14) and (24) are the α-extrapolation of Equation (13); they are equivalent.

3. On a Modification of the Splitting Iterative Scheme and Determining η_k

Inspired by Equation (3), the iterative scheme in Equation (13) is orthogonal if its descent and residual vectors satisfy

$$\mathbf{A}{\mathbf{u}}_k·{\mathbf{r}}_{k+1}=0.$$

Usually this is not true. For the original splitting iterative scheme in Equation (13), we have

$${\mathbf{r}}_{k+1}={\mathbf{r}}_k-\mathbf{A}{\mathbf{u}}_k,$$

(25)

which results to

$${\mathbf{r}}_{k+1}^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k={\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k-{\parallel \mathbf{A}{\mathbf{u}}_k\parallel }^2\ne 0.$$

(26)

We define a quantity ${\eta }_k$ to measure the degree of the non-orthogonality of the iterative scheme:

$${\eta }_k={\displaystyle \frac{{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k}{\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2}}.$$

(27)

Now the iterative scheme in Equation (13) is orthogonal if and only if ${\eta }_k=1$.

Rather than the constant value of $\alpha$ in Equation (14), in this section we derive a more powerful version of the splitting iterative scheme (5) with a stepwise varying factor ${\eta }_k$ in Equation (27).

Theorem 2. 

If a factor ${\eta }_k$ is inserted preceding ${\mathbf{u}}_k$ in the first one of Equation (13), then a new splitting iteration method for solving Equation (1) reads as

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\eta }_k{\mathbf{u}}_k,\mathbf{M}{\mathbf{u}}_k={\mathbf{r}}_k.$$

(28)

The optimal value for the orthogonalization factor ${\eta }_k$ at each step is given by Equation (27).

Proof. 

Multiplying the first one in Equation (28) by $\mathbf{A}$ and using Equation (11) yields

$${\mathbf{r}}_{k+1}={\mathbf{r}}_k-{\eta }_k\mathbf{A}{\mathbf{u}}_k;$$

the squared norm is

$$\parallel {\mathbf{r}}_{k+1}{\parallel }^2=\parallel {\mathbf{r}}_k{\parallel }^2-2{\eta }_k{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k+{\eta }_k^2{\parallel \mathbf{A}{\mathbf{u}}_k\parallel }^2.$$

Let

$$g=2{\eta }_k{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k-{\eta }_k^2{\parallel \mathbf{A}{\mathbf{u}}_k\parallel }^2;$$

(29)

we have

$$\parallel {\mathbf{r}}_{k+1}{\parallel }^2={\parallel {\mathbf{r}}_k\parallel }^2-g.$$

(30)

The following maximization problem is established to maximally decrease the residual vector norm at each step by

$$\underset{{\eta }_k}{max}\{g=2{\eta }_k{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k-{\eta }_k^2\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2\}.$$

(31)

Using $dg/d{\eta }_k=0$ for the maximality of g in Equation (31), we can derive Equation (27).  □

Remark 2. 

In contrast to the α-extrapolation in Equation (14), Equation (28) is an ${\eta }_k$-extrapolation of Equation (13). When α is a constant in Equation (14), ${\eta }_k$ is a varying factor per step.

Theorem 3. 

The splitting iterative scheme in Equation (28) is absolute convergence, i.e.,

$$\parallel {\mathbf{r}}_{k+1}\parallel <\parallel {\mathbf{r}}_k\parallel ,\forall k=0,1,\dots .$$

(32)

Proof. 

Inserting Equation (27) for the optimal value of ${\eta }_k$ into Equation (29) yields

$$g={\displaystyle \frac{{\left({\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k\right)}^2}{\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2}}.$$

It follows from Equation (30) that

$${\parallel {\mathbf{r}}_{k+1}\parallel }^2={\parallel {\mathbf{r}}_k\parallel }^2-{\displaystyle \frac{{\left({\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k\right)}^2}{{\parallel \mathbf{A}{\mathbf{u}}_k\parallel }^2}}<{\parallel {\mathbf{r}}_k\parallel }^2,$$

due to ${\left({\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k\right)}^2/{\parallel \mathbf{A}{\mathbf{u}}_k\parallel }^2>0$. Hence, we proved Equation (32).  □

Theorem 4. 

In the splitting iterative scheme in Equations (28) and (27), we have

$${\mathbf{r}}_{k+1}^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k=0,$$

(33)

which means that the consecutive residual vector ${\mathbf{r}}_{k+1}$ is perpendicular to $\mathbf{A}{\mathbf{u}}_k$.

Proof. 

By means of Equations (28) and (27), we have

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\displaystyle \frac{{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k}{\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2}}{\mathbf{u}}_k.$$

(34)

Multiplying Equation (34) by $\mathbf{A}$ and using Equation (11) yields

$${\mathbf{r}}_{k+1}={\mathbf{r}}_k-{\displaystyle \frac{{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k}{\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2}}\mathbf{A}{\mathbf{u}}_k,$$

which being taken the inner product to $\mathbf{A}{\mathbf{u}}_k$ renders

$${\mathbf{r}}_{k+1}^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k={\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k-{\displaystyle \frac{{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k}{\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2}}{\parallel \mathbf{A}{\mathbf{u}}_k\parallel }^2=0.$$

The proof of Equation (33) for Theorem 4 is complete.  □

Equation (26) indicates that the consecutive residual vectors of the original splitting iterative scheme are not perpendicular to $\mathbf{A}{\mathbf{u}}_k$. Hence, the absolute convergence is not guaranteed for the original splitting iterative scheme.

The property of

$${\mathbf{r}}_{k+1}\perp \mathbf{A}{\mathbf{u}}_k$$

is well-known in the literature. The orthogonal property guarantees that the residual vector norm is decreased per step. Refer to [7] for a detailed development of the re-orthogonalized GMRES technique for solving the linear systems.

4. New Form of QAOR and Optimal Value of r

Wu and Liu [17] proposed a quasi accelerated overrelaxation method (QAOR) by taking

$$[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=r\mathbf{b}+[\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k.$$

(35)

Dividing by $1+r$ and renaming the parameters as

$$w_0={\displaystyle \frac{w}{1+r}},r_0={\displaystyle \frac{r}{1+r}},$$

Equation (35) can be recast to

$$(\mathbf{D}-w_0\mathbf{L}){\mathbf{x}}_{k+1}=r_0\mathbf{b}+[(1-r_0)\mathbf{D}+r_0\mathbf{U}+(r_0-w_0)\mathbf{L}]{\mathbf{x}}_k.$$

It is just the AOR method in Equation (7). However, Equation (35) has an advantage that the parameter r is permitted in a large range.

4.1. A New Form of QAOR

The original form of quasi accelerated overrelaxation method (QAOR) in Equation (35) is not easy to handle. For the purpose of comparison we derive a new form of QAOR in terms of descent and residual vectors as follows.

Theorem 5. 

A new iterative form of the QAOR method is

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{u}}_k=r{\mathbf{r}}_k.$$

(36)

Proof. 

We rewrite Equation (35) to

$$\begin{array}{c}[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k+r\mathbf{b}\hfill \\ +[\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k-[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k,\hfill \end{array}$$

and arrange it to

$$[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k+r\mathbf{b}-r[\mathbf{D}-\mathbf{U}-\mathbf{L}]{\mathbf{x}}_k.$$

With the aids of Equations (8) and (11), we have

$$[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k+r{\mathbf{r}}_k,$$

and also

$$[(1+r)\mathbf{D}-w\mathbf{L}]({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)=r{\mathbf{r}}_k.$$

The proof of Equation (36) is complete upon setting ${\mathbf{x}}_{k+1}-{\mathbf{x}}_k={\mathbf{u}}_k$.  □

4.2. Determining r in QAOR

Theorem 6. 

Given an initial value ${\mathbf{x}}_0$, ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$ is known. For the QAOR method in Equation (36) if w is given, then

$$r={\displaystyle \frac{{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0}{\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2}}$$

(37)

is the optimal value of r, where ${\mathbf{v}}_0$ is obtained from a forward solution of

$$[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{v}}_0={\mathbf{r}}_0.$$

(38)

Proof. 

Upon letting

$${\mathbf{v}}_0={\displaystyle \frac{{\mathbf{u}}_0}{r}},$$

Equation (38) is a direct result of the second one in Equation (36) with $k=0$. The first one in Equation (36) with $k=0$ is

$${\mathbf{x}}_1={\mathbf{x}}_0+r{\mathbf{v}}_0.$$

(39)

Multiplying Equation (39) by $\mathbf{A}$ and using Equation (11) yields

$${\mathbf{r}}_1={\mathbf{r}}_0-r\mathbf{A}{\mathbf{v}}_0,$$

the squared norm of which is

$${\parallel {\mathbf{r}}_1\parallel }^2={\parallel {\mathbf{r}}_0\parallel }^2-2r{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0+r^2{\parallel \mathbf{A}{\mathbf{v}}_0\parallel }^2.$$

Let

$$f=2r{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0-r^2{\parallel \mathbf{A}{\mathbf{v}}_0\parallel }^2$$

be a decreasing function of the residual vector norm at the first step; we have

$${\parallel {\mathbf{r}}_1\parallel }^2={\parallel {\mathbf{r}}_0\parallel }^2-f.$$

We encounter a maximization problem to determine r via

$$\underset{r}{max}\{f=2r{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0-r^2\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2\}.$$

Adopting $df/dr=0$, Equation (37) is derived.  □

Equation (37) is a nonlinear equation for r, because ${\mathbf{v}}_0={[(1+r)\mathbf{D}-w\mathbf{L}]}^{-1}{\mathbf{r}}_0$ obtained from Equation (38) is also a function of r. It is a fixed point problem to seek the solution of r, which can be solved by the iteration method to determine the optimal value of r.

4.3. Accelerated QAOR

According to Theorem 1, we can accelerate the convergence speed of QAOR by utilizing the following corollary.

Corollary 1. 

A new iterative form of the accelerated QAOR method is

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+\alpha {\mathbf{u}}_k,[(1+r)\mathbf{D}-w\mathbf{L}]{\mathbf{u}}_k=r{\mathbf{r}}_k,$$

(40)

where $\alpha >1$, and r is determined by Equation (37).

5. Reaccelerated over Relaxation Method

Vatti et al. [30] proposed a reaccelerated over relaxation (ROR) method as follows:

$$(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=r\mathbf{b}+[(1-r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k+rw(\mathbf{A}{\mathbf{x}}_k-\mathbf{b}).$$

(41)

Theorem 7. 

A new iterative form of the ROR method is

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,(\mathbf{D}-w\mathbf{L}){\mathbf{u}}_k=r(1-w){\mathbf{r}}_k.$$

(42)

Proof. 

With the aid of Equation (11), we rewrite Equation (41) to

$$(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_k+r\mathbf{b}+[(1-r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k-(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_k-rw{\mathbf{r}}_k,$$

and then to

$$(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_k+r\mathbf{b}-r[\mathbf{D}-\mathbf{U}-\mathbf{L}]{\mathbf{x}}_k-rw{\mathbf{r}}_k.$$

It follows from Equations (8) and (11) that

$$\begin{array}{c}(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_{k+1}=(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_k+r\mathbf{b}-r\mathbf{A}{\mathbf{x}}_k-rw{\mathbf{r}}_k\hfill \\ =(\mathbf{D}-w\mathbf{L}){\mathbf{x}}_k+r{\mathbf{r}}_k-rw{\mathbf{r}}_k,\hfill \end{array}$$

and that

$$(\mathbf{D}-w\mathbf{L})({\mathbf{x}}_{k+1}-{\mathbf{x}}_k)=r{\mathbf{r}}_k-rw{\mathbf{r}}_k;$$

Upon using ${\mathbf{u}}_k={\mathbf{x}}_{k+1}-{\mathbf{x}}_k$, Equation (42) is proven.  □

Theorem 8. 

The AOR method can be recast to

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,(\mathbf{D}-w\mathbf{L}){\mathbf{u}}_k=r{\mathbf{r}}_k.$$

(43)

The ROR method is obtained from the AOR method upon applying the extrapolation technique (19) and (20) by a parameter $\beta =1-w$ on Equation (43); it generates the following iterative form:

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+(1-w){\mathbf{u}}_k,(\mathbf{D}-w\mathbf{L}){\mathbf{u}}_k=r{\mathbf{r}}_k.$$

(44)

Proof. 

Equation (43) is obtained from Equation (42) by deleting $rw{\mathbf{r}}_k$ in the right-hand side of the second equation. Applying the extrapolation technique (19) and (20) on Equation (43) leads to

$$\begin{array}{c}\beta {\mathbf{x}}_{k+1}=\beta {\mathbf{x}}_k+\beta {\mathbf{u}}_k,\hfill \\ \beta \left[{\displaystyle \frac{1}{\beta }}{\mathbf{x}}_{k+1}+\left(1-{\displaystyle \frac{1}{\beta }}\right){\mathbf{x}}_k\right]=\beta {\mathbf{x}}_k+\beta {\mathbf{u}}_k,\hfill \\ {\mathbf{x}}_{k+1}={\mathbf{x}}_k+\beta {\mathbf{u}}_k,\hfill \end{array}$$

which is the first one in Equation (44), if we take $\beta =1-w$.

In Equation (42), let

$${\mathbf{w}}_k={\displaystyle \frac{{\mathbf{u}}_k}{1-w}};$$

we have

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+(1-w){\mathbf{w}}_k,(\mathbf{D}-w\mathbf{L}){\mathbf{w}}_k=r{\mathbf{r}}_k.$$

(45)

Comparing Equation (45), with ${\mathbf{w}}_k$ replaced by ${\mathbf{u}}_k$, to Equation (44), they are the same.  □

Theorem 9. 

Given an initial value ${\mathbf{x}}_0$, ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$ is known. For the ROR method in Equation (42) if w is given, then

$$r={\displaystyle \frac{{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0}{\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2}}$$

(46)

is the optimal value of r, where ${\mathbf{v}}_0$ is obtained from a forward solution of

$$(\mathbf{D}-w\mathbf{L}){\mathbf{v}}_0=(1-w){\mathbf{r}}_0.$$

(47)

Proof. 

Letting

$${\mathbf{v}}_0={\displaystyle \frac{{\mathbf{u}}_0}{r}},$$

Equation (47) is a direct result of the second one in Equation (42) with $k=0$. The first one in Equation (42) with $k=0$ is

$${\mathbf{x}}_1={\mathbf{x}}_0+r{\mathbf{v}}_0.$$

Other processes are similar to that in Theorem 6.  □

Equation (46) is a simple equation for r, because ${\mathbf{v}}_0=(1-w){(\mathbf{D}-w\mathbf{L})}^{-1}{\mathbf{r}}_0$ is not a function of r. We can compute the optimal value of r by Equation (46) directly.

Remark 3. 

We have mentioned in Remark 1 that AOR is the $\alpha =r/w$-extrapolation of SOR. It is interesting that ROR is the $\beta =1-w$-extrapolation of AOR. Consequently, ROR is the secondary extrapolation of SOR by two parameters $\alpha =r/w$ and $\beta =1-w$. An advantage of ROR is that the optimal value of r can be obtained from Equation (46) directly without any iteration.

6. Three-Parameter Splitting Iterative Scheme

A generalization of Equation (35) was proposed in [31,36] as follows:

$$[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=r\mathbf{b}+[(1+\sigma -r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k,$$

(48)

which is named a parametric-AOR (POR), accompanied by an extra parameter $\sigma$. The QAOR method in Equation (35) is a special case of Equation (48) with $\sigma =r$.

Theorem 10. 

A new iterative form of the POR method is

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{u}}_k=r{\mathbf{r}}_k.$$

(49)

Proof. 

Equation (48) is rewritten as

$$\begin{array}{c}[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k\hfill \\ +r\mathbf{b}+[(1+\sigma -r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k-[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k,\hfill \end{array}$$

which can be rearranged to

$$[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}+r\mathbf{b}-r[\mathbf{D}-\mathbf{U}-\mathbf{L}]{\mathbf{x}}_k.$$

Equations (8) and (11) lead to

$$[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k+r{\mathbf{r}}_k.$$

The proof for Equation (49) is complete, upon moving $[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_k$ to the left-hand side, and setting ${\mathbf{x}}_{k+1}-{\mathbf{x}}_k={\mathbf{u}}_k$.  □

Isah et al. [31] further generalized Equation (48) to

$$[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{x}}_{k+1}=r\mathbf{b}+[(1+\sigma -r)\mathbf{D}+r\mathbf{U}+(r-w)\mathbf{L}]{\mathbf{x}}_k-rw{\mathbf{r}}_k,$$

(50)

which is named a parametric reaccelerated overrelaxation (PROR) method. Like that in Theorem 9, we can prove the following result.

Theorem 11. 

The PROR method is obtained from the POR method by applying the extrapolation technique (19) and (20) by a parameter $\beta =1-w$; it generates the following iterative form:

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+(1-w){\mathbf{u}}_k,[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{u}}_k=r{\mathbf{r}}_k.$$

(51)

Proof. 

As that done in Theorem 10, Equation (50) can be expressed as

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{w}}_k,[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{w}}_k=r(1-w){\mathbf{r}}_k.$$

(52)

If we take

$${\mathbf{u}}_k={\displaystyle \frac{{\mathbf{w}}_k}{1-w}},$$

Equation (52) changes to Equation (51).

By applying the extrapolation technique (19) and (20) on Equation (49) by a parameter $\beta =1-w$, Equation (51) can be derived again.  □

When ${\mathbf{w}}_k$ in Equation (52) is replaced by ${\mathbf{u}}_k$, the PROR method can also be written as

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{u}}_k=r(1-w){\mathbf{r}}_k.$$

Remark 4. 

QAOR is a special case of POR with $\sigma =w$. The PROR method is the $\beta =1-w$-extrapolation of POR.

7. Algorithms

The iterative schemes can be adjust to have the same iterative form:

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k+{\mathbf{u}}_k,$$

but the governing equations for ${\mathbf{u}}_k$ are different. In Table 1, we summarize the iterative schemes discussed in this paper.

Table 1. Comparing the governing equations for ${\mathbf{u}}_k$ for different iterative schemes; $\mathbf{P}:=\mathbf{D}-w\mathbf{L}$ and $\mathbf{Q}:=(1+\sigma )\mathbf{D}-w\mathbf{L}$; QAOR is a special case of POR with $\sigma =r$.

SOR	AOR	ROR	POR	PROR
$\mathbf{P}{\mathbf{u}}_k=w{\mathbf{r}}_k$	$\mathbf{P}{\mathbf{u}}_k=r{\mathbf{r}}_k$	$\mathbf{P}{\mathbf{u}}_k=r(1-w){\mathbf{r}}_k$	$\mathbf{Q}{\mathbf{u}}_k=r{\mathbf{r}}_k$	$\mathbf{Q}{\mathbf{u}}_k=r(1-w){\mathbf{r}}_k$

According to the different methods of the splitting iterative scheme we have the following iterative algorithms.

Algorithms 1 and 2 can be applied to the above iterative schemes in Table 1. For saving notations we abbreviate them as A1-SOR and A2-SOR, and so on; they are Algorithms 1 and 2 for SOR, and so on. In Algorithm 2, $\alpha$ is added to further accelerate the convergence speed. which needs to satisfy the constraint (15). For most cases we take $\alpha =1$, otherwise specified.

We can reduce the number of unknown parameters by using the maximization techniques, like as Theorem 6 for the QAOR method with the optimal value of r.

Algorithm 1: $\alpha$-acceleration

1: Give $\mathbf{A}$, $\mathbf{M}$, ${\mathbf{x}}_0$, $\alpha >1$, and $\epsilon$ 2: Do $k=0,1,\dots$, until $\parallel {\mathbf{r}}_k\parallel <\epsilon$ 3: ${\mathbf{r}}_k=\mathbf{b}-\mathbf{A}{\mathbf{x}}_k$ 4: Solve $\mathbf{M}{\mathbf{u}}_k={\mathbf{r}}_k$ 5: ${\mathbf{x}}_{k+1}={\mathbf{x}}_k+\alpha {\mathbf{u}}_k$

Algorithm 2: ${\eta }_k$-acceleration

1: Give $\mathbf{A}$, $\mathbf{M}$, ${\mathbf{x}}_0$, $\alpha$, and $\epsilon$ 2: Do $k=0,1,\dots$, until $\parallel {\mathbf{r}}_k\parallel <\epsilon$ 3: ${\mathbf{r}}_k=\mathbf{b}-\mathbf{A}{\mathbf{x}}_{c}k$ 4: Solve $\mathbf{M}{\mathbf{u}}_k={\mathbf{r}}_k$ 5: ${\eta }_k={\displaystyle \frac{{\mathbf{r}}_k^{\mathrm{T}}\mathbf{A}{\mathbf{u}}_k}{\parallel \mathbf{A}{\mathbf{u}}_k{\parallel }^2}}$ 6: ${\mathbf{x}}_{k+1}={\mathbf{x}}_k+\alpha {\eta }_k{\mathbf{u}}_k$

In Algorithm 3 for w of SOR, and Algorithm 4 for r of QAOR, we need some iterations with a loose convergence criterion say ${\epsilon }_1={10}^{-2}$; in Algorithm 5 for r of AOR and POR, and Algorithm 6 for r of ROR and PROR, they do not need any iteration.

Algorithm 3: for w of SOR

1: Give ${\mathbf{x}}_0$, $w^{\left(0\right)}$ and ${\epsilon }_1$ 2: ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$ 3: Do $k=0,1,\dots$ 4: Solve $(\mathbf{D}-w^{\left(k\right)}\mathbf{L}){\mathbf{v}}_0={\mathbf{r}}_0$ 5: $w^{(k+1)}={\displaystyle \frac{{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0}{\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2}}$ 6: Enddo, if $|w^{(k+1)}-w^{\left(k\right)}|<{\epsilon }_1$

Algorithm 4: for r of QAOR

1: Give ${\mathbf{x}}_0$, w, $r^{\left(0\right)}$ and ${\epsilon }_1$ 2: ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$ 3: Do $k=0,1,\dots$ 4: Solve $[(1+r^{\left(k\right)})\mathbf{D}-w\mathbf{L}]{\mathbf{v}}_0={\mathbf{r}}_0$ 5: $r^{(k+1)}={\displaystyle \frac{{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0}{\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2}}$ 6: Enddo, if $|r^{(k+1)}-r^{\left(k\right)}|<{\epsilon }_1$

Algorithm 5: for r of AOR and POR

1: Give ${\mathbf{x}}_0$, w, $\sigma$ ($\sigma =0$ for AOR) 2: ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$ 3: Solve $[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{v}}_0={\mathbf{r}}_0$ 4: $r={\displaystyle \frac{{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0}{\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2}}$

Algorithm 6: for r of ROR and PROR

1: Give ${\mathbf{x}}_0$, w, $\sigma$ ($\sigma =0$ for ROR) 2: ${\mathbf{r}}_0=\mathbf{b}-\mathbf{A}{\mathbf{x}}_0$ 3: Solve $[(1+\sigma )\mathbf{D}-w\mathbf{L}]{\mathbf{v}}_0=(1-w){\mathbf{r}}_0$ 4: $r={\displaystyle \frac{{\mathbf{r}}_0^{\mathrm{T}}\mathbf{A}{\mathbf{v}}_0}{\parallel \mathbf{A}{\mathbf{v}}_0{\parallel }^2}}$

In Algorithm 5 for AOR if the optimization of r is carried out at all iteration steps it is equivalent to A2-SOR with $r={\eta }_k$.

8. Results and Discussion

For a given linear problem $\mathbf{A}\mathbf{x}=\mathbf{b}$ with size n, the first step is constructing $\mathbf{D}=\left\{D_{ij}\right\}$, $\mathbf{U}=\left\{U_{ij}\right\}$ and $\mathbf{L}=\left\{L_{ij}\right\}$ from $\mathbf{A}$ (refer to Algorithm 7), which are used in all algorithms. The number of operations is $n^2$.

Algorithm 7: for $\mathbf{D}$, $\mathbf{U}$ and $\mathbf{L}$

1: Give $\mathbf{A}=\left\{A_{ij}\right\}$ 2: Do $i=1,\dots ,n$ 3: Do $j=1,\dots ,n$ 4: If $j=i$, $D_{ij}=A_{ij}$; otherwise $D_{ij}=0$ 5: If $j<i$, $L_{ij}=-A_{ij}$; otherwise $L_{ij}=0$ 6: If $j>i$, $U_{ij}=-A_{ij}$; otherwise $U_{ij}=0$ 7: Enddo

In order to compare the proposed NSIS to the methods in the existent literature, some simple examples are taken and the sizes n of the linear systems are the same to the literature.

8.1. Example 1

Consider an example of Equation (1) with $\mathbf{A}=\left[a_{ij}\right],i,j=1,\dots ,n$; $a_{ij}=1/\left(10j\right)-1/20,i>j$, $a_{ij}=1/\left[10(i-j)\right]-1/20,i<j$, and $a_{ij}=1,i=j$ [17]. We take $b_i=1$, $i=1,\dots ,n$ and $x_i^0=0,i=1,\dots ,n$, and fix $n=20$.

With the convergence criterion $\epsilon ={10}^{-12}$ Table 2 compares the number of iterations (NI) for SOR, A2-SOR and A2-AOR, and that obtained in [17] by using Quasi-AOR (QAOR) and Quasi-SOR (QSOR), which are subjected to a loose convergence criterion ${10}^{-6}$. In SOR the optimal value of w is computed from Algorithm 3; through three iterations $w=0.715496$ is obtained; the spectral radius of the iteration matrix is small with 0.2898. If an ad hoc value $w=0.5$ is used in SOR, it needs 269 iterations.

Table 2. Example 1, NI for QAOR, QSOR, AOR, SOR, A2-SOR and A2-AOR; op. r means the optimal value of r, and op. w means the optimal value of w.

	QAOR	QSOR	AOR	AOR (op. r)	SOR (op. w)	A2-SOR	A2-AOR
NI	155	141	161	83	167	72	54

In A2-SOR, we take $w=0.715496$ and $\alpha =1.6$; $w=0.15$ and $r=0.759399$ are used in A2-AOR; $w=0.3$ and $r=0.9$ are used in AOR and QAOR. In AOR we take $w=1.5$, and if the optimal $r=0.641305$ is computed from Algorithm 5, then NI reduces to 83.

In Table 3, we demonstrate the usefulness of A1-SOR method with $w=0.5$. A1-SOR with $\alpha =1.98$ can improve the convergence speed twice than the original SOR.

Table 3. Example 1, NI and spectral radius obtained by A1-SOR for different values of $\alpha$.

$\mathit{\alpha }$	1	1.2	1.5	1.8	1.98
$\rho$	0.504	0.407	0.267	0.154	0.147
NI	269	223	178	147	133

In Table 4 the test is for the AOR method with $w=0.3$ and $r=0.9$. The convergence is improved by increasing the value of $\alpha$.

Table 4. Example 1, NI obtained by A1-AOR for different values of $\alpha$.

$\mathit{\alpha }$	1	1.2	1.3	1.4	1.5	1.6	1.7
NI	161	133	121	111	103	96	89

Comparing Table 2, Table 3 and Table 4, the Algorithm 2 type iterative schemes are convergent faster than the Algorithm 1 type iterative schemes.

As that given in [17], we take $r=0.9$ and $w=0.3$ in the original QAOR; it needs NI = 421 to satisfy a stringent convergence criterion $\epsilon ={10}^{-15}$, where ${\rho }_{\mathrm{QAOR}}=0.5341$. If Algorithm 4 for r is adopted to find the optimal value $r=3.22823$, ${\rho }_{\mathrm{QAOR}}=0.5341$ is greatly reduced to ${\rho }_{\mathrm{QAOR}}=0.2843$. Under the same convergence criterion, NI is reduced to 264.

In Table 5, the NI obtained by Algorithm 1 for the QAOR method are compared for different values of $\alpha$.

Table 5. Example 1, NI and spectral radius $\rho$ obtained by A1-QAOR for different values of $\alpha$.

$\mathit{\alpha }$	1	1.2	1.5	1.8	2
$\rho$	0.2843	0.2070	0.2775	0.4891	0.6495
NI	264	218	171	140	124

In Table 6, NI obtained by Algorithm 2 for the QAOR is compared for different values of r. Under a loose convergence criterion ${\epsilon }_1={10}^{-2}$, r is obtained from Algorithm 4 for r with $r^{\left(0\right)}=1$ and different values of w. The best value of w is $w=0.3$ for the QAOR method.

Table 6. Example 1, the value of r and NI obtained by A2-QAOR for different values of w.

w	0.1	0.3	0.5	0.8	1
r	3.28909	3.22941	3.16874	3.06835	3.00518
NI	91	72	88	94	111

We take $w=1.5$; r is computed by Algorithm 5 for r of AOR, and r is computed by Algorithm 6 for r of ROR. In Table 7, NI obtained by different methods are compared. $\alpha =2$ is used in A1-ROR; $w=0.1$ is used in A2-ROR.

Table 7. Example 1, the value of r and NI obtained by different methods of AOR and ROR.

	AOR	ROR	A1-ROR	A2-ROR
r	0.64131	−1.28261	−1.28261	0.84773
NI	102	102	40	74

In Table 8, NI obtained by POR and A1-POR are compared for different values of $\sigma$. r is obtained from Algorithm 5. The best value of $\sigma$ is $\sigma =0.1$ for the POR method.

Table 8. Example 1, NI obtained by POR and A1-POR for different values of $\sigma$; $w=1.9$ fixed, and $\alpha =2$ for A1-POR.

$\mathit{\sigma }$	0.1	0.5	0.8	1.2
POR	117	152	178	198
A1-POR	58	67	80	90

In Table 9, NI obtained by PROR and A1-PROR are compared for different values of $\sigma$. r is obtained from Algorithm 6.

Table 9. Example 1, NI for PROR and A1-PROR with different values of $\sigma$; $w=1.5$ fixed, and $\alpha =2$ for A1-PROR.

$\mathit{\sigma }$	0.1	0.3	0.5	0.8
PROR	137	167	185	200
A1-PROR	59	75	83	93

Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8 and Table 9 demonstrate that the proposed NSIS and the optimization technique for determining the optima value of parameter can significantly reduce the number of iterations (NI).

8.2. Example 2

In Equation (1), we take [37]

$$\mathbf{A}=\left[\begin{array}{cccc}1& 0& {\displaystyle \frac{1}{5}}& {\displaystyle \frac{1}{5}}\\ 0& 1& -{\displaystyle \frac{71}{10}}& {\displaystyle \frac{113}{10}}\\ {\displaystyle \frac{16}{5}}& {\displaystyle \frac{1}{5}}& 1& 0\\ 2& {\displaystyle \frac{1}{5}}& 0& 1\end{array}\right],\mathbf{b}=\left[\begin{array}{c}1.4\\ 5.2\\ 4.4\\ 3.2\end{array}\right],\mathbf{x}=\left[\begin{array}{c}1\\ 1\\ 1\\ 1\end{array}\right].$$

Under ${\mathbf{x}}_0=\mathbf{0}$ and $\epsilon ={10}^{-10}$, we take $w=5/3$ and $r=14/3$ for AOR, $w=5/3$ and $r=-4.03$ for ROR, and $\alpha =0.6$ used in A1-AOR. For AOR if $r=1.97713$ is computed from Algorithm 5 with $w=5/3$, then NI is reduced to 62. The values of $w=5/3$ and $r=14/3$ for AOR were claimed in [30,31] to be the optimal ones; however, the AOR method with the values of $w=5/3$ and $r=1.97713$ converges faster.

For A1-AOR if $r=1.97713$ is computed from Algorithm 5 with $w=5/3$, and take $\alpha =1.45$, then NI is reduced to 45 as shown in Table 10. For A1-ROR if $r=-2.965691$ is computed from Algorithm 6, and $w=5/3$ and $\alpha =1.45$ were taken, then NI is reduced to 45.

Table 10. Example 2, NI for AOR and ROR.

	AOR	AOR (op. r)	A1-AOR	Al-AOR (op. r)	ROR	Al-ROR (op. r)
NI	96	62	47	45	46	45

Table 11 demonstrates the A1-PROR method for different values of $\alpha$. $r=0.23726$ is computed from Algorithm 6 with $w=1/6$ and $\sigma =-0.9$. The optimal value $r=0.3446$ is derived in [31].

Table 11. Example 2, comparing NI for A1-PROR with different values of $\alpha$.

$\mathit{\alpha }$	1	1.2	1.3	1.4	1.5	1.6
NI	62	51	49	48	46	48

8.3. Example 3

In Equation (1), we take

$$\mathbf{A}=\left[\begin{array}{ccccc}1& 2& 3& 4& 5\\ 2& 3& 4& 5& 1\\ 3& 4& 5& 1& 2\\ 4& 5& 1& 2& 3\\ 5& 1& 2& 3& 4\end{array}\right],\mathbf{x}=\left[\begin{array}{c}1\\ 1\\ 1\\ 1\\ 1\end{array}\right].$$

$\mathbf{A}$ is a symmetric positive definite matrix. When $\mathbf{A}$ is a positive definite matrix,

$$w_{\mathrm{opt}}={\displaystyle \frac{2}{1+\sqrt{1-{\rho }^2({\mathbf{I}}_n-{\mathbf{D}}^{-1}\mathbf{A})}}}$$

(53)

is the optimal value for the SOR method [38], where $\rho$ is the spectral radius of ${\mathbf{I}}_n-{\mathbf{D}}^{-1}\mathbf{A}$.

We find that the spectral radius of ${\mathbf{I}}_n-{\mathbf{D}}^{-1}\mathbf{A}$ is 2.5233, which according to Equation (53) means that the SOR method for this problem is divergent. Indeed for this problem other algorithms AOR, QAOR, ROR, POR, and PROR are divergent.

We take ${\mathbf{x}}_0=\mathbf{0}$. Under $\epsilon ={10}^{-5}$, Table 12 demonstrates the NI of the A2-SOR method for different values of $\alpha$. $w=0.20078$ is computed from Algorithm 3.

Table 12. Example 3, NI for A2-SOR with different values of $\alpha$.

$\mathit{\alpha }$	1.3	1.4	1.45	1.5	1.6
NI	113	128	109	140	165

Table 13 demonstrates the A2-PROR method for different values of $\alpha$. $r=0.865643$ is computed from Algorithm 6 with $w=0.5$ and $\sigma =0.5$.

Table 13. Example 3, NI for A2-PROR with different values of $\alpha$.

$\mathit{\alpha }$	1.25	1.3	1.4	1.45	1.5	1.6
NI	114	77	100	85	98	142

There exists an optimal value of $\alpha$, for which NI is minimal. In general the relation between $\alpha$ to NI is not simple.

Table 14 demonstrates the A2-ROR method for different values of w but $\alpha =1.3$ fixed. r is computed from Algorithm 6 at each iteration step.

Table 14. Example 3, NI for A2-ROR with different values of w.

w	0.3	0.4	0.5	0.6	0.7	0.8
$\alpha$	1.3	1.4	1.3	1.5	1.5	1.6
NI	97	88	81	74	58	76

This problem shows that the factor ${\eta }_k$ in the splitting iterative schemes can stabilize the original splitting iterative schemes, which are unstable for this problem.

8.4. Example 4

We solve a complex Helmholtz equation:

$$-\Delta u(x,y)+\sigma u(x,y)=f(x,y),0<x<1,0<y<1,$$

(54)

with homogeneous boundary condition, where $\sigma$ is a complex-valued impedance coefficient.

After five-point discretization, Equation (54) becomes a complex linear system:

$$(\mathbf{W}+i\mathbf{T})(\mathbf{w}+i\mathbf{t})=\mathbf{f}+i\mathbf{g},$$

(55)

where $\mathbf{W},\mathbf{T}\in {\mathbb{R}}^{N\times N}$ are symmetric positive semi-definite with $\mathbf{W}>\mathbf{0}$ positive, and $i^2=-1$. Upon letting

$$\mathbf{A}=\left[\begin{array}{cc}\mathbf{W}& -\mathbf{T}\\ \mathbf{T}& \mathbf{W}\end{array}\right],\mathbf{y}=\left[\begin{array}{c}\mathbf{w}\\ \mathbf{t}\end{array}\right],\mathbf{h}=\left[\begin{array}{c}\mathbf{f}\\ \mathbf{g}\end{array}\right],$$

(56)

Equation (55) becomes

$$\mathbf{A}\mathbf{x}=\mathbf{b},\mathbf{x},\mathbf{b}\in {\mathbb{R}}^n,\mathbf{A}\in {\mathbb{R}}^{n\times n},$$

(57)

where $n=2N$.

Equation (57) is a two-block linear system; in general, special splitting techniques are designed to effectively solve this sort linear system, for instance, the optimal two-block splitting iterative scheme [8], the generalized successive overrelaxation (GSOR) method [39]; the symmetric block triangular splitting (SBTS) iteration method [40], the Hermitian and skew-Hermitian splitting (HSS) iteration method [41], and the modified Hermitian and skew-Hermitian splitting (MHSS) iteration method [42].

Suppose that $\mathbf{K}={\mathbf{I}}_{n_0}\otimes \mathbf{C}+\mathbf{C}\otimes {\mathbf{I}}_{n_0}$ be the centered difference matrix approximation of the negative Laplacian operator in Equation (54); the mesh size is $h=1/(n_0+1)$, and $\mathbf{C}=\mathrm{tridiag}(-1,2,-1)/h^2$; then we have

$$\left[\mathbf{K}+{\displaystyle \frac{3-\sqrt{3}}{h}}{\mathbf{I}}_N+i\left(\mathbf{K}+{\displaystyle \frac{3+\sqrt{3}}{h}}{\mathbf{I}}_N\right)\right]u=\mathbf{f}+i\mathbf{g},$$

where $N=n_0^2$ and $n=2n_0^2$;

$$\begin{array}{c}\mathbf{W}=\mathbf{K}+{\displaystyle \frac{3-\sqrt{3}}{h}}{\mathbf{I}}_N,\mathbf{T}=\mathbf{K}+{\displaystyle \frac{3+\sqrt{3}}{h}}{\mathbf{I}}_N,\hfill \\ \mathbf{f}={\displaystyle \frac{j}{h{(j+1)}^2}},\mathbf{g}=-{\displaystyle \frac{j}{h{(j+1)}^2}},j=1,\dots ,N.\hfill \end{array}$$

We apply the AOR in Equation (43) with $w=0.6$ and r being optimized by Algorithm 5. Table 15 compares the optimal value of r, the number of steps (NI) and CPU obtained by AOR, under $\epsilon ={10}^{-4}$ and ${\mathbf{x}}_0=\mathbf{0}$.

Table 15. Example 4, r, NI and CPU obtained by AOR with different n.

n	200	450	800	1250	1800	2450
r	0.737113	0.740454	0.738657	0.736292	0.734101	0.732191
NI	64	93	98	133	148	158
CPU	0.35	0.67	2.44	7.05	15.86	33.89

We apply the AOR in Equation (43) with $w=1.2$ and r being optimized by Algorithm 5 at every iteration steps. It is equivalent to A2-SOR. Table 16 compares NI and CPU under $\epsilon ={10}^{-4}$ and ${\mathbf{x}}_0=\mathbf{0}$. Comparing to Table 15 the A2-SOR is saving CPU time and also with less number of NI.

Table 16. Example 4, NI and CPU obtained by A2-SOR with different n.

n	200	450	800	1250	1800	2450
NI	40	57	73	86	92	100
CPU	0.34	0.67	1.59	4.41	10.12	20.85

Under $n_0=32$ ($n=2048$), $\parallel \mathbf{r}\parallel /\parallel \mathbf{b}\parallel \le \epsilon ={10}^{-6}$, where $\parallel \mathbf{b}\parallel =\sqrt{{\parallel {\mathbf{b}}_1\parallel }^2+{\parallel {\mathbf{b}}_2\parallel }^2}$, ${\mathbf{b}}_1=\mathbf{f}+\mathbf{g}=\mathbf{0}$ and ${\mathbf{b}}_2=\mathbf{g}$, we apply the A1-AOR with $\alpha =1.95$, $w=1.6$ and r being optimized by Algorithm 5 at every iteration steps to solve this problem.

We compare NI obtained by different methods in Table 17, some of which is listed in [40] with HSS in [41], and MHSS in [42]. The GSOR was obtained from Table 1 in [39]. Owing to its easy formulation and with a low computational cost, the A1-AOR is good even it is slower than other iterative methods in Table 17. Without needing of the complicated spectral analysis to determine the optimal value of parameter the A1-AOR is a competitive method.

Table 17. Example 4, NI obtained by different methods.

Method	HSS	MHSS	SBTS	GSOR	A1-AOR
NI	65	54	31	22	106

8.5. Example 5

We solve a Hilbert linear problem with the following coefficient matrix:

$$A_{ij}={\displaystyle \frac{1}{i+j-1}},i,j=1,\dots ,n.$$

(58)

In a practical application the problem is finding an $(n-1)$-order polynomial function $p\left(x\right)=a_0+a_{1}x+\dots +a_{n-1}{x}^{n-1}$ to best match a continuous function $f\left(x\right)$ in the interval of $x\in [0,1]$:

$$\underset{\mathrm{deg}\left(p\right)\le n-1}{min}{\int }_0^1|f\left(x\right)-p\left(x\right)|dx,$$

which leads to a problem governed by Equation (1). $\mathbf{A}$ is the $n\times n$ Hilbert matrix defined by Equation (58), $\mathbf{x}$ is composed of the n coefficients $a_0,a_1,\dots ,a_{n-1}$ appearing in $p\left(x\right)$, and

$$\mathbf{b}=\left[\begin{array}{c}{\int }_0^{1}f\left(x\right)dx\\ {\int }_0^{1}xf\left(x\right)dx\\ ⋮\\ {\int }_0^1{x}^{n-1}f\left(x\right)dx\end{array}\right]$$

is uniquely determined by the function $f\left(x\right)$.

The Hilbert matrix is notoriously ill-conditioned, which can be seen from Table 18. The condition number of Hilbert matrix grows as $e^{3.5n}$.

Table 18. The condition numbers of Hilbert matrix.

n	cond(A)	n	cond(A)
3	$5.24\times {10}^2$	7	$4.57\times {10}^8$
4	$1.55\times {10}^4$	8	$1.53\times {10}^{10}$
5	$4.77\times {10}^5$	9	$4.93\times {10}^{11}$
6	$1.50\times {10}^7$	10	$1.60\times {10}^{13}$

Because the Hilbert matrix is seriously ill-conditioned, a loose convergence criterion with $\epsilon ={10}^{-3}$ is considered. We suppose that $x_j=1,j=1,\dots ,n$ are the solutions. We first apply the Gauss–Seidel method to solve this problem. Table 19 lists NI, the maximum error (ME) = ${max}_{j=1,\dots ,n}|x_j-1|$, and the root-mean-square-error (RMSE) = $\sqrt{{\displaystyle {\sum }_{j=1}^n}{(x_j-1)}^2/n}$ obtained by the Gauss–Seidel method.

Table 19. Example 5, NI, ME and RMSE obtained by Gauss–Seidel method with different n.

n	20	40	60	80	100
NI	328	514	448	725	757
ME	$1.08\times {10}^{-1}$	$9.79\times {10}^{-2}$	$2.68\times {10}^{-1}$	$1.38\times {10}^{-1}$	$1.42\times {10}^{-1}$
RMSE	$5.92\times {10}^{-2}$	$5.10\times {10}^{-2}$	$6.83\times {10}^{-2}$	$5.00\times {10}^{-2}$	$4.89\times {10}^{-2}$

Table 20 lists NI, ME and RMSE obtained by the SOR with w determined by Algorithm 3 at the first step. It is apparent that the SOR is superior than the Gauss–Seidel method.

Table 20. Example 5, NI, ME and RMSE obtained by the SOR with different n.

n	20	40	60	80	100
w	0.156725	0.111880	0.082611	0.073455	0.063021
NI	76	248	299	298	294
ME	$1.31\times {10}^{-1}$	$1.11\times {10}^{-1}$	$9.59\times {10}^{-2}$	$9.78\times {10}^{-2}$	$1.00\times {10}^{-1}$
RMSE	$5.91\times {10}^{-2}$	$4.43\times {10}^{-2}$	$3.42\times {10}^{-2}$	$3.57\times {10}^{-2}$	$3.79\times {10}^{-2}$

9. Conclusions

In this paper the splitting iterative schemes were reformulated in terms of descent and residual vectors. In the new formulation for the new splitting iterative scheme (NSIS) the extrapolation technique becomes easy to follow. The NSIS can be obtained from the original splitting iterative scheme by either multiplying the descent vector by a parameter $\alpha$, or multiplying the right-hand side of the governing equation for the descent vector by a parameter $\alpha$. Different splitting iterative schemes can be unified to have the same iterative form, but they are different in the governing equations for the descent vector.

We proved that the spectral radius of the NSIS is smaller than the original one if $\alpha >1$. In addition to a constant value of $\alpha$, varying values of $\alpha$ were examined by preserving the orthogonality and maximizing the residual vectors’ decreasing norm. A varying orthogonalization factor was introduced in the second NSIS to enhance the stability, and the property of absolute convergence was proven.

Main points for the novel contributions are summarized as follows.

• The splitting iterative schemes were unified in terms of descent and residual vectors.
• An extrapolation parameter $\alpha >1$ in the splitting iterative scheme can improve the convergence speed.
• The NSISs were developed to maximally decrease the residual and to preserve the orthogonal property.
• The second method by adding a stepwise varying factor ${\eta }_k$ can stabilize the splitting iterative scheme, even the original one is unstable.
• The acceleration parameter r can be obtained readily by a maximization technique.
• The improvement of convergence speed was observed by adopting the proposed NSISs together with the optimization technique for determining the optima value of parameter.