Improved Convergence Analysis of Gauss-Newton-Secant Method for Solving Nonlinear Least Squares Problems

Abstract

We study an iterative differential-difference method for solving nonlinear least squares problems, which uses, instead of the Jacobian, the sum of derivative of differentiable parts of operator and divided difference of nondifferentiable parts. Moreover, we introduce a method that uses the derivative of differentiable parts instead of the Jacobian. Results that establish the conditions of convergence, radius and the convergence order of the proposed methods in earlier work are presented. The numerical examples illustrate the theoretical results.

1. Introduction

Nonlinear least squares problems often arise while solving overdetermined systems of nonlinear equations, parameter estimation of physical processes by measurement results, constructing nonlinear regression models for solving engineering problems, etc.

The nonlinear least squares problem has the form

$$\underset{x\in R^p}{min}\frac{1}{2}F{(x)}^{T}F(x),$$

(1)

where the residual function $F:R^p\to R^m$ ($m\ge p$) is nonlinear in x; F is a continuously differentiable function. Effective methods for solving nonlinear least squares problems is the Gauss-Newton method [1,2,3]

$$x_{n+1}=x_n-{\left[F^{\prime }{(x_n)}^T{F}^{\prime }(x_n)\right]}^{-1}{F}^{\prime }{(x_n)}^{T}F(x_n),n=0,1\dots$$

(2)

However, in practice, there are often problems with the calculation of derivatives. Hence, one can use the iterative-difference methods. These methods do not require calculation of derivatives. Moreover, they do not perform worse than Gauss-Newton method in terms of the convergence rate and the number of iterations. In some cases, nonlinear functions consist of differentiable and nondifferentiable parts. However, it is possible to use iterative-difference methods [4,5,6,7]

$$x_{n+1}=x_n-{(A_n^T{A}_n)}^{-1}{A}_n^{T}F(x_n),n=0,1,\dots ,$$

(3)

where

$$A_n=F(x_n,x_{n-1}),$$

$$A_n=F(2x_n-x_{n-1},x_{n-1}),$$

or

$$A_n=F(x_n,x_{n-1})+F(x_{n-2},x_n)-F(x_{n-2},x_{n-1}).$$

It is desirable to build iterative methods that take into account properties of the problem. In particular, we can use only derivative of differentiable part of operator instead of full Jacobian, which in fact, does not exist. The methods obtained using this approach converge slowly. More efficient methods use sum of the derivatives of the differentiable part and divided difference of the nondifferentiable part of the operator instead of the Jacobian. Such an approach shows great results in the case of solving nonlinear equations.

In this work we study a combined method for solving nonlinear least squares problem, based on the Gauss-Newton, secant methods. We also use a method, requiring only derivative from the differentiable part of operator. We prove the local convergence and show efficiency on test cases when comparing with secant type methods [5,6]. The convergence region of iterative methods is small in general. This fact limits the number of initial approximations. It is therefore important to extend this region without requiring additional hypotheses. The new approach [8] leads to larger convergence radius than before [9]. We achieve this goal by locating an at least as small region as before containing the iterates. Then, the new Lipschitz constants are at least as tight as the old Lipschitz constants. Moreover, using more precise estimates on the distances involved, under weaker hypotheses, and under the same computational cost, we provide an analysis of the Gauss-Newton-Secant method with the following advantages over the corresponding results in [9]: larger convergence region; finer error estimates on the distances involved, and an at least as precise information on the location of the solution.

The rest of the paper is given as follows. Section 2 contains the statement of the problem, in Section 3 and Section 4, we present the local convergence analysis of the first and second method, respectively. In Section 5, we provide the numerical examples. The article ends with some conclusions.

2. Description of the Problem

Consider the nonlinear least squares problem

$$\underset{x\in R^p}{min}\frac{1}{2}{(F(x)+G(x))}^T(F(x)+G(x)),$$

(4)

where residual function $F+G:R^p\to R^m$ ($m\ge p$) is nonlinear in x; F is continuously differentiable function; G is continuous function, differentiability of which, in general, is not required.

We propose a modification of the Gauss-Newton method to find a solution of problem (4):

$$\begin{array}{c}{x}_{n+1}=x_n-{(A_n^T{A}_n)}^{-1}{A}_n^T(F(x_n)+G(x_n)),\\ A_n=F^{\prime }(x_n)+G(x_n,x_{n-1}),n=0,1,\dots \end{array}$$

(5)

Here, $F^{\prime }(x_n)$ is Fréchet derivative by $F(x)$; $G(x_n,x_{n-1})$ is a divided difference of order one for function $G\left(x\right)$ [10], where vectors $x_n$, $x_{n-1}$ and $x_0$, $x_{-1}$ are given initial approximations, satisfying $G(x,y)(x-y)=G(x)-G(y)$ for $x\ne y$ and $G(x,x)=G^{\prime }(x)$, if G is differentiable. Setting $A_n=F^{\prime }(x_n)$, from method (5) we get Gauss-Newton type iterative method for solving problem (4)

$$x_{n+1}=x_n-{(F^{\prime }{(x_n)}^T{F}^{\prime }(x_n))}^{-1}{F}^{\prime }{(x_n)}^T(F(x_n)+G(x_n)),n=0,1,\dots$$

(6)

In case of $m=p$, problem (4) turns into a system of nonlinear equations

$$F(x)+G(x)=0.$$

(7)

Then, it is well known ([3], p. 267) that techniques for minimizing problem (4) are techniques for finding a solution $x^{*}$ of Equation (7). In this case (5) transforms into the Newton-Secant combined method [11,12]

$$x_{n+1}=x_n-{(F^{\prime }(x_n)+G(x_n,x_{n-1}))}^{-1}(F(x_n)+G(x_n)),n=0,1,\dots ,$$

(8)

and method (6) into Newton’s-type method for solving nonlinear Equation (7) [13]

$$x_{n+1}=x_n-{(F^{\prime }(x_n))}^{-1}(F(x_n)+G(x_n)),n=0,1,\dots$$

We assume from now on that function G is differentiable at $x=x^{*}$.

3. Local Convergence Analysis (5)

Sufficient conditions and the convergence order of the iterative process (5) are presented. However first, we need some crucial definitions. They are needed to provide a clear relationship between the Lipschitz constants appearing in the local convergence analysis and the relationship between them.

Definition 1.

The Fréchet derivative $F^{\prime }$ satisfies the center-Lipschitz condition on D, if there exists $L_0>0$ such that for each $x\in D$

$$\left|\right|F^{\prime }(x)-F^{\prime }(x^{*})\left|\right|\le L_0\left|\right|x-x^{*}\left|\right|.$$

(9)

Definition 2.

The divided difference $G(x,y)$ satisfies the center-Lipschitz condition $D\times D$, if there exists $M_0>0$ such that for each $x,y\in D$

$$\left|\right|G(x,y)-G(x^{*},x^{*})\left|\right|\le M_0(||x-x^{*}\left|\right|+\left|\right|y-x^{*}\left|\right|).$$

(10)

Let $B>0$ and $\alpha >0$. Define function $\phi :[0,+\infty )\to [0,+\infty )$ by

$$\phi (r)=B[2\alpha +(L_0+2M_0)r](L_0+2M_0)r.$$

(11)

Let $U(x^{*},r_{*})=\{x:||x-x^{*}\left|\right|\le r_{*}\},r_{*}>0$. Suppose that equation $\phi (r)=1$ has at least one positive solution. Denote by γ the smallest such solution. Define

$$D_0=D\cap U(x^{*},\gamma ).$$

(12)

Definition 3.

The Fréchet derivative $F^{\prime }$ satisfies the restricted Lipschitz condition on $D_0$, if there exists $L>0$ such that for each $x,y\in D_0$

$$\left|\right|F^{\prime }(x)-F^{\prime }(y)\left|\right|\le L\left|\right|x-y\left|\right|.$$

(13)

Definition 4.

The first order divided difference $G(x,y)$ satisfies the restricted Lipschitz condition on $D_0\times D_0$, if there exists $M>0$ such that for each $x,y,u\in D_0$

$$\left|\right|G(x,y)-G(u,x^{*})\left|\right|\le M(||x-u\left|\right|+\left|\right|y-x^{*}\left|\right|).$$

(14)

Next, we also state the definitions given in [9], so we can compare them to preceding ones.

Definition 5.

The Fréchet derivative $F^{\prime }$ satisfies the Lipschitz condition on D, if there exists $L_1>0$ such that for each $x,y\in D$

$$\left|\right|F^{\prime }(x)-F^{\prime }(y)\left|\right|\le L_1\left|\right|x-y\left|\right|.$$

(15)

Definition 6.

The first order divided difference $G(x,y)$ satisfies the Lipschitz condition on $D\times D$, if there exists $M_1>0$ such that for each $x,y,u,v\in D$

$$\left|\right|G(x,y)-G(u,v)\left|\right|\le M_1(||x-u\left|\right|+\left|\right|y-v\left|\right|).$$

(16)

Remark 1.

It follows from the preceding definitions that $L=L(L_0,M_0)$, $M=M(L_0,M_0)$

$$L_0\le L_1,$$

(17)

$$L\le L_1,$$

(18)

$$M_0\le M_1,$$

(19)

and

$$M\le M_1,$$

(20)

since $D_0\subseteq D$. If any of (17)–(20) are strict inequalities, then the following advantages are obtained over the work in [9] using $L_1$ and $M_1$ instead of the new constants:

$(a_1)$ At least as large convergence domain leading to at least as many initial choices.

$(a_2)$ At least as tight upper bounds on the distances $\left|\right|x_n-x^{*}\left|\right|$, so at most as many iterations are needed to obtain a desired error tolerance.

It is always true that $D_0$ is at least as small and included in D by (12). Here lies the new idea and the reason for the advantages. Notice that these advantages are obtained under the same computational cost, as in [9], since the new constants $L_0,M_0,L$ and M are special cases of constants $L_1$ and $M_1$. This technique of using the center Lipschitz condition in combination with the restricted convergence region has been used on Newton’s, Secant and Newton-like methods [14] and can be used on other methods in order to extend their applicability.

The Euclidean norm, and the corresponding matrix norm are used in this study which has the advantage $\parallel A^T\parallel =\parallel A\parallel$.

The proof of the next result follows the corresponding one in [9] but there are crucial differences where we use $(L_0,L)$ instead of $L_1$ and $M_0,M$ instead of $M_1$.

Theorem 1.

Let $F+G:R^p\to R^m$ be continuous on set $D\subseteq R^p$, F be continuously differentiable in this set, and $G(·,·):D\times D\to \mathcal{L}(R^p,R^m)$ be a divided difference of order one. Suppose, the problem (4) has a solution $x^{*}$ on set D, and the inverse operator ${(A_{*}^T{A}_{*})}^{-1}={\left[{(F^{\prime }(x^{*})+G(x^{*},x^{*}))}^T(F^{\prime }(x^{*})+G(x^{*},x^{*}))\right]}^{-1}$ exists, $\left|\right|{(A_{*}^T{A}_{*})}^{-1}\left|\right|\le B$, (9), (10), (13), (14) hold, and γ defined in (11) exists. Moreover,

$$\left|\right|F(x^{*})+G(x^{*})\left|\right|\le \eta ,\left|\right|F^{\prime }(x^{*})+G(x^{*},x^{*})\left|\right|\le \alpha ;$$

(21)

$$B(L_0+2M_0)\eta <1,$$

(22)

and $U(x^{*},r_{*})\subseteq D$, where $r_{*}$ is the unique positive zero of function q, defined by

$$\begin{array}{c}q(r)=B[(\alpha +(L_0+2M_0)r)(L+2M)r/2+(L_0+2M_0)\eta ]\\ +B[2\alpha +(L_0+2M_0)r](L_0+2M_0)r-1.\end{array}$$

(23)

Then, for $x_0,x_{-1}\in U(x^{*},r_{*})$ method (5) is well defined and generates the sequence $\{x_n\},n=0,1,\dots ,$ which belongs to set $U(x^{*},r_{*})$, and converges to the solution $x^{*}$. Moreover, the following error bounds hold

$$\begin{array}{c}\left|\right|x_{n+1}-x^{*}\left|\right|\le C_1\left|\right|x_{n-1}-x^{*}\left|\right|+C_2\left|\right|x_n-x^{*}\left|\right|+C_3\left|\right|x_{n-1}-x^{*}\left|\right|\left|\right|x_n-x^{*}\left|\right|\\ +C_4\left|\right|x_{n-1}-x^{*}{\left|\right|}^2\left|\right|x_n-x^{*}\left|\right|+C_5\left|\right|x_n-x^{*}{\left|\right|}^2\\ +C_6\left|\right|x_{n-1}-x^{*}\left|\right|\left|\right|x_n-x^{*}{\left|\right|}^2+C_7\left|\right|x_n-x^{*}{\left|\right|}^3,\end{array}$$

(24)

where

$$\begin{array}{c}g(r)=B{[1-B(2\alpha +(L_0+2M_0)r)(L_0+2M_0)r]}^{-1};\\ C_1=g(r_{*})M_0\eta ;C_2=g(r_{*})(L_0+M_0)\eta ;C_3=g(r_{*})\alpha M;C_4=g(r_{*})M_{0}M;\\ C_5=g(r_{*})\alpha \frac{L}{2};C_6=g(r_{*})\left(L_{0}M+M_{0}M+\frac{M_{0}L}{2}\right);C_7=g(r_{*})\frac{L}{2}\left(L_0+M_0\right).\end{array}$$

(25)

Proof. 

According to the intermediate value theorem on $[0,r]$ for sufficiently large r and in view of (22) function q has at least one positive zero. Denote by $r_{*}$ the least such positive zero. Moreover, we have $q^{\prime }(r)\ge 0$ for $r\ge 0.$ Indeed, this zero is unique on $[0,r]$.

We shall show estimate (24) by first showing that sequence $x_n$ is well defined.

Let $A_n=F^{\prime }(x_n)+G(x_n,x_{n-1})$, and set $n=0$. We need to show that linear operator $A_0$ is invertible. By assuming, $x_0,x_{-1}\in U(x^{*},r_{*})$, we obtain the following estimation:

$$\begin{array}{c}\left|\right|I-{(A_{*}^T{A}_{*}^{})}^{-1}{A}_0^T{A}_0\left|\right|=\left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}(A_{*}^T{A}_{*}^{}-A_0^T{A}_0)\left|\right|\\ =\left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}(A_{*}^T(A_{*}^{}-A_0)+(A_{*}^T-A_0^T)(A_0-A_{*}^{})+(A_{*}^T-A_0^T)A_{*})\left|\right|\\ \le \left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}\left|\right|(||A_{*}^T\left|\right|\left|\right|A_{*}^{}-A_0\left|\right|+\left|\right|A_{*}^T-A_0^T\left|\right|\left|\right|A_0-A_{*}^{}\left|\right|+\left|\right|A_{*}^T-A_0^T\left|\right|\left|\right|A_{*}^{}\left|\right|)\\ \le B(\alpha ||A_{*}^{}-A_0||+||A_{*}^T-A_0^T|\left|\right||A_0-A_{*}^{}||+\alpha ||A_{*}^T-A_0^T||).\end{array}$$

(26)

By (9) and (10), we have in turn the estimate

$$\begin{array}{c}\left|\right|A_0-A_{*}\left|\right|=\left|\right|(F^{\prime }(x_0)+G(x_0,x_{-1}))-(F^{\prime }(x^{*})+G(x^{*},x^{*}))\left|\right|\\ =\left|\right|F^{\prime }(x_0)-F^{\prime }(x^{*})+G(x_0,x_{-1})-G(x^{*},x^{*})\left|\right|\\ \le \left|\right|F^{\prime }(x_0)-F^{\prime }(x^{*})\left|\right|+\left|\right|G(x_0,x_{-1})-G(x^{*},x^{*})\left|\right|\\ \le L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|).\end{array}$$

(27)

Then from inequality (26), definition of $r_{*}$ (23), we get

$$\begin{array}{c}\left|\right|I-{(A_{*}^T{A}_{*}^{})}^{-1}{A}_0^T{A}_0\left|\right|\le B[2\alpha +L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)]\\ \times [L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)]\\ \le B[2\alpha +(L_0+2M_0)r_{*}](L_0+2M_0)r_{*}=\phi (r_{*})<1.\end{array}$$

(28)

By the Banach Lemma on invertible operators [3], and (28) $A_0^T{A}_0^{-1}$ is invertible. Then from (26), (27) and (28), we get in turn that

$$\begin{array}{c}\left|\right|{(A_0^T{A}_0)}^{-1}\left|\right|\le g_0=B\{1-B[2\alpha +(L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)]\\ \times (L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}{\left|\right|)\}}^{-1}\\ \le g(r_{*})=B{\{1-B[2\alpha +(L_0+2M_0)r_{*}](L_0+2M_0)r_{*}\}}^{-1}.\end{array}$$

Hence, iterate $x_1$ is well defined by method (5) for $n=0$. Next, we will show that $x_1\in U(x^{*},r_{*})$. First of all, we get the estimation

$$\begin{array}{c}\left|\right|x_1-x^{*}\left|\right|=\left|\right|x_0-x^{*}-{(A_0^T{A}_0)}^{-1}(A_0^T(F(x_0)+G(x_0))-A_{*}^T(F(x^{*})+G(x^{*}))\left|\right|\\ \le \left|\right|-{(A_0^T{A}_0)}^{-1}\left|\right|\left|\right|[-A_0^T(A_0-{\int }_0^1{F}^{\prime }(x^{*}+t(x_0-x^{*}))dt\\ -G(x_0,x^{*}))(x_0-x^{*})+(A_0^T-A_{*}^T)(F(x^{*})+G(x^{*}))\left]\right||.\end{array}$$

Moreover, using (9), (10), (13), (14) and (21), we obtain in turn

$$\begin{array}{c}\left|\right|A_0-{\int }_0^1{F}^{\prime }(x^{*}+t(x_0-x^{*}))dt-G(x_0,x^{*})\left|\right|\\ =\left|\right|F^{\prime }(x_0)-{\int }_0^1{F}^{\prime }(x^{*}+t(x_0-x^{*}))dt+G(x_0,x_{-1})-G(x_0,x^{*})\left|\right|\\ =\left|\right|{\int }_0^1(F^{\prime }(x_0)-F^{\prime }(x^{*}+t(x_0-x^{*}))dt+G(x_0,x_{-1})-G(x_0,x^{*})\left|\right|\\ \le \frac{1}{2}L\left|\right|x_0-x^{*}\left|\right|+M\left|\right|x_{-1}-x^{*}\left|\right|=\frac{1}{2}\left(L\left|\right|x_0-x^{*}\left|\right|+2M\left|\right|x_{-1}-x^{*}\left|\right|\right),\\ \left|\right|A_0\left|\right|\le \left|\right|A_{*}\left|\right|+\left|\right|A_0-A_{*}\left|\right|\le \alpha +L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|).\end{array}$$

Then, by method (5) for $n=0$ and the preceding estimate, we have in turn that

$$\begin{array}{c}\left|\right|x_1-x^{*}\left|\right|\le B\{\left(\alpha +L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)\right)\\ \times \frac{1}{2}\left(L∥x_0-x^{*}∥+2M∥x_{-1}-x^{*}∥\right)\left|\right|x_0-x^{*}\left|\right|\\ +\eta \left(L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)\right)\}\\ /\{1-B[2\alpha +L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)]\\ \times (L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|))\}\\ \le g_0\{(\alpha +L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|))\\ \times \frac{1}{2}\left(L∥x_0-x^{*}∥+2M∥x_{-1}-x^{*}∥\right)\left|\right|x_0-x^{*}\left|\right|\\ +\eta \left(L_0\left|\right|x_0-x^{*}\left|\right|+M_0(||x_0-x^{*}\left|\right|+\left|\right|x_{-1}-x^{*}\left|\right|)\right)\}\\ <g(r_{*})[(\alpha +(L_0+2M_0)r_{*})(L+2M)r_{*}/2+(L_0+2M_0)\eta ]r_{*}\\ =p(r_{*})r_{*}=r_{*},\end{array}$$

where $p(r)=g(r)[(\alpha +(L_0+2M_0)r)(L+2M)r/2+(L_0+2M_0)\eta ]$. That is $x_1\in U(x^{*},r_{*})$ and estimate (24) holds for $n=0$.

Suppose that $x_n\in U(x^{*},r_{*})$ for $n=0,1,\dots ,k$ and estimate (24) holds for $n=0,1,\dots ,k-1$, where $k\ge$ 1 is integer. We shall show that $x_{n+1}\in U$ and estimate (24) holds for $n=k$.

As in the derivation of (28), using (9), (21) and the definition of function $\phi$, we get in turn that

$$\begin{array}{c}\left|\right|I-{(A_{*}^T{A}_{*}^T)}^{-1}{A}_{{}^k}^T{A}_k\left|\right|=\left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}(A_{*}^T{A}_{*}^{}-A_{{}^k}^T{A}_k)\left|\right|\\ =\left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}(A_{*}^T(A_{*}^{}-A_k)+(A_{*}^T-A_{{}^k}^T)(A_k-A_{*}^{})+(A_{*}^T-A_{{}^k}^T)A_{*})\left|\right|\\ \le \left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}\left|\right|(||A_{*}^T\left|\right|\left|\right|A_{*}^{}-A_k\left|\right|+\left|\right|A_{*}^T-A_{{}^k}^T\left|\right|\left|\right|A_k-A_{*}^{}\left|\right|+\left|\right|A_{*}^T-A_{{}^k}^T\left|\right|\left|\right|A_{*}^{}\left|\right|)\\ \le B(\alpha ||A_{*}^{}-A_k||+||A_{*}^T-A_{{}^k}^T|\left|\right||A_k-A_{*}^{}||+\alpha ||A_{*}^T-A_{{}^k}^T||)\\ \le B[2\alpha +L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|)]\\ \times [L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|)]\\ \le B[2\alpha +(L_0+2M_0)r_{*}](L_0+2M_0)r_{*}<1.\end{array}$$

Hence, ${\left(A_{{}^k}^T{A}_k\right)}^{-1}$ exists and

$$\begin{array}{c}\left|\right|{(A_{k+1}^T{A}_{k+1})}^{-1}\left|\right|\le g_k=B\{1-B[2\alpha +(L_0\left|\right|x_k-x^{*}\left|\right|\\ +M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|))]\\ \times (L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}{\left|\right|))\}}^{-1}\le g(r_{*}).\end{array}$$

Therefore, iteration $x_{k+1}$ is well defined, and the following estimate holds

$$\begin{array}{c}\left|\right|x_{k+1}-x^{*}\left|\right|=\left|\right|x_k-x^{*}-{(A_k^T{A}_k)}^{-1}(A_k^T(F(x_k)+G(x_k))-A_{*}^T(F(x^{*})+G(x^{*}))\left|\right|\\ \le \left|\right|-{(A_k^T{A}_k)}^{-1}\left|\right|\left|\right|([-A_k^T(A_k-{\int }_0^1{F}^{\prime }(x^{*}+t(x_k-x^{*}))dt\\ -G(x_k,x^{*}))(x_k-x^{*})+(A_k^T-A_{*}^T)(F(x^{*})+G(x^{*}))\left]\right||\\ \le g_k\{\left(\alpha +L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|)\right)\\ \times \frac{1}{2}\left(L∥x_k-x^{*}∥+2M∥x_{k-1}-x^{*}∥\right)\left|\right|x_k-x^{*}\left|\right|\\ +\eta \left(L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|)\right)\}\\ \le g(r_{*})\{\left(\alpha +L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|)\right)\\ \times \frac{1}{2}\left(L∥x_k-x^{*}∥+2M∥x_{k-1}-x^{*}∥\right)\left|\right|x_k-x^{*}\left|\right|\\ +\eta \left(L_0\left|\right|x_k-x^{*}\left|\right|+M_0(||x_k-x^{*}\left|\right|+\left|\right|x_{k-1}-x^{*}\left|\right|)\right)\}\\ <p(r_{*})r_{*}=r_{*}.\end{array}$$

That proves $x_{k+1}\in U(x^{*},r_{*})$ and estimate (24) for $n=k.$

Thus, method (5) is well defined, $x_n\in U(x^{*},r_{*})$ for all $n\ge 0$ and estimate (24) holds for all $n\ge 0$. It remains to prove that $x_n\to x^{*}$ for $n\to \infty$.

Define a and b on $[0,r_{*}]$ by

$$a(r)=g(r)((L_0+M_0)\eta +\alpha Lr/2+L(L_0+M_0)r^2/2)$$

(29)

and

$$b(r)=g(r)(M_0\eta +\alpha Mr+(2M_{0}M+L_{0}M+\frac{M_{0}L}{2})r^2).$$

(30)

According to $r_{*}$, we get

$$a(r_{*})\ge 0,b(r_{*})\ge 0,a(r_{*})+b(r_{*})=1.$$

(31)

Using estimate (24), the definitions of constants $C_i,i=1,2,\dots ,7$, and functions a and b, for $n\ge 0$ we get following

$$\begin{array}{c}\left|\right|x_{n+1}-x^{*}\left|\right|\le C_1\left|\right|x_{n-1}-x^{*}\left|\right|+C_2\left|\right|x_n-x^{*}\left|\right|+C_3{r}_{*}\left|\right|x_{n-1}-x^{*}\left|\right|\\ +C_4{r}_{*}^2\left|\right|x_{n-1}-x^{*}\left|\right|+C_5{r}_{*}\left|\right|x_n-x^{*}\left|\right|+C_6{r}_{*}^2\left|\right|x_{n-1}-x^{*}\left|\right|+C_7{r}_{*}^2\left|\right|x_n-x^{*}\left|\right|\\ =a(r_{*})\left|\right|x_n-x^{*}\left|\right|+b(r_{*})\left|\right|x_{n-1}-x^{*}\left|\right|.\end{array}$$

(32)

As it was shown in [1], under conditions (29)–(32) sequence {$x_n$} converges to $x^{*}$, as $n\to \infty$. □

Corollary 1.

In case of $\eta =0$, we have a nonlinear least squares problem with zero residual. Then, $C_1=0$ and $C_2=0$, and estimate (24) reduces to

$$\left|\right|x_{n+1}-x^{*}\left|\right|\le (C_3+C_4{r}_{*})\left|\right|x_{n-1}-x^{*}\left|\right|\left|\right|x_n-x^{*}\left|\right|+(C_5+C_6{r}_{*}+C_7{r}_{*})\left|\right|x_n-x^{*}{\left|\right|}^2.$$

That is method (5) converges with order $\frac{1+\sqrt{5}}{2}$.

Let $G(x)\equiv 0$ in (4), corresponding to the residual functions being differentiable. Then, from Theorem 1, we obtain the following corollary.

Corollary 2.

If $G(x)\equiv 0$, then in the conditions of theorem, we set, $M=0$, $C_3=0$, $C_4=0$, and estimate (24) reduces to:

$$\left|\right|x_{n+1}-x^{*}\left|\right|\le (C_5+C_6{r}_{*}+C_7{r}_{*})\left|\right|x_n-x^{*}{\left|\right|}^2.$$

Hence method (5) has a convergence order two.

Remark 2.

If $L_0=L=L_1$ and $M_0=M=M_1$ our results specialize to the corresponding ones in [9]. Otherwise, they constitute an improvement as already noted in the Remark 1. As an example let the $a_1,q_1,C_1^1,C_2^1,C_3^1,C_4^1,r_{*}^1$ denote the functions and parameter where $L_0,L,M_0,M$ are replaced by $L_1,L_1,M_1,M_1$ respectively. Then we have in view of (17)–(20) that

$$q(r)\le q_1(r),$$

$$g(r)\le g_1(r),$$

$$C_1\le C_1^1,$$

$$C_2\le C_2^1,$$

$$C_3\le C_3^1,$$

$$C_4\le C_4^1,$$

so

$$r_{*}^1\le r_{*},$$

$$B(L_1+2M_1)\eta <1\Rightarrow B(L_0+2M_0)\eta <1.$$

Consequently, the new sufficient convergence criteria are weaker than the ones in [9], unless, if $L_0=L_1$ and $M_0=M_1$. And moreover, the new error bounds are tighter than the corresponding ones in [9] and the rest of the advantages already mentioned in Remark 1 hold true.

The results can be improved even further, if (10) and (14) are replaced by

$$\left|\right|G(x,y)-G(x^{*},x^{*})\left|\right|\le K_0\left|\right|x-x^{*}\left|\right|+\overline{K_0}\left|\right|y-x^{*}\left|\right|,$$

(33)

and

$$\left|\right|G(x,y)-G(u,x^{*})\left|\right|\le N_0\left|\right|x-u\left|\right|+\overline{N_0}\left|\right|y-x^{*}\left|\right|,$$

(34)

respectively, since $K_0\le M_0$, $\overline{K_0}\le M_0$, $N_0\le M$ and $\overline{N_0}\le M$. We leave the details to the motivated reader.

4. Local Convergence Analysis (6)

Sufficient conditions and the rate of local converges of method (6) are defined in the following theorem.

Theorem 2.

Let $F+G:R^p\to R^m$ be continuous on set $D\subseteq R^p$, F be continuously differentiable in this set, and G be a function on D. Suppose, the problem (4) has a solution $x^{*}$ on set D, $F(x^{*})+G(x^{*})=0$ and the inverse operator ${(A_{*}^T{A}_{*})}^{-1}={\left[F^{\prime }{(x^{*})}^T{F}^{\prime }(x^{*})\right]}^{-1}$ exists and $\left|\right|{(A_{*}^T{A}_{*})}^{-1}\left|\right|\le B$. Fréchet derivative $F^{\prime }$ and function G satisfy Lipschitz conditions on set $D_0$

$$∥F^{\prime }(x)-F^{\prime }(x^{*})∥\le L_0∥x-x^{*}∥,$$

(35)

$$∥F^{\prime }(x)-F^{\prime }(y)∥\le L∥x-y∥,$$

(36)

$$∥G(x)-G(x^{*})∥\le M_0∥x-x^{*}∥.$$

(37)

Moreover,

$$∥F^{\prime }(x^{*})∥\le \alpha ;$$

(38)

$$B{M}_0\alpha <1$$

(39)

and $U(x^{*},r_{*})\subseteq D,$ where $r_{*}$ is unique positive zero of function q, defined by

$$q(r)=B[(\alpha +L_{0}r)(Lr+2M_0)/2]+B[2\alpha +L_{0}r]L_{0}r-1.$$

(40)

Then, for $x_0\in U(x^{*},r_{*})$ method (6) is well defined and generates the sequence $\{x_n\},n=0,1,\dots$ which belongs to set $U(x^{*},r_{*})$, and converges to the solution $x^{*}$. Moreover, the following error bounds hold

$$\left|\right|x_{n+1}-x^{*}\left|\right|\le C_1\left|\right|x_n-x^{*}\left|\right|+C_2\left|\right|x_n-x^{*}{\left|\right|}^2+C_3\left|\right|x_n-x^{*}{\left|\right|}^3,$$

(41)

where

$$\begin{array}{c}g(r)=B{[1-B(2\alpha +L_{0}r)L_{0}r]}^{-1};\\ C_1=g(r_{*})M_0\alpha ;C_2=g(r_{*})\left(L_0{M}_0+\frac{\alpha L}{2}\right);C_3=g(r_{*})\frac{L_{0}L}{2}.\end{array}$$

(42)

Proof. 

According to intermediate value theorem on $[0,r]$ for sufficiently large r and in view of (39) function q has a least positive zero, denoted by $r_{*}$, and $q^{\prime }(r)\ge 0$ for $r\ge 0.$ Indeed, this zero is unique on $[0,r]$. The proof analogous to the one given in Theorem 1.

Let $A_n=F^{\prime }(x_n)$, and set $n=0$. By assuming $x_0,x_{-1}\in U(x^{*},r_{*})$. By analogy to (26) in Theorem 1, we get

$$\left|\right|I-{(A_{*}^T{A}_{*}^{})}^{-1}{A}_0^T{A}_0\left|\right|\le B(\alpha ||A_{*}^{}-A_0\left|\right|+\left|\right|A_{*}^T-A_0^T\left|\right|\left|\right|A_0-A_{*}^{}\left|\right|+\alpha \left|\right|A_{*}^T-A_0^T\left|\right|).$$

(43)

Taking into account, that

$$\left|\right|A_0-A_{*}\left|\right|=\left|\right|F^{\prime }(x_0)-F^{\prime }(x^{*})\left|\right||\le L_0\left|\right|x_0-x^{*}\left|\right|,$$

(44)

from inequality (43), definition of $r_{*}$ given in (40), we get

$$\begin{array}{c}\left|\right|I-{(A_{*}^T{A}_{*}^{})}^{-1}{A}_0^T{A}_0\left|\right|\le B[2\alpha +L_0\left|\right|x_0-x^{*}\left|\right|]L_0\left|\right|x_0-x^{*}\left|\right|\\ \le B[2\alpha +L_0{r}_{*}]L_0{r}_{*}=\phi (r_{*})<1.\end{array}$$

(45)

From the Banach Lemma on invertible operators [3], and (45) $A_0^T{A}_0$ is invertible. Then, from (43)–(45), we get

$$\begin{array}{c}\left|\right|{(A_0^T{A}_0)}^{-1}\left|\right|\le g_0=B\{1-B[2\alpha +L_0\left|\right|x_0-x^{*}\left|\right|](L_0\left|\right|x_0-x^{*}{\left|\right|\}}^{-1}\\ \le g(r_{*})=B{\{1-B[2\alpha +L_0{r}_{*}]L_0{r}_{*}\}}^{-1}.\end{array}$$

Hence, iteration $x_1$ is well defined.

Next, we will show that $x_1\in U(x^{*},r_{*})$. We have the estimate

$$\begin{array}{c}\left|\right|x_1-x^{*}\left|\right|=\left|\right|x_0-x^{*}-{(A_0^T{A}_0)}^{-1}(A_0^T(F(x_0)+G(x_0))-A_{*}^T(F(x^{*})+G(x^{*}))\left|\right|\\ \le \left|\right|-{(A_0^T{A}_0)}^{-1}\left|\right|\left|\right|[-A_0^T(A_0-{\int }_0^1{F}^{\prime }(x^{*}+t(x_0-x^{*}))dt(x_0-x^{*})\\ -(G(x_0)-G(x^{*})))+(A_0^T-A_{*}^T)(F(x^{*})+G(x^{*}))\left]\right||.\end{array}$$

In view of the estimates

$$\begin{array}{c}\left|\right|(A_0-{\int }_0^1{F}^{\prime }(x^{*}+t(x_0-x^{*}))dt)(x_0-x^{*})-(G(x_0)-G(x^{*}))\left|\right|\\ =\left|\right|{\int }_0^1(F^{\prime }(x_0)-F^{\prime }(x^{*}+t(x_0-x^{*}))dt(x_0-x^{*})-(G(x_0)-G(x^{*}))\left|\right|\\ \le \left(\frac{1}{2}L\left|\right|x_0-x^{*}\left|\right|+M_0\right)\left|\right|x_0-x^{*}\left|\right|=\frac{1}{2}\left(L\left|\right|x_0-x^{*}\left|\right|+2M_0\right)\left|\right|x_0-x^{*}\left|\right|,\\ \left|\right|A_0\left|\right|\le \left|\right|A_{*}\left|\right|+\left|\right|A_0-A_{*}\left|\right|\le \alpha +L_0\left|\right|x_0-x^{*}\left|\right|,\end{array}$$

we obtain in turn that

$$\begin{array}{c}\left|\right|x_1-x^{*}\left|\right|\le B\{\left(\alpha +L_0\left|\right|x_0-x^{*}\left|\right|\right)\frac{1}{2}\left(L∥x_0-x^{*}∥+2M_0\right)\left|\right|x_0-x^{*}\left|\right|\}\\ /\{1-B[2\alpha +L_0\left|\right|x_0-x^{*}\left|\right|]L_0\left|\right|x_0-x^{*}\left|\right|\}\\ \le g_0\{\left(\alpha +L_0\left|\right|x_0-x^{*}\left|\right|\right)\frac{1}{2}\left(L∥x_0-x^{*}∥+2M_0\right)\left|\right|x_0-x^{*}\left|\right|\}\\ \le g(r_{*})\{\left(\alpha +L_0\left|\right|x_0-x^{*}\left|\right|\right)\frac{1}{2}\left(L∥x_0-x^{*}∥+2M_0\right)\left|\right|x_0-x^{*}\left|\right|\}\\ <g(r_{*})[(\alpha +L_0{r}_{*})(L{r}_{*}+2M_0)/2]r_{*}=r_{*}.\end{array}$$

Hence, $x_1\in U(x^{*},r_{*})$ and inequality (41) holds for $n=0$.

Suppose $x_n\in U(x^{*},r_{*})$ for $n=0,1,\dots ,k$ and estimate (41) holds for $n=0,1,\dots ,k-1$, where $k\ge 1$ is integer. Next, we show that $x_{n+1}\in U(x^{*},r_{*})$ and estimate (41) holds for $n=k$.

Then, we obtain

$$\begin{array}{c}\left|\right|I-{(A_{*}^T{A}_{*}^T)}^{-1}{A}_{{}^k}^T{A}_k\left|\right|=\left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}(A_{*}^T{A}_{*}^{}-A_{{}^k}^T{A}_k)\left|\right|\\ =\left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}(A_{*}^T(A_{*}^{}-A_k)+(A_{*}^T-A_{{}^k}^T)(A_k-A_{*}^{})+(A_{*}^T-A_{{}^k}^T)A_{*})\left|\right|\\ \le \left|\right|{(A_{*}^T{A}_{*}^{})}^{-1}\left|\right|(||A_{*}^T\left|\right|\left|\right|A_{*}^{}-A_k\left|\right|+\left|\right|A_{*}^T-A_{{}^k}^T\left|\right|\left|\right|A_k-A_{*}^{}\left|\right|+\left|\right|A_{*}^T-A_{{}^k}^T\left|\right|\left|\right|A_{*}^{}\left|\right|)\\ \le B(\alpha ||A_{*}^{}-A_k||+||A_{*}^T-A_{{}^k}^T|\left|\right||A_k-A_{*}^{}||+\alpha ||A_{*}^T-A_{{}^k}^T||)\\ \le B[2\alpha +L_0\left|\right|x_k-x^{*}\left|\right|]L_0\left|\right|x_k-x^{*}\left|\right|\le B[2\alpha +L_0{r}_{*}]L_0{r}_{*}<1.\end{array}$$

Hence, ${\left(A_{{}^k}^T{A}_k\right)}^{-1}$ exists and

$$\left|\right|{(A_{k+1}^T{A}_{k+1})}^{-1}\left|\right|\le g_k=B\{1-B[2\alpha +L_0\left|\right|x_k-x^{*}\left|\right|]L_0\left|\right|x_k-x^{*}{\left|\right|\}}^{-1}\le g(r_{*}).$$

Therefore iteration $x_{k+1}$ is well defined, and we get in turn that

$$\begin{array}{c}\left|\right|x_{k+1}-x^{*}\left|\right|=\left|\right|x_k-x^{*}-{(A_k^T{A}_k)}^{-1}(A_k^T(F(x_k)+G(x_k))-A_{*}^T(F(x^{*})+G(x^{*}))\left|\right|\\ \le \left|\right|-{(A_k^T{A}_k)}^{-1}\left|\right|\left|\right|([-A_k^T(A_k-{\int }_0^1{F}^{\prime }(x^{*}+t(x_k-x^{*}))dt\\ -G(x_k,x^{*}))(x_k-x^{*})+(A_k^T-A_{*}^T)(F(x^{*})+G(x^{*}))\left]\right||\\ \le g_k\{\left(\alpha +L_0\left|\right|x_k-x^{*}\left|\right|\right)\frac{1}{2}\left(L∥x_k-x^{*}∥+2M_0\right)\left|\right|x_k-x^{*}\left|\right|\}\\ \le g(r_{*})\{\left(\alpha +L_0\left|\right|x_k-x^{*}\left|\right|\right)\frac{1}{2}\left(L∥x_k-x^{*}∥+2M_0\right)\left|\right|x_k-x^{*}\left|\right|\}<r_{*}.\end{array}$$

That proves $x_k\in U(x^{*},r_{*})$, and estimate (41) for $n=k$.

Thus, iterative process (6) is well defined, $x_n\in U(x^{*},r_{*})$ for all $n\ge 0$ and estimate (41) holds for all $n\ge 0$.

Define function a on $[0,r_{*}]$

$$a(r)=g(r)(M_0\alpha +(\alpha L/2+L_0{M}_0)r+L_{0}L{r}^2/2).$$

(46)

Using estimate (41), the definitions of constants $C_i,i=1,2,3$ and function a, for $n\ge 0$, we get the following

$$\begin{array}{c}\left|\right|x_{n+1}-x^{*}\left|\right|\le C_1\left|\right|x_n-x^{*}\left|\right|+C_2{r}_{*}\left|\right|x_n-x^{*}\left|\right|+C_3{r}_{*}^2\left|\right|x_n-x^{*}\left|\right|\\ =a(r_{*})\left|\right|x_n-x^{*}\left|\right|.\end{array}$$

(47)

For any $r_{*}>0$ and initial point $x_0\in U(x^{*},r_{*})$, $r^{\prime }$ exists and $0<r^{\prime }<r_{*}$ such that $x_0\in U(x^{*},r^{\prime })$. Similarly to the proof that all iterates stay in $U(x^{*},r_{*})$, we show that all iterates stay in $U(x^{*},r^{\prime })$. So, estimation (47) holds, if $r_{*}$ is replaced by $r^{\prime }$. In particular, from (47) for $n\ge 0$, we get

$$\left|\right|x_{n+1}-x^{*}\left|\right|\le a\left|\right|x_n-x^{*}\left|\right|,$$

where $a=a(r^{\prime })$. Obviously $a\ge 0$, $a<a(r_{*})=1$. Therefore, we obtain

$$\left|\right|x_{n+1}-x^{*}\left|\right|\le a\left|\right|x_n-x^{*}\left|\right|\le \dots \le a^{n+1}\left|\right|x_0-x^{*}\left|\right|.$$

However, $a^{n+1}\to 0$ for $n\to \infty$. Hence, sequence {$x_n$} converges to $x^{*}$ as $n\to \infty$, with a rate of geometric progression. □

The same type of improvements as in Theorem 1 are obtained for Theorem 2 (see Remark 2).

Remark 3.

As we can see from estimations (41) and (42), convergence of method (6) depends on α, $L_0$, L and M. For problems with weak nonlinearity (α, $L_0$, L and $M_0$– “small”) convergence rate of iterative process is linear. In case of strongly nonlinear problems (α, $L_0$, L and/or $M_0$ – “large”) method (6) may not converge at all.

5. Numerical Experiments

Let us compare the convergence rate of combined method (5), Gauss-Newton type method (6) Secant-type method for solving nonlinear least squares problem [5,6] on some test cases with

$$\begin{array}{c}{x}_{n+1}=x_n-{\left(A_n^T{A}_n\right)}^{-1}{A}_n^T(F\left(x_n\right)+G(x_n)),\\ A_n=F\left(x_n,x_{n-1}\right)+G\left(x_n,x_{n-1}\right),n=0,1,\dots \end{array}$$

(48)

Testing is carried out on nonlinear systems with a nondifferentiable operator with zero and non-zero residual. Classic Gauss-Newton and Newton methods can not be used for solving such a problem. Results are searched with an accuracy $\epsilon ={10}^{-8}$. Calculations are performed until the following conditions are satisfied

$$\left|\right|x_{n+1}-x_n\left|\right|\le \epsilon and\left|\right|A_n^T(F\left(x_n\right)+G(x_n))\left|\right|\le \epsilon ,$$

in this case $f(x)=\underset{x\in R^n}{min}\frac{1}{2}{(F(x)+G(x))}^T(F(x)+G(x)).$

Example 1.

[11,12].

$$\left\{\begin{array}{c}3x^{2}y+y^2-1+|x-1|=0,\hfill \\ x^4+x{y}^3-1+\left|y\right|=0,\hfill \end{array}\right.$$

$$(x^{*},y^{*})\approx (0.89465537,0.32782652),f(x^{*})=0.$$

Example 2.

$n=2,m=3;$

$$\left\{\begin{array}{c}3x^{2}y+y^2-1+|x-1|=0,\hfill \\ x^4+x{y}^3-1+\left|y\right|=0,\hfill \\ |x^2-y|=0,\hfill \end{array}\right.$$

$$(x^{*},y^{*})\approx (0.74862800,0.43039151),f(x^{*})\approx 4.0469349·{10}^{-2}.$$

Remark 4.

The results of the numerical experiments are shown in the Table 1. In particular, we compare studied methods with respect to the number of iterations needed to find the solution with given accuracy. In Example 1, all methods converge to one solution. In Example 2 Gauss-Newton type method (6) converges to point $(x^{*},y^{*})\approx (0.89465537,0.32782652)$ with residual $f(x^{*})\approx 1.11666739·{10}^{-1}$, with the same number of iterations. Such iterations are marked with * symbol in the table. Other methods find the point $(x^{*},y^{*})\approx (0.74862800,0.43039151)$ with smaller residual $f(x^{*})\approx 4.0469349·{10}^{-2}$. Additional initial approximation $(x_{-1},y_{-1})$ is chosen as:

$$(x_{-1},y_{-1})=(x_0-{10}^{-4},y_0-{10}^{-4}).$$

Table 1. Number of iteration made to solve test problem.

Example	$({\mathit{x}}_0,{\mathit{y}}_0)$	Gauss-Newton Type (6)	Secant Type (48)	Combined Method (5)
1	(1, 0)	19	7	7
	(3, 1)	22	11	10
	(0.5, 0.5)	21	18	10
2	(1, 0)	19*	22	12
	(3, 1)	22*	25	15
	(0.5, 0.5)	21*	19	13

6. Conclusions

Based on the theoretical studies, the numerical experiments, and the comparison of obtained results, we can argue that the combined differential-difference method (5) converges faster than Gauss-Newton type method (6) and Secant type method (48). Moreover, the method has high convergence order $(1+\sqrt{5})/2$ in case of zero residual and does not require calculation of derivatives of the nondifferentiable part of operator. Therefore, the proposed method (5) solves the problem efficiently and fast.