Dimensionality of Iterative Methods: The Adimensional Scale Invariant Steffensen (ASIS) Method

Abstract

The dimensionality of parameters and variables is a fundamental issue in physics but is mostly ignored from a mathematical point of view. Difficulties arising from dimensional inconsistency are overcome by scaling analysis and, often, both concepts, dimensionality and scaling, are confused. In the particular case of iterative methods for solving nonlinear equations, dimensionality and scaling affect their robustness: While some classical methods, such as Newton’s, are adimensional and scale independent, some other iterations such as Steffensen’s are not; their convergence depends on scaling, and their evaluation needs a dimensional congruence. In this paper, we introduce the concept of an adimensional form of a function in order to study the behavior of iterative methods, thus correcting, if possible, some pathological features. From this adimensional form, we will devise an adimensional and scale invariant method based on Steffensen’s, which we will call the ASIS method.

1. Introduction

The main difference between absolute and relative errors in approximation is that while absolute errors share the dimension of the values they approximate, relative errors are adimensional and, therefore, independent from scaling. This trivial idea is often ignored when the parameter we want to approximate is the solution of an equation. From a numerical point of view, scaling is a fundamental feature, but, often, dimensional congruence is ignored. As a consequence of disregarding dimensionality, some methods devised originally for scalar, real, or complex equations are not useful or are hard to adapt to general vector Banach spaces. This is a reason why many iterative methods are not used in optimization (of course, storage and computational costs are also important).

In [1], a nice introduction to dimensional and scaling analysis is found. Although dimensional analysis goes back as far as Fourier in 1822, it was boosted in the second decade of last century, especially since the works of E. Buckingham ([2], for example) and most of them are from a physical rather than mathematical point of view. In fact, dimensionality is a fundamental concept in physics in order to determine the consistency of equations and magnitudes, but it is not so important in mathematics. The simple example of coefficients of polynomials illustrates this assertion.

Given $p\left(x\right)=a_0+a_{1}x+a_2{x}^2+\cdots +a_n{x}^n$, if x and $p\left(x\right)$ have dimensions denoted by $\left[x\right]$ and $\left[p\right]$ respectively, then the coefficients must have dimension $\left[a_k\right]=\left[p\right]{\left[x\right]}^{-k}=\left[p^{\left(k\right)}\left(x\right)\right]$. This obvious remark implies that $a_k$ is a constant magnitude but not a constant number (a change of scale both in x or in $p\left(x\right)$ modifies its value). Numerically, this means that we must be careful with scaling (if x is a length, a change from centimeters to inches, for instance, implies a change of coefficients), and one must take into account that $a_k=\frac{p^{\left(k\right)}\left(0\right)}{k!}$. Dimensional considerations do not come from the physical units we use but from the kind of magnitude we are measuring (let us say time, length, weight, etc.). To put it shortly, dimensionality is inherent to the variable while scaling is accidental and subject to changes.

In general, dimensional analysis spans over different and apparently unrelated topics in mathematics. It reaches its strength in nonlinear equations, where dimensionality and scaling are clearly separated. The purpose of this paper is the application of dimensional analysis of iterative methods for nonlinear equations.

We start by some basic notions about convergent sequences. It is well known [3] that a convergent sequence, ${\left\{x_k\right\}}_{k\ge 0}$, in a Banach space $(E,\Vert ·\Vert )$ with limit $x^{\ast }\in E$ has Q-order of convergence at least p if there exists a constant $K>0$ such that the following is the case.

$$\frac{\Vert x_{n+1}-x^{\ast }\Vert }{\Vert x_n-x^{\ast }{\Vert }^p}\le K$$

(1)

Therefore, K is not an adimensional parameter when $p\ne 1$. It has dimension $\left[K\right]={\left[x\right]}^{1-p}$. Any change of scale $y=cx$ changes the value of K. This fact, which could become dangerous, is annoying because there exists another weaker definition, R-order of convergence [3], that is scale invariant, where the sequence has an R-order of at least $p>1$ if there exists a constant K and an $n_0>0$ such that the following is the case.

$$\underset{n⟶\infty }{lim\; sup}\Vert x_n-x^{\ast }{\Vert }^{1/p^n}\le K$$

(2)

In this definition, K is adimensional ($\left[K\right]=\left[1\right]$), and there are no issues derived from scales or dimensions. In addition to the evident differences between both definitions, there is a conceptual motivation: Q-order controls the evolution of every term n, while R-order focuses on the global (limit) behavior of the sequence. From this point of view, it is not surprising that the Q-order is a more restrictive definition than that of R-order.

Here, we need to establish an order independent from dimensionality, as it happens with R-order, but that is still able to analyze convergence for any step n as Q-order. Thus, we will introduce an adimensional Q-order, which is scale independent and adimensional and is weaker than the Q-order but we retain its essential core. As a consequence, we will obtain an adimensional R-order, which is, in fact, a reformulation of the traditional R-order.

Now, we are ready to focus on iterative methods for solving nonlinear equations such as $F\left(x\right)=0$, when $F:E⟶V$, E, and V Banach spaces. The idea behind these methods is to correct a previous approximation $x_k$ to the solution $x^{\ast }$ by the following:

$$x_{k+1}=x_k+{\Delta }_k=x_k·(1+{\delta }_k)$$

(3)

$\left[{\Delta }_k\right]=\left[x\right]$. Therefore, ${\Delta }_k$, the absolute error, depends on x: Any change of scale or dimension in x leads to a similar change on ${\Delta }_k$. The relative error, ${\delta }_k$, is adimensional, and ${\delta }_k={\Delta }_k/x_k$ relates both definitions. As we said above, it is not surprising that, numerically, the first option is preferred, although ${\delta }_k$ is free from any changes of magnitude.

One of the simplest and best known methods is the first order one obtained when ${\Delta }_k=-cF\left(x_k\right)$ is under certain restrictions on c. Numerically, these restrictions are due to scale: $\Vert I-c{F}^{\prime }\left(x\right)\Vert <1$ for x in the domain. From a dimensional point of view, the following is the case.

$$\left[{\Delta }_k\right]=\left[c\right]\left[F\right]\Rightarrow \left[c\right]=\left[x\right]{\left[F\right]}^{-1}$$

Restrictions on the size of c are a consequence of its dimension, considering that c dependent of $F^{\prime }$ is more clarifying than thinking of c as a constant.

On the other hand, ${\Delta }_k=-F^{\prime }{\left(x_k\right)}^{-1}F\left(x_k\right)$ yields Newton’s, which is dimensionally consistent.

$$\left[{\Delta }_k\right]=\left[F\right]\left[F^{\prime }{}^{-1}\right]=\left[x\right](\mathrm{because}\left[F^{\prime }\right]=\left[F\right]{\left[x\right]}^{-1}).$$

Another second order method such as Steffensen’s is defined as follows:

$${\Delta }_k=-F{[x_k+F\left(x_k\right),x_k]}^{-1}F\left(x_k\right)$$

where $F[·,·]$ is the divided difference operator (bracket notation should not be confused with the dimension). Scaling is indeed a concern, but the most important feature is the inconsistency of dimensions. The sum $x+F\left(x\right)$ in the operator only makes sense if $E=V$, because $\left[x\right]$ and $\left[F\right]$ are not equal in general. This is one of the reasons why, although competitive with respect to Newton’s, Steffensen’s is less popular.

However, Steffensen’s has a clear advantage over Newton’s method: it is not necessary to evaluate Jacobians or gradients for Steffensen’s, because divided differences are usually obtained by the following interpolating condition:

$$F[x,y](x-y)=F\left(x\right)-F\left(y\right)$$

which does not require explicit derivatives.

The rest of this study will focus on two main goals. The first one is the concept of the adimensional form of a function, a tool we introduce, which keeps all information of the function, but it is invariant to changes of scales and dimensions of the variables. We will use these adimensional functions in a theoretical manner to obtain semilocal convergent results and, in practical applications, to obtain versions of nonrobust methods in order to make them scale independent. The second part of this paper deals with the analysis, development, and features of these modified methods.

In particular, we will prove the power of adimensional functions in the particular case of Steffensen’s method. We will devise an adimensional and scale invariant (ASIS) correction of Steffensen’s method and we will settle some sufficient semilocal conditions of convergence, from which optimal estimates of errors will be obtained. As a difference with previous published results, sufficient conditions of convergence will not be more restrictive than those of Newton’s. Furthermore, so far, optimality has not been obtained for Steffensen’s method in the literature. Thus, our estimates improve previous works in some aspects.

This paper is organized as follows: The next section will be divided into two subsections in which we review some basic and classical results on dimensionality and we introduce the adimensional form of functions; Section 3 will focus on iterative methods for nonlinear equations in Banach spaces; there, we also point out a fundamental theorem relating both worlds (dimensionality and iterative methods). In Section 4, we will apply the adimensional form of polynomials and functions to the analysis of semilocal convergence of iterative methods in Section 5. In Section 6, we devise new variations of classic scale-dependent iterative methods, such as Steffensen’s, in order to eliminate their scale and dimensional dependence. Section 7 will illustrate, by using examples, the theoretical results in this paper. Finally, we draw conclusions in Section 8.

2. Basic Concepts on Dimensionality

Dimension of a variable is a physical concept related to the units in which that variable is measured. It is one of the concepts that is easier to understand than to define. A physical quantity may represent time, length, weight, or any other magnitude. Equations in physics relate one or more of these dimensions and they must show consistency: dimensions in both sides of the equation must be equal. Nevertheless, this trivial remark is often ignored because either there is no evident method to obtain the dimension of any term or, as it happens in mathematics, good scaling may overcome the potential risks of dimensionality.

However, dimensionality and scale are not always equivalent. For example, changing the measure of a length from, say, centimeters to inches is a change of scale (and very dangerous indeed), but their dimensions are the same: length. Inconsistency happens when there are different measures involved in each term of the equation (time versus length, for instance). The problems increase when the equation is not scalar but is defined in Banach spaces. In this case, it is easy to confuse dimensionality and dimension (two vectors of the same dimension may belong to different Banach spaces and have different dimensionality even if there exists an isomorphism between both), leading to some problems in areas such as optimization.

In the rest of this section, we look over some aspects of dimensional analysis in Section 2.1, which will drive us to introduce the adimensional form of functions in Section 2.2.

2.1. Dimensionality

From this point onward, we will use classical notation, $\left[x\right]$, for the dimension of a variable x. If z is an adimensional parameter, we will denote it by $\left[z\right]=\left[1\right]$. Unlike classic dot notation in physics, we will denote the derivatives of a function $f\left(x\right)$ by $f^{\prime }\left(x\right)$, $f^{″}\left(x\right)$, … as is normally used in mathematics. We will omit variables unless we shall explicate them due to the context.

For vectors and Banach spaces in general, the functions will be denoted by capital letters ($F\left(x\right)$, $G\left(x\right)$, …), and the vectors x, y, … will use the same notation as for scalars (there shall not be any confusion). $F^{\prime }\left(x\right)$ and $F^{″}\left(x\right)$ will denote the Jacobian and Hessian of $F\left(x\right)$, respectively.

$F:E⟶V$, where $(E,\Vert ·\Vert )$ and $(V,\Vert ·\Vert )$ are Banach spaces. Their norms, although different, will be denoted with the same notation (once again, it shall not be confusing).

It is not intuitive to introduce dimensionality in nonscalar Banach spaces. When the space is finite dimensional, we define $\left[x\right]$ as a vector where every component has the dimension of the corresponding element of the base. Thus, if $x=(x_1,\cdots ,x_m)$, then $\left[x\right]=(\left[x_1\right],\cdots ,\left[x_m\right])$. We must consider other operations in addition to the sum and multiplication by scalars in the context of nonlinear equations. Hence, nonlinear operations must be carried pointwise (component by component). For example, $x\mathrm{o}y=(x_1\mathrm{o}{y}_1,\cdots ,x_m\mathrm{o}{y}_m)$, and $\left[x\mathrm{o}y\right]=(\left[x_1\mathrm{o}{y}_1\right],\cdots ,\left[x_m\mathrm{o}{y}_m\right])$, for any operation $\mathrm{o}$.

For infinite dimensional Banach spaces, dimensionality loses its physical sense, and it is a more complicated concept. For the purpose of this paper, we consider dimensionality in a similar manner as finite dimensional vector spaces: here, $\left[x\right]$ is an infinite vector, and its operations are also performed pointwise. There is no misunderstanding when all the bases have the same dimensionality although, in general, the dimensional vectors may not be homogeneous.

Elementary rules of dimensionality apply.

$$[x·y]=\left[x\right]\left[y\right];[x/y]=\left[x\right]{\left[y\right]}^{-1};\left[x^n\right]={\left[x\right]}^n$$

Rules for differentiability and integration are particularly interesting.

$$\left[F^{\prime }\right]=\left[F\right]/\left[x\right]=\left[F\right]{\left[x\right]}^{-1};\left[\int F(x)\mathrm{d}x\right]=\left[F\right]\left[x\right]$$

2.2. Adimensional Polynomials and Functions

Dimensionality may affect the evaluation of variables, functions, and their derivatives. Any equation with terms involving them must be carefully managed in order to not lose physical meaning. For example, terms such as $x+f\left(x\right)$ or $cf\left(x\right)-f^{\prime }\left(x\right)$ can only make sense, from a dimensional point of view, if both terms in the sums are dimensionally homogeneous.

One of the consequences of careless handling of dimensionality is that stability of equations may be affected by rescaling, if the terms do not share the same dimensionality.

In order to avoid possible difficulties due to dimensionality, we propose confronting adimensional equations: Any function and variable will be reduced to adimensional forms by linear transforms; all the terms will have dimensionality [1]. Thus, dimensional and scaling issues will appear only in linear transforms and not in the resolution of the equation.

The following definitions will clear the concept and process of the above proposal.

Definition 1.

Given a (real or complex) scalar nonlinear function $f\left(x\right)$, such that both $f\left(0\right)\ne 0$ and $f^{\prime }\left(0\right)\ne 0$, theadimensional normalized formof $f\left(x\right)$ is a function $g\left(y\right)$ such that $g\left(0\right)=1$, $g^{\prime }\left(0\right)=-1$, and $y=-(f^{\prime }\left(0\right)/f\left(0\right))x$.

Then, given $f\left(x\right)$ such that $f\left(0\right)\ne 0$ and $f^{\prime }\left(0\right)\ne 0$, the following is the case.

$$f\left(x\right)=f\left(0\right)\left(g\left(-\frac{f^{\prime }\left(0\right)}{f\left(0\right)}x\right)\right)$$

We notice that $\left[f\right(0\left)\right]\left[g\right]=\left[f\right]$ and, therefore, $\left[g\right]=\left[1\right]$ and $\left[y\right]=\left[f^{\prime }\right]{\left[f\right]}^{-1}\left[x\right]=\left[1\right]$. Thus, both y and $g\left(y\right)$ are adimensional scale independent, which is the goal of this definition.

Some remarks on this definition are provided here. In the first place, it is not necessary to normalize the function in $x_0=0$. If $f\left(0\right)=0$, $f^{\prime }\left(0\right)=0$, or it is convenient not to use $x_0=0$, any other value of $x_0$ can be useful. On the other hand, the choice of the minus sign on $g^{\prime }\left(0\right)$ and y is not relevant, but it simplifies developments in the rest of the paper. Finally, let us say that normalization is not a strict requirement. Approximate normalization ($g\left(0\right)\approx 1$, $g^{\prime }\left(0\right)\approx -1$) is a realistic setting when errors appear in finite precision systems. Furthermore, in some cases, normalization will not be required at all.

For general Banach spaces, the definition above can be easily generalized.

Definition 2.

Let $E,V$ be Banach spaces, $U\subset E$, $F:U⟶V$, a differentiable function, and $x_0\in U$ such that $F^{\prime }\left(x_0\right)$ is invertible, theadimensional normalized formof $F\left(x\right)$ is the following function:

$$G\left(y\right)=\frac{F\left(x\right)}{\Vert F\left(x_0\right)\Vert }$$

(4)

where the following is the case:

$$y=-\frac{F^{\prime }\left(x_0\right)}{\Vert F\left(x_0\right)\Vert }x$$

and the following is obtained.

$$y_0=-\frac{F^{\prime }\left(x_0\right)}{\Vert F\left(x_0\right)\Vert }{x}_0$$

(5)

It is verified: $\Vert G\left(y_0\right)\Vert =1$ and $G^{\prime }\left(y_0\right)=-I$.

The remarks we pointed out in the scalar case are also valid here.

Let us notice that the change of variables from x to y is also a change of space from E to an adimensional space $\tilde{V}$ homeomorphic to V. $\tilde{V}$ is an example showing that dimension and dimensionality are different: it has the same dimension as V but, for any $y\in \tilde{V}$, $\left[y\right]=1$. As $G:\tilde{V}⟶\tilde{V}$ is an endomorphism, $y+G\left(y\right)$ is a consistent sum in $\tilde{V}$ as opposed to the general case. As we do not have to worry about dimensionality or scales, adimensional functions may speed up and regularize the computations.

Adimensional functions are not only useful for iterative approximations. For example, the well-known pathological zig-zag optimization problem ([4], p. 348) consisting of minimizing function $H(x,y)=(x^2+b{y}^2)/2$ is implied to solve, via the steepest descent (a first-order method similar to (6) system: $F(x,y)=(F_1(x,y),F_2(x,y))=(x,by)=(0,0)$), a very slow convergence when b is small. Actually, its rate of convergence is as follows.

$$H(x_{n+1},y_{n+1})={\left(\frac{1-b}{1+b}\right)}^{2}H(x_n,y_n)$$

However, its adimensional form is $G(u,v)=-(x,y)/\Vert F(x_0,y_0)\Vert$, being $u=x/\Vert F(x_0,y_0)\Vert$, $v=b·y/\Vert F(x_0,y_0)\Vert$, and it converges in only one step.

Another example of how adimensionality can help improve the performance of some algorithms is the evaluation of matrix functions. In particular, Newton’s iteration for approximating the square root of a matrix A, from an initial invertible guess $X_0$, provides the sequence of approximates ${\left\{X_k\right\}}_{k\ge 0}$ for $k\ge 0$.

$$X_{k+1}=\frac{1}{2}(X_k+X_k^{-1}A)$$

Although, theoretically, this sequence converges to $A^{1/2}$, it is well known ([5]) that it may not converge computationally if the condition number of A is greater than 9. If $X_0$ commutes with A, the adimensional form of the sequence becomes the following.

$$Y_{k+1}=\frac{1}{2}(Y_k+Y_k^{-1}{X}_0^{2}A)$$

$$X_k=X_0^{-1}{Y}_k$$

It is seen that this adimensional recurrence is equivalent to a preconditioning of the matrix A by multiplying times $X_0^2$. Adimensioning is equivalent to regularization, and some preconditioning techniques appearing in ([6]) can be considered as particular cases of adimensional matrices. Furthermore, in the adimensional case, it is not necessary to consider $X_0$ as the regularizer. Any other matrix C can be considered and the following is also a valid adimensional recurrence.

$$Y_{k+1}=\frac{1}{2}(Y_k+Y_k^{-1}{C}^{2}A)$$

$$X_k=C^{-1}{Y}_k$$

If A is symmetric and C is diagonal, commutativity is given. If commutativity is not granted, a Lyapunov-type of equation may arise and some deeper analysis must be performed, although, even in such noncommutative case, adimensionality translates also as a preconditioning of A.

Similar reasoning follows for the other matrix functions.

Before proceeding, we will review some basics on iterative methods. This classic topic has been treated in most elementary books on iterative methods ([7,8], for instance).

3. Iterative Methods for Nonlinear Equations

Iterative methods for solving a nonlinear scalar equation $f\left(x\right)=0$ are based on the simple idea stated in the Introduction: We start by approximating value $x^{\ast }$ such that $f\left(x^{\ast }\right)=0$ by an initial guess $x_0$, and we correct the approximations by recurrence, $x_{k+1}=x_k+{\Delta }_k$. Thus, a sequence ${\left\{x_n\right\}}_{n\ge 0}$ is obtained. If that sequence converges to a limit $x^{\ast }$, we obtain the solution of the equation.

Of course, this idea becomes more and more complicated in practice. In order to obtain convergence, the corrections ${\Delta }_k$ must be well devised (depending on function $f\left(x\right)$ and, often, its derivatives, if they exist), and some restrictions must apply on the own function and the initial guess. On the other side, a stopping criterion is needed in order to obtain a good approximation relative to the root.

The accuracy of the method depends not only on the design (consistence) of the recurrence but on the stability (sensitivity to errors) and the speed of convergence. There are different methods to define the speed of a sequence, but we point out Q-order and R-order defined in (1) and (2), respectively.

Perhaps the easiest iterative method is the bisection. Its approximates not values but intervals. However, a point-wise version of this method can be obtained if we consider the middle point of the interval where the function $f\left(x\right)$ switches signs. As the intervals are divided every step, a choice (an if) must be performed every step. No other information about $f\left(x\right)$ is needed but the sign. When there is a change of sign, it has first-order convergence.

$$\Vert x_{n+1}-x^{\ast }\Vert \approx \frac{1}{2}\Vert x_n-x^{\ast }\Vert$$

The more we know about $f\left(x\right)$, the more sophisticated methods can be obtained. By carefully choosing a constant $c\in \mathbb{R}$, the recurrence of the following:

$${\Delta }_k=-c·f\left(x_k\right)$$

(6)

is a first-order method.

As we saw in the introduction, we can eliminate dimensionality of parameter c by introducing this first-order method:

$$x_{n+1}=x_n-\frac{\lambda }{f^{\prime }\left(x_0\right)}f\left(x_k\right)$$

(7)

where $\lambda$ is adimensional, but some scale requirements are still needed: $0<\left|{\displaystyle \lambda \frac{f^{\prime }\left(x\right)}{f^{\prime }\left(x_0\right)}}\right|<2$. Improvement comes from the fact that, once $\lambda$ is fixed, any rescaling on x or $f\left(x\right)$ does not affect $\lambda$.

However, the main method around which all theory has evolved is Newton’s, where the following is the case.

$${\Delta }_k=-\frac{f\left(x_k\right)}{f^{\prime }\left(x_k\right)}$$

(8)

This needs more requirements than the other two ($f\left(x\right)$ must be differentiable and its derivative must be invertible), and its convergence is usually local, but when it works, it is a second order method. More importantly, it provides insight on this topic. A great amount of papers have been published on Newton’s method. Since Kantorovich settled conditions for convergence of this method in Banach spaces ([9]), a lot of papers have appeared that analyzed this method, and some of them are classics ([10,11,12] among others) while new and different forms of variants and analysis are also frequent ([13,14,15,16] to cite just a few). In [14], an updated bibliography can also be found).

In the cited bibliography, some variants of Newton provided in the literature can be found: quasi-Newton in optimization [17] and approximated Newton in computational analysis are some of these versions [18,19].

Let us focus on quasi-Newton methods. They are based on the approximation of the derivative by means of divided differences. For scalar functions, a divided difference operator in two nodes x and y, denoted by $f[x,y]$, is defined as follows:

$$f[x,y]=\frac{f\left(x\right)-f\left(y\right)}{x-y}$$

(9)

where $f[x,y]$ verifies the interpolatory condition.

$$f[x,y](x-y)=f\left(x\right)-f\left(y\right)$$

If $f\left(x\right)$ is at least twice differentiable, divided differences approximate the derivative of $f\left(x\right)$:

$$f[x,y]=f^{\prime }\left(x\right)+f^{″}(x+\xi (y-x))(y-x)$$

for $\xi \in [0,1]$.

Thus, an adequate choice of the nodes is a good strategy to obtain an approximation to Newton’s method when derivatives are unknown or costly.

By choosing $x_{k-1},x_k$ as nodes, the secant method is obtained.

$${\Delta }_k=-f{[x_{k-1},x_k]}^{-1}f\left(x_k\right)$$

(10)

In spite of a small reduction in the order of convergence (it is a superlinear method with order of convergence $(1+\sqrt{5})/2$) and an increase in storage in memory (a previous approximation must be kept), the fact that derivatives are not explicitly needed and that it only requires one additional evaluation of $f\left(x\right)$ for every iteration constitutes reasons why this is a very popular method.

On the other side, the dimension of $f[x_{n-1},x_n]$ is $\left[f\right]{\left[x\right]}^{-1}=\left[f^{\prime }\right]$, and the secant method does not introduce issues on scales and dimensions.

Finally, our goal in this paper is addressed to Steffensen’s method. It is a quasi-Newton method in which the nodes are x and $x+f\left(x\right)$.

$${\Delta }_k=-f{[x_k+f\left(x_k\right),x_k]}^{-1}f\left(x_k\right)$$

(11)

This method is second order, as Newton’s, and it requires only two evaluations of $f\left(x\right)$, but the derivative is not needed. However, in spite of the amount of literature it has generated (as in [20,21,22,23,24]), it is not very popular and is often dismissed as an alternative to other classical methods. The reasons, even though not always explicit, are related to dimensionality.

As opposed to the other methods we have outlined in this section (except perhaps (6)), Steffensen’s is scale dependent. The sum $x+f\left(x\right)$ depends strongly on the scales of both x and $f\left(x\right)$. A change of scale on x and $cx$ or $f\left(x\right)$ and $kf\left(x\right)$ transforms $x+f\left(x\right)$ in $cx+kf\left(x\right)$, which is not a change of scale in the node.

Some variations have been proposed in order to adjust this method. One of the most rediscovered is the modification of node $x+f\left(x\right)$ by shrinking (or dilating) the function times a constant $\alpha$: $x+\alpha f\left(x\right)$. The idea behind this is the same useful resource after many scaling problems: a small value of $\left|\alpha \right|$ provides a better approximation of $f[x+\alpha f(x),x]$ to the exact derivative $f^{\prime }\left(x\right)$ (see [20]).

While being a good alternative to the method, the problem is deeper. It is not a scaling but a dimensional problem: $\left[x\right]$ and $\left[f\right]$ are usually different, and their sum is inconsistent.

Part of the problem can be solved by considering $\alpha$ not as an adimensional factor but as $\left[\alpha \right]=\left[x\right]{\left[f\right]}^{-1}$ (that is, with the same dimension as $f^{\prime }{\left(x\right)}^{-1}$). In this case, difficulties may arise from the definitions of $\alpha$, which can vary greatly if the derivative also varies. A possibility is to use the same idea we used in (7).

$$f[x,x+\lambda f\left(x\right)/f^{\prime }\left(x_0\right)]$$

Here, $\lambda$ is an adimensional constant, and it is not influenced by a change of scale.

To conclude, bisection and Newton’s and secant methods are robust due to their scale independence, while the first-order one (6) and Steffensen’s are not and some adjustments must be performed in order to keep them practical.

Emphasis on the importance of Newton’s is not a subjective appreciation. The following theorem shows a dimensional aspect of this method, which deserves to be highlighted:

Theorem 1.

Under the notation in this paragraph, if $f\left(x\right)$ is at least once differentiable, then, any iterative method in the form (3) must verify the following dimensional relation.

$$\left[{\Delta }_k\right]=\left[\frac{f\left(x_k\right)}{f^{\prime }\left(x_k\right)}\right]$$

As a consequence, any iterative method can be represented as follows:

$${\Delta }_k=-h\left(x_k\right)\frac{f\left(x_k\right)}{f^{\prime }\left(x_k\right)}$$

(12)

where $h\left(x\right)$ is an adimensional function, $\left[h\right]=\left[1\right]$.

The proof is evident because $\left[{\Delta }_k\right]=\left[x\right]=[f/f^{\prime }]$. However, some of its consequences are not evident: for instance, it is not trivial to think of bisection as an adimensional modification of Newton’s method.

Moreover, by taking into account that $[f/f^{\prime }]=\left[x\right]$, its derivative of the following:

$${\left(\frac{f\left(x\right)}{f^{\prime }\left(x\right)}\right)}^{\prime }=1-\frac{f^{″}\left(x\right)f\left(x\right)}{f^{\prime }{\left(x\right)}^2}=1-L_f\left(x\right)$$

is adimensional. $L_f\left(x\right)$ is called the degree of logarithmic convexity, and (12) can be expressed as follows.

$${\Delta }_k=-h\left(L_f\left(x_k\right)\right)\frac{f\left(x_k\right)}{f^{\prime }\left(x_k\right)}$$

(13)

By differentiating (13) and canceling derivatives, we obtain this well-known characterization of second and third-order algorithms [25]:

Corollary 1.

An iterative method such as (13) is second order if $h\left(0\right)=1$ and third order if, in addition, $h^{\prime }\left(0\right)=1/2$.

From Theorem 1, higher-order characterizations can be obtained, depending on higher derivatives of $L_f\left(x\right)$.

4. Adimensional Semilocal Analysis for Iterative Methods

Previous sections provide some insight on the relation between iterative methods, scaling, and dimension. How do linear changes in x or $f\left(x\right)$ affect iterations?

As we stated above, Newton and secant are independent of the scale.

If $y=cx$, $g\left(y\right)=kf\left(y\right)$, and $y_k=c{x}_k$, then the following is the case.

$$y_{k+1}=y_k-\frac{g\left(y_k\right)}{g^{\prime }\left(y_k\right)}=c\left(x_k-\frac{f\left(x_k\right)}{f^{\prime }\left(x_k\right)}\right)=c{x}_{k+1}$$

Rescaling $f\left(x\right)$ does not modify the iteration in the sense that a change of scale in variable x produces the same rescaling on the iterates.

Similar behavior shows the secant method and even bisection. However, as we saw above, first order (6) and Steffensen’s are different, and we must redefine the variables or the parameters in order to be rid of scaling troubles.

In order to analyze Steffensen’s method, some preliminary operations must be performed in the estimation of the errors of iterative methods.

There are two principles to ensure the convergence of the methods in Banach spaces. One of them is the majorizing sequence and the other one is the majorizing function.

Definition 3.

Given a sequence ${\left\{x_n\right\}}_{n\ge 0}\subset E$ (E a Banach space), an increasing sequence ${\left\{t_n\right\}}_{n\ge 0}\subset \mathbb{R}$ is a majorizing sequence if, for all $k\ge 0$, the following is the case.

$$\Vert x_{k+1}-x_k\Vert \le t_{k+1}-t_k$$

Convergence of ${\left\{t_n\right\}}_n$ implies convergence of ${\left\{x_n\right\}}_n$ and, if $t^{\ast }$ and $x^{\ast }$ are their limits, $\Vert x^{\ast }-x_k\Vert \le t^{\ast }-t_k$.

Majorizing sequences are often obtained by applying the iterative method to majorizing functions, which are defined as follows.

Definition 4.

Given $F:E⟶V$, a real function $f\left(t\right)$ is a majorizing function of $F\left(x\right)$ if for an $x_0\in E$, for any x in a neighborhood of $x_0$, $\Vert F\left(x\right)\Vert \le \left|f\right(\Vert x-x_0\Vert \left)\right|$.

Newton’s method ($x_{n+1}=x_n-F^{\prime }{\left(x_n\right)}^{-1}F\left(x_n\right)$) illustrates semilocal convergence analysis. Kantorovich established conditions on the initial guess, $x_0$, in order to obtain convergence of Newton’s. These conditions were related to the boundness of the following variables.

Theorem 2.

Let $F:U⟶V$ be a differentiable function defined in a neighborhood, $U\subset E$, and $x_0\in U$ such that $F^{\prime }\left(x_0\right)$ is invertible, and there exist three positive constants $K_2,\eta ,B>0$ verifying the following:

1. 

$\Vert F^{″}\left(x\right)\Vert \le K_2$, for all $x\in U$;

2. 

$\Vert F^{\prime }{\left(x_0\right)}^{-1}\Vert \le B$;

3. 

$\Vert F^{\prime }{\left(x_0\right)}^{-1}F\left(x_0\right)\Vert \le \eta$.

Then, if $K_{2}B\eta \le 1/2$, Newton’s method from $x_0$ converges.

The proof can be found in classical textbooks [7,8]. For our purposes, we point out a key step in this proof:

$$p\left(t\right)=\frac{K_2}{2}{t}^2-\frac{1}{B}t+\frac{\eta }{B}$$

where it is a majorizing function, when starting from $t_0=0$. Condition $K_{2}B\eta \le \frac{1}{2}$ means that the discriminant of $p\left(t\right)$ is greater or equal to zero; therefore, $p\left(t\right)$ has two simple positive roots or a double positive root.

Dimensional analysis simplifies this polynomial. The adimensional form of $p\left(t\right)$ is the following:

$$q\left(s\right)=\frac{a}{2}{s}^2-s+1$$

where $a=K_{2}B\eta$, ${\displaystyle s=\frac{t}{\eta }}$, and ${\displaystyle p\left(t\right)=\frac{B}{\eta }q\left(s\right)}$.

As $q\left(s\right)$ is invariant from scales, any change of scale implies changes on $K_2$, B, or $\eta$ but not on a and, hence, on $q\left(s\right)$. Thus, control of the convergence of the method depends on parameter a.

For third-order iterative methods, such as Halley’s, Chebyshev’s, or Euler’s, it is useful to consider the cubic majorizing polynomial of the following ([26,27]):

$$p\left(t\right)=\frac{K_3}{6}{t}^3+\frac{K_2}{2}{t}^2-\frac{1}{B}t+\frac{\eta }{B}$$

assuming that the third-order derivative is bounded $|F^{‴}\left(x\right)|\le K_3$ for all x in the domain.

The adimensional form of $p\left(t\right)$ is as follows: ${\displaystyle q\left(s\right)=\frac{b}{6}{s}^3+\frac{a}{2}{s}^2-s+1}$.

$$(b=K_{3}B{\eta }^2,a=K_{2}B\eta ).$$

Convergence is obtained, as for Newton’s, when $q\left(s\right)$ has two simple positive roots or one positive double root (there is always a negative root).

In [26,27], a different method to study convergence is introduced. The idea is to obtain a system of sequences, two of them being fundamental because they control both iterates $x_k$ and their values $F\left(x_k\right)$ and some auxiliary sequences controlling different aspects of the iteration, if needed. The main advantage of these systems is that the sequences are adimensional, and the error estimates do not depend on the scales. We will use that technique in the next section applied to Steffensen’s method.

5. A System of a Priori Error Estimates for Newton’s Method

We start by illustrating the use of systems of bounds in the last paragraph and applying it to Newton’s method. We refer the interested reader to [26] for details. Let us consider the following system of sequences.

Let $a>0$, $a_0=1$, and $d_0=1$ for all $n\ge 0$. By recurrence, we define the following sequences.

$$\begin{array}{cccc}\hfill \left(\mathbf{i}\right)& a_{n+1}=\frac{a_n}{1-a{a}_n{d}_n}\hfill & \hfill \hspace{1em}\hspace{1em}\hspace{1em}\left(\mathbf{ii}\right)& d_{n+1}=\frac{a}{2}{a}_{n+1}{d}_n^2\hfill \end{array}$$

(14)

This system is said to be positive if $a_n>0$ for all $n\ge 0$. It is easy to prove that the system is positive if and only if $0<a<1/2$. Positivity and convergence of ${\sum }_{n\ge 0}{d}_n$ are equivalent.

It can be checked that, for all $n\ge 0$, the following is the case.

$${\left(\frac{1}{a_n}\right)}^2-a\frac{d_n}{a_n}=1-2a$$

(15)

Therefore, $a_n$ depends on $d_n$, and the following is the case.

$$d_{n+1}=\frac{a}{2}\frac{d_n^2}{a{d}_n+\sqrt{{\left(a{d}_n\right)}^2+(1-2a)}}$$

(16)

This function relating $d_n$ and $d_{n+1}$ is called the rate of convergence in [28].

Thus, we obtain the following.

$$\sum _{n=0}^{+\infty }{d}_n=\frac{1-\sqrt{1-2a}}{a}=:s^{\ast }$$

(17)

Now, this theorem provides semilocal conditions for the convergence of Newton’s method in Banach spaces ($(E,\Vert ·\Vert )$, $(V,\Vert ·\Vert )$).

Theorem 3.

Given a function $F:E⟶V$ that is at least twice differentiable, $x_0\in E$ such that $F^{\prime }\left(x_0\right)$ is invertible, and there exists positive constants $K_2,B,\eta$ verifying:

$$\begin{array}{cc}\hfill \left(\mathbf{i}\right)& a=K_{2}B\eta \le 1/2\hfill \\ \hfill \left(\mathbf{ii}\right)& \Vert F^{″}\left(x\right)\Vert \le K_{2}forallx\in E\hfill \\ \hfill \left(\mathbf{iii}\right)& \Vert F^{\prime }{\left(x_0\right)}^{-1}\Vert \le B\hfill \\ \hfill \left(\mathbf{iv}\right)& \Vert F^{\prime }{\left(x_0\right)}^{-1}F\left(x_0\right)\Vert \le \eta \hfill \end{array}$$

Then, Newton’s method ($x_{n+1}=x_n-F^{\prime }{\left(x_n\right)}^{-1}F\left(x_n\right)$) converges to a root, $x^{\ast }$, of $F\left(x\right)$ and the following inequalities hold:

1. 

$\Vert F^{\prime }{\left(x_n\right)}^{-1}\Vert \le a_{n}B$, for all $n\ge 0$;

2. 

$\Vert x_{n+1}-x_n\Vert \le d_n\eta$, for all $n\ge 0$;

3. 

$\Vert x^{\ast }-x_0\Vert \le s^{\ast }\eta$;

4. 

$\Vert x^{\ast }-x_n\Vert \le (s^{\ast }-{\sum }_{k=0}^{n-1}{d}_k)\eta$, for all $n\ge 1$.

$\{a_n,d_n\}$ is the system generated by (14) from a. Furthermore, $x^{\ast }$ is the only root of $F\left(x\right)$ in the ball $\mathcal{B}(x_0,s^{\ast \ast }\eta )$, where the following is the case.

$$s^{\ast \ast }:=\frac{1+\sqrt{1-2a}}{a}$$

The proof of this theorem is a direct consequence of the adimensional polynomial $q\left(d\right)=(a/2)s^2-s+1$ related to the majorizing one: $p\left(t\right)=(K_2/2)t^2-t/B+\eta /B$. Under the hypothesis of the theorem, $q\left(s\right)$ has two positive roots, $s^{\ast }\le s^{\ast \ast }$, and Newton’s sequence is as follows: $s_0=0$, $s_{n+1}=s_n-q\left(s_n\right)/q^{\prime }\left(s_n\right)$ converges monotonically to $s^{\ast }$ and verifies $s_{n+1}-s_n=d_n$ and $-1/q^{\prime }\left(s_n\right)=a_n$.

The adimensional polynomial explains also why (15) and (17) are verified: The value $q^{\prime }{\left(s\right)}^2-4q^{″}\left(s\right)q\left(s\right)$ is an invariant for any quadratic polynomial, and its value is the discriminant, which in this case is $1-2a$. Moreover, $s^{\ast }=lim\left\{s_n\right\}={\sum }_n{d}_n$.

On the other side, it is easy to check that if $a\ne 0$:

$$\sum _{k=0}^{n-1}{d}_k=\frac{1}{a}(1-\frac{1}{a_n})$$

given that $-1/a_n=q^{\prime }\left(s_n\right)=a{s}_n-1$ and $s_n={\sum }_{k=0}^{n-1}{d}_k$.

As we said in the Introduction, assymptotical constants of convergence from classical Q- and R-orders of convergence depend on dimensionality and scaling of the equation. In order to analyze the speed of convergence, adimensional functions and sequences provide a new characterization of order of convergence, which is stronger than classical R-order but weaker than Q-order, with the advantage that their definitions are adimensional and independent from scales.

Definition 5.

Let us assume that $(E,\Vert ·\Vert )$ is a Banach space and ${\left\{x_n\right\}}_n\subset E$. Then, p is theadimensional Q-order(respectively,adimensional R-order) of ${\left\{x_n\right\}}_n$ if there exists an adimensional scalar sequence ${\left\{d_n\right\}}_n$ with Q-order (respectively, R-order) p and a constant η with $\left[\eta \right]=\left[x_0\right]$ such that the following is the case.

$$\Vert x_{n+1}-x_n\Vert \le d_n\eta$$

As expected, R-order and adimensional R-order (AR-order) are equivalent (R-order itself is always adimensional), but adimensional Q-order (AQ-order) lies between classic Q and R-order (Q-order implies AQ-order, and AQ-order implies R-order). In fact, AQ-order behaves as classic Q-order but is applied to relative instead of absolute errors.

From (5) and 2., it is deduced that Newton’s method has second AR-order when the root is simple ($a<1/2$), and first AR-order when $a=1/2$. Actually, discriminant $1-2a$ marks the boundary between fast and slow convergence [28].

This is an example illustrating how adimensional polynomials and dimensional analysis make easier the study of the iterative method. In the next paragraph, we proceed further in order to devise robust methods.

6. Steffensen’s Method

As it was defined above, Steffensen’s method for a scalar function $f\left(x\right)$ is defined, from $x_0$, by the following iteration.

$$x_{n+1}=x_n-f{[x_n+f\left(x_n\right),x_n]}^{-1}f\left(x_n\right)\mathrm{for}\mathrm{all}n\ge 0$$

(18)

We consider nodes $x_n$ and $x_n+f\left(x_n\right)$. The key idea is that if the root $x^{\ast }$ is simple, by the mean value theorem, $f\left(x_n\right)=O(x_n-x^{\ast })$; therefore, $f[x_n,x_n+f\left(x_n\right)]$ is a first-order approximation to $f^{\prime }\left(x_n\right)$. In fact, when it converges to a simple root, Steffensen’s is a second-order method. In addition to sharing with the secant method and the fact that no derivatives are needed, Steffensen’s method has the advantage of a greater convergence order and no additional storage required.

As we said before, a large number of papers have appeared that are related to Steffensen’s method, but it is not widely used because of its scale dependence and dimension inconsistency. From a dimensional point of view, $\left[x\right]$ and $\left[f\right(x\left)\right]$ are different, and it is not a good idea to add both terms. Even in those cases where $\left[x\right]=\left[f\right]$, a different rescaling of the variable $cx$ and the function $kf$ makes the sum $cx+kf\left(x\right)$ unrelated to the original $x+f\left(x\right)$. In Section 3, some of the best known proposals in the literature to overcome possible troubles can be found.

Adimensional functions provide a different point of view, which will be explained in the rest of this paper. We begin by finding a system of bounds for Steffensen’s, which is similar to that of Newton’s.

6.1. A System of a Priori Error Estimates for Steffensen’s Method

Let us try to understand the behaviour of Steffensen’s method with a simple example. We consider the adimensional quadratic polynomial $q\left(s\right)=(a/2)s^2-s+1$ and we remind the reader that this polynomial does not have scale or dimensional difficulties. For these polynomials, Steffensen’s converges at least as fast as Newton’s, and both are of second-order.

Theorem 4.

If $0\le a\le 1/2$, $s_0=t_0=0$ and we denote the sequence of Steffensen’s approximations as follows:

$$s_{n+1}=s_n-q{[s_n,s_n+q\left(s_n\right)]}^{-1}q\left(s_n\right)$$

then $\left\{s_n\right\}$ converges monotonically to $s^{\ast }$. Furthermore, if we denote the following Newton’s approximations:

$$t_{n+1}=t_n-\frac{q\left(t_n\right)}{q^{\prime }\left(t_n\right)}$$

it is verified that, for all $n\ge 0$, $0\le t_n\le s_n\le s^{\ast }$.

Proof. 

First, we observe that Steffensen’s iteration is well-defined. For all $0<s<s^{\ast }$, we have the following:

$$q[s,s+q\left(s\right)]=\frac{q(s+q(s\left)\right)-q\left(s\right)}{q\left(s\right)}=q^{\prime }\left(s\right)+\frac{a}{2}q\left(s\right)=:g\left(s\right)$$

where $g\left(s\right)$ is an increasing function in $[0,s^{\ast }]$.

$$g^{\prime }\left(s\right)=q^{″}\left(s\right)+\frac{a}{2}{q}^{\prime }\left(s\right)=a+\frac{a}{2}(as-1)=\frac{a}{2}(1+as)>0$$

Moreover, $-1+a/2=g\left(0\right)<g\left(s\right)<g\left(s^{\ast }\right)=(a/2)q^{\prime }\left(s^{\ast }\right)\le 0$.

Therefore, $q[s,s+q(s\left)\right]\ne 0$, and it is invertible.

On the other side, denoting by ${\Delta }_S$ Steffensen’s correction and by ${\Delta }_N$ for Newton’s one, we have the following.

$${\Delta }_S\left(s\right)=-q{[s,s+q\left(s\right)]}^{-1}q\left(s\right)=-\frac{q\left(s\right)}{q^{\prime }\left(s\right)+\frac{a}{2}q\left(s\right)}\ge -\frac{q\left(s\right)}{q^{\prime }\left(s\right)}={\Delta }_N\left(s\right)$$

(19)

$${\Delta }_S\left(s_0\right)={\Delta }_S\left(0\right)=\frac{1}{1-\frac{a}{2}}\le \frac{1-\sqrt{1-2a}}{a}=s^{\ast }=s^{\ast }-s_0$$

$${\Delta }_S\left(s^{\ast }\right)=0$$

Differentiating ${\Delta }_S\left(s\right)$ with respect to s, we obtain the following.

$${\Delta }_S^{\prime }\left(s\right)=-\frac{q^{\prime }{\left(s\right)}^2-aq\left(s\right)}{{(q^{\prime }\left(s\right)+\frac{a}{2}q\left(s\right))}^2}$$

As the following is the case:

$$q^{\prime }{\left(s\right)}^2-aq\left(s\right)=(q^{\prime }{\left(s\right)}^2-2aq\left(s\right))+aq\left(s\right)=(1-2a)+aq\left(s\right)\ge 0$$

${\Delta }_S\left(s\right)$ is decreasing and positive, and if $0\le s\le s^{\ast }$, then $s+{\Delta }_S\left(s\right)\le s^{\ast }$.

As ${\Delta }_S\left(s\right)\ge 0$, sequence ${\left\{s_n\right\}}_n$ is increasing and bounded; hence, it converges. The limit is, obviously, $s^{\ast }$.

On the other side, monotony and (19) prove $0\le t_n\le s_n$ for all $n\ge 0$. □

However, if we take $\eta >2/a$ and $K_2=a/\eta$, polynomial $p\left(t\right)=(K_2/2)t^2-t+\eta$ has the same adimensional form $q\left(s\right)=(a/2)s^2-s+1$, but $p[0,p(0\left)\right]>0$ and, if $t_0=0$, then $t_1<t_0$ and the sequence of Steffensen’s iterates does not converge. If $\eta =2/a$, it becomes worse, because $p[0,p(0\left)\right]=0$ and as it is not invertible; the sequence is not even defined.

These phenomena do not happen in scale and dimensional independent methods such as Newton’s or secant, but they may become a real problem in Steffensen’s.

We construct now a system of estimates for Steffensen’s applied to adimensional polynomials. Given a positive $a\ge 0$ and $c_0=a_0=1$, $r_0=0$, we define for all $n\ge 0$ the following.

$$\begin{array}{cc}\left(\mathbf{i}\right)\hfill & b_n=\frac{a_n}{1-\frac{a}{2}{a}_n{c}_n}\hfill \\ \left(\mathbf{ii}\right)\hfill & d_n=b_n{c}_n\hfill \\ \left(\mathbf{iii}\right)\hfill & a_{n+1}=\frac{a_n}{1-a{a}_n{d}_n}\hfill \\ \left(\mathbf{iv}\right)\hfill & c_{n+1}=\frac{a^2}{2}{d}_n^2(r_n+\frac{1}{2}{c}_n)=\left\{\left(\frac{1}{a_n}+1\right)+\frac{a}{2}{c}_n\right\}\frac{a}{2}{d}_n^2\hfill \\ \left(\mathbf{v}\right)\hfill & r_{n+1}=r_n+d_n\hfill \end{array}$$

Now, our system consists of five sequences: two of them fundamental, ${\left\{a_n\right\}}_n$ and ${\left\{c_n\right\}}_n$, and the other three are auxiliaries only for simplification.

The following theorem states that the optimality of the system holds.

Theorem 5.

Given $q\left(s\right)$, adimensional polynomial, and the sequence ${\left\{s_n\right\}}_{n\ge 0}$ generated by $s_0=0$, $s_{n+1}=s_n-q[s_n,s_n+q\left(s_n\right)]$, $n\ge 0$, then we have the following.

$$\begin{array}{cc}\left(\mathbf{I}\right)\hfill & |q^{\prime }{\left(s_n\right)}^{-1}|=a_n.\hfill \\ \left(\mathbf{II}\right)\hfill & |q{[s_n,s_n+q\left(s_n\right)]}^{-1}|=b_n.\hfill \\ \left(\mathbf{III}\right)\hfill & |q\left(s_n\right)|=c_n.\hfill \\ \left(\mathbf{IV}\right)\hfill & |s_{n+1}-s_n|=d_n.\hfill \\ \left(\mathbf{V}\right)\hfill & s_n=r_n.\hfill \end{array}$$

Proof. 

By induction, if (III) and (IV) are valid for n, then the following is the case.

$$\begin{array}{ccc}\hfill & \left(\mathbf{I}\right)\hfill & \hfill q^{\prime }\left(s_{n+1}\right)=q^{\prime }\left(s_n\right)+a(s_{n+1}-s_n)\mathrm{implies}\end{array}$$

$$\frac{1}{q^{\prime }\left(s_{n+1}\right)}=\frac{1}{q^{\prime }\left(s_n\right)+a(s_{n+1}-s_n)}=\frac{1}{\frac{-1}{a_n}+a{d}_n}=\frac{-a_n}{1-a{a}_n{d}_n}=-a_{n+1}$$

$$\begin{array}{ccc}\hfill & \left(\mathbf{II}\right)\hfill & \hfill q[s_n,s_n+q\left(s_n\right)]=q^{\prime }\left(s_n\right)+\frac{a}{2}q\left(s_n\right)=\frac{-1}{a_n}+\frac{a}{2}{c}_n\end{array}$$

Then, we obtain the following.

$$q{[s_n,s_n+q\left(s_n\right)]}^{-1}=\frac{-a_n}{1-\frac{a}{2}{a}_n{c}_n}=-b_n$$

$$\begin{array}{ccc}\hfill & \left(\mathbf{III}\right)\hfill & \hfill q\left(s_{n+1}\right)=q\left(s_n\right)+q^{\prime }\left(s_n\right)(s_{n+1}-s_n)+\frac{a}{2}{(s_{n+1}-s_n)}^2\end{array}$$

The following expression must be 0.

$$q\left(s_n\right)+q[s_n,s_n+q\left(s_n\right)](s_{n+1}-s_n)=q\left(s_n\right)+q^{\prime }\left(s_n\right)(s_{n+1}-s_n)+\frac{a}{2}q\left(s_n\right)(s_{n+1}-s_n)$$

(20)

We have the following.

$$q\left(s_{n+1}\right)=\frac{a}{2}{d}_n(d_n-c_n)$$

(21)

From (20), $q\left(s_n\right)=-q^{\prime }\left(s_n\right)d_n-(a/2)q\left(s_n\right)d_n$.

Moreover, (21) becomes the following:

$$q\left(s_{n+1}\right)=\frac{a}{2}{d}_n^2(1+q^{\prime }\left(s_n\right)+\frac{a}{2}q\left(s_n\right))=-\frac{a}{2}{d}_n^2((q^{\prime }\left(s_n\right)-q^{\prime }\left(0\right))+\frac{a}{2}{c}_n)$$

because $q^{\prime }\left(0\right)=-1$. Taking into account that $s_n=r_n$, we finally approach the following.

$$q\left(s_{n+1}\right)=-\frac{a^2}{2}{d}_n^2(s_n+\frac{1}{2}{c}_n)=c_{n+1}$$

The rest of the theorem is obvious. □

System (I)–(V) show that sequence ${\left\{d_n\right\}}_n$ (similarly, ${\left\{c_n\right\}}_n$) is second AQ-order when $1-2a>0$ and first AQ-order when $2a=1$, as it happened with Newton. The rate of convergence, however, is not so simple. It is long and tedious but involves not so complicated algebraic manipulations, and the invariance of $(1/a_n^2)-2a{c}_n=1-2a$ provides the following.

$$d_{n+1}=\frac{a}{2}·\frac{(1+{\alpha }_n)+\frac{a}{2}\frac{{\alpha }_n{d}_n}{1+\frac{a}{2}{d}_n}}{{\alpha }_n\left(1-\frac{a}{2}\frac{d_n}{1+\frac{a}{2}{d}_n}\right)}·d_n^2$$

$$\begin{array}{ccc}\hfill & \mathrm{with}\hfill & \hfill {\alpha }_n=\frac{a{d}_n}{1+\frac{a}{2}{d}_n}+\sqrt{{\left(\frac{a{d}_n}{1+\frac{a}{2}{d}_n}\right)}^2+(1-2a)}\end{array}$$

As in Newton, $1-2a$ is the boundary separating low and high speed of convergence. However, while this region of high-order convergence is reached by ${\left(a{d}_n\right)}^2$ in Newton, in Steffensen’s, it is reached by ${(a{d}_n/(1+(a/2)d_n))}^2$. This shows a faster (although with the same order) convergence of Steffensen’s versus Newton’s method.

Now, we are ready to introduce the system of error bounds for Steffensen’s method in Banach spaces. First, we must remind the reader which kind of operator is the divided differences in Banach spaces. As opposed to a scalar, there is no unique divided difference operator for vectors. The characterization is based on interpolation.

Definition 6.

Given $F:E⟶V$, $x,y\in E$, a linear operator $H:E⟶V$ is a divided difference of F with nodes $x,y$ if $H(x-y)=F\left(x\right)-F\left(y\right)$. We denote $F[x,y]:=H$.

If F is twice differentiable, with $\Vert F^{″}\left(\xi \right)\Vert \le K_2$ for every $\xi \in E$, the following is verified.

$$\Vert F[x,y]-F^{\prime }\left(x\right)\Vert \le K_2\Vert x-y\Vert$$

Further insight on divided differences in Banach spaces can be found in [28]. There, we can find one example of differentiable divided difference as follows.

$$F[x,y]={\int }_0^1{F}^{\prime }(x+\theta (y-x))\mathrm{d}\theta$$

In practice, this integral is not explicitly evaluated, and divided differences are obtained from scalar-divided differences.

The key point of the rest of this paragraph is to consider Steffensen’s method applied to G, the adimensional form of F, defined in Section 2.2.

As the sum $y+G\left(y\right)$ is not only consistent but also independent from scales and dimensions, two of the main concerns about Steffensen’s are avoided. We can consider G as a preprocessing step of the nonlinear operator and, certainly, its performance is similar to that of preprocessing of matrices to solve linear systems: it reduces computational costs after small additional work before starting the method.

Instead of obtaining a root of F, we must find a root of G. Any method is admissible. Even if the method is scale and dimensional independent, such as Newton’s, adimensioning the function may be useful to regularize the system, as we saw in the square root example in Section 2.1. For Steffensen’s, adimensioning is indeed very useful. For example, it is easier to ensure convergence. In addition, the error bounds are more accurate in the sense that they are optimal (optimality does not always guarantee better estimates, but under certain conditions they cannot be improved). Optimal bounds have not been found in general for the secant method. We will prove that the system of estimates as defined above provides optimal error bounds for Steffensen’s method applied to adimensional equations.

6.2. Adimensional Scale Invariant Steffensen’s Method (ASIS Method) and Its Convergence

ASIS (adimensional scale invariant Steffensen’s) is the method where Steffensen’s method applied to the adimensional form of the function.

By the linear transformation from F to G, as it was explained in (4) and $y_0$ in (5), we define the following for $n\ge 0$.

$$y_{n+1}=y_n-G{[y_n,y_n+G\left(y_n\right)]}^{-1}G\left(y_n\right)$$

$$x_n=\Vert F\left(x_0\right)\Vert F^{\prime }{\left(x_0\right)}^{-1}{y}_n$$

Theorem 6.

Let us assume that there exists $G^{″}$ in V and $\Vert G^{″}\left(\xi \right)\Vert \le a$ for $\xi \in V$ and $a\le 1/2$. Then, from the system of bounds (i)–(v) in the last subsection, we have the following.

$$\begin{array}{cc}\hfill \left(\mathbf{a}\right)& \Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert \le a_n.\hfill \\ \hfill \left(\mathbf{b}\right)& \Vert G{[y_n,y_n+G\left(y_n\right)]}^{-1}\Vert \le b_n.\hfill \\ \hfill \left(\mathbf{c}\right)& \Vert G\left(y_n\right)\Vert \le c_n.\hfill \\ \hfill \left(\mathbf{d}\right)& \Vert y_{n+1}-y_n\Vert \le d_n.\hfill \\ \hfill \left(\mathbf{e}\right)& \Vert y_{n+1}-y_0\Vert \le r_n.\hfill \end{array}$$

Proof. 

It is similar to that of theorem 5 by using the Banach invertibility criterion.

First, we note that $\Vert G^{″}\left(y\right)\Vert \le a$ for all y.

By induction, if (a) and (c) hold for $n=0$, then, for $n\ge 0$, we have the following:

$$\Vert G^{\prime }\left(y_n\right)-G[y_n,y_n+G\left(y_n\right)]\Vert \le \frac{a}{2}\Vert G\left(y_n\right)\Vert <\frac{1}{\Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert }$$

which is the last inequality by the induction hypothesis. Then, by the Banach invertibility criterion, $G[y_n,y_n+G\left(y_n\right)]$ is invertible and the following is the case:

$$\Vert G{[y_n,y_n+G\left(y_n\right)]}^{-1}\Vert \le \frac{\Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert }{1-\frac{a}{2}\Vert G\left(y_n\right)\Vert ·\Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert }$$

and (b) is verified.

(d) and (e) are evident.

By the same Banach criterion, the following is the case.

$$\Vert G^{\prime }\left(y_{n+1}\right)-G^{\prime }\left(y_n\right)\Vert \le a\Vert y_{n+1}-y_n\Vert <\frac{1}{\Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert }$$

Thus, $G^{\prime }\left(y_{n+1}\right)$ is invertible and (a) is verified because the following is the case.

$$\Vert G^{\prime }{\left(y_{n+1}\right)}^{-1}\Vert \le \frac{\Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert }{1-\Vert G^{\prime }{\left(y_n\right)}^{-1}\Vert ·\Vert y_{n+1}-y_n\Vert }$$

Finally, from the following:

$$G\left(y_n\right)+G[y_n,y_n+G\left(y_n\right)](y_{n+1}-y_n)=0$$

$$G[y_n,y_n+G\left(y_n\right)]=G^{\prime }\left(y_n\right)+G^{″}\left(\xi \right)G\left(y_n\right),\mathrm{for}\xi \in \tilde{V}$$

and the following:

$$\Vert G\left(y_{n+1}\right)-G\left(y_n\right)-G^{\prime }\left(y_n\right)(y_{n+1}-y_n)\Vert \le \frac{a}{2}\Vert y_{n+1}-y_n{\Vert }^2$$

we obtain the following result.

$$\Vert G\left(y_n\right)\Vert \le \frac{a}{2}\Vert y_{n+1}-y_n\Vert \Vert (y_{n+1}-y_n)-G\left(y_n\right)(y_{n+1}-y_n)\Vert$$

$$\le \frac{a}{2}\Vert y_{n+1}-y_n{\Vert }^2\Vert I-G\left(y_n\right)\Vert \le \frac{a}{2}{d}_n^2\Vert G\left(y_n\right)+G^{\prime }\left(y_0\right)\Vert$$

Moreover, it follows from the following.

$$\Vert G\left(y_n\right)+G^{\prime }\left(y_0\right)\Vert =\Vert G^{\prime }\left(y_0\right)-G^{\prime }\left(y_n\right)-\frac{G^{″}\left(\xi \right)}{2}(y_{n+1}-y_n)\Vert$$

$$\le \frac{a}{2}(a\Vert y_n-y_0\Vert +\frac{a}{2}\Vert y_{n+1}-y_n\Vert )$$

□

The estimates of this theorem are optimal, because they are reached for the adimensional polynomial $q\left(s\right)=(a/2)s^2-s+1$.

As a consequence of Theorem 6, the following main result proves the convergence of Steffensen’s method in Banach spaces.

Theorem 7.

Let $F:E⟶V$ be a twice differentiable nonlinear function, $x_0\in E$ such that $F\left(x_0\right)\ne 0$ and $F^{\prime }\left(x_0\right)$ are invertible. Let $K_2,B,\eta >0$ verifying $\Vert F\left(x_0\right)\Vert \le \eta /B$, $\Vert F^{\prime }{\left(x_0\right)}^{-1}\Vert \le B$, and, for all $x\in E$, $\Vert F^{″}\left(x\right)\Vert \le K_2$.

If $a=K_{2}B\eta \le 1/2$, then the sequence generated by the ASIS method converges to a root $y^{\ast }$, $G\left(y^{\ast }\right)=0$. Furthermore, if for any $n\ge 0$, $x_n=(\eta /B)F^{\prime }{\left(x_0\right)}^{-1}{y}_n$, the following inequalities hold.

$$\Vert x_{n+1}-x_n\Vert \le d_n\eta$$

$$\Vert x^{\ast }-x_n\Vert \le (s^{\ast }-s_n)\eta$$

$$F^{\prime }\left(x_n\right)isinvertibleand\Vert F^{\prime }{\left(x_n\right)}^{-1}\Vert \le a_{n}B$$

This theorem improves previous results. So far, sufficient conditions in literature implied the existence of a strictly positive constant $h>0$ depending on $K_2,B,\eta$ such that $a(1+h)<1/2$. These conditions are more restrictive than those of Newton and, hence, than those we propose in Theorem 7, where $a\le 1/2$ suffices to ensure convergence.

We remark that not only invertibility but the existence of $F[x_n,x_n+F\left(x_n\right)]$ is not ensured in general for the classic Steffensen’s method for $a\le 1/2$; even $x_n+F\left(x_n\right)$ may not be evaluated. However, differences of the adimensional form do exist and they are invertible.

7. Numerical Examples

Our first example is a simple scalar equation:

$$f_1\left(x\right)=exp(x-1)-1=0$$

with the obvious solution, $x^{\ast }=1$.

We want to compare the performance of Newton’s, Steffensen’s, and ASIS methods. Both x and $f\left(x\right)$ are real.

This is a case where Kantorovich conditions are more restrictive than the actual Newtonian iteration. Kantorovich ($a\le 1/2$) guarantees convergence for $x_0\ge 1-log\left(\frac{1+\sqrt{3}}{2}\right)\approx 0.6881$. However, the convexity of $f_1\left(x\right)$ enlarges the region of convergence, as we will see.

We start Newton’s and Steffensen’s methods from $x_0=0$. This initial guess does not verify Kantorovich conditions, but both methods converge. In the figure, we represent the logarithms of the errors, where it can be seen that Newton’s converges faster than Steffensen’s. Although both are second order, Steffensen’s first iterations are clearly slower, because it takes some steps to enter into the second-order region. Steffensen’s errors are not only greater than those of Newton’s, but it requires some more iterations in order to obtain computational accuracy.

The setting varies for ASIS. The adimensional version of $f\left(x\right)$ is as follows:

$$g\left(s\right)=\frac{1}{exp(-1)-1}(exp((e-1)s-1)-1)$$

(22)

and $x=(e-1)s$. Therefore, $s_0=0$. In Figure 1, we represent the logarithms of the errors $|1-(e-1)s_n|$. It can be seen that these errors are smaller than Newton’s. That is, in this case, ASIS does not only improves Steffensen’s but also Newton’s, as expected from the theoretical results above.

A second example shows the consequences of rescaling the equation. We consider the following.

$$f_2\left(x\right)=e^{2x-1}-1=0$$

This function is obtained by the following: $f_2\left(x\right)=f_1\left(2x\right)$. The root of $f_2\left(x\right)$, $x^{\ast }=\frac{1}{2}$ is $\frac{1}{2}$ times the root of $f_1\left(x\right)$. Hence, Newton’s errors for $f_2$ are one-half the errors of $f_1$ due to the invariance with respect to scales in the variable. The adimensional version of $f_2\left(x\right)$ is exactly the same as for $f_1\left(x\right)$ and $g\left(s\right)$, and the errors of ASIS are also one-half of the errors obtained for $f_1\left(x\right)$. As a consequence, ASIS and Newton’s behave in the same manner as in the first example (ASIS converges faster than Newton’s).

However, Steffensen’s applied to $f_2\left(x\right)$ needs 3705 iterations to obtain an error that is less than $0.5$. We remark that $0.5$ is the initial error. Although it proceeds faster after that (iteration 3716 provides an approximation less than ${10}^{-16}$, double precision accuracy), it is clearly an unpractical method in this case. In Figure 2, we show the logarithms of the errors of classic Steffensen’s (not adimensional): In the first 3700 iterations, the errors look similar to a constant, and after that, it behaves as a second-order method. Its region of quadratic convergence is not so small, but it takes a long period to reach it from the initial guess. It can become worse for finer rescalings: The method can blow up due to roundoff errors for $f\left(x\right)=f_1\left(cx\right)$ for large c. This behavior is normal for Steffensen’s method: its convergence and its speed of convergence, when it converges, depend strongly on the scaling of both the variable and the own function.

The last example in this section is a system of two nonlinear equations with two variables.

$$\begin{array}{cc}\hfill F_1(x,y)=& -4x(y-x^2+2)-2(1-x)=0\hfill \\ \hfill F_2(x,y)=& 2(y-x^2+2)=0\hfill \end{array}$$

This system is obtained from an optimization problem by unrestricted minimization of the following function:

$$F(x,y)={(y-x^2+2)}^2+{(1-x^2)}^2$$

which is the Euclidean norm of a quadratic curve.

Starting from $x_0=y_0=0$, both Newton and Steffensen converge. As it happens with our first one dimensional equation, Steffensen’s is clearly slower than Newton. Among the possible choices of the divided difference operator, we used the one in [28].

Once again, we try the adimensional version of the system, which is normalized via the Euclidean norm, and we obtain the following:

$$\begin{array}{cc}\hfill g_1(s,t)=& -\frac{2}{3}s(\sqrt{5}t-\frac{5}{9}{s}^2+2)+(1+\frac{\sqrt{5}}{3}s)=0\hfill \\ \hfill g_2(s,t)=& -(t-\frac{5}{9}{s}^2+\frac{2}{\sqrt{5}})=0\hfill \end{array}$$

with the change of the following variables.

$$s=-\frac{3}{\sqrt{5}}x;t=\frac{1}{\sqrt{5}}y$$

The fact that $\Delta F(0,0)$ is diagonal simplifies the change of variables (which becomes a diagonal transformation). On the other side, some good features of the original system, such as the symmetry of the gradient, are lost in the adimensional system (therefore, the divided differences operator is no longer quasi-symmetric.

Its behavior is similar to that of the scalar case: ASIS is faster than Newton’s, and Steffensen’s is clearly slower, as observed in Figure 1.

As systems with greater order behave in a similar manner (ASIS is competitive with Newton’s, even sometimes with better performance, and often overcomes stability and convergence problems arising in classic Steffensen’s method), the last example was included as a representative for the sake of simplicity.

However, for really large systems, other issues appear, such as conditioning and dependence among variables and the advantages of ASIS method are not so evident (advantages of Newton’s method are not so evident either). There, ASIS can be a supporting method for solving equations. For instance, the BFGS formula for updating gradients in quasi-Newton methods is a strategy where ASIS improves the performance of the secant method, which is traditionally used in optimization.

The increase in computational cost of ASIS versus classical Steffensen’s is due to the evaluation of preconditioning $F^{\prime }{\left(x_0\right)}^{-1}$. As normalization of the adimensional form of F is not necessary, a simpler matrix $B\approx F^{\prime }{\left(x_0\right)}^{-1}$ is used such that $\left[B\right]=\left[F^{\prime }{\left(x_0\right)}^{-1}\right]$ is an option. A good choice for B can simplify the problem, as we saw in the square root example in Section 2.2.

8. Conclusions

This paper was motivated by the study of quasi-Newton methods in optimization. It is somehow surprising that despite Steffensen’s method being a well-known method (not as popular as Newton or secant, but well known), which can be implemented without explicit knowledge of derivatives, it is not used or even proposed as a feasible method in this topic. Some insight makes it clear that the weakness of Steffensen’s in scalar equations increases when dealing with equations in several variables. Nevertheless, most of these weaknesses are related to dimensional troubles. Independence from dimensions makes Steffensen’s method (and other methods) more robust and liable.

After some considerations about dimensionality, in this paper, we introduced the adimensional form of functions and polynomials in order to analyze the convergence of iterative methods. These adimensional forms make possible the construction of methods based on classical ones that are adapted from scale dependent methods and overcome all concerns due to dimensionality and scaling. Here, we produced the ASIS method based on the classic Steffensen’s method.

Semilocal conditions are settled in order to ensure convergence of ASIS. The ASIS method has the same convergence conditions as Newton’s method. This is a really interesting result, because, in general, conditions so far are more restrictive for classic Steffensen’s than for Newton’s. The obtained estimates are optimal in the sense that adimensional polynomials reach estimates. This optimality is also an important improvement with respect to other estimators in the literature, which are not optimal. Optimality improves the theoretical region of convergence. In fact, ASIS is at least as fast as Newton’s for polynomials.

These semilocal conditions are not only introduced for theoretical knowledge but also for practical reasons: hybrid methods using different algorithms that slower for the initial iterations with the goal of obtaining approximations into the regions of convergence and faster ones when the approximations are close enough to the exact solution are often devised. In order to obtain good performance, well fitted estimators are needed.

The price to pay in order to obtain adimensional functions is an increase in computational cost due to a change of variable that must be implemented. However, this cost can be reduced because, in general, $F^{\prime }{\left(x_0\right)}^{-1}$ is not an accurate requirement, although the analysis of softening conditions is outside the scope of this paper. It is remarkable that even random numerical errors do have dimension. Randomness is a numerical condition, but it affects the measure of the parameters.

Finally, we remark that some conditions in the theorems in this paper can be relaxed. Not only the exact evaluation of the inverse of $F^{\prime }\left(x_0\right)$ but also the conditions on the second derivative can be relaxed, because it is enough to ask for Lipschitz continuity of the first derivative or some other weaker conditions appearing often in the literature.

Last but not least, ASIS is a good alternative method for quasi-Newton methods in optimization because of its interpolating conditions (no need to evaluate derivatives), robustness, and speed of convergence.

Some features of the method, such as difficulties from ill conditioning, or large systems are actually under research, and the results so far are promising.

The techniques developed here can be extended and generalized to other kind of algorithms (numerical solution of differential and integral equations, regression, interpolation, etc.).

Summing up: although it is often disregraded, dimensional analysis is a powerful tool that helps to improve the performance of algorithms and helps in obtaining a better understanding of their theoretical properties. Adimensional functions rid us of the drawbacks induced by heterogeneous data.