Next Article in Journal
Design of an Optimized Fuzzy Classifier for the Diagnosis of Blood Pressure with a New Computational Method for Expert Rule Optimization
Next Article in Special Issue
On the Existence of Solutions of Nonlinear Fredholm Integral Equations from Kantorovich’s Technique
Previous Article in Journal
New Methodology to Approximate Type-Reduction Based on a Continuous Root-Finding Karnik Mendel Algorithm
Previous Article in Special Issue
Expanding the Applicability of Some High Order Househölder-Like Methods

Algorithms 2017, 10(3), 78; https://doi.org/10.3390/a10030078

Article
An Efficient Algorithm for the Separable Nonlinear Least Squares Problem
Department of Mathematics, Western Washington University, Bellingham, WA 98225-9063, USA
*
Authors to whom correspondence should be addressed.
Received: 7 June 2017 / Accepted: 1 July 2017 / Published: 10 July 2017

Abstract

:
The nonlinear least squares problem m i n y , z A ( y ) z + b ( y ) , where A ( y ) is a full-rank ( N + ) × N matrix, y R n , z R N and b ( y ) R N + with n , can be solved by first solving a reduced problem m i n y f ( y ) to find the optimal value y * of y, and then solving the resulting linear least squares problem m i n z A ( y * ) z + b ( y * ) to find the optimal value z * of z. We have previously justified the use of the reduced function f ( y ) = C T ( y ) b ( y ) , where C ( y ) is a matrix whose columns form an orthonormal basis for the nullspace of A T ( y ) , and presented a quadratically convergent Gauss–Newton type method for solving m i n y C T ( y ) b ( y ) based on the use of QR factorization. In this note, we show how LU factorization can replace the QR factorization in those computations, halving the associated computational cost while also providing opportunities to exploit sparsity and thus further enhance computational efficiency.
Keywords:
separable equations; nonlinear least squares; full-rank matrices; QR factorization; over-determined systems; Gauss–Newton method; least squares solutions; LU factorization; quadratic convergence
MSC:
65H10

1. Introduction

Many applications [1,2,3] lead to the need to find a solution ( y * , z * ) of the separable nonlinear overdetermined least squares problem
m i n y , z F ( y , z ) m i n y , z A ( y ) z + b ( y ) ,
where the full-rank matrix A ( y ) R ( N + ) × N and the vector b ( y ) R N + are twice Lipschitz continuously differentiable functions of y R n , z R N and n . Here, we are using the Euclidean norm. The classic Golub–Pereyra variable projection method [4] replaces the problem (1) by the simpler least squares problem
m i n y f ( y ) m i n y [ I A ( y ) A + ( y ) ] b ( y ) .
Once y * has been obtained by solving (2), z * can be found by solving the resulting linear least squares problem
m i n z A ( y * ) z + b ( y * ) ;
formally,
z * = A + ( y * ) b ( y * ) ,
where A + ( y * ) is the pseudo-inverse of A ( y * ) . An alternative method that reduces (1) to a smaller least squares problem m i n y f ( y ) was proposed in [5,6] for the case = n , and the associated iterative technique was then applied to the case > n in [7] but without a complete analytical justification. The reduced system is
m i n y f ( y ) m i n y C T ( y ) b ( y ) ,
where C ( y ) is an ( N + ) × N matrix whose columns form an orthonormal basis for N u l l ( A T ( y ) ) . In [8], we presented a quadratically convergent Gauss–Newton type iterative method, incorporating second derivative terms, to solve this problem, and provided a complete theoretical justification for our technique. In particular, we showed in [8] that our Gauss–Newton type method, and also the standard Gauss–Newton method [9,10,11], which omits the second derivative terms and is not quadratically convergent, are both invariant with respect to the specific basis matrix C ( y ) that is used at any particular point in the iteration. This makes it possible to substitute, at each point of the iteration, any easily computed local orthonormal basis for the nullspace of A T ( y ) . This observation also makes it possible to compute both the first and the second derivatives involved in the computation very cheaply.
Many instances of (1), such as those arising from the discretization of linear differential equations with only a few nonlinear parameters, involve N , while and n are fixed and small. In this case, the main computational cost in each step of our quadratically convergent Gauss–Newton type method of [8] is the QR factorization of A ( y ) , costing approximately 4 N 3 / 3 flops [12] per iteration. In this note, we show how the relevant computations can instead be performed by using one LU factorization of A ( y ) per iteration, so that the computational cost of the computation is essentially halved to 2 N 3 / 3 flops [12]. We note that, in the case of discretizations of differential equations, the matrix A ( y ) is typically very sparse and strongly patterned. Since LU factorization more easily exploits and preserves such sparsity and patterns than does QR factorization, this technique also opens up opportunities for further reductions in the computational cost in such applications.
In a previous paper [3], we presented the use of LU factorizations in the context of solving separable systems of nonlinear equations and zero-residual separable nonlinear least squares problems. We mentioned there that extending that approach to nonzero residual problems remained an open question. The current paper resolves that question, while also improving on the technique of [3] in other ways. In particular, our new method does not involve the singular value decomposition, and in contrast to the use of the standard Gauss–Newton method in [3], it retains quadratic convergence even in the case of nonzero residual problems. The latter is achieved by retaining the second derivative terms, and we show here how those terms can be computed very efficiently. This is in contrast to the usual use of the Gauss–Newton method, which sacrifices the quadratic convergence for the convenience of not computing the second derivatives.
In the next section, we present the relevant results from [8] relating to our technique for solving (5) by our Gauss–Newton type method using QR factorization, and show how those results and methods can be modified to use LU factorization. The resulting algorithm and some numerical examples to illustrate the method are given in the following two sections. The numerical results exhibit the quadratic convergence expected from our Gauss–Newton type method.

2. Analysis

We begin by presenting the relevant background results from [8]. Assume that A ( y ) and b ( y ) are twice Lipschitz continuously differentiable in a neighborhood of the least squares solution y * of (1). As shown in [5,6,7,8], there exists a smoothly differentiable ( N + ) × matrix C ( y ) whose columns form an orthonormal basis of N u l l ( A T ( y ) ) in a neighborhood of y * . Then, finding the least squares solution of (1) can be reduced to finding the least squares solution of (5). Our Gauss–Newton type method for solving (5) takes the form
[ ( f ( y ( m ) ) ) T f ( y ( m ) ) + Σ i = 1 f i ( y ( m ) ) H i ( y ( m ) ) ] ( y ( m + 1 ) y ( m ) ) = ( f ( y ( m ) ) ) T f ( y ( m ) ) ,
where f ( y ) C T ( y ) b ( y ) and f ( y ) , f i ( y ) and H i ( y ) denote the derivative of f ( y ) , the i-th component of f ( y ) , and the Hessian matrix of f i ( y ) , respectively.
In [8], we proved that, at any point y ( m ) in the iteration (6), the particular matrix C ( y ( m ) ) whose columns form an orthonormal basis for the nullspace of A T ( y ( m ) ) can be replaced by any other orthonormal basis C ˜ ( y ( m ) ) for this space, without changing y ( m + 1 ) . Thus, instead of using the function f ( y ) C T ( y ) b ( y ) in (6), we may use a different function g ( y ) C ˜ T ( y ) b ( y ) at each iteration point y ( m ) . This freedom to use any orthonormal basis for the nullspace of A T ( y ) at the point y ( m ) is key to our numerical technique. It also permits us to write simply C ( y ( m ) ) and f ( y ( m ) ) instead of C ˜ ( y ( m ) ) and g ( y ( m ) ) in the rest of this note, regardless of the particular matrix C ( y ) being used, and for simplicity we write y ¯ instead of y ( m ) . The analysis below centers on the efficient computation of the terms required to use (6) at the particular point y ( m ) = y ¯ .
The following results are derived and used in [5,6,7,8] and are required for our analysis below. With the terms as defined above, and writing C i ( y ) for the i-th column of the matrix C ( y ) ,
C T ( y ) A ( y ) = 0 , C T ( y ) C ( y ) = I ,
f ( y ) = C T ( y ) b ( y ) + [ C T ( y ) ] 1 b ( y ) . . . [ C T ( y ) ] n b ( y ) ,
and the ( j , k ) -th entry of the n × n Hessian matrix H i ( y ) is
[ C i T ( y ) ] j , k b ( y ) + [ C i T ( y ) ] j [ b ( y ) ] k + [ C i T ( y ) ] k [ b ( y ) ] j + C i T ( y ) [ b ( y ) ] j , k .
Here, we use [ C ( y ) ] j and [ C ( y ) ] j , k to denote the termwise first derivative w.r.t. y j and second derivative w.r.t. y j , y k , respectively, of C ( y ) , for j , k = 1 , , n .
Define the ( N + ) × ( N + ) nonsingular matrix
M ( y ) A ( y ) C ( y ) .
Then, we showed in [8] that
[ C ( y ¯ ) ] j M ( y ¯ ) = C T ( y ¯ ) [ A ( y ¯ ) ] j 0 ( N + ) ×
and
[ C T ( y ¯ ) ] j , k M ( y ¯ ) = [ C T ( y ¯ ) ] j [ A ( y ¯ ) ] k [ C T ( y ¯ ) ] k [ A ( y ¯ ) ] j C T ( y ¯ ) [ A ( y ¯ ) ] j , k [ K ( y ¯ ) ] j , k ,
where the ( p , q ) -entry K ( y ¯ ) ] j , k ( p , q ) of the upper triangular matrix K ( y ¯ ) ] j , k is
K ( y ¯ ) ] j , k ( p , q ) = 0 if   p > q , [ C p T ( y ¯ ) ] j [ C p ( y ¯ ) ] k if   p = q , [ C p T ( y ¯ ) ] j [ C q ( y ¯ ) ] k [ C p T ( y ¯ ) ] k [ C q ( y ¯ ) ] j if   p < q .
Throughout the rest of this paper, we assume not only that A ( y ) has full rank N, but also that it is sufficiently well-conditioned that LU factorization with partial pivoting can be performed safely. For separable nonlinear systems with a rank-deficient A ( y ) , the technique of bordered matrices [13,14,15,16] may be applied to produce a full rank matrix.
The following theorem lays the foundation for our approach to using the LU factorization.
Theorem 1.
Let the LU factorization of the rectangular matrix A ( y ¯ ) with some permutation matrix P ( y ¯ ) be
P ( y ¯ ) A ( y ¯ ) = L ( y ¯ ) U ( y ¯ ) ,
where A ( y ¯ ) is ( N + ) × N , P ( y ¯ ) is an ( N + ) × ( N + ) permutation matrix, L ( y ¯ ) is an ( N + ) × N unit lower triangular matrix and U ( y ¯ ) is an N × N nonsingular upper triangular matrix.
(a) 
The matrix
M ¯ ( y ¯ ) [ A ( y ¯ ) | S ( y ¯ ) ] ,
where
S ( y ¯ ) P T ( y ¯ ) 0 I ,
is nonsingular.
(b) 
Define
Ψ ( y ¯ ) [ M ¯ T ( y ¯ ) ] 1 0 I .
Then, the thin positive QR factorization
Ψ ( y ¯ ) C ( y ¯ ) R ( y ¯ )
produces a thin orthonormal ( N + ) × matrix C ( y ¯ ) and a small × nonsingular upper triangular matrix R ( y ¯ ) , and
C T ( y ¯ ) A ( y ¯ ) = 0 , C T ( y ¯ ) C ( y ¯ ) = I .
Proof. 
(a) By direct computation, M ¯ ( y ¯ ) is a product of nonsingular matrices:
M ¯ ( y ¯ ) = P T ( y ¯ ) L ( y ¯ ) U ( y ¯ ) 0 I = P T ( y ¯ ) L ( y ¯ ) 0 I U ( y ¯ ) 0 0 I .
(b) By (15) and (16),
C T ( y ¯ ) A ( y ¯ ) = R T ( y ¯ ) Ψ T ( y ¯ ) A ( y ¯ ) = R T ( y ¯ ) 0 I [ M ¯ ( y ¯ ) ] 1 ( y ¯ ) A ( y ¯ ) = R T ( y ¯ ) 0 I I N 0 = 0 .
From (17), we see that the matrix C ( y ¯ ) constructed in the theorem satisfies (7) at y ¯ , and thus, as shown in [8], Equations (11) and (12) hold for this matrix at y ¯ .
To determine the computational cost, recall that we assume that N , while and n are fixed and small. Then, the thin QR factorization in (16) is cheap since the matrix Ψ ( y ¯ ) has only columns. More specifically, the LU factorization of the ( N + ) × N dimensional A ( y ¯ ) in  (13) costs ( N + ) N 2 N 3 / 3 2 N 3 / 3 flops, with an additional O ( N 2 ) comparisons to produce P ( y ¯ ) [12], while computing  S ( y ¯ ) only costs O ( N ) flops and the thin QR factorization of the ( N + ) × dimensional matrix S ( y ¯ ) takes only ( N + ) 2 3 = O ( N ) flops [12].
Notice that solving (11) and (12) involves solving several equations of the form u T M ( y ¯ ) = v T . However, the matrix M ( y ) defined in (10) and which is used in (11) and (12), differs from the matrix M ¯ ( y ¯ ) defined in (14) whose LU factors are given in (18). To reduce the cost of solving (11) and (12) from O ( N 3 ) to O ( N 2 ) operations, we can exploit the factorization (18) of the matrix M ¯ ( y ¯ ) , as below. This result is essentially an application of the Sherman–Morrison–Woodbury formula.
Theorem 2.
Let P ( y ¯ ) A ( y ¯ ) = L ( y ¯ ) U ( y ¯ ) be the LU factorization of A ( y ¯ ) with a permutation matrix P ( y ¯ ) as in  (13), and let the matrices M ¯ ( y ¯ ) , S ( y ¯ ) and C ( y ¯ ) be defined as in  (14)–(16), while M ( y ) is defined by (10). Denote
W ( y ¯ ) = I N + [ C ( y ¯ ) S ( y ¯ ) ] C T ( y ¯ ) .
Then,
(a) 
M 1 ( y ¯ ) = M ¯ 1 ( y ¯ ) W ( y ¯ ) ;
(b) 
For u , v , w R N + , u T M ( y ¯ ) = v T if and only if
w T M ¯ ( y ¯ ) = v T , u T = w T W ( y ¯ ) .
Proof. 
( a ) By (18),
W ( y ¯ ) M ( y ¯ ) = W ( y ¯ ) A ( y ¯ ) C ( y ¯ ) = { I N + [ C ( y ¯ ) S ( y ¯ ) ] C T ( y ¯ ) } A ( y ¯ ) { I N + [ C ( y ¯ ) S ( y ¯ ) ] C T ( y ¯ ) } C ( y ¯ ) = A ( y ¯ ) C ( y ¯ ) [ C ( y ¯ ) S ( y ¯ ) ] = [ A ( y ¯ ) S ( y ¯ ) ] = M ¯ ( y ¯ ) .
Then,
M 1 ( y ¯ ) = M ¯ 1 ( y ¯ ) [ M ¯ ( y ¯ ) ] M 1 ( y ¯ ) = M ¯ 1 ( y ¯ ) [ W ( y ¯ ) M ( y ¯ ) ] M 1 ( y ¯ ) = M ¯ 1 ( y ¯ ) W ( y ¯ ) .
Part (b) follows directly from (a). ☐
Finally, we show how the value z * defined by (3) can be computed efficiently using the LU factorization of A ( y * ) .
Theorem 3.
z * = A + ( y * ) b ( y * ) if and only if
z * = I N 0 N × A ( y * ) C ( y * ) 1 b ( y * ) .
Proof. 
Let the QR factorization of A ( y * ) be
A ( y * ) = Q R ( Q 1 ) ( N + ) × N ( Q 2 ) ( N + ) × R 1 0 .
Since A ( y * ) is full rank, we see that
A + ( y * ) = [ A T ( y * ) A ( y * ) ] 1 A T ( y * ) = ( R 1 ) T 0 Q T Q R 1 0 1 ( R 1 ) T 0 Q T = R 1 1 ( ( R 1 ) T ) 1 ( R 1 ) T 0 Q T = R 1 1 0 Q T .
Since C ( y * ) is an orthonormal matrix, there exists an × orthogonal matrix Θ such that
I N 0 N × A ( y * ) C ( y * ) 1 = I N 0 N × Q R 1 0 0 Θ × 1 = I N 0 N × R 1 1 0 0 Θ × T Q T = R 1 1 0 Q T .
Hence, the conclusion follows. ☐
Computing z * via (21) involves the matrix M ( y * ) of (10), but the technique of Theorem 2 may again be used to replace this by the known LU factorization (18) of M ¯ ( y * ) obtained from the LU factorization of A ( y * ) . Thus, the cost of this step is also only 2 N 3 / 3 + O ( N 2 ) flops.

3. Algorithm

We now give a detailed Algorithm 1 based on the analysis from the last section.
Algorithm 1: Given the function F ( y , z ) = A ( y ) z + b ( y ) with a matrix A ( y ) that is full rank in a neighborhood of y * , a small positive real number ϵ , a positive integer m 0 , and a point y ( 0 ) near the solution y * . For m = 0 , 1 , 2 , . . . , m 0 , do steps ( a ) ( l ) :
(a)
Compute the LU factorization (13) of A ( y ( m ) ) :
P ( y ( m ) ) A ( y ( m ) ) = L ( y ( m ) ) U ( y ( m ) ) ;
(b)
Form the matrix S ( y ( m ) ) and the matrix M ¯ ( y ( m ) ) as in (14);
(c)
Use (16) and the factorization (18) to form the matrix C ( y ( m ) ) ;
(d)
Compute f ( y ( m ) ) = C T ( y ( m ) ) b ( y ( m ) ) ;
(e)
Use (11) and (20) to solve for [ C ( y ( m ) ) ] j , j = 1 , . . . , n ;
(f)
Use (8) to form f ( y ( m ) ) ;
(g)
Use (12) and (20) to solve for [ C T ( y ( m ) ) ] j , k , j , k = 1 , ... , n ;
(h)
Use (9) to find the ( j , k ) -th entry of H i ( y ( m ) ) , i = 1 , , ;
(i)
Form the matrix [ ( f ( y ( m ) ) ) T f ( y ( m ) ) + Σ i = 1 f i ( y ( m ) ) H i ( y ( m ) ) ] for the left-hand side of (6);
(j)
Form the vector ( f ( y ( m ) ) ) T f ( y ( m ) ) for the right side of (6);
(k)
Compute y ( m + 1 ) by solving (6);
(l)
If | | y ( m + 1 ) y ( m ) | | < ϵ stop, output y * y ( m + 1 ) and z * from (21) using (19) and (20). Otherwise, if m < m 0 , replace m by m + 1 and go to ( a ) ; if m = m 0 , output that the method fails to obtain y * within given m 0 and ϵ .
Note that the cost of solving (6) for y ( m + 1 ) y ( m ) is only O ( 1 ) since the matrix involved is only × and we assume = O ( 1 ) . Thus, the overall cost per iteration remains 2 N 3 / 3 + O ( N 2 ) , the primary cost being the LU factorization of A ( y ( m ) ) . The standard Gauss–Newton method, which omits the second derivative terms, omits steps (g) and (h), but the resulting method is no longer quadratically convergent.

4. Examples

We present three examples to illustrate our algorithm for computing the least squares solution of overdetermined separable equations. Our examples have large N and small n , , approximating the theoretical characteristic that n , = O ( 1 ) as N . Similar results hold if we use much larger N.
Example 1.
Consider the least squares solution of the overdetermined system
A ( y ) z + b ( y ) ( A 1 ( y ) ) ( 2 k + 1 ) × ( 2 k + 1 ) + I 2 k + 1 ( A 2 ( y ) ) × ( 2 k + 1 ) z + ( b 1 ( y ) ) ( 2 k + 1 ) × 1 ( b 2 ( y ) ) × 1 = 0 ,
where
A 1 ( y ) = 2 1 1 2 1 1 2 1 1 2 , b 1 ( y ) = 0 ( 2 k + 1 ) × 1 ,
A 2 ( y ) = e T k + 1 0 , b 2 ( y ) = 1 0 . 02 α ( y , y * , β ) ,
with
α ( y , y * , β ) = ( y y * ) 2 ( y y * ) s i n ( 2 ( y y * ) ) 0.5 c o s ( 2 ( y y * ) ) + β
and y R (thus n = 1 ), and where e k + 1 is the ( k + 1 ) -th standard unit vector in R 2 k + 1 . The selected value of y * R is the value of the exact solution ( y * , z * ) . Here, = 2 .
The problem can be regarded as a constrained generalized eigenvalue problem. We choose n = 21 , β = 9.5 , and y * = 0.25 c s c 2 ( π / 44 ) 49.1228712506303890 . For the eigenvalues of matrices of the form of A 1 ( y ) , see [17]. Beginning with y ( 0 ) = 48 and using Algorithm 1, we obtain the results in Table 1 for y ( m ) , m = 1 , 2 , 3 , 4 , which shows quadratic convergence of the method.
After y ( 4 ) is obtained, we compute z ( 4 ) z * using (21), where z * R 21 , z j * = s i n ( j * π / 22 ) , j = 1 , , 21 . We find that z ( 4 ) z * = 5.2774 × 10 15 , and the 2-norm of the residual of A ( y ( 4 ) ) z ( 4 ) + b ( y ( 4 ) ) is ≈0.06.
Example 2.
Consider the least squares solution of the overdetermined system
A ( y ) z + b ( y ) A 1 ( y ) A 2 ( y ) z + b 1 ( y ) b 2 ( y ) = 0 ,
where
A 1 ( y ) = 1 1 1 1 1 N × N , b 1 ( y ) = 1 y 1 0 0 N × 1 ,
A 2 ( y ) = y 1 0 0 0 0 y 2 y 1 0 0 0 0 y 2 y 1 0 0 3 × N , b 2 ( y ) = 1 y 1 + y 1 2 1 + y 1 y 2 + y 1 y 2 2 y 1 + y 2 y 2 2 .
Here, we use y = ( y 1 , y 2 ) , thus n = 2 , and = 3 .
We choose n = 23 . The least squares solution of (23) is ( y * ) T = [ 0 , 0 ] . Beginning with [ y ( 0 ) ] T = [ 0.1 , 0.1 ] and using Algorithm 1, we obtain the results in Table 2 for y ( m ) , m = 1 , 2 , 3 , 4 . These results show the expected quadratic convergence. After y ( 4 ) is obtained, we compute z ( 4 ) z * using (21). Here, z * R 23 , z j * = ( 1 ) j + 1 , j = 1 , , 23 , and the exact residual is A ( y * ) z * + b ( y * ) = e 24 + e 25 + 2 e 26 . We find that z ( 4 ) z * = 5.0974 × 10 15 , while 2-norm of the residual of A ( y ( 4 ) ) z ( 4 ) + b ( y ( 4 ) ) is 2.4494897427831779 6 .
Our last example arises from a discretization, using the finite element method, of the one-dimensional elliptic interface problem ( β u x ) x + 2 = 0 , for x ( 0 , 1 ) , where β = β for 0 < x < α and β = β + for α < x < 1 . The boundary conditions are u ( 0 ) = 0 and u ( 1 ) = 1 / β + + ( 1 / β 1 / β + ) α 2 , and the interface conditions are l i m x α u ( x ) = l i m x α + u ( x ) and l i m x α β u x ( x ) = l i m x α + β + u x ( x ) . For more on interface problems, see [18]. In this discretization, the slopes of the basis tent functions on either side of the interface x = α are modified from 1 / h and 1 / h to s 1 , s 2 and s 1 , s 2 on the respective line segments. In our setting, β and s 2 are given, and ( y 1 , y 2 ) = ( s 1 , β + ) T represent the nonlinear variables.
Example 3.
Consider the least squares solution of the overdetermined system
A ( y ) z + b ( y ) = 0 ,
where the N × ( N + 3 ) matrix A ( y ) is defined as
A ( y ) β h F k 1 β h e k 1 0 ( k 1 ) × 1 0 ( k 1 ) × ( N k 1 ) β h e k 1 T β h + β y 1 β y 1 0 1 × ( N k 1 ) 0 1 × ( k 1 ) s 2 y 2 y 2 h + s 2 y 2 y 2 h e 1 T 0 ( N k 2 ) × ( k 1 ) 0 ( N k 2 ) × 1 y 2 h e 1 y 2 h F N k 1 0 1 × ( k 1 ) a N + 1 , k ( y 1 ) 0 0 1 × ( N k 1 ) 0 1 × ( k 1 ) 0 β y 1 s 2 y 2 0 1 × ( N k 1 ) 0 1 × ( k 1 ) 0 0 0 1 × ( N k 1 ) ,
with a N + 1 , k ( y 1 ) = ( α x k ) y 1 + ( x k + 1 α ) s 2 1 and
b ( y ) 2 h E ( k 1 ) × 1 2 h + ( α x k ) y 1 ( α x k ) 2 ( x k + 1 α ) + s 2 ( x k + 1 α ) 2 2 h ( α x k ) + y 1 ( α x k ) 2 + ( x k + 1 α ) s 2 ( x k + 1 α ) 2 2 h E ( N k 1 ) × 1 ( 1 α 2 ) / h + ( α 2 / h ) y 2 / β e N k 1 0 0 ( a N + 1 , k ( y ) ) 2 + ( β 1 y 1 s 2 y 2 ) 2 + γ 3 2 ,
where h = 1 / ( N + 1 ) and x j = j h , x k < α < x k + 1 , and the square matrices F are tridiagonal of the form
2 1 1 2 1 1 2 1 1 2 ,
the vectors E have all entries one, e j are the jth standard unit vectors, and
γ 1 = 1 ( x k + 1 α ) s 2 α x k , γ 2 = [ 1 ( x k + 1 α ) s 2 ] β ( α x k ) s 2 .
The exact least squares solution is y * = ( γ 1 , γ 2 ) T .
We use n = 39 , k = 26 , = 3 , n = 2 , β = 1 , s 2 = 8 , α = 2 / 3 and γ 3 = 4.775 π . Using the Algorithm 1 with y ( 0 ) = ( 56.5 , 7.2 ) T , we obtain the results in Table 3 for y ( m ) , m = 1 , 2 , 3 , 4 , which shows quadratic convergence of the method. After y ( 4 ) is obtained, we obtain z ( 4 ) using (21). Here, z j * = ( j h ) 2 / β , j = 1 , 2 , . . . , k and z j * = ( j h ) 2 / γ 2 + ( 1 / β 1 / γ 2 ) α 2 , j = k + 1 , ... , N , and the exact residual is A ( y * ) z * + b ( y * ) = γ 3 2 e N + 3 . We find that z ( 4 ) z * = 1.5081 × 10 15 and the 2-norm of the residual of A ( y ( 4 ) ) z ( 4 ) + b ( y ( 4 ) ) is 225.03314884758808 ( 4.775 π ) 2 .

5. Conclusions

We have shown how LU factorization can be used to solve separable nonlinear least squares problems with nonzero residuals. Our Gauss-Newton type method retains the second derivative terms and is quadratically convergent. The technique is most efficient for situations in which there are few nonlinear terms and variables. In that context we show how the second derivative terms can be computed relatively cheaply. Our numerical examples demonstrate the effectiveness of the technique in some areas of application.

Acknowledgments

Publication costs were provided by Western Washington University, Bellingham, WA 98225, USA.

Author Contributions

Yunqiu Shen conceived the key ideas and drafted the paper; Tjalling Ypma refined the details and the presentation.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Golub, G.H.; Pereyra, V. Separable nonlinear least squares: the variable projection method and its applications. Topic Review. Inverse Probl. 2003, 19, R1–R26. [Google Scholar] [CrossRef]
  2. Mullen, K.M.; van Stokkum, I.M. The variable projection algorithm in time-resolved spectroscopy, microscopy and mass spectrometry applications. Numer. Algorithms 2009, 51, 319–340. [Google Scholar] [CrossRef]
  3. Shen, Y.-Q.; Ypma, T.J. Solving separable nonlinear equations using LU factorization. ISRN Math. Anal. 2013, 2013, 258072. [Google Scholar] [CrossRef] [PubMed]
  4. Golub, G.H.; Pereyra, V. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal. 1973, 10, 413–432. [Google Scholar] [CrossRef]
  5. Shen, Y.-Q.; Ypma, T.J. Solving nonlinear systems of equations with only one nonlinear variable. J. Comput. Appl. Math. 1990, 30, 235–246. [Google Scholar] [CrossRef]
  6. Ypma, T.J.; Shen, Y.-Q. Solving N+m nonlinear equations with only m nonlinear variables. Computing 1990, 44, 259–271. [Google Scholar] [CrossRef]
  7. Lukeman, G.G. Separable Overdetermined Nonlinear Systems: An Application of the Shen-Ypma Algorithm; VDM Verlag: Saarbrucken, Germany, 2009. [Google Scholar]
  8. Shen, Y.; Ypma, T.J. Solving Separable Least Squares Problems using QR factorization. J. Comp. Appl. Math. submitted.
  9. Dennis, J.E., Jr.; Schnabel, R.B. Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Corrected Reprint of the 1983 Original; SIAM: Philadelphia, PA, USA, 1996. [Google Scholar]
  10. Deuflhard, P.; Hohmann, A. Numerical Analysis in Modern Scientific Computing, 2nd ed.; Springer: New York, NY, USA, 2003. [Google Scholar]
  11. Ortega, J.M.; Rheinboldt, W.C. Iterative Solution of Nonlinear Equations in Several Variables; Academic Press: New York, NY, USA, 1970. [Google Scholar]
  12. Golub, G.H.; Van Loan, C.F. Matrix Computations, 3rd ed.; Johns Hopkins: Baltimore, MD, USA, 1996. [Google Scholar]
  13. Shen, Y.-Q.; Ypma, T.J. Newton’s method for singular nonlinear equations using approximate left and right nullspaces of the Jacobian. Appl. Numer. Math. 2005, 54, 256–265. [Google Scholar] [CrossRef]
  14. Shen, Y.-Q.; Ypma, T.J. Solving rank-deficient separable nonlinear equations. Appl. Numer. Math. 2007, 57, 609–615. [Google Scholar] [CrossRef]
  15. Shen, Y.-Q.; Ypma, T.J. Numerical bifurcation of separable parameterized equations. Elect. Trans. Numer. Anal. 2009, 34, 31–43. [Google Scholar]
  16. Shen, Y.-Q.; Ypma, T.J. Rank deficiency and bifurcation into affine subspaces for separable parametrized equations. Math. Comp. 2016, 85, 271–293. [Google Scholar] [CrossRef]
  17. Smith, G.D. Numerical Solution of Partial Differential Equations: Finite Difference Methods, 3rd ed.; Oxford University Press: New York, NY, USA, 1985. [Google Scholar]
  18. Li, Z.; Ito, K. The Immersed Interface Method; SIAM: Philadelphia, PA, USA, 2006. [Google Scholar]
Table 1. Gauss–Newton type iterations for Example 1.
Table 1. Gauss–Newton type iterations for Example 1.
m y ( m ) | | y ( m ) y * | |
04.80000000000000e+0011.1229e+000
14.90312546237768e+0019.1617e−002
24.91243552951534e+0011.4840e−003
34.91228716369764e+0013.8635e−007
44.91228712506304e+0012.8422e−014
Table 2. Gauss–Newton type iterations for Example 2.
Table 2. Gauss–Newton type iterations for Example 2.
m y 1 ( m ) y 1 * y 2 ( m ) y 2 * | | y ( m ) y * | |
0 1.0000e−001 1.0000e−0011.4142e−001
1−3.2975e−002−1.7299e−0023.7238e−002
2 8.5227e−004 3.5533e−0049.2338e−004
3−1.5227e−008−6.1721e−0091.6890e−008
4−4.3532e−016−1.8608e−0164.7342e−016
Table 3. Gauss–Newton type iterations for Example 3.
Table 3. Gauss–Newton type iterations for Example 3.
m y 1 ( m ) y 1 * y 2 ( m ) y 2 * | | y ( m ) y * | |
0 5.0000e−001 2.0000e−0015.3852e−001
1 1.4160e+000 1.7845e−0011.4272e+000
2 2.0255e−002 2.5279e−0032.0412e−002
3−1.3048e−006−1.6292e−0071.3149e−006
4 1.4921e−013 1.8652e−0141.5038e−013

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Back to TopTop