An Efficient Algorithm for the Separable Nonlinear Least Squares Problem

Abstract

The nonlinear least squares problem $mi{n}_{y,z}\parallel A\left(y\right)z+b\left(y\right)\parallel$, where $A\left(y\right)$ is a full-rank $(N+\ell )\times N$ matrix, $y\in {\mathbf{R}}^n$, $z\in {\mathbf{R}}^N$ and $b\left(y\right)\in {\mathbf{R}}^{N+\ell }$ with $\ell \ge n$, can be solved by first solving a reduced problem $mi{n}_y\parallel f\left(y\right)\parallel$ to find the optimal value $y^{*}$ of y, and then solving the resulting linear least squares problem $mi{n}_z\parallel A\left(y^{*}\right)z+b\left(y^{*}\right)\parallel$ to find the optimal value $z^{*}$ of z. We have previously justified the use of the reduced function $f\left(y\right)=C^T\left(y\right)b\left(y\right)$, where $C\left(y\right)$ is a matrix whose columns form an orthonormal basis for the nullspace of $A^T\left(y\right)$, and presented a quadratically convergent Gauss–Newton type method for solving $mi{n}_y\parallel C^T\left(y\right)b\left(y\right)\parallel$ based on the use of QR factorization. In this note, we show how LU factorization can replace the QR factorization in those computations, halving the associated computational cost while also providing opportunities to exploit sparsity and thus further enhance computational efficiency.

1. Introduction

Many applications [1,2,3] lead to the need to find a solution $(y^{*},z^{*})$ of the separable nonlinear overdetermined least squares problem

$$mi{n}_{y,z}\parallel F(y,z)\parallel \equiv mi{n}_{y,z}\parallel A\left(y\right)z+b\left(y\right)\parallel ,$$

(1)

where the full-rank matrix $A\left(y\right)\in {\mathbf{R}}^{(N+\ell )\times N}$ and the vector $b\left(y\right)\in {\mathbf{R}}^{N+\ell }$ are twice Lipschitz continuously differentiable functions of $y\in {\mathbf{R}}^n$, $z\in {\mathbf{R}}^N$ and $\ell \ge n$. Here, we are using the Euclidean norm. The classic Golub–Pereyra variable projection method [4] replaces the problem (1) by the simpler least squares problem

$$mi{n}_y\parallel f\left(y\right)\parallel \equiv mi{n}_y\parallel [I-A\left(y\right)A^{+}\left(y\right)]b\left(y\right)\parallel .$$

(2)

Once $y^{*}$ has been obtained by solving (2), $z^{*}$ can be found by solving the resulting linear least squares problem

$$mi{n}_z\parallel A\left(y^{*}\right)z+b\left(y^{*}\right)\parallel ;$$

(3)

formally,

$$z^{*}=-A^{+}\left(y^{*}\right)b\left(y^{*}\right),$$

(4)

where $A^{+}\left(y^{*}\right)$ is the pseudo-inverse of $A\left(y^{*}\right)$. An alternative method that reduces (1) to a smaller least squares problem $mi{n}_y\parallel f\left(y\right)\parallel$ was proposed in [5,6] for the case $\ell =n$, and the associated iterative technique was then applied to the case $\ell >n$ in [7] but without a complete analytical justification. The reduced system is

$$mi{n}_y\parallel f\left(y\right)\parallel \equiv mi{n}_y\parallel C^T\left(y\right)b\left(y\right)\parallel ,$$

(5)

where $C\left(y\right)$ is an $(N+\ell )\times N$ matrix whose columns form an orthonormal basis for $Null\left(A^T\left(y\right)\right)$. In [8], we presented a quadratically convergent Gauss–Newton type iterative method, incorporating second derivative terms, to solve this problem, and provided a complete theoretical justification for our technique. In particular, we showed in [8] that our Gauss–Newton type method, and also the standard Gauss–Newton method [9,10,11], which omits the second derivative terms and is not quadratically convergent, are both invariant with respect to the specific basis matrix $C\left(y\right)$ that is used at any particular point in the iteration. This makes it possible to substitute, at each point of the iteration, any easily computed local orthonormal basis for the nullspace of $A^T\left(y\right)$. This observation also makes it possible to compute both the first and the second derivatives involved in the computation very cheaply.

Many instances of (1), such as those arising from the discretization of linear differential equations with only a few nonlinear parameters, involve $N\to \infty$, while ℓ and n are fixed and small. In this case, the main computational cost in each step of our quadratically convergent Gauss–Newton type method of [8] is the QR factorization of $A\left(y\right)$, costing approximately $4N^3/3$ flops [12] per iteration. In this note, we show how the relevant computations can instead be performed by using one LU factorization of $A\left(y\right)$ per iteration, so that the computational cost of the computation is essentially halved to $2N^3/3$ flops [12]. We note that, in the case of discretizations of differential equations, the matrix $A\left(y\right)$ is typically very sparse and strongly patterned. Since LU factorization more easily exploits and preserves such sparsity and patterns than does QR factorization, this technique also opens up opportunities for further reductions in the computational cost in such applications.

In a previous paper [3], we presented the use of LU factorizations in the context of solving separable systems of nonlinear equations and zero-residual separable nonlinear least squares problems. We mentioned there that extending that approach to nonzero residual problems remained an open question. The current paper resolves that question, while also improving on the technique of [3] in other ways. In particular, our new method does not involve the singular value decomposition, and in contrast to the use of the standard Gauss–Newton method in [3], it retains quadratic convergence even in the case of nonzero residual problems. The latter is achieved by retaining the second derivative terms, and we show here how those terms can be computed very efficiently. This is in contrast to the usual use of the Gauss–Newton method, which sacrifices the quadratic convergence for the convenience of not computing the second derivatives.

In the next section, we present the relevant results from [8] relating to our technique for solving (5) by our Gauss–Newton type method using QR factorization, and show how those results and methods can be modified to use LU factorization. The resulting algorithm and some numerical examples to illustrate the method are given in the following two sections. The numerical results exhibit the quadratic convergence expected from our Gauss–Newton type method.

2. Analysis

We begin by presenting the relevant background results from [8]. Assume that $A\left(y\right)$ and $b\left(y\right)$ are twice Lipschitz continuously differentiable in a neighborhood of the least squares solution $y^{*}$ of (1). As shown in [5,6,7,8], there exists a smoothly differentiable $(N+\ell )\times \ell$ matrix $C\left(y\right)$ whose columns form an orthonormal basis of $Null\left(A^T\left(y\right)\right)$ in a neighborhood of $y^{*}$. Then, finding the least squares solution of (1) can be reduced to finding the least squares solution of (5). Our Gauss–Newton type method for solving (5) takes the form

$$\begin{array}{cc}& [{(f^{\prime }(y^{(m)}))}^T{f}^{\prime }(y^{(m)})+{\mathrm{\Sigma }}_{i=1}^{\ell }{f}_i(y^{(m)})H_i(y^{(m)})](y^{(m+1)}-y^{(m)})\hfill \\ =& -{(f^{\prime }(y^{(m)}))}^{T}f(y^{(m)}),\hfill \end{array}$$

(6)

where $f\left(y\right)\equiv C^T\left(y\right)b\left(y\right)$ and $f^{\prime }\left(y\right)$, $f_i\left(y\right)$ and $H_i\left(y\right)$ denote the derivative of $f\left(y\right)$, the i-th component of $f\left(y\right)$, and the Hessian matrix of $f_i\left(y\right)$, respectively.

In [8], we proved that, at any point $y^{\left(m\right)}$ in the iteration (6), the particular matrix $C(y^{\left(m\right)})$ whose columns form an orthonormal basis for the nullspace of $A^T(y^{\left(m\right)})$ can be replaced by any other orthonormal basis $\tilde{C}(y^{\left(m\right)})$ for this space, without changing $y^{(m+1)}$. Thus, instead of using the function $f\left(y\right)\equiv C^T\left(y\right)b\left(y\right)$ in (6), we may use a different function $g\left(y\right)\equiv {\tilde{C}}^T\left(y\right)b\left(y\right)$ at each iteration point $y^{\left(m\right)}$. This freedom to use any orthonormal basis for the nullspace of $A^T\left(y\right)$ at the point $y^{\left(m\right)}$ is key to our numerical technique. It also permits us to write simply $C(y^{\left(m\right)})$ and $f(y^{\left(m\right)})$ instead of $\tilde{C}(y^{\left(m\right)})$ and $g(y^{\left(m\right)})$ in the rest of this note, regardless of the particular matrix $C\left(y\right)$ being used, and for simplicity we write $\overline{y}$ instead of $y^{\left(m\right)}$. The analysis below centers on the efficient computation of the terms required to use (6) at the particular point $y^{\left(m\right)}=\overline{y}$.

The following results are derived and used in [5,6,7,8] and are required for our analysis below. With the terms as defined above, and writing $C_i\left(y\right)$ for the i-th column of the matrix $C\left(y\right)$,

$$C^T\left(y\right)A\left(y\right)=0,C^T\left(y\right)C\left(y\right)=I,$$

(7)

$$f^{\prime }\left(y\right)=C^T\left(y\right)b^{\prime }\left(y\right)+\left[\begin{array}{ccc}{\left[C^T\left(y\right)\right]}_1^{\prime }b\left(y\right)& ...& {\left[C^T\left(y\right)\right]}_n^{\prime }b\left(y\right)\end{array}\right],$$

(8)

and the $(j,k)$-th entry of the $n\times n$ Hessian matrix $H_i\left(y\right)$ is

$${\left[C_i^T\left(y\right)\right]}_{j,k}^{″}b\left(y\right)+{\left[C_i^T\left(y\right)\right]}_j^{\prime }{\left[b\left(y\right)\right]}_k^{\prime }+{\left[C_i^T\left(y\right)\right]}_k^{\prime }{\left[b\left(y\right)\right]}_j^{\prime }+C_i^T\left(y\right){\left[b\left(y\right)\right]}_{j,k}^{″}.$$

(9)

Here, we use ${\left[C\left(y\right)\right]}_j^{\prime }$ and ${\left[C\left(y\right)\right]}_{j,k}^{″}$ to denote the termwise first derivative w.r.t. $y_j$ and second derivative w.r.t. $y_j,y_k$, respectively, of $C\left(y\right)$, for $j,k=1,\dots ,n$.

Define the $(N+\ell )\times (N+\ell )$ nonsingular matrix

$$M\left(y\right)\equiv \left[\begin{array}{cc}A\left(y\right)& C\left(y\right)\end{array}\right].$$

(10)

Then, we showed in [8] that

$${\left[C\left(\overline{y}\right)\right]}_j^{\prime }M\left(\overline{y}\right)=\left[\begin{array}{cc}-C^T\left(\overline{y}\right){\left[A\left(\overline{y}\right)\right]}_j^{\prime }& 0_{(N+\ell )\times \ell }\end{array}\right]$$

(11)

and

$$\begin{array}{c}{\left[C^T\left(\overline{y}\right)\right]}_{j,k}^{″}M\left(\overline{y}\right)=\\ \left[-{\left[C^T\left(\overline{y}\right)\right]}_j^{\prime }{\left[A\left(\overline{y}\right)\right]}_k^{\prime }-{\left[C^T\left(\overline{y}\right)\right]}_k^{\prime }{\left[A\left(\overline{y}\right)\right]}_j^{\prime }-C^T\left(\overline{y}\right){\left[A\left(\overline{y}\right)\right]}_{j,k}^{″}\mid {\left[K\left(\overline{y}\right)\right]}_{j,k}^{″}\right],\end{array}$$

(12)

where the $(p,q)$-entry ${\left[K\left(\overline{y}\right){]}_{j,k}^{″}\right]}_{(p,q)}$ of the upper triangular matrix $\left[K\left(\overline{y}\right){]}_{j,k}^{″}\right]$ is

$${\left[K\left(\overline{y}\right){]}_{j,k}^{″}\right]}_{(p,q)}=\left\{\begin{array}{cc}0\hfill & \mathrm{if}p>q,\hfill \\ -{\left[C_p^T\left(\overline{y}\right)\right]}_j^{\prime }{\left[C_p\left(\overline{y}\right)\right]}_k^{\prime }\hfill & \mathrm{if}p=q,\hfill \\ -{\left[C_p^T\left(\overline{y}\right)\right]}_j^{\prime }{\left[C_q\left(\overline{y}\right)\right]}_k^{\prime }-{\left[C_p^T\left(\overline{y}\right)\right]}_k^{\prime }{\left[C_q\left(\overline{y}\right)\right]}_j^{\prime }\hfill & \mathrm{if}p<q.\hfill \end{array}\right.$$

Throughout the rest of this paper, we assume not only that $A\left(y\right)$ has full rank N, but also that it is sufficiently well-conditioned that LU factorization with partial pivoting can be performed safely. For separable nonlinear systems with a rank-deficient $A\left(y\right)$, the technique of bordered matrices [13,14,15,16] may be applied to produce a full rank matrix.

The following theorem lays the foundation for our approach to using the LU factorization.

Theorem 1.

Let the LU factorization of the rectangular matrix $A\left(\overline{y}\right)$ with some permutation matrix $P\left(\overline{y}\right)$ be

$$P\left(\overline{y}\right)A\left(\overline{y}\right)=L\left(\overline{y}\right)U\left(\overline{y}\right),$$

(13)

where $A\left(\overline{y}\right)$ is $(N+\ell )\times N$, $P\left(\overline{y}\right)$ is an $(N+\ell )\times (N+\ell )$ permutation matrix, $L\left(\overline{y}\right)$ is an $(N+\ell )\times N$ unit lower triangular matrix and $U\left(\overline{y}\right)$ is an $N\times N$ nonsingular upper triangular matrix.

(a) 

The matrix

$$\overline{M}\left(\overline{y}\right)\equiv \left[A\left(\overline{y}\right)\right|S\left(\overline{y}\right)],$$

(14)

where

$$S\left(\overline{y}\right)\equiv P^T\left(\overline{y}\right)\left[\begin{array}{c}0\\ I_{\ell }\end{array}\right],$$

is nonsingular.

(b) 

Define

$$\mathsf{\Psi }\left(\overline{y}\right)\equiv {\left[{\overline{M}}^T\left(\overline{y}\right)\right]}^{-1}\left[\begin{array}{c}0\\ I_{\ell }\end{array}\right].$$

(15)

Then, the thin positive QR factorization

$$\mathsf{\Psi }\left(\overline{y}\right)\equiv C\left(\overline{y}\right)R\left(\overline{y}\right)$$

(16)

produces a thin orthonormal $(N+\ell )\times \ell$ matrix $C\left(\overline{y}\right)$ and a small $\ell \times \ell$ nonsingular upper triangular matrix $R\left(\overline{y}\right)$, and

$$C^T\left(\overline{y}\right)A\left(\overline{y}\right)=0,C^T\left(\overline{y}\right)C\left(\overline{y}\right)=I_{\ell }.$$

(17)

Proof. 

(a) By direct computation, $\overline{M}\left(\overline{y}\right)$ is a product of nonsingular matrices:

$$\begin{array}{cc}& \overline{M}\left(\overline{y}\right)=P^T\left(\overline{y}\right)\left[\begin{array}{cc}L\left(\overline{y}\right)U\left(\overline{y}\right)& \left|\begin{array}{c}0\\ I_{\ell }\end{array}\right.\end{array}\right]\hfill \\ =& P^T\left(\overline{y}\right)\left[\begin{array}{cc}L\left(\overline{y}\right)& \left|\begin{array}{c}0\\ I_{\ell }\end{array}\right.\end{array}\right]\left[\begin{array}{cc}U\left(\overline{y}\right)& 0\\ 0& I_{\ell }\end{array}\right].\hfill \end{array}$$

(18)

(b) By (15) and (16),

$$\begin{array}{cc}& C^T\left(\overline{y}\right)A\left(\overline{y}\right)=R^{-T}\left(\overline{y}\right){\mathsf{\Psi }}^T\left(\overline{y}\right)A\left(\overline{y}\right)=R^{-T}\left(\overline{y}\right)\left[\begin{array}{cc}0& I_{\ell }\end{array}\right]{\left[\overline{M}\left(\overline{y}\right)\right]}^{-1}\left(\overline{y}\right)A\left(\overline{y}\right)\hfill \\ =& R^{-T}\left(\overline{y}\right)\left[\begin{array}{cc}0& I_{\ell }\end{array}\right]\left[\begin{array}{c}{I}_N\\ 0\end{array}\right]=0.\hfill \end{array}$$

☐

From (17), we see that the matrix $C\left(\overline{y}\right)$ constructed in the theorem satisfies (7) at $\overline{y}$, and thus, as shown in [8], Equations (11) and (12) hold for this matrix at $\overline{y}$.

To determine the computational cost, recall that we assume that $N\to \infty$, while ℓ and n are fixed and small. Then, the thin QR factorization in (16) is cheap since the matrix $\mathsf{\Psi }\left(\overline{y}\right)$ has only ℓ columns. More specifically, the LU factorization of the $(N+\ell )\times N$ dimensional $A\left(\overline{y}\right)$ in  (13) costs $(N+\ell )N^2-N^3/3\approx 2N^3/3$ flops, with an additional $O\left(N^2\right)$ comparisons to produce $P\left(\overline{y}\right)$ [12], while computing $S\left(\overline{y}\right)$ only costs $O\left(N\right)$ flops and the thin QR factorization of the $(N+\ell )\times \ell$ dimensional matrix $S\left(\overline{y}\right)$ takes only $(N+\ell ){\ell }^2-{\ell }^3=O\left(N\right)$ flops [12].

Notice that solving (11) and (12) involves solving several equations of the form $u^{T}M\left(\overline{y}\right)=v^T$. However, the matrix $M\left(y\right)$ defined in (10) and which is used in (11) and (12), differs from the matrix $\overline{M}\left(\overline{y}\right)$ defined in (14) whose LU factors are given in (18). To reduce the cost of solving (11) and (12) from $O\left(N^3\right)$ to $O\left(N^2\right)$ operations, we can exploit the factorization (18) of the matrix $\overline{M}\left(\overline{y}\right)$, as below. This result is essentially an application of the Sherman–Morrison–Woodbury formula.

Theorem 2.

Let $P\left(\overline{y}\right)A\left(\overline{y}\right)=L\left(\overline{y}\right)U\left(\overline{y}\right)$ be the LU factorization of $A\left(\overline{y}\right)$ with a permutation matrix $P\left(\overline{y}\right)$ as in  (13), and let the matrices $\overline{M}\left(\overline{y}\right)$, $S\left(\overline{y}\right)$ and $C\left(\overline{y}\right)$ be defined as in  (14)–(16), while $M\left(y\right)$ is defined by (10). Denote

$$W\left(\overline{y}\right)=I_{N+\ell }-[C\left(\overline{y}\right)-S\left(\overline{y}\right)]C^T\left(\overline{y}\right).$$

(19)

Then,

(a) 

$M^{-1}\left(\overline{y}\right)={\overline{M}}^{-1}\left(\overline{y}\right)W\left(\overline{y}\right)$;

(b) 

For $u,v,w\in {\mathbf{R}}^{N+\ell }$, $u^{T}M\left(\overline{y}\right)=v^T$ if and only if

$$\begin{array}{c}\hfill w^T\overline{M}\left(\overline{y}\right)=v^T,u^T=w^{T}W\left(\overline{y}\right).\end{array}$$

(20)

Proof. 

$\left(a\right)$ By (18),

$$\begin{array}{cc}& W\left(\overline{y}\right)M\left(\overline{y}\right)=W\left(\overline{y}\right)\left[\begin{array}{cc}A\left(\overline{y}\right)\mid & C\left(\overline{y}\right)\end{array}\right]\hfill \\ =& \left[\{I_{N+\ell }-[C\left(\overline{y}\right)-S\left(\overline{y}\right)]C^T\left(\overline{y}\right)\}A\left(\overline{y}\right)\mid \{I_{N+\ell }-[C\left(\overline{y}\right)-S\left(\overline{y}\right)]C^T\left(\overline{y}\right)\}C\left(\overline{y}\right)\right]\hfill \\ =& \left[A\left(\overline{y}\right)\mid C\left(\overline{y}\right)-[C\left(\overline{y}\right)-S\left(\overline{y}\right)]\right]=[A\left(\overline{y}\right)\mid S\left(\overline{y}\right)]=\overline{M}\left(\overline{y}\right).\hfill \end{array}$$

Then,

$$\begin{array}{cc}& M^{-1}\left(\overline{y}\right)={\overline{M}}^{-1}\left(\overline{y}\right)\left[\overline{M}\left(\overline{y}\right)\right]M^{-1}\left(\overline{y}\right)\hfill \\ =& {\overline{M}}^{-1}\left(\overline{y}\right)\left[W\left(\overline{y}\right)M\left(\overline{y}\right)\right]M^{-1}\left(\overline{y}\right)={\overline{M}}^{-1}\left(\overline{y}\right)W\left(\overline{y}\right).\hfill \end{array}$$

Part (b) follows directly from (a). ☐

Finally, we show how the value $z^{*}$ defined by (3) can be computed efficiently using the LU factorization of $A\left(y^{*}\right)$.

Theorem 3.

$z^{*}=-A^{+}\left(y^{*}\right)b\left(y^{*}\right)$ if and only if

$$z^{*}=-\left[\begin{array}{cc}{I}_N& 0_{N\times \ell }\end{array}\right]{\left[\begin{array}{cc}A\left(y^{*}\right)& C\left(y^{*}\right)\end{array}\right]}^{-1}b\left(y^{*}\right).$$

(21)

Proof. 

Let the QR factorization of $A\left(y^{*}\right)$ be

$$A\left(y^{*}\right)=QR\equiv \left[\begin{array}{cc}{\left(Q_1\right)}_{(N+\ell )\times N}& {\left(Q_2\right)}_{(N+\ell )\times \ell }\end{array}\right]\left[\begin{array}{c}{R}_1\\ 0\end{array}\right].$$

Since $A\left(y^{*}\right)$ is full rank, we see that

$$\begin{array}{c}{A}^{+}\left(y^{*}\right)={\left[A^T\left(y^{*}\right)A\left(y^{*}\right)\right]}^{-1}{A}^T\left(y^{*}\right)\\ ={\left(\left[\begin{array}{cc}{\left(R_1\right)}^T& 0\end{array}\right]Q^{T}Q\left[\begin{array}{c}{R}_1\\ 0\end{array}\right]\right)}^{-1}\left[\begin{array}{cc}{\left(R_1\right)}^T& 0\end{array}\right]Q^T\\ =R_1^{-1}{\left({\left(R_1\right)}^T\right)}^{-1}\left[\begin{array}{cc}{\left(R_1\right)}^T& 0\end{array}\right]Q^T=\left[\begin{array}{cc}{R}_1^{-1}& 0\end{array}\right]Q^T.\end{array}$$

Since $C\left(y^{*}\right)$ is an orthonormal matrix, there exists an $\ell \times \ell$ orthogonal matrix $\mathsf{\Theta }$ such that

$$\begin{array}{c}\left[\begin{array}{cc}{I}_N& 0_{N\times \ell }\end{array}\right]{\left[\begin{array}{cc}A\left(y^{*}\right)& C\left(y^{*}\right)\end{array}\right]}^{-1}=\left[\begin{array}{cc}{I}_N& 0_{N\times \ell }\end{array}\right]{\left(Q\left[\begin{array}{cc}{R}_1& 0\\ 0& {\mathsf{\Theta }}_{\ell \times \ell }\end{array}\right]\right)}^{-1}\\ =\left[\begin{array}{cc}{I}_N& 0_{N\times \ell }\end{array}\right]\left(\left[\begin{array}{cc}{R}_1^{-1}& 0\\ 0& {\mathsf{\Theta }}_{\ell \times \ell }^T\end{array}\right]Q^T\right)=\left[\begin{array}{cc}{R}_1^{-1}& 0\end{array}\right]Q^T.\end{array}$$

Hence, the conclusion follows. ☐

Computing $z^{*}$ via (21) involves the matrix $M\left(y^{*}\right)$ of (10), but the technique of Theorem 2 may again be used to replace this by the known LU factorization (18) of $\overline{M}\left(y^{*}\right)$ obtained from the LU factorization of $A\left(y^{*}\right)$. Thus, the cost of this step is also only $2N^3/3+O\left(N^2\right)$ flops.

3. Algorithm

We now give a detailed Algorithm 1 based on the analysis from the last section.

Algorithm 1: Given the function $F(y,z)=A\left(y\right)z+b\left(y\right)$ with a matrix $A\left(y\right)$ that is full rank in a neighborhood of $y^{*}$, a small positive real number $ϵ$, a positive integer $m_0$, and a point $y^{\left(0\right)}$ near the solution $y^{*}$. For $m=0,1,2,...,m_0$, do steps $\left(a\right)$–$\left(l\right)$:

(a) Compute the LU factorization (13) of $A(y^{(m)})$ : $$P(y^{(m)})A(y^{(m)})=L(y^{(m)})U(y^{(m)});$$ (b) Form the matrix $S(y^{(m)})$ and the matrix $\overline{M}(y^{(m)})$ as in (14); (c) Use (16) and the factorization (18) to form the matrix $C(y^{(m)})$; (d) Compute $f(y^{(m)})=C^T(y^{(m)})b(y^{(m)})$; (e) Use (11) and (20) to solve for ${\left[C(y^{(m)})\right]}_j^{\prime },j=1,...,n$; (f) Use (8) to form $f^{\prime }(y^{(m)})$; (g) Use (12) and (20) to solve for ${\left[C^T(y^{(m)})\right]}_{j,k}^{″},j,k=1,...,n$; (h) Use (9) to find the $(j,k)$-th entry of $H_i(y^{(m)}),i=1,\dots ,\ell$; (i) Form the matrix $[{(f^{\prime }(y^{(m)}))}^T{f}^{\prime }(y^{(m)})+{\mathrm{\Sigma }}_{i=1}^{\ell }{f}_i(y^{(m)})H_i(y^{(m)})]$ for the left-hand side of (6); (j) Form the vector $-{(f^{\prime }(y^{(m)}))}^{T}f(y^{(m)})$ for the right side of (6); (k) Compute $y^{(m+1)}$ by solving (6); (l) If $\left|\right|y^{(m+1)}-y^{(m)}\left|\right|<ϵ$ stop, output $y^{*}\approx y^{(m+1)}$ and $z^{*}$ from (21) using (19) and (20). Otherwise, if $m<m_0$, replace m by $m+1$ and go to $(a)$; if $m=m_0$, output that the method fails to obtain $y^{*}$ within given $m_0$ and $ϵ$.

Note that the cost of solving (6) for $y^{(m+1)}-y^{\left(m\right)}$ is only $O\left(1\right)$ since the matrix involved is only $\ell \times \ell$ and we assume $\ell =O\left(1\right)$. Thus, the overall cost per iteration remains $2N^3/3+O\left(N^2\right)$, the primary cost being the LU factorization of $A\left(y^{\left(m\right)}\right)$. The standard Gauss–Newton method, which omits the second derivative terms, omits steps (g) and (h), but the resulting method is no longer quadratically convergent.

4. Examples

We present three examples to illustrate our algorithm for computing the least squares solution of overdetermined separable equations. Our examples have large N and small $n,\ell$, approximating the theoretical characteristic that $n,\ell =O\left(1\right)$ as $N\to \infty$. Similar results hold if we use much larger N.

Example 1.

Consider the least squares solution of the overdetermined system

$$A\left(y\right)z+b\left(y\right)\equiv \left[\begin{array}{c}{\left(A_1\left(y\right)\right)}_{(2k+1)\times (2k+1)}+I_{2k+1}\\ {\left(A_2\left(y\right)\right)}_{\ell \times (2k+1)}\end{array}\right]z+\left[\begin{array}{c}{\left(b_1\left(y\right)\right)}_{(2k+1)\times 1}\\ {\left(b_2\left(y\right)\right)}_{\ell \times 1}\end{array}\right]=0,$$

(22)

where

$$A_1\left(y\right)=\left[\begin{array}{cccc}\hfill -2& \hfill 1& \hfill & \hfill \\ \hfill 1& \hfill -2& \hfill 1& \hfill \\ \hfill & \hfill \cdots & \hfill \\ \hfill & \hfill 1& \hfill -2& \hfill 1\\ \hfill & \hfill & \hfill 1& \hfill -2\end{array}\right],b_1\left(y\right)=0_{(2k+1)\times 1},$$

$$A_2\left(y\right)=\left[\begin{array}{c}{e^T}_{k+1}\\ 0\end{array}\right],b_2\left(y\right)=\left[\begin{array}{c}-1\hfill \\ 0.02\sqrt{\alpha (y,y^{*},\beta )}\hfill \end{array}\right],$$

with

$$\alpha (y,y^{*},\beta )={(y-y^{*})}^2-(y-y^{*})sin\left(2(y-y^{*})\right)-0.5cos\left(2(y-y^{*})\right)+\beta$$

and $y\in \mathbf{R}$ (thus $n=1$), and where $e_{k+1}$ is the $(k+1)$-th standard unit vector in ${\mathbf{R}}^{2k+1}$. The selected value of $y^{*}\in \mathbf{R}$ is the value of the exact solution $(y^{*},z^{*})$. Here, $\ell =2$.

The problem can be regarded as a constrained generalized eigenvalue problem. We choose $n=21$, $\beta =9.5$, and $y^{*}=0.25cs{c}^2(\pi /44)\approx 49.1228712506303890$. For the eigenvalues of matrices of the form of $A_1\left(y\right)$, see [17]. Beginning with $y^{\left(0\right)}=48$ and using Algorithm 1, we obtain the results in Table 1 for $y^{\left(m\right)},m=1,2,3,4$, which shows quadratic convergence of the method.

Table 1. Gauss–Newton type iterations for Example 1.

m	${\mathit{y}}^{\left(\mathit{m}\right)}$	$\left|\right|{\mathit{y}}^{\left(\mathit{m}\right)}-{\mathit{y}}^{*}\left|\right|$
0	4.80000000000000e+001	1.1229e+000
1	4.90312546237768e+001	9.1617e−002
2	4.91243552951534e+001	1.4840e−003
3	4.91228716369764e+001	3.8635e−007
4	4.91228712506304e+001	2.8422e−014

After $y^{\left(4\right)}$ is obtained, we compute $z^{\left(4\right)}\approx z^{*}$ using (21), where $z^{*}\in {\mathbf{R}}^{21}$, $z_j^{*}=sin(j*\pi /22),$$j=1,\dots ,21$. We find that $\parallel z^{\left(4\right)}-z^{*}\parallel =5.2774\times {10}^{-15}$, and the 2-norm of the residual of $A\left(y^{\left(4\right)}\right)z^{\left(4\right)}+b\left(y^{\left(4\right)}\right)$ is ≈0.06.

Example 2.

Consider the least squares solution of the overdetermined system

$$A\left(y\right)z+b\left(y\right)\equiv \left[\begin{array}{c}{A}_1\left(y\right)\\ A_2\left(y\right)\end{array}\right]z+\left[\begin{array}{c}{b}_1\left(y\right)\\ b_2\left(y\right)\end{array}\right]=0,$$

(23)

where

$$A_1\left(y\right)={\left[\begin{array}{cccc}1& & & \\ 1& 1& & \\ & \ddots & \ddots & \\ & & 1& 1\end{array}\right]}_{N\times N},b_1\left(y\right)={\left[\begin{array}{c}-1-y_1\\ 0\\ \cdots \\ 0\end{array}\right]}_{N\times 1},$$

$$A_2\left(y\right)={\left[\begin{array}{cccccc}{y}_1& 0& 0& 0& \cdots & 0\\ y_2& y_1& 0& 0& \cdots & 0\\ 0& y_2& y_1& 0& \cdots & 0\end{array}\right]}_{3\times N},b_2\left(y\right)=\left[\begin{array}{c}1-y_1+y_1^2\\ 1+y_1-y_2+y_1{y}_2\\ 2-y_1+y_2-y_2^2\end{array}\right].$$

Here, we use $y=(y_1,y_2)$, thus $n=2$, and $\ell =3$.

We choose $n=23$. The least squares solution of (23) is ${\left(y^{*}\right)}^T=[0,0]$. Beginning with ${\left[y^{\left(0\right)}\right]}^T=[0.1,0.1]$ and using Algorithm 1, we obtain the results in Table 2 for $y^{\left(m\right)},m=1,2,3,4$. These results show the expected quadratic convergence. After $y^{\left(4\right)}$ is obtained, we compute $z^{\left(4\right)}\approx z^{*}$ using (21). Here, $z^{*}\in {\mathbf{R}}^{23}$, $z_j^{*}={(-1)}^{j+1},j=1,\dots ,23$, and the exact residual is $A\left(y^{*}\right)z^{*}+b\left(y^{*}\right)=e_{24}+e_{25}+2e_{26}$. We find that $\parallel z^{\left(4\right)}-z^{*}\parallel =5.0974\times {10}^{-15}$, while 2-norm of the residual of $A\left(y^{\left(4\right)}\right)z^{\left(4\right)}+b\left(y^{\left(4\right)}\right)$ is $2.4494897427831779\approx \sqrt{6}$.

Table 2. Gauss–Newton type iterations for Example 2.

m	${\mathit{y}}_{\mathbf{1}}^{\left(\mathit{m}\right)}-{\mathit{y}}_{\mathbf{1}}^{*}$	${\mathit{y}}_{\mathbf{2}}^{\left(\mathit{m}\right)}-{\mathit{y}}_{\mathbf{2}}^{*}$	$\left|\right|{\mathit{y}}^{\left(\mathit{m}\right)}-{\mathit{y}}^{*}\left|\right|$
0	1.0000e−001	1.0000e−001	1.4142e−001
1	−3.2975e−002	−1.7299e−002	3.7238e−002
2	8.5227e−004	3.5533e−004	9.2338e−004
3	−1.5227e−008	−6.1721e−009	1.6890e−008
4	−4.3532e−016	−1.8608e−016	4.7342e−016

Our last example arises from a discretization, using the finite element method, of the one-dimensional elliptic interface problem $-{\left(\beta u_x\right)}_x+2=0$, for $x\in (0,1)$, where $\beta ={\beta }^{-}$ for $0<x<\alpha$ and $\beta ={\beta }^{+}$ for $\alpha <x<1$. The boundary conditions are $u\left(0\right)=0$ and $u\left(1\right)=1/{\beta }^{+}+(1/{\beta }^{-}-1/{\beta }^{+}){\alpha }^2$, and the interface conditions are $li{m}_{x\to {\alpha }^{-}}u\left(x\right)=li{m}_{x\to {\alpha }^{+}}u\left(x\right)$ and $li{m}_{x\to {\alpha }^{-}}{\beta }^{-}{u}_x\left(x\right)=li{m}_{x\to {\alpha }^{+}}{\beta }^{+}{u}_x\left(x\right)$. For more on interface problems, see [18]. In this discretization, the slopes of the basis tent functions on either side of the interface $x=\alpha$ are modified from $-1/h$ and $1/h$ to $-s_1$, $-s_2$ and $s_1$, $s_2$ on the respective line segments. In our setting, ${\beta }^{-}$ and $s_2$ are given, and $(y_1,y_2)={(s_1,{\beta }^{+})}^T$ represent the nonlinear variables.

Example 3.

Consider the least squares solution of the overdetermined system

$$A\left(y\right)z+b\left(y\right)=0,$$

(24)

where the $N\times (N+3)$ matrix $A\left(y\right)$ is defined as

$$A\left(y\right)\equiv \left[\begin{array}{cccc}\frac{{\beta }^{-}}{h}{F}_{k-1}& -\frac{{\beta }^{-}}{h}{e}_{k-1}& 0_{(k-1)\times 1}& 0_{(k-1)\times (N-k-1)}\\ -\frac{{\beta }^{-}}{h}{e}_{k-1}^T& \frac{{\beta }^{-}}{h}+{\beta }^{-}{y}_1& -{\beta }^{-}{y}_1& 0_{1\times (N-k-1)}\\ 0_{1\times (k-1)}& -s_2{y}_2& \frac{y_2}{h}+s_2{y}_2& -\frac{y_2}{h}{e}_1^T\\ 0_{(N-k-2)\times (k-1)}& 0_{(N-k-2)\times 1}& -\frac{y_2}{h}{e}_1& \frac{y_2}{h}{F}_{N-k-1}\\ 0_{1\times (k-1)}& a_{N+1,k}\left(y_1\right)& 0& 0_{1\times (N-k-1)}\\ 0_{1\times (k-1)}& 0& {\beta }^{-}{y}_1-s_2{y}_2& 0_{1\times (N-k-1)}\\ 0_{1\times (k-1)}& 0& 0& 0_{1\times (N-k-1)}\end{array}\right],$$

with $a_{N+1,k}\left(y_1\right)=(\alpha -x_k)y_1+(x_{k+1}-\alpha )s_2-1$ and

$$b\left(y\right)\equiv \left[\begin{array}{c}2h{E}_{(k-1)\times 1}\hfill \\ 2h+(\alpha -x_k)-y_1{(\alpha -x_k)}^2-(x_{k+1}-\alpha )+s_2{(x_{k+1}-\alpha )}^2\hfill \\ 2h-(\alpha -x_k)+y_1{(\alpha -x_k)}^2+(x_{k+1}-\alpha )-s_2{(x_{k+1}-\alpha )}^2\hfill \\ 2h{E}_{(N-k-1)\times 1}-\left[(1-{\alpha }^2)/h+({\alpha }^2/h)y_2/{\beta }^{-}\right]e_{N-k-1}\hfill \\ 0\hfill \\ 0\hfill \\ {\left(a_{N+1,k}\left(y\right)\right)}^2+{({\beta }_1{y}_1-s_2{y}_2)}^2+{{\gamma }_3}^2\hfill \end{array}\right],$$

where $h=1/(N+1)$ and $x_j=jh$, $x_k<\alpha <x_{k+1}$, and the square matrices F are tridiagonal of the form

$$\left[\begin{array}{cccc}\hfill 2& \hfill -1& \hfill & \hfill \\ \hfill -1& \hfill 2& \hfill -1& \hfill \\ \hfill & \hfill \cdots & \hfill \\ \hfill & \hfill -1& \hfill 2& \hfill -1\\ \hfill & \hfill & \hfill -1& \hfill 2\end{array}\right],$$

the vectors E have all entries one, $e_j$ are the jth standard unit vectors, and

$${\gamma }_1=\frac{1-(x_{k+1}-\alpha )s_2}{\alpha -x_k},{\gamma }_2=\frac{[1-(x_{k+1}-\alpha )s_2]{\beta }^{-}}{(\alpha -x_k)s_2}.$$

(25)

The exact least squares solution is $y^{*}={({\gamma }_1,{\gamma }_2)}^T$.

We use $n=39$, $k=26$, $\ell =3$, $n=2$, ${\beta }^{-}=1$, $s_2=8$, $\alpha =2/3$ and ${\gamma }_3=4.775\pi$. Using the Algorithm 1 with $y^{\left(0\right)}={(56.5,7.2)}^T$, we obtain the results in Table 3 for $y^{\left(m\right)},m=1,2,3,4$, which shows quadratic convergence of the method. After $y^{\left(4\right)}$ is obtained, we obtain $z^{\left(4\right)}$ using (21). Here, $z_j^{*}={\left(jh\right)}^2/{\beta }^{-},j=1,2,...,k$ and $z_j^{*}={\left(jh\right)}^2/{\gamma }_2+(1/{\beta }^{-}-1/{\gamma }_2){\alpha }^2,j=k+1,...,N$, and the exact residual is $A\left(y^{*}\right)z^{*}+b\left(y^{*}\right)={\gamma }_3^2{e}_{N+3}$. We find that $\parallel z^{\left(4\right)}-z^{*}\parallel =1.5081\times {10}^{-15}$ and the 2-norm of the residual of $A\left(y^{\left(4\right)}\right)z^{\left(4\right)}+b\left(y^{\left(4\right)}\right)$ is $225.03314884758808\approx {\left(4.775\pi \right)}^2$.

Table 3. Gauss–Newton type iterations for Example 3.

m	${\mathit{y}}_{\mathbf{1}}^{\left(\mathit{m}\right)}-{\mathit{y}}_{\mathbf{1}}^{*}$	${\mathit{y}}_{\mathbf{2}}^{\left(\mathit{m}\right)}-{\mathit{y}}_{\mathbf{2}}^{*}$	$\left|\right|{\mathit{y}}^{\left(\mathit{m}\right)}-{\mathit{y}}^{*}\left|\right|$
0	5.0000e−001	2.0000e−001	5.3852e−001
1	1.4160e+000	1.7845e−001	1.4272e+000
2	2.0255e−002	2.5279e−003	2.0412e−002
3	−1.3048e−006	−1.6292e−007	1.3149e−006
4	1.4921e−013	1.8652e−014	1.5038e−013

5. Conclusions

We have shown how LU factorization can be used to solve separable nonlinear least squares problems with nonzero residuals. Our Gauss-Newton type method retains the second derivative terms and is quadratically convergent. The technique is most efficient for situations in which there are few nonlinear terms and variables. In that context we show how the second derivative terms can be computed relatively cheaply. Our numerical examples demonstrate the effectiveness of the technique in some areas of application.