Roundoff Error Analysis of an Algorithm Based on Householder Bidiagonalization for Total Least Squares Problems ^†

Abstract

For large-scale problems, how to establish an algorithm with high accuracy and stability is particularly important. In this paper, the Householder bidiagonalization total least squares (HBITLS) algorithm and nonlinear iterative partial least squares for total least squares (NIPALS-TLS) algorithm were established, by which the same approximate TLS solutions was obtained. In addition, the propagation of the roundoff error for the process of the HBITLS algorithm was analyzed, and the mixed forward-backward stability of these two algorithms was proved. Furthermore, an upper bound of roundoff error was derived, which presents a more detailed and clearer approximation of the computed solution.

1. Introduction

Consider estimating $\mathit{x}$ from the overdetermined linear system

$$\begin{array}{c}\hfill A\mathit{x}\approx \mathit{b}\mathrm{for}A\in {\mathbb{R}}^{m\times n},\mathit{b}\in {\mathbb{R}}^m\mathrm{and}\mathit{x}\in {\mathbb{R}}^n,\end{array}$$

(1)

where the error exists in both the right-hand side b and the data matrix A and $m\ge n+1$. In this case, the total least squares (TLS) model should be appropriate to adopt (cf. [1,2]). The TLS approach is just to find a perturbation with the minimum Frobenius norm to make the system (1) a compatible system

$$\begin{array}{c}\hfill {min\parallel (E,\mathit{r})\parallel }_F,\mathrm{subject}\mathrm{to}\mathit{b}+\mathit{r}\in \mathrm{Range}(A+E).\end{array}$$

(2)

The TLS method is widely used in various scientific fields, such as physics, automatic control, signal processing, statistics, economics, biology, medicine etc. In essence, a solution of a TLS problem can be expressed by a singular value decomposition of the augmented matrix $(A,\mathit{b}).$ When the dimensions of A are not too large, one can use the truncated-SVD (TSVD) method. When the dimensions of A become large, this approach becomes prohibitive because the SVD algorithm is of complexity $\mathcal{O}\left(m{n}^2\right)$. The above considerations lead us to consider Krylov iterative methods, that do not alter the matrix A. The methods have the attractive feature just like the Lanczos methods—that when n increases, the computed extreme singular elements rapidly become good approximations to the exact ones, and are satisfactorily accurate even if k is far less than n theoretically [1]. Nevertheless, the orthonormal properties of the Krylov basis strongly support the use of these Householder matrix-based algorithms. This is particularly true when we need to be sure that the perturbed problem we are solving has to conserve some spectral similarity properties. This will be especially relevant when we need to compute approximations of the TLS problems. In view of this, we consider applying the Householder bidiagonalization algorithm and the NIPALS PLS algorithm posed by Å. Bj$\ddot{o}$rck [3] to TLS problems, the formed Householder bidiagonalization total least squares (HBITLS) algorithm, and NIPALS-TLS algorithm, respectively. Furthermore, we find that the HBITLS and NIPALS-TLS algorithms also compute the same approximate solutions for the TLS problems.

When it comes to practical problems, the arithmetic will be inaccurate and there will be errors in each step of the calculation. Arithmetic operations running on the computer have finite precision, so there will be rounding errors as long as there are numerical computations. These rounding errors cause the calculation quantities to be different from their theoretical values. One of the design principles of the floating-point operation is that it should encourage experts to develop robust, efficient, and portable numerical programs, enable the handling of arithmetic exceptions, and provide for the development of transcendental functions and high-precision arithmetic [4]. The results in the roundoff error analysis in Lanczos-type methods obtained by Paige [5,6,7] played an important role in interpreting the behavior of the Lanczos method in finite-precision computations. Parlett and Scott  [8] used the results of the roundoff error analysis as the basis for suggesting a modification of the Lanczos method, which they called selective orthogonalization  [8,9,10]. In addition, in many practical problems, the stop criterion can be safely selected on the basis of the rounding error analysis of the original problem, thereby diminishing the need for an extremely precise approximation of the algebraic problem solution [4]. As far as we know, the roundoff error analysis of the approximation TLS solutions obtained by using the Householder bidiagonalization procedure was not systematically performed in the literature. Hence, in this paper, we analyzed the propagation of the roundoff error during the process of the HBITLS algorithm and found that the HBITLS algorithm and NIPALS-TLS algorithm are mixed forward–backward stable.

The paper is organized as follows. The HBITLS algorithm and NIPALS-TLS algorithm were established, by which the same approximate TLS solution was obtained in Section 2. Section 3 analyzes the propagation of the roundoff error during the process of the HBITLS algorithm. A brief conclusion is shown in the last section.

2. HBITLS Algorithm and NIPALS-TLS Algorithm

It is well known that algorithms based on a sequence of orthogonal transformations with Householder matrices have very good stability properties; see Higham [4]. Based on this, this paper gives the HBITLS and NIPALS-TLS algorithms and finds that they both compute the same approximate TLS solutions.

2.1. HBITLS Algorithm

Let us first describe the Householder bidiagonalization process just as shown in [3]. However, in this paper, the process is used in the augmented matrix $(\mathit{b},A)$ for the TLS problem. The idea is to compute orthogonal matrices $U\in {\mathbb{R}}^{m\times (n+1)}$ and $V\in {\mathbb{R}}^{(n+1)\times (n+1)}$, such that

$$\begin{array}{c}\hfill U^T(\mathit{b},A)V=\left(\begin{array}{ccccc}{\beta }_1& {\alpha }_1& & & \\ & {\beta }_2& {\alpha }_2& & \\ & & {\beta }_3& \ddots & \\ & & & \ddots & {\alpha }_n\\ & & & & {\beta }_{n+1}\end{array}\right)=({\beta }_1{\mathit{e}}_1,B_n)\equiv C_n.\end{array}$$

(3)

$U=G_1\cdots G_n=[{\mathit{u}}_1,{\mathit{u}}_2,\cdots ,{\mathit{u}}_{n+1}]$ and $V=H_1\cdots H_n=[{\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_{n+1}]$ can be determined as a product of Householder matrices in each iteration. Generally, $G_k$ introduces zero in the kth column, while $H_k$ sets zero for the appropriate entries in the kth row. This can be done by an algorithm named Householder. Given the reason of space, and known to all, we omit it here, see algorithm 5.4.2 in [1] for details.

From the above process of Householder bidiagonalization, we know that V can be rewritten as $V=\left(\begin{array}{cc}1& 0\\ 0& V_{n-1}\end{array}\right),$ $V_{n-1}=[{\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_n].$

In detail, from (3), we have

$$\begin{array}{c}\hfill U^T\mathit{b}=U^T(\mathit{b},A)V{\mathit{e}}_1=U^T(\mathit{b},A)\left(\begin{array}{cc}1& 0\\ 0& V_{n-1}\end{array}\right){\mathit{e}}_1=C_n{\mathit{e}}_1={\beta }_1{\mathit{e}}_1\end{array}$$

and

$$\begin{array}{ccc}\hfill A{\mathit{v}}_i& =& {\alpha }_i{\mathit{u}}_i+{\beta }_{i+1}{\mathit{u}}_{i+1},\mathrm{for}i=1,\cdots ,n.\hfill \end{array}$$

(4)

From (3), we have

$${(\mathit{b},A)}^{T}U=\left(\begin{array}{cc}1& 0\\ 0& V_{n-1}\end{array}\right)C_n^T$$

and there comes

$$\begin{array}{c}\hfill A^T{\mathit{u}}_1={\alpha }_1{\mathit{v}}_1,\end{array}$$

$$\begin{array}{ccc}\hfill A^T{\mathit{u}}_i& =& {\alpha }_i{\mathit{v}}_i+{\beta }_i{\mathit{v}}_{i-1},\mathrm{for}i=2,\cdots ,n-1.\hfill \end{array}$$

(5)

Let $V_k=H_1{H}_2\cdots H_k\left(\begin{array}{c}{I}_k\\ 0\end{array}\right),$ $U_k=G_k\cdots G_2{G}_1\left(\begin{array}{c}{I}_k\\ 0\end{array}\right),$ and $C_k=({\beta }_1{\mathit{e}}_1,B_k)$ is a leading principal submatrix of order $k+1$ of the final bidiagonal matrix $C_n$. As we all know, if the exact arithmetic is used, we have $U_{k+1}^T{U}_{k+1}=I$, $V_k^T{V}_k=I$. However, in any case, the previous equations remain within machine precision. Then (4) and (5) can be rewritten as

$$\begin{array}{c}\hfill (\mathit{b},A)V_k=U_{k+1}{C}_k,\end{array}$$

(6)

$$\begin{array}{c}\hfill {(\mathit{b},A)}^T{U}_{k+1}=V_k{C}_k^T+{\alpha }_{k+1}{\mathit{v}}_{k+1}{\mathit{e}}_{k+1}^T=V_{k+1}{\overline{C}}_k^T,\end{array}$$

(7)

where ${\overline{C}}_k=(C_k,{\alpha }_{k+1}{\mathit{e}}_k)\in {\mathbb{R}}^{k\times (k+1)}.$

After performing the k-step Householder bidiagonalization iterations, the TLS problem can be reduced onto the subspace generated by $U_{k+1}$ and $V_k$. Then the reduced TLS problem (also see [11]) is as follows

$$\begin{array}{c}\hfill min\parallel U_{k+1}^T((\mathit{b},A)-({\widehat{\mathit{b}}}_k,{\widehat{A}}_k))\left(\begin{array}{cc}1& 0\\ 0& V_k\end{array}\right){\parallel }_F\mathrm{subject}\mathrm{to}{U}_{k+1}^T{\widehat{A}}_k{V}_k\mathit{y}=U_{k+1}^T{\widehat{\mathit{b}}}_k,\end{array}$$

(8)

or

$$\begin{array}{c}\hfill min\parallel ({\beta }_1{\mathit{e}}_1,B_k)-({\widehat{\mathit{e}}}_k,{\widehat{B}}_k){\parallel }_F\mathrm{subject}\mathrm{to}{\widehat{B}}_k\mathit{y}={\widehat{\mathit{e}}}_k,\end{array}$$

(9)

where ${\mathit{e}}_1={(1,0,\cdots ,0)}^T$, and ${\widehat{B}}_k$ and ${\widehat{\mathit{e}}}_k$ are generally full. As in LSQR, seek an approximate TLS solution

$$\begin{array}{c}\hfill {\mathit{x}}_k=V_k{\mathit{y}}_k\in {\mathcal{K}}_k(A^{T}A,A^T\mathit{b}),\end{array}$$

where ${\mathcal{K}}_k(B,\mathit{y})$ denotes the Krylov subspace span$\{\mathit{y},B\mathit{y},\dots ,B^{k-1}\mathit{y}\}$.

Let the SVD of $({\beta }_1{\mathit{e}}_1,B_k)={\overline{W}}_k{\overline{\Sigma }}_k{\overline{Z}}_k^T$ and if let

$$\begin{array}{c}\hfill \left(\begin{array}{c}{\gamma }_k\\ {\mathit{z}}_k\end{array}\right)=\left(\begin{array}{cc}1& 0\\ 0& V_k\end{array}\right)\left({\overline{Z}}_k{\mathit{e}}_{k+1}\right)\end{array}$$

(10)

with

$${\overline{Z}}_k=\left(\begin{array}{cc}{\overline{Z}}_1& {\overline{Z}}_2\end{array}\right)=\begin{array}{cc}k& 1\\ {\overline{Z}}_{11}& {\overline{\mathit{z}}}_{12}& 1\\ {\overline{Z}}_{21}& {\overline{\mathit{z}}}_{22}& k\end{array}$$

(11)

then the approximate TLS solution is given by

$$\begin{array}{c}\hfill {\mathit{x}}_k=-\frac{{\mathit{z}}_k}{{\gamma }_k}\in {\mathcal{K}}_k(A^{T}A,A^T\mathit{b}).\end{array}$$

(12)

Note that we only need the last singular vector ${\overline{Z}}_k{\mathit{e}}_{k+1}$ to compute ${\mathit{x}}_k$. To this extent, summarizing the above process, we can get the Householder bidiagonalization TLS (HBITLS) algorithm as follows:

Remark 1.

A variant of Algorithm 1 can also be given, in which the product of the Householder transformations applying to vectors are replaced by operations that can be performed concurrently, to a large extent. This variation gives an efficient method for developing parallelism in the case of parallel computing matrix vector products. In regard to this variation of Algorithm 1, one can refer to [12] and we omit it here.

Algorithm 1 HBITLS

1: Initialize: $C=(\mathit{b},A),$ $U:=I$, $V:=I$. 2: for $j=1,\cdots ,n+1$   $[\mathit{v},\beta ]=\mathrm{Householder}\left(C\right(j:m,j\left)\right);$   $P=I_{m-j+1}-\beta \mathit{v}{\mathit{v}}^T;$   $C(j:m,j:n+1)=PC(j:m,j:n+1);$   $G=\mathrm{diag}\{I_{j-1},P\};$   $U=UG;$   ${\mathit{u}}_j=U(:,j);$ if $j\le n-1$   $[\mathit{v},\beta ]=\mathrm{Householder}\left(C\right(j,j+1:n+1\left)\right);$   $Q=I_{n-j}-\beta \mathit{v}{\mathit{v}}^T;$   $C(j:m,j+1:n+1)=C(j:m,j+1:n+1)Q;$   $H=\mathrm{diag}\{I_j,Q\};$   $V=VH;$   ${\mathit{v}}_j=V(:,j)$.   end 3: Compute the last singular triplet for matrix $C_k$ by employing the implicit zero-shift QR algorithm. 4: the approximate TLS solution is given by ${\mathit{x}}_k=-\frac{{\mathit{z}}_k}{{\gamma }_k}$, see (10). 5: end

Using the recursions (4) and (5), the following properties of $\{{\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_i\}$ and $\{{\mathit{u}}_1,{\mathit{u}}_2,\cdots ,{\mathit{u}}_i\}$ can be proved.

Lemma 1.

The sets $\{v_1,v_2,\cdots ,v_i\}$ and $\{u_1,u_2,\cdots ,u_i\}$ generated by Algorithm 1 are the orthonormal basis of ${\mathcal{K}}_i(A^{T}A,A^T\mathit{b})$ and ${\mathcal{K}}_i(A{A}^T,A{A}^T\mathit{b})$ respectively.

Proof. 

As a result of the facts that ${\beta }_1{\mathit{v}}_1=A^T\mathit{b}$, ${\alpha }_1{\mathit{u}}_1=A{\mathit{v}}_1=A{A}^T\mathit{b}/{\beta }_1$, $H_1{\mathit{v}}_1={\mathit{e}}_1$ and the process of Householder bidiagonalization, for $1\le k\le i,$ it’s easy to know that ${\mathit{v}}_j=H_1\cdots H_k{\mathit{e}}_j$ for $j=1,\cdots ,k,$ i.e., that

$$\begin{array}{c}\hfill H_1\cdots H_k=[{\mathit{v}}_1,\cdots ,{\mathit{v}}_k,\cdots ],1\le k\le i.\end{array}$$

(13)

Certainly ${\mathcal{K}}_1(A^{T}A,A^T\mathit{b})=\mathbb{R}\left({\mathit{v}}_1\right)$. It clearly holds if $i=1$. Suppose for some $i>1$ that the iteration has produced $V_i=[{\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_i]$ with orthonormal columns such that

$${\mathcal{K}}_i(A^{T}A,A^T\mathit{b})=\mathrm{span}\{{\mathit{v}}_1,\left(A^{T}A\right){\mathit{v}}_1,\cdots ,{\left(A^{T}A\right)}^{i-1}{\mathit{v}}_1\}=\mathrm{span}\{{\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_i\}.$$

It is easy to see from (4) that

$$V_i^T{A}^{T}A{V}_i=B_i^T{B}_i$$

and we have $V_i^T{\mathit{r}}_i=0$, where ${\mathit{r}}_i=A^{T}A{\mathit{v}}_i-({\alpha }_i^2+{\beta }_{i+1}^2){\mathit{v}}_i-{\alpha }_{i-1}{\beta }_{i-1}{\mathit{v}}_{i-1}$. If ${\mathit{r}}_i\ne 0$, then ${\mathit{v}}_{i+1}={\mathit{r}}_i/{\parallel {\mathit{r}}_i\parallel }_2$ is orthogonal to ${\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_i$. It follows that ${\mathit{v}}_{i+1}\notin {\mathcal{K}}_i(A^{T}A,A^T\mathit{b})$ and

$${\mathit{v}}_{i+1}\in \mathrm{span}\{A^{T}A{\mathit{v}}_i,{\mathit{v}}_i,{\mathit{v}}_{i-1}\}\subseteq {\mathcal{K}}_{i+1}(A^{T}A,A^T\mathit{b}).$$

Thus, $V_{i+1}^T{V}_{i+1}^T=I$ and

$$\mathrm{span}\left(V_{i+1}\right)={\mathcal{K}}_{i+1}(A^{T}A,A^T\mathit{b}).$$

On the other hand, if ${\mathit{r}}_i=0$, then $A^{T}A{V}_i=V_i{B}_i^T{B}_i$. This says that $\mathrm{span}\left(V_i\right)={\mathcal{K}}_i(A^{T}A,A^T\mathit{b})$ is invariant for $A^{T}A$ and the induction is complete. The proof of $\mathrm{span}\left(U_i\right)={\mathcal{K}}_i(A{A}^T,A{A}^T\mathit{b})$ is in a similar way. □

The Householder matrices $H_i$ and $G_i$ need not be formed explicitly. In other words, the matrices $V_k$ and $U_k$ can also remain in product form in the HBITLS algorithm. In floating-point operations, the Householder transformation does not have to worry too much about the loss of orthogonality.

2.2. The NIPALS-TLS Algorithm

For the NIPALS PLS algorithm, one can see in [3,13]. In this paper, we want to use it to solve the TLS problems and then form the NIPALS-TLS algorithm. We can find that the HBITLS algorithm and NIPALS-TLS algorithm generate the same sequences, orthonormal base $V_k$. From the uniqueness of this base, and combined with the relationship between the two algorithms, we conclude that the two algorithms generate the same numerical solution ${\mathit{x}}_k.$

In [3], it tells us that we can set $A_0=A,$ ${\mathit{b}}_0=\mathit{b},$ for $k=1,2,\cdots$, we can produce sequences ${\mathit{u}}_k$ and ${\mathit{v}}_k$ according to the following form:

$$\begin{array}{ccc}\hfill {\mathit{v}}_k& =& A_{k-1}^T{\mathit{b}}_{k-1}/{\mu }_k,{\mu }_k={\parallel A_{k-1}^T{\mathit{b}}_{k-1}\parallel }_2,\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill {\mathit{p}}_k& =& A_{k-1}{\mathit{v}}_{k-1}/{\rho }_k,{\rho }_k={\parallel A_{k-1}{\mathit{v}}_{k-1}\parallel }_2,\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill (A_k,{\mathit{b}}_k)& =& (I-{\mathit{p}}_k{\mathit{p}}_k^T)(A_{k-1},{\mathit{b}}_{k-1}).\hfill \end{array}$$

(16)

In (16) $A_k$ and ${\mathit{b}}_k$ are formed by deflated $A_{k-1}$ and ${\mathit{b}}_{k-1}$ by subtracting their orthogonal projections onto ${\mathit{p}}_k.$ We know that this operation uses elementary orthogonal transformations, such that $S=I-\mathit{p}{\mathit{p}}^T$, ${\parallel \mathit{p}\parallel }_2=1.$ The deflation in () can also be written as

$$\begin{array}{ccc}\hfill A_k& =& A_{k-1}-{\mathit{p}}_k{\mathit{s}}_k^T,{\mathit{s}}_k=A_{k-1}^T{\mathit{p}}_k,\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill {\mathit{b}}_k& =& {\mathit{b}}_{k-1}-{\mathit{p}}_k{\zeta }_k^T,{\zeta }_k={\mathit{b}}_{k-1}^T{\mathit{p}}_k.\hfill \end{array}$$

(18)

The process is terminated when it meets either $\parallel A_{k-1}^T{\mathit{b}}_{k-1}{\parallel }_2=0$ or $\parallel A_{k-1}{\mathit{v}}_i{\parallel }_2=0$. We note that if ${\mathit{p}}_k^T{A}_{k-1}{\mathit{v}}_k\ne 0$, then the rank of the matrix $A_k$ is one less than that of $A_{k-1}$ exactly.

Using exact arithmetic, the sets $\{{\mathit{v}}_1,{\mathit{v}}_2,\cdots ,{\mathit{v}}_k\}$ and $\{{\mathit{p}}_1,{\mathit{p}}_2,\cdots ,{\mathit{p}}_k\}$ generated by (14) and (15) are the unique orthogonal bases for the Krylov sub-spaces ${\mathcal{K}}_k(A^{T}A,A^T\mathit{b})$ and ${\mathcal{K}}_k(A{A}^T,A{A}^T\mathit{b})$, respectively. Summing (17) and (18) generates

$$\begin{array}{c}\hfill A=P_k{S}_k^T+A_k,\mathit{b}=P_k{\mathit{z}}_k+{\mathit{b}}_k,\end{array}$$

(19)

where $P_k=[{\mathit{p}}_1,{\mathit{p}}_2,\cdots ,{\mathit{p}}_k],$ $S_k=[{\mathit{s}}_1,{\mathit{s}}_2,\cdots ,{\mathit{s}}_k],$ and ${\mathit{z}}_k={({\zeta }_1,{\zeta }_2,\cdots ,{\zeta }_k)}^T$. These relationships maintain working accuracy and do not depend on orthogonality. The matrix $P_k{S}_k^T$ is a rank-k approximation to the data matrix A. From [3], we have $S_k^T=P_k^{T}A$ and $S_k^T{V}_k=R_k$. Thus, in exact arithmetic, the matrix $S_k^T{V}_k$ is upper bidiagonal with its elements

$$\begin{array}{c}\hfill {\theta }_k={\mathit{s}}_{k-1}^T{\mathit{v}}_k{\rho }_k={\mathit{s}}_k^T{\mathit{v}}_k={\parallel A_{k-1}{\mathit{v}}_k\parallel }_2,\end{array}$$

(20)

and

$$R_k\equiv \left(\begin{array}{ccccc}{\rho }_1& {\theta }_2& & & \\ & {\rho }_2& {\theta }_3& & \\ & & \ddots & \ddots & \\ & & & {\rho }_{k-1}& {\theta }_k\\ & & & & {\rho }_k\end{array}\right).$$

By Paige [14], we know that $R_k$ must be identical to the matrix that would be obtained from the conventional QR factorization of $B_k$, such that

$$Q_k{B}_k=\left(\begin{array}{c}{R}_k\\ 0\end{array}\right).$$

Then we have

$$\begin{array}{c}\hfill \left(\begin{array}{cc}{B}_k& {\beta }_1{\mathit{e}}_1\end{array}\right)=Q_k^T\left(\begin{array}{cc}{R}_k& {\mathit{z}}_k\\ & \parallel {\mathit{b}}_k{\parallel }_2\end{array}\right)\equiv Q_k^T{N}_k.\end{array}$$

(21)

Let $N_k={\check{U}}_k{\check{\Sigma }}_k{\check{V}}_k$, and ${\check{V}}_k{\mathit{e}}_{k+1}\equiv {\check{\mathit{v}}}_{k+1}={\left(\begin{array}{cc}{\left({\check{\mathit{v}}}_{k+1}^{\left(1\right)}\right)}^T& {\check{\mathit{v}}}_{k+1}^{\left(2\right)}\end{array}\right)}^T,$ then the solution of the projected TLS problem (9) is ${\mathit{y}}_k=-\frac{{\check{\mathit{v}}}_{k+1}^{\left(1\right)}}{{\check{v}}_{k+1}^{\left(2\right)}}$, and the TLS solution is ${\mathit{x}}_k=-V_k\frac{{\check{\mathit{v}}}_{k+1}^{\left(1\right)}}{{\check{v}}_{k+1}^{\left(2\right)}}$. And the following theorem comes by (12) and (21)

Theorem 1.

the HBITLS and NIPALS-TLS algorithms compute the same approximate solutions ${\mathit{x}}_k$.

3. Roundoff Error Analysis

In this section, we analyze the propagation of roundoff error during the process of the HBITLS algorithm and get the mixed forward–backward stability of the HBITLS algorithm and NIPALS-TLS algorithm naturally. The total roundoff error during the process of the HBITLS algorithm can be divided into the following four parts:

First, we can find that the HBITLS algorithm solves the original TLS problem (2) to a perturbed TLS problem. The propagation of the roundoff error of a Householder matrix in the HBITLS algorithm is advantageous when performing numerical computations.

From now one, we will denote by $\epsilon$ the machine precision under consideration. In [15], it shows that the computed Householder matrix $fl\left(H\right)$ comes near the exact Householder matrix H itself:

$$\begin{array}{c}\hfill {\parallel fl\left(H\right)-H\parallel }_2\le 84\epsilon +\mathcal{O}\left({\epsilon }^2\right).\end{array}$$

Moreover, for a vector $\mathit{y}$, the computed updates with $fl\left(H\right)$ are very close to the exact updates with H:

$$\begin{array}{c}\hfill fl\left(fl\left(H\right)\mathit{y}\right)={H(\mathit{y}+\mathit{w}),\parallel \mathit{w}\parallel }_2\le 87\epsilon {\parallel \mathit{y}\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).\end{array}$$

and, in general,

$$\begin{array}{c}\hfill fl(fl\left(H_1\right)\cdots fl\left(H_j\right)\mathit{y})=H_1\cdots H_j(\mathit{y}+\mathit{z}){,\parallel \mathit{z}\parallel }_2\le 87j\epsilon {\parallel \mathit{y}\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).\end{array}$$

(22)

The following lemma tells us that the reduced system calculated by the HBITLS algorithm is equivalent to the system formed after the original system has been disturbed. ${\overline{B}}_k$, ${\widehat{Q}}_k$ and ${\widehat{P}}_{k+1}$ are the floating-point computation of the matrices $B_k$, $V_k$ and $U_{k+1}$ in HBITLS algorithm, respectively.

Lemma 2.

Let ${\overline{B}}_k$ be the computed bidiagonalization matrix $(k+1)\times k$ matrix obtained by the HBITLS algorithm. Then, there comes a perturbation matrix E and exists two column orthogonal matrices ${\widehat{Q}}_k$ and ${\widehat{P}}_{k+1}$ s.t.

$$\begin{array}{c}\hfill (A+E){\widehat{Q}}_k={\widehat{P}}_{k+1}{\overline{B}}_k,\end{array}$$

and

$${\parallel E\parallel }_2\le \sqrt{n}(174n+3\sqrt{n}+87)\epsilon {\parallel A\parallel }_2+\mathcal{O}\left({\epsilon }^2\right),$$

where n is the number of columns of matrix A. Furthermore, the matrix ${\widehat{Q}}_k$ is an orthonormal basis of ${\mathcal{K}}_k\left({(A+E)}^T(A+E),{(A+E)}^T(b+e)\right)$ with a perturbation vector $e$, where ${\parallel e\parallel }_2\le 87\epsilon {\parallel b\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).$

Proof. 

We prove this theorem by induction. The key point is that we should show the computed matrix, which will be shown by introduction from (3), for $G_k^T\cdots G_1^T[{\mathit{v}}_1,A{\mathit{v}}_1,\cdots ,A{\mathit{v}}_k]$ as follows

$${\widehat{P}}_{k+1}\cdots {\widehat{P}}_1[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,(A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1,\cdots ,(A+G_k){\overline{\mathit{v}}}_k+{\mathit{w}}_k].$$

For $k=1$, first, let ${\mathit{u}}_1=\mathit{b}/{\parallel \mathit{b}\parallel }_2$, ${\overline{\mathit{u}}}_1=fl\left({\mathit{u}}_1\right)$, a Householder matrix $P_1$ is found s.t. $P_1{\mathit{u}}_1={\mathit{e}}_1$. Set ${\overline{P}}_1=fl\left(P_1\right),$ ref. [16] tells us that, corresponding to matrix ${\overline{P}}_1$, we can find a Householder matrix to make

$$fl\left({\overline{P}}_1{\overline{\mathit{v}}}_1\right)={\widehat{P}}_1({\overline{\mathit{v}}}_1+{\mathit{g}}_1),$$

with

$$\parallel {\mathit{g}}_1{\parallel }_2\le 87\epsilon {\parallel {\overline{\mathit{v}}}_1\parallel }_2+\mathcal{O}\left({\epsilon }^2\right)=87\epsilon +\mathcal{O}\left({\epsilon }^2\right).$$

Next, let ${\mathit{v}}_1=A^T{\mathit{u}}_1/{\parallel A^T{\mathit{u}}_1\parallel }_2,$ $fl\left(A^T{\mathit{u}}_1\right)={(A+G_0)}^T{\overline{\mathit{u}}}_1,$ where $\parallel G_0{\parallel }_2\le \epsilon 3\sqrt{n}{\parallel A\parallel }_2.$ Similarly, for $A{\mathit{v}}_1,$ the computed result can be written as $fl\left(A{\mathit{v}}_1\right)=(A+G_1){\overline{\mathit{v}}}_1,$ where $\parallel G_1{\parallel }_2\le \epsilon 3\sqrt{n}{\parallel A\parallel }_2.$ Now, we set the Householder matrix $P_2$ s.t. $P_2{P}_1[{\mathit{v}}_1,A{\mathit{v}}_1]$ is upper bidiagonal matrix. We know $P_2$ only works the vector $P_{1}A{\mathit{v}}_1,$ so there’s no change for the 1st column of $fl\left({\overline{P}}_1[fl\left({\mathit{v}}_1\right),fl\left(A{\mathit{v}}_1\right)]\right)$ when producted by ${\overline{P}}_2$ and ${\widehat{P}}_2$. Likely, there’s a Householder matrix ${\widehat{P}}_2$ s.t. ${\widehat{P}}_{2}fl({\overline{P}}_1[{\overline{\mathit{v}}}_1,A{\overline{\mathit{v}}}_1]$ is bidiagonal matrix in theory, but the algorithm computes a matrix ${\overline{P}}_2$ in practice [16] such that

$$fl\left({\overline{P}}_2{\overline{P}}_{1}fl\left(A{\mathit{v}}_1\right)\right)={\widehat{P}}_2{\widehat{P}}_1((A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1),$$

where

$$\parallel {\mathit{w}}_1{\parallel }_2\le {174\epsilon (\parallel A\parallel }_2+\parallel G_1{\parallel }_2)\parallel {\overline{\mathit{v}}}_1{\parallel }_2=174\epsilon {\parallel A\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).$$

Finally, we have

$$\begin{array}{ccc}\hfill fl\left({\overline{P}}_2{\overline{P}}_1[{\overline{\mathit{v}}}_1,fl\left(A{\mathit{v}}_1\right)]\right)& =& \left[fl\left({\overline{P}}_1{\overline{\mathit{v}}}_1\right),fl\left({\overline{P}}_{2}fl\left({\overline{P}}_{1}fl\left(A{\mathit{v}}_1\right)\right)\right)\right]\hfill \\ & =& \left[{\widehat{P}}_1({\overline{\mathit{v}}}_1+{\mathit{g}}_1),{\widehat{P}}_2{\widehat{P}}_1((A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1)\right]\hfill \\ & =& {\widehat{P}}_2{\widehat{P}}_1\left[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,(A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1\right].\hfill \end{array}$$

For the kth step, assume that the HBITLS algorithm has calculated the matrices ${\overline{P}}_1,\cdots ,{\overline{P}}_k,$ associated with the Householder matrices ${\widehat{P}}_1,\cdots ,{\widehat{P}}_k$. Then, after k steps, we can get the following result:

$$\begin{array}{ccc}& & fl({\overline{P}}_k\cdots {\overline{P}}_1[{\overline{\mathit{v}}}_1,fl\left(A{\mathit{v}}_1\right),\cdots ,fl\left(A{\mathit{v}}_k\right)])\hfill \\ & & ={\widehat{P}}_k\cdots {\widehat{P}}_1\left[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,(A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1,\cdots ,(A+G_k){\overline{\mathit{v}}}_k+{\mathit{w}}_k\right],\hfill \end{array}$$

where $\parallel {\mathit{w}}_i{\parallel }_2\le 87(i+1)\epsilon {\parallel A\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).$

We know ${\mathit{v}}_k=Q_1\cdots Q_k{\mathit{e}}_k,$ and ${\overline{\mathit{v}}}_k=fl({\overline{Q}}_1\cdots {\overline{Q}}_k{\mathit{e}}_k)={\widehat{Q}}_1\cdots {\widehat{Q}}_k({\mathit{e}}_k+{\mathit{f}}_k),$ where $\parallel {\mathit{f}}_k{\parallel }_2\le 87k\epsilon +\mathcal{O}\left({\epsilon }^2\right).$ For $A{\mathit{v}}_k,$ the floating-point vector is $fl\left(A{\mathit{v}}_k\right)=(A+G_k){\overline{\mathit{v}}}_k,$ where $\parallel G_k{\parallel }_2\le \epsilon 3\sqrt{n}{\parallel A\parallel }_2.$ Likely, there is a Householder matrix $P_{k+1}$, which only works on the vector $P_k\cdots P_{1}A{\mathit{v}}_k$, s.t. $P_{k+1}{P}_k\cdots P_1[{\mathit{v}}_1,A{\mathit{v}}_1,\cdots ,A{\mathit{v}}_k]$ is the upper bidiagonal matrix. The algorithm computes a matrix ${\overline{P}}_{k+1}$ in practice so that

$$fl({\overline{P}}_{k+1}\cdots {\overline{P}}_{1}fl\left(A{\mathit{v}}_k\right))={\widehat{P}}_{k+1}\cdots {\widehat{P}}_1((A+G_k){\overline{\mathit{v}}}_k+{\mathit{w}}_k),$$

where

$$\parallel {\mathit{w}}_k{\parallel }_2\le {87(k+1)\epsilon (\parallel A\parallel }_2+\parallel G_k{\parallel }_2\left)\right(1+\parallel {\mathit{f}}_k{\parallel }_2{)\le 87(k+1)\epsilon \parallel A\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).$$

Then the floating-point matrix is obtained, such that

$${\widehat{P}}_{k+1}\cdots {\widehat{P}}_1[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,(A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1,\cdots ,(A+G_k){\overline{\mathit{v}}}_k+{\mathit{w}}_k].$$

Let ${\widehat{P}}^{\left(k\right)}$, ${\widehat{Q}}^{\left(k\right)}$ be the matrices, such that ${\left({\widehat{P}}^{\left(k\right)}\right)}^T={\widehat{P}}_k\cdots {\widehat{P}}_1$ and ${\left({\widehat{Q}}^{\left(k\right)}\right)}^T={\widehat{Q}}_k\cdots {\widehat{Q}}_1$, respectively, we find that the first $(i-1)$ rows of each ${\widehat{Q}}_i$ is ${\mathit{e}}_1,\cdots ,{\mathit{e}}_{i-1}.$ Let ${\mathit{q}}_i^{\left(j\right)}$ be the i-th column of ${\widehat{Q}}^{\left(j\right)}$, then the results are as follows

$${\mathit{q}}_i^{\left(j\right)}={\mathit{q}}_i^{\left(k\right)},\forall k\ge j,i=1,\cdots ,j.$$

Then there comes

$${\left({\widehat{P}}^{(k+1)}\right)}^T[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,(A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1,\cdots ,(A+G_k){\overline{\mathit{v}}}_k+{\mathit{w}}_k],$$

it is an $n\times (k+1)$ upper bidiagonal matrix. And, $\forall j\le k,$ we obtain

$$\begin{array}{ccc}\hfill (A+G_j){\overline{\mathit{v}}}_j+{\mathit{w}}_j& =& (A+G_j){\widehat{Q}}_1\cdots {\widehat{Q}}_j({\mathit{e}}_j+{\mathit{f}}_j)+{\mathit{w}}_j\hfill \\ & =& A{\widehat{Q}}^{\left(j\right)}{\mathit{e}}_j+G_j{\widehat{Q}}^{\left(j\right)}{\mathit{e}}_j+A{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+G_j{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+{\mathit{w}}_j\hfill \\ & =& A{q}_j^{\left(j\right)}+G_j{q}_j^{\left(j\right)}+A{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+G_j{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+{\mathit{w}}_j.\hfill \end{array}$$

Since $\forall k\le j,$ ${\mathit{q}}_j^{\left(j\right)}$ is the j-th column ${\mathit{q}}_j$ of the matrix ${\widehat{Q}}^{\left(j\right)}$, if we denote by ${\mathit{y}}_j=G_j{\mathit{q}}_j^{\left(j\right)}+A{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+G_j{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+{\mathit{w}}_j,$ we have

$$(A+G_j){\overline{\mathit{v}}}_j+{\mathit{w}}_j=A{\mathit{q}}_j+{\mathit{y}}_j,$$

and so we can obtain

$$\begin{array}{ccc}& & {\left({\widehat{P}}^{(k+1)}\right)}^T[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,(A+G_1){\overline{\mathit{v}}}_1+{\mathit{w}}_1,\cdots ,(A+G_k){\overline{\mathit{v}}}_k+{\mathit{w}}_k]\hfill \\ & =& {\left({\widehat{P}}^{(k+1)}\right)}^T[{\overline{\mathit{v}}}_1+{\mathit{g}}_1,A{\mathit{q}}_1+{\mathit{y}}_1,\cdots ,A{\mathit{q}}_k+{\mathit{y}}_k]\hfill \\ & =& {\left({\widehat{P}}^{(k+1)}\right)}^T\left[({\overline{\mathit{v}}}_1,A{\mathit{q}}_1,\cdots ,A{\mathit{q}}_k)+({\mathit{g}}_1,{\mathit{y}}_1,\cdots ,{\mathit{y}}_k)\right].\hfill \end{array}$$

If we cut off the first column of the matrix, we can set ${\tilde{B}}_k$ with $n\times k$ such that ${\tilde{B}}_k={({\overline{B}}_k^T,0)}^T$, here ${\overline{B}}_k$ is an $(k+1)\times k$ upper bidiagonal matrix. If we denote $F_i=[{\mathit{y}}_l,\cdots ,{\mathit{y}}_i],$ $\forall i,$ then ${\tilde{B}}_k={\left({\widehat{P}}^{(k+1)}\right)}^T\left(A[{\mathit{q}}_1,\cdots ,{\mathit{q}}_k]+F_k\right).$ In addition, $\forall j\le k,$ let ${\widehat{Q}}_j$ be the matrix made up of the first j columns of ${\widehat{Q}}^{\left(k\right)}$, we obtain ${\widehat{P}}^{(k+1)}{\tilde{B}}_k=A{\widehat{Q}}^{\left(k\right)}+F_k,$ and, from the structure of ${\tilde{B}}_k,$ there comes ${\widehat{P}}_{k+1}{\overline{B}}_k=A{\widehat{Q}}_k+F_k.$ Then we can write $F_k=F_n{\widehat{Q}}_n^T{\widehat{Q}}_k$ owing to ${\widehat{Q}}_n^T{\widehat{Q}}_k=\left(\begin{array}{c}{I}_k\\ 0\end{array}\right),$ and so

$${\widehat{P}}_{k+1}{\overline{B}}_k=(A+F_n{\widehat{Q}}_n^T){\widehat{Q}}_k.$$

If $E=F_n{\widehat{Q}}_n^T,$ we can finish the proof of the first part of the lemma, because

$$\begin{array}{ccc}\hfill {\parallel E\parallel }_2& \le & {\parallel E\parallel }_F={\parallel F_n\parallel }_F\hfill \\ & =& \sqrt{\sum _{j=1}^n{\parallel {\mathit{y}}_j\parallel }_2^2}\hfill \\ & \le & \sqrt{\sum _{j=1}^n{\parallel G_j{\mathit{q}}_j^{\left(j\right)}+A{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+{\mathit{w}}_j\parallel }_2^2}+\mathcal{O}\left({\epsilon }^2\right)\hfill \\ & \le & \sqrt{n}\underset{j}{max}{\parallel G_j{\mathit{q}}_j^{\left(j\right)}+A{\widehat{Q}}^{\left(j\right)}{\mathit{f}}_j+{\mathit{w}}_j\parallel }_2+\mathcal{O}\left({\epsilon }^2\right)\hfill \\ & \le & \sqrt{n}(174n+3\sqrt{n}+87)\epsilon {\parallel A\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).\hfill \end{array}$$

Finally, we prove that the subspace spanned by the columns of the matrix ${\widehat{Q}}_k$ is an orthogonal basis of a Krylov space. Let $\tilde{A}=A+E$, $\tilde{b}={\parallel A^T\mathit{b}\parallel }_2{\widehat{Q}}_k{\mathit{e}}_1$ and form ${\mathcal{K}}_k({\tilde{A}}^T\tilde{A},{\tilde{A}}^T\tilde{\mathit{b}})$. We know that $\tilde{\mathit{b}}=\parallel A^T{\mathit{b}\parallel }_2{\widehat{Q}}_k{\mathit{e}}_1={\parallel A^T\mathit{b}\parallel }_2{\widehat{Q}}_1{\mathit{e}}_1$ and set $e=\tilde{\mathit{b}}-A^T\mathit{b}$, then we have ${\parallel \mathit{e}\parallel }_2=\parallel A^T{\mathit{b}\parallel }_2\parallel ({\widehat{Q}}_1-Q_1){\mathit{e}}_1{\parallel }_2\le 87\epsilon {\parallel A^T\mathit{b}\parallel }_2+\mathcal{O}\left({\epsilon }^2\right).$

We still prove the rest of the theorem by induction; that is, to prove

$${\left({\tilde{A}}^T\tilde{A}\right)}^i{\tilde{A}}^T\tilde{\mathit{b}}={\widehat{Q}}_k{\mathit{r}}_i,\forall i\le k-1,$$

where each vector ${\mathit{r}}_i$ has only the first $(i+1)$ components, which are different from zero.

For $i=1$, we have

$${\tilde{A}}^T\tilde{A}{\tilde{A}}^T\tilde{\mathit{b}}={\tilde{A}}^T\tilde{A}\parallel A^T{\mathit{b}\parallel }_2{\widehat{Q}}_k{\mathit{e}}_1={\parallel A^T\mathit{b}\parallel }_2{\widehat{Q}}_k{\overline{B}}_k^T{\overline{B}}_k{\mathit{e}}_1={\widehat{Q}}_k{\mathit{r}}_1,$$

since the last component of the vector ${\overline{B}}_k^T{\overline{B}}_k{\mathit{e}}_1$ is zero, in addition, except for the first two components, the rest of the components of vector ${\mathit{r}}_1$ are all zero.

Suppose for a given i the following relation is true,

$${\left({\tilde{A}}^T\tilde{A}\right)}^{i+1}{\tilde{A}}^T\tilde{\mathit{b}}=\left({\tilde{A}}^T\tilde{A}\right){\widehat{Q}}_k{\mathit{r}}_i={\widehat{Q}}_{k+1}{\overline{B}}_k^T{\overline{B}}_k{\mathit{r}}_i,$$

and in the next step, we will show it is true for $i+1$.

From the inductive hypothesis, we know that only the first $(i+1)$ components of ${\mathit{r}}_i$ are not zero; therefore, the last component is zero of the vector ${\overline{B}}_k^T{\overline{B}}_k{\mathit{r}}_i$ with $(i+2)$ non-zero elements. Then there comes a conclusion that

$${\left({\tilde{A}}^T\tilde{A}\right)}^{i+1}{\tilde{A}}^T\tilde{\mathit{b}}={\widehat{Q}}_k{\mathit{r}}_{i+1}$$

and, hence, the lemma is proved. □

Based on Lemma 2 and Algorithm 1, for the bidiagonalization matrix ${\overline{B}}_k$ obtained by the HBITLS algorithm, one can find an orthonormal matrix ${\widehat{Q}}_k$ s.t.

$$(A+E){\widehat{Q}}_k={\widehat{P}}_{k+1}{\overline{B}}_k,$$

where ${\widehat{Q}}_k$ is just an orthogonal basis of ${\mathcal{K}}_k({(A+E)}^T(A+E),{(A+E)}^T(\mathit{b}+\mathit{e}))$. Based on this, we know that the first part of HBITLS, in exact arithmetic, gives the exact basis of the perturbed Krylov space ${\mathcal{K}}_k({(A+E)}^T(A+E),{(A+E)}^T(\mathit{b}+\mathit{e}))$. A perturbation bound for TLS solutions is given by Xie and Wei [17], see Lemma 3, which is related to the smallest singular value ${\sigma }_{n+1}$ of $(A,\mathit{b})$, the TLS solution $\mathit{x}$, and the residual $\mathit{r}=\mathit{b}-A\mathit{x}$. Let $\widehat{A}=A+\Delta A$ and $\widehat{\mathit{b}}=\mathit{b}+\Delta \mathit{b}$. Then, the unique solution of the perturbed TLS problem can be expressed as $\widehat{x}$. Denote ${\kappa }_{\mathit{b}}=\frac{{\parallel \mathit{b}\parallel }_2}{{\parallel \mathit{x}\parallel }_2}{\parallel B_{\lambda }^{-1}{A}^T\parallel }_2$ and ${\kappa }_A=\frac{{\parallel A\parallel }_2}{{\parallel \mathit{x}\parallel }_2}\left({\parallel \mathit{r}\parallel }_2\parallel B_{\lambda }^{-1}{\parallel }_2+{\parallel x\parallel }_2{\parallel B_{\lambda }^{-1}{A}^T\parallel }_2\right)$ with $B_{\lambda }=A^{T}A-{\sigma }_{n+1}^{2}I.$ The perturbation bound is obtained under the genericity condition ${\overline{\sigma }}_n>{\sigma }_{n+1}$, where ${\overline{\sigma }}_n$ is the smallest singular value of A

Lemma 3

([17]). Consider the TLS problem (2) and assume that the genericity condition holds. If ${\parallel (\Delta A,\Delta \mathit{b})\parallel }_F$ is sufficiently small, then we obtain that

$$\begin{array}{ccc}\hfill \frac{\parallel \mathit{x}-\widehat{\mathit{x}}{\parallel }_2}{{\parallel \mathit{x}\parallel }_2}& \lesssim & {\kappa }_{\mathit{b}}\frac{{\parallel \Delta \mathit{b}\parallel }_2}{{\parallel \mathit{b}\parallel }_2}+{\kappa }_A\frac{{\parallel \Delta A\parallel }_2}{{\parallel A\parallel }_2}.\hfill \end{array}$$

Suppose that $\mathit{x}$ and $\widehat{\mathit{x}}$ are the exact TLS solutions of $A\mathit{x}\approx \mathit{b}$ and $(A+E)\widehat{\mathit{x}}\approx \mathit{b}+\mathit{e}$ respectively. The error introduced in this part of the HBITLS algorithm is the inherent error, so we can give $\parallel \mathit{x}-\widehat{\mathit{x}}{\parallel }_2$ by Lemma 3 and Lemma 2 easily, see Theorem 2.

Secondly, let us consider the error between the TLS solution of the system $(A+E)\widehat{\mathit{x}}\approx \mathit{b}+\mathit{e}$ and the approximation solution ${\widehat{\mathit{x}}}_k={\widehat{V}}_k{\widehat{\mathit{y}}}_k$ of the system computed by the HBITLS algorithm at step k with the exact arithmetic, i.e., ${\widehat{\mathit{y}}}_k$ is the exact solution of the reduced TLS problem

$$\begin{array}{c}\hfill min\parallel ({\beta }_1{\mathit{e}}_1,{\overline{B}}_k)-({\check{\mathit{e}}}_k,{\check{B}}_k){\parallel }_F\mathrm{subject}\mathrm{to}{\check{B}}_k{\widehat{\mathit{y}}}_k={\check{\mathit{e}}}_k,\end{array}$$

(23)

For convenience, define ${\overline{V}}_k=\left(\begin{array}{cc}1& 0\\ 0& V_k\end{array}\right)$ and let

$$\begin{array}{c}\hfill (\mathit{b},A)=W\Sigma Z^T,\end{array}$$

(24)

where W and $Z=[{\mathit{z}}_1,{\mathit{z}}_2,\cdots ,{\mathit{z}}_{n+1}]$ are orthogonal matrices of dimension m and $n+1$, respectively, $\Sigma$ is an $m\times (n+1)$ diagonal matrix whose diagonal entries ${\sigma }_i$ are the singular values of $(\mathit{b},A)$, sorted in non-increasing order. Let $\theta (\mathit{u},\mathit{v})$ be the subspace angle between $\mathbb{R}\left(\mathit{u}\right)$ and $\mathbb{R}\left(\mathit{v}\right)$.

Lemma 4.

Let $\mathit{x}$ denotes the essential TLS solution to the linear system (2) satisfying genericity condition and ${\mathit{x}}_k$ be the approximation solution obtained from Algorithm 1. Then

$$\begin{array}{c}\hfill sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)})\le \parallel \mathit{x}-{\mathit{x}}_k\parallel \le sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)})\sqrt{1+{\parallel \mathit{x}\parallel }^2}\sqrt{1+\parallel {\mathit{x}}_k{\parallel }^2},\end{array}$$

(25)

where

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathit{z}& \equiv {\mathit{z}}_{n+1}=\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right)\in {\mathbb{R}}^{n+1},\hfill \\ \hfill {\mathit{z}}^{\left(k\right)}& \equiv \left(\begin{array}{c}{\mathit{z}}_{12}^{\left(k\right)}\\ {\mathit{z}}_{22}^{\left(k\right)}\end{array}\right)={\overline{V}}_k{\overline{\mathit{z}}}_{k+1}=\left(\begin{array}{cc}1& 0\\ 0& V_k\end{array}\right)\left(\begin{array}{c}{\overline{\mathit{z}}}_{12}\\ {\overline{\mathit{z}}}_{22}\end{array}\right)\in {\mathbb{R}}^{n+1}.\hfill \end{array}\end{array}$$

(26)

Proof. 

It is easy to know that

$$\begin{array}{c}\hfill \left(\begin{array}{c}-1\\ \mathit{x}\end{array}\right)-\left(\begin{array}{c}-1\\ {\mathit{x}}_k\end{array}\right)=\left(\begin{array}{c}{\mathit{z}}_{12}^{\left(k\right)}\\ {\mathit{z}}_{22}^{\left(k\right)}\end{array}\right){\left({\mathit{z}}_{12}^{\left(k\right)}\right)}^{-1}-\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right){\mathit{z}}_{12}^{-1}.\end{array}$$

(27)

Then we have an orthonormal matrix $G\in {\mathbb{R}}^{(n+1)\times n}$ with the partition

$$G=\stackrel{n}{\left[\begin{array}{c}{G}_1\\ G_2\end{array}\right]}\begin{array}{c}1\\ n\end{array}$$

such that $G^T{\left({\left({\mathit{z}}_{12}^{\left(k\right)}\right)}^T{\left({\mathit{z}}_{22}^{\left(k\right)}\right)}^T\right)}^T=0$ (i.e., G “forms a complete space”). From Equation (27) there comes

$$\begin{array}{c}\hfill G^T\left[\left(\begin{array}{c}-1\\ \mathit{x}\end{array}\right)-\left(\begin{array}{c}-1\\ {\mathit{x}}_k\end{array}\right)\right]=-G^T\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right){\mathit{z}}_{12}^{-1},\end{array}$$

(28)

and, therefore

$$\begin{array}{c}\hfill G_2^T(\mathit{x}-{\mathit{x}}_k)=-G^T\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right){\mathit{z}}_{12}^{-1}.\end{array}$$

(29)

From the CS theorem [18], we know that ${\sigma }_{min}{\left(G_2\right)}^{-1}={\left({\mathit{z}}_{12}^{\left(k\right)}\right)}^{-1}$. Then

$$\begin{array}{ccc}\hfill {\sigma }_{min}\left(G_2\right)\parallel (\mathit{x}-{\mathit{x}}_k)\parallel & \le & \parallel G_2^T(\mathit{x}-{\mathit{x}}_k)\parallel \hfill \\ & =& \parallel G^T\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right){\mathit{z}}_{12}^{-1}\parallel \hfill \\ & \le & sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)}){\mathit{z}}_{12}^{-1}.\hfill \end{array}$$

It was noticed that

$$sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)})=\parallel G^T\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right)\parallel$$

denotes the sine of the subspace angle between $\mathbb{R}\left(\mathit{z}\right)$ and $\mathbb{R}\left({\mathit{z}}^{\left(k\right)}\right).$ Hence, the upper bound can be proved as follows

$$\begin{array}{ccc}\hfill \parallel \mathit{x}-{\mathit{x}}_k\parallel & \le & sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)}){\mathit{z}}_{12}^{-1}{\left({\mathit{z}}_{12}^{\left(k\right)}\right)}^{-1}\hfill \\ & =& sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)})\sqrt{1+{\parallel \mathit{x}\parallel }^2}\sqrt{1+\parallel {\mathit{x}}_k{\parallel }^2}.\hfill \end{array}$$

For the lower bound, we have

$$\begin{array}{ccc}\hfill \parallel \mathit{x}-{\mathit{x}}_k\parallel & \ge & \parallel G_2^T(\mathit{x}-{\mathit{x}}_k)\parallel \hfill \\ & =& \parallel G^T\left(\begin{array}{c}{\mathit{z}}_{12}\\ {\mathit{z}}_{22}\end{array}\right){\mathit{z}}_{12}^{-1}\parallel \hfill \\ & \ge & {\mathit{z}}_{12}^{-1}sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)})\hfill \\ & \ge & sin\theta (\mathit{z},{\mathit{z}}^{\left(k\right)}).\hfill \end{array}$$

Since ${\mathit{z}}_{12}^{-1}\ge 1$, and this proves the upper bound case. □

Thirdly, we need to consider how to solve problem (9) and show that the error is between the solution obtained by this method and the theoretical solution. Let the computed solution be

• ${\overline{\mathit{x}}}_k={\widehat{V}}_k{\overline{\mathit{y}}}_k$, where ${\overline{\mathit{y}}}_k$ is the computed solution of the problem (23).

In [19], James and Kahan posed an algorithm named QR iteration with a zero shift, which guaranteed forward stability. Furthermore, an implicit algorithm about it is given. Error analysis including the singular values and singular vectors are also given, which is just what we’ve needed.

Lemma 5

([19]). Let the matrix $\overline{B}$ obtained by running the implicit zero-shift QR algorithm on a bidiagonal matrix B with $n\times n$. Suppose that all perturbation angles θ emerged from the operations of the algorithm satisfy ${sin}^2\theta \le \tau <1$. Let ${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_n$ and ${\overline{\sigma }}_1\ge {\overline{\sigma }}_2\ge \cdots \ge {\overline{\sigma }}_n$ are the singular values of B and $\overline{B}$ respectively. If

$$\begin{array}{c}\hfill \omega \equiv \frac{88n\epsilon }{{(1-\tau )}^2}<1,\end{array}$$

(30)

then we have:

$$|{\sigma }_i-{\overline{\sigma }}_i|\le \frac{\omega }{1-\omega }{\sigma }_i.$$

Moreover, let ${\sigma }_{k1}\ge {\sigma }_{k2}\ge \cdots \ge {\sigma }_{kn}$ be the singular values of $B_k$ produced after k steps of the implicit zero-shift QR algorithm. Then if condition (30) holds, and all perturbation angles θ satisfy ${sin}^2\theta \le \tau <1$, we obtain

$$|{\sigma }_i-{\sigma }_{ki}|\le \left(\frac{1}{{(1-\omega )}^k-1}{\sigma }_i\right)\approx \frac{88kn\epsilon }{{(1-\tau )}^2}{\sigma }_i.$$

James and Kahan [19] also give the relative differences between the singular vectors of B and the ones of $\overline{B}$.

Lemma 6

([19]). Let ${\sigma }_i$ be the singular value of be an unreduced bidiagonal matrix B with ${\mathit{u}}_i$ and ${\mathit{v}}_i$ being its corresponding left and right singular vectors, respectively. Let ${\overline{\mathit{u}}}_i$ and ${\overline{\mathit{v}}}_i$ be the singular vectors computed by the implicit zero-shift QR algorithm. Then the bound of the errors in ${\overline{\mathit{v}}}_i$ are shown by

$$\begin{array}{c}\hfill \theta ({\overline{\mathit{v}}}_i,{\mathit{v}}_i)\le p\left(n\right)\epsilon /re{l}_{gap}\equiv p\left(n\right)\epsilon /min\left(\right|{\sigma }_i-{\sigma }_{i+1}|/{\sigma }_i).\end{array}$$

(31)

Then, combining with the perturbation bound of TLS given in [20] as shown in Lemma 7, we can give the error estimate $\parallel {\widehat{\mathit{y}}}_k-{\overline{\mathit{y}}}_k{\parallel }_2$.

If let $(\tilde{A},\tilde{\mathit{b}})$ is a rank-k matrix approximation to $(A,\mathit{b})$, and $(\Delta \tilde{A},\Delta \tilde{\mathit{b}})=(A,\mathit{b})-(\tilde{A},\tilde{\mathit{b}})$. Let $(\overline{A},\overline{\mathit{b}})=(A,\mathit{b})+(\Delta A,\Delta \mathit{b})$ represent a perturbation of $(A,\mathit{b})$, $(\check{A},\check{\mathit{b}})$ denote a rank-k matrix approximation to $(\overline{A},\overline{\mathit{b}})$ and define $(\Delta \check{A},\Delta \check{\mathit{b}})=(\overline{A},\overline{\mathit{b}})-(\check{A},\check{\mathit{b}})$, then

Lemma 7

([20]). Let $\mathit{x}$ and $\overline{\mathit{x}}$ denote the TLS solution and the perturbed TLS solution. If $max(\parallel \Delta \tilde{A}\parallel ,\parallel \Delta \check{A}\parallel +\parallel \Delta A\parallel <{\sigma }_k^{{}^{\prime }}$(the k-th singular value of A) may be provided. Then

$$\begin{array}{c}\hfill sin\theta (\mathit{v},\overline{\mathit{v}})\le \parallel \mathit{x}-\overline{\mathit{x}}\parallel \le sin\theta (\mathit{v},\overline{\mathit{v}})\sqrt{1+{\parallel \mathit{x}\parallel }^2}\sqrt{1+\parallel \overline{\mathit{x}}{\parallel }^2},\end{array}$$

(32)

where $\mathit{v}$ and $\overline{\mathit{v}}$ are the smallest right singular vectors of $(A,\mathit{b})$ and $(\overline{A},\overline{\mathit{b}})$ respectively.

In summary, if let ${\tilde{\mathit{x}}}_k=fl\left({\overline{V}}_k{\overline{\mathit{y}}}_k\right)$ be the final computed solution at the k-th step, then roundoff error analysis of HBITLS algorithm for TLS problem can be shown as follows

Theorem 2.

Considering the HBITLS algorithm at step k, the roundoff error emerged during the algorithm can be bounded as follows:

$$\begin{array}{ccc}\hfill \parallel \mathit{x}-{\tilde{\mathit{x}}}_k\parallel & \le & {\kappa }_{\mathit{b}}87\epsilon +{\kappa }_A\sqrt{n}(174n+3\sqrt{n}+87)\epsilon \hfill \\ & +& sin\theta (\widehat{\mathit{z}},{\widehat{\mathit{z}}}^{\left(k\right)})\sqrt{1+\parallel \widehat{\mathit{x}}{\parallel }^2}\sqrt{1+\parallel {\widehat{\mathit{x}}}_k{\parallel }^2}\hfill \\ & +& sin\theta ({\widehat{\mathit{z}}}^{\left(k\right)},\check{\mathit{z}})\sqrt{1+\parallel {\widehat{\mathit{y}}}_k{\parallel }^2}\sqrt{1+\parallel {\overline{\mathit{y}}}_k{\parallel }^2}\hfill \\ & +& 87k\epsilon \parallel {\overline{\mathit{y}}}_k\parallel +\mathcal{O}\left({ϵ}^2\right),\hfill \end{array}$$

where $\widehat{\mathit{z}}$ and ${\widehat{\mathit{z}}}^{\left(k\right)}$ are defined in (26) similarly, $\check{\mathit{z}}$ is the computed smallest right singular vector of $({\beta }_1{\mathit{e}}_1,{\overline{B}}_k)$.

Proof. 

The roundoff error can be composed of the following parts

$$\mathit{x}-{\tilde{\mathit{x}}}_k=(\mathit{x}-\widehat{\mathit{x}})+(\widehat{\mathit{x}}-{\widehat{\mathit{x}}}_k)+({\widehat{\mathit{x}}}_k-{\widehat{V}}_k{\overline{\mathit{y}}}_k)+({\widehat{V}}_k{\overline{\mathit{y}}}_k-fl\left({\overline{V}}_k{\overline{\mathit{y}}}_k\right)$$

and we analyze these errors separately.

For the first part, $\mathit{x}$ and $\widehat{\mathit{x}}$ are the TLS solutions of the systems $A\mathit{x}\approx \mathit{b}$ and $(A+E)\widehat{\mathit{x}}\approx \mathit{b}+\mathit{e}$, respectively, in line with Lemma 2, so the error of this part is the inherent error. Then, combining with Lemma 3, we have

$$\parallel \mathit{x}-\widehat{\mathit{x}}\parallel \le {\kappa }_{\mathit{b}}87\epsilon +{\kappa }_A\sqrt{n}(174n+3\sqrt{n}+87)\epsilon ,$$

where ${\kappa }_A$ and ${\kappa }_{\mathit{b}}$ see Lemma 3.

For the second part, this error is owing to the approximate solution of $(A+E)\widehat{\mathit{x}}\approx \mathit{b}+\mathit{e}$ obtained by using HBITLS algorithm after k steps with the exact arithmetic. Lemma 4 tells us that

$$\parallel \widehat{\mathit{x}}-{\widehat{\mathit{x}}}_k\parallel \le sin\theta (\widehat{\mathit{z}},{\widehat{\mathit{z}}}^{\left(k\right)})\sqrt{1+\parallel \widehat{\mathit{x}}{\parallel }^2}\sqrt{1+\parallel {\widehat{\mathit{x}}}_k{\parallel }^2}.$$

For the third part, it is noticed that ${\widehat{\mathit{x}}}_k={\widehat{V}}_k{\widehat{\mathit{y}}}_k,$ we have that $\parallel {\widehat{\mathit{x}}}_k-{\widehat{V}}_k{\overline{\mathit{y}}}_k\parallel =\parallel {\widehat{\mathit{y}}}_k-{\overline{\mathit{y}}}_k\parallel ,$ where $\parallel {\widehat{\mathit{y}}}_k-{\overline{\mathit{y}}}_k\parallel$ is the roundoff error stem from the projected TLS solution. Since $B_k$ is a special form of bidiagonal matrices, we consider using the implicit zero-shift QR algorithm to perform singular value decomposition. (31) gives an upper bound of the angle between the solution vectors, and combining Lemma 7, we know

$$\parallel {\widehat{\mathit{x}}}_k-{\widehat{V}}_k{\overline{\mathit{y}}}_k\parallel =\parallel {\widehat{\mathit{y}}}_k-{\overline{\mathit{y}}}_k\parallel \le sin\theta ({\widehat{\mathit{z}}}^{\left(k\right)},\check{\mathit{z}})\sqrt{1+\parallel {\widehat{\mathit{y}}}_k{\parallel }^2}\sqrt{1+\parallel {\overline{\mathit{y}}}_k{\parallel }^2},$$

where $\theta ({\widehat{\mathit{z}}}^{\left(k\right)},\check{\mathit{z}})$ is the subspace angle between the sub-spaces produced by the smallest right singular vector and the computed smallest right singular vector of $({\beta }_1{\mathit{e}}_1,{\overline{B}}_k)$, respectively.

For the last part, we know $V_k=H_1{H}_2\cdots H_k\left(\begin{array}{c}{I}_k\\ 0\end{array}\right),$ where $H_1{H}_2\cdots H_k$ is the product of k Householder matrices. So, on the basis of (22), we obtain

$$\parallel {\widehat{V}}_k{\overline{\mathit{y}}}_k-fl\left({\overline{V}}_k{\overline{\mathit{y}}}_k\right)\parallel \le 87k\epsilon \parallel {\overline{\mathit{y}}}_k\parallel +\mathcal{O}\left({ϵ}^2\right).$$

□

By theorem 2, we get the mixed forward–backward stability of the HBITLS algorithm and NIPALS-TLS algorithm naturally. The backward stability will generate perturbation that will marginally influence the theoretical convergence of the residual to zero.

Remark 2.

The bound we introduced in Theorem 2 shows that the total roundoff errors are dominated by the approximation errors $\parallel \widehat{\mathit{x}}-{\widehat{\mathit{x}}}_k\parallel$. From this, we can know that, in many practical problems, we can safely select the stopping criteria required by the algorithm based on the theoretical nature of the original problem. This shows that, in a great deal of practical studies, the stopping criteria may be effectively selected based on the theoretical properties of the problem itself, thereby reducing the cost required to pursue an extremely accurate approximate solution to the original problem.

4. Conclusions

For large-scale problems, how to give an algorithm with good accuracy and stability is particularly important. In this paper, the Householder bidiagonalization total least squares (HBITLS) algorithm and nonlinear iterative partial least squares (NIPALS-TLS) algorithm are given. The HBITLS uses the Householder bidiagonalization algorithm for reducing $(b,A)$ to upper bidiagonal form and then runs the implicit zero-shift QR algorithm to compute the smallest right singular vector of the reduced form for the approximation solutions. The NIPALS-TLS is based on rank-reducing orthogonal projections. The two algorithms compute the same approximate TLS solutions. By analyzing the propagation of the roundoff error during the process of the HBITLS algorithm, we find that the HBITLS algorithm and the NIPALS-TLS algorithm are to be mixed forward–backward stable. In addition, in many practical problems, the stop criterion can be safely selected on the basis of the rounding error analysis of the original problem. The upper bound of our roundoff error gives a more detailed and clearer approximation of the computed solution.