The Smallest Singular Value Anomaly and the Condition Number Anomaly

Abstract

Let A be an arbitrary matrix in which the number of rows, m, is considerably larger than the number of columns, n. Let the submatrix $A_i,i=1,\dots ,m$, be composed of the first i rows of A. Let ${\beta }_i$ denote the smallest singular value of $A_i$, and let $k_i$ denote the condition number of $A_i$. In this paper, we examine the behavior of the sequences ${\beta }_1,\dots ,{\beta }_m$, and $k_1,\dots ,k_m$. The behavior of the smallest singular values sequence is somewhat surprising. The first part of this sequence, ${\beta }_1,\dots ,{\beta }_n$, is descending, while the second part, ${\beta }_n,\dots ,{\beta }_m$, is ascending. This phenomenon is called “the smallest singular value anomaly”. The sequence of the condition numbers has a different character. The first part of this sequence, $k_1,\dots ,k_n$, always ascends toward $k_n$, which can be very large. The condition number anomaly occurs when the second part, $k_n,\dots ,k_m$, descends toward a value of $k_m$, which is considerably smaller than $k_n$. It is shown that this is likely to happen whenever the rows of A satisfy two conditions: all the rows are about the same size, and the directions of the rows scatter in some random way. These conditions hold in a wide range of random matrices, as well as other types of matrices. The practical importance of these phenomena lies in the use of iterative methods for solving large linear systems, since several iterative solvers have the property that a large condition number results in a slow rate of convergence, while a small condition number yields fast convergence. Consequently, a condition number anomaly leads to a similar anomaly in the number of iterations. The paper ends with numerical experiments that illustrate the above phenomena.

1. Introduction

Let $A\in {\mathbb{R}}^{m\times n}$ be a given matrix in which the number of rows, m, is considerably larger than the number of columns, n. Let the rows of A be denoted as ${\mathbf{a}}_i^T,i=1,\dots ,m$, where ${\mathbf{a}}_i\in {\mathbb{R}}^n$. That is, $A={[{\mathbf{a}}_1,\dots ,{\mathbf{a}}_m]}^T$. Let

$$A_i={[{\mathbf{a}}_1,\dots ,{\mathbf{a}}_i]}^T\in {\mathbb{R}}^{i\times n}$$

(1)

be a submatrix of A, which is composed from the first i rows of A. Let ${\alpha }_i$ denote the largest singular value of $A_i$, let ${\beta }_i$ denote the smallest singular value of $A_i$, and let $k_i$ denote the condition number of this matrix. In this paper, we investigate the behavior of the sequences ${\alpha }_1,\dots ,{\alpha }_m$, ${\beta }_1,\dots ,{\beta }_m$ and $k_1,\dots ,k_m$. We start by showing that adding rows causes the largest singular value to increase,

$${\alpha }_i\le {\alpha }_{i+1}\mathrm{for}i=1,\dots ,m-1,$$

(2)

and study the reasons for the large, or small, increase. Next, we consider the behavior of the smallest singular values, which is somewhat surprising: at first, adding rows causes decreasing,

$${\beta }_i\ge {\beta }_{i+1}\mathrm{for}i=1,\dots ,n-1.$$

(3)

Then, as i passes n, adding rows increases the smallest singular value. That is,

$${\beta }_i\le {\beta }_{i+1}\mathrm{for}i=n,\dots ,m-1.$$

(4)

This behavior is called “the smallest singular value anomaly”. The study of this phenomenon explains the reasons for large, or small, difference between ${\beta }_{i+1}$ and ${\beta }_i$.

The last observation implies that ${\beta }_n$ is the smallest number in the sequence ${\beta }_1,\dots ,{\beta }_m$. Assume for simplicity that ${\beta }_n>0$. In this case, ${\beta }_i\ge {\beta }_n>0$ for $i=1,\dots ,m$, and the ratio $k_i={\alpha }_i/{\beta }_i$ is the condition number of $A_i$. This number affects the results of certain computations, such as the solution of linear equations, e.g., [1,2,3]. It is interesting, therefore, to examine the behavior of the sequence $k_i,i=1,\dots ,m$. The inequalities (2) and (3) show that as i moves from 1 to n, the value of $k_i$ increases. That is,

$$k_i\le k_{i+1}\mathrm{for}i=1,\dots ,n-1.$$

(5)

However, as i passes n, both ${\alpha }_i$ and ${\beta }_i$ are increasing, and the behavior of $k_i$ is not straightforward. The fact that the sequence ${\beta }_i,i=n,\dots ,m$ is increasing tempts one to expect that the sequence $k_i,i=n,\dots ,m$ will decrease. That is,

$$k_i\ge k_{i+1}\mathrm{for}i=n,\dots ,m-1.$$

(6)

The situation in which (6) holds is called the condition number anomaly. In this case, the sequence $k_i,i=1,\dots ,n$ increases toward $k_n$, which can be quite large, while the sequence $k_i,i=n,\dots ,m$ decreases toward a value of $k_m$, which is considerably smaller than $k_n$. The inequalities that assess the increase in the sequences (2) and (4) enable us to derive a useful bound on the ratio $k_{i+1}/k_i$. The bound explains the reasons behind the condition number anomaly, and characterizes situations that invite (or exclude) such behavior.

One type of matrices that exhibits the condition number anomaly is that of dense random matrices in which each element of the matrix is independently sampled from the same probability distribution. In particular, if each element of A comes from an independent standard normal distribution, then A is a Gaussian random matrix, and $A{A}^T$ is a Wishart matrix. The problem of estimating the largest and the smallest singular values of large Gaussian matrices has been studied by several authors. See [4,5,6,7,8,9,10,11,12,13,14] and the references therein. In this case, when n is very large and $i>n$, we have the estimates

$${\alpha }_i/\sqrt{i}\approx 1+\sqrt{n/i},$$

(7)

$${\beta }_i/\sqrt{i}\approx 1-\sqrt{n/i},$$

(8)

and

$$k_i\approx (1+\sqrt{n/i})/(1-\sqrt{n/i}),$$

(9)

which means that very large Gaussian matrices possess the condition number anomaly (for very large n and $i=n$ we have ${\beta }_n\approx 1/\sqrt{n}$; see [7,12]).

Our analysis shows that the condition number anomaly is not restricted to large Gaussian matrices. It is shared by a wide range of matrices, from small random matrices to large sparse matrices. The bounds that we derive have a simple geometric interpretation that helps to see what makes $k_n$ large and what forces the sequence $k_n,k_{n+1},\dots ,k_m$ to decrease. Roughly speaking, the condition number anomaly is expected whenever all the rows of the matrix have about the same size and the directions of the rows are randomly scattered. The paper brings several numerical examples that illustrate this feature.

The practical interest in the condition number anomaly comes from the use of iterative methods for solving large sparse linear systems, e.g., [15,16,17,18]. Some of these methods have the property that the asymptotic rate of convergence depends on the condition number of the related matrix. That is, a large condition number results in slow convergence, while a small condition number yields fast convergence. Assume now that such a method is used to solve a linear system whose matrix has the condition number anomaly. Then the last property implies a similar anomaly in the number of iterations. This phenomenon is called “iterations anomaly”. The discussion in Section 5 demonstrates this property in the methods of Richardson, Cimmino, and Jacobi. See Table 12.

2. The Ascending Behavior of the Largest Singular Values

In this section, we investigate the behavior of the sequence ${\alpha }_1,\dots ,{\alpha }_m$. The first assertion establishes the ascending property of this sequence.

Theorem 1.

The sequence ${\alpha }_1,\dots ,{\alpha }_m$ satisfies

$${\alpha }_i^2\le {\alpha }_{i+1}^2\mathrm{for}i=1,\dots ,m-1.$$

(10)

Proof. 

Observe that ${\alpha }_i^2$ is the largest eigenvalue of the matrix $A_i{A}_i^T$, which is a principal submatrix of $A_{i+1}{A}_{i+1}^T$. Hence, (10) is a direct consequence of the Cauchy interlace theorem. For statements and proofs of this theorem, see, for example, Refs. [2] (p. 441), [19] (p. 185), [20] (p. 149), [21] and [22] (p. 186). A second way to prove (10) is given below. This approach enables a closer inspection of the ascending process.

Here, we use the fact that ${\alpha }_i^2$ is the largest eigenvalue of the cross-product matrix $A_i^T{A}_i$. Let the unit vector ${\mathbf{u}}_i$ denote the corresponding dominant eigenvector of $A_i^T{A}_i$. Then

$$A_i^T{A}_i{\mathbf{u}}_i={\alpha }_i^2{\mathbf{u}}_i$$

(11)

and

$${\mathbf{u}}_i^T{A}_i^T{A}_i{\mathbf{u}}_i={\alpha }_i^2=\max \left\{{\mathbf{x}}^T{A}_i^T{A}_i\mathbf{x}|\mathbf{x}\in {\mathbb{R}}^n\mathrm{and}{\parallel \mathbf{x}\parallel }_2=1\right\},$$

(12)

where ${\parallel \mathbf{x}\parallel }_2={\left({\mathbf{x}}^T\mathbf{x}\right)}^{1/2}$ denotes the Euclidean vector norm. Note also that

$$A_i^T{A}_i=\sum _{j=1}^i{\mathbf{a}}_j{\mathbf{a}}_j^T,$$

(13)

$$A_{i+1}^T{A}_{i+1}=A_i^T{A}_i+{\mathbf{a}}_{i+1}{\mathbf{a}}_{i+1}^T,$$

(14)

and

$${\mathbf{u}}_i^T{A}_{i+1}^T{A}_{i+1}{\mathbf{u}}_i\le {\mathbf{u}}_{i+1}^T{A}_{i+1}^T{A}_{i+1}{\mathbf{u}}_{i+1}={\alpha }_{i+1}^2.$$

(15)

Consequently,

$$\begin{array}{cc}\hfill {\alpha }_{i+1}^2& \ge {\mathbf{u}}_i^T{A}_{i+1}^T{A}_{i+1}{\mathbf{u}}_i={\mathbf{u}}_i^T{A}_i^T{A}_i{\mathbf{u}}_i+\left({\mathbf{u}}_i^T{\mathbf{a}}_{i+1}\right)\left({\mathbf{a}}_{i+1}^T{\mathbf{u}}_i\right)\hfill \\ \hfill & ={\alpha }_i^2+{\left({\mathbf{u}}_i^T{\mathbf{a}}_{i+1}\right)}^2\ge {\alpha }_i^2,\hfill \end{array}$$

(16)

which proves (10). □

Next, we provide an upper bound on the increase in ${\alpha }_{i+1}^2$.

Theorem 2.

The inequality

$${\alpha }_{i+1}^2\le {\alpha }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2$$

(17)

holds for $i=1,\dots ,m-1$.

Proof. 

Observe that

$${\mathbf{u}}_{i+1}^T{A}_i^T{A}_i{\mathbf{u}}_{i+1}\le {\mathbf{u}}_i^T{A}_i^T{A}_i{\mathbf{u}}_i={\alpha }_i^2.$$

(18)

Hence a further use of (14) gives

$$\begin{array}{cc}\hfill {\alpha }_{i+1}^2& ={\mathbf{u}}_{i+1}^T{A}_{i+1}^T{A}_{i+1}{\mathbf{u}}_{i+1}={\mathbf{u}}_{i+1}^T{A}_i^T{A}_i{\mathbf{u}}_{i+1}+{\left({\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2\hfill \\ \hfill & \le {\alpha }_i^2+|{\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}{|}^2\le {\alpha }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2,\hfill \end{array}$$

(19)

which proves (17). The last inequality in (19) is due to the Cauchy–Schwartz inequality

$$|{\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}|\le \parallel {\mathbf{u}}_{i+1}{\parallel }_2{\parallel {\mathbf{a}}_{i+1}\parallel }_2$$

(20)

and the fact that $\parallel {\mathbf{u}}_{i+1}{\parallel }_2=1$. □

Combining (10) and (17) shows that

$${\alpha }_i^2\le {\alpha }_{i+1}^2\le {\alpha }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2.$$

(21)

This raises the question of for which directions of ${\mathbf{a}}_{i+1}$ the value of ${\alpha }_{i+1}^2$ attains its bounds. The key to answering this question lies in the following observation.

Lemma 1.

Assume for a moment that ${\mathbf{a}}_{i+1}$ is an eigenvector of the matrix $A_i^T{A}_i$. That is,

$$A_i^T{A}_i{\mathbf{a}}_{i+1}=\tau {\mathbf{a}}_{i+1}$$

(22)

where $\tau \in \mathbb{R}$ is a nonnegative scalar. In this case, the matrix $A_{i+1}^T{A}_{i+1}$ has the same set of eigenvectors as $A_i^T{A}_i$. The eigenvector ${\mathbf{a}}_{i+1}$ satisfies

$$A_{i+1}^T{A}_{i+1}{\mathbf{a}}_{i+1}=(\tau +\parallel {\mathbf{a}}_{i+1}{\parallel }_2^2){\mathbf{a}}_{i+1},$$

(23)

while all the other eigenpairs remain unchanged.

Proof. 

Since ${\mathbf{a}}_{i+1}$ is an eigenvector of $A_i^T{A}_i$, substituting the spectral decomposition of $A_i^T{A}_i$ into (14) yields the spectral decomposition of $A_{i+1}^T{A}_{i+1}$. □

The possibility that ${\alpha }_{i+1}^2$ achieves its upper bound is characterized by the following assertion.

Theorem 3.

Assume for a moment that ${\mathbf{a}}_{i+1}$ is a dominant eigenvector of $A_i^T{A}_i$. In this case

$${\alpha }_{i+1}^2={\alpha }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2.$$

(24)

Otherwise, when ${\mathbf{a}}_{i+1}$ is not pointing toward a dominant eigenvector,

$${\alpha }_{i+1}^2<{\alpha }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2.$$

(25)

Proof. 

The first claim is a direct consequence of Lemma 1. To prove the second claim, we consider two cases. The first one occurs when ${\mathbf{u}}_{i+1}={\mathbf{u}}_i$. Since ${\mathbf{a}}_{i+1}$ is not at the direction of ${\mathbf{u}}_{i+1}$, in this case, there is a strict inequality in (20), which yields a strict inequality in (19). In the second case, ${\mathbf{u}}_{i+1}\ne {\mathbf{u}}_i$, so now we have a strict inequality in (18), which leads to a strict inequality in (19). □

Finally, we consider the possibility that ${\alpha }_{i+1}^2={\alpha }_i^2$.

Theorem 4.

Assume that $i\ge n$ and that ${\mathbf{a}}_{i+1}$ is an eigenvector of $A_i^T{A}_i$, which corresponds to the smallest eigenvalue of this matrix. That is,

$$A_i^T{A}_i{\mathbf{a}}_{i+1}={\beta }_i^2{\mathbf{a}}_{i+1}.$$

(26)

If

$$\parallel {\mathbf{a}}_{i+1}{\parallel }_2^2\le {\alpha }_i^2-{\beta }_i^2$$

(27)

then ${\alpha }_{i+1}^2={\alpha }_i^2$. Otherwise, when

$$\parallel {\mathbf{a}}_{i+1}{\parallel }_2^2>{\alpha }_i^2-{\beta }_i^2$$

(28)

the value of ${\alpha }_{i+1}^2$ satisfies

$${\alpha }_{i+1}^2={\beta }_i^2+{\parallel {\mathbf{a}}_i\parallel }_2^2>{\alpha }_i^2.$$

(29)

Proof. 

From Lemma 1, we obtain that ${\mathbf{a}}_{i+1}$ is an eigenvector of $A_{i+1}^T{A}_{i+1}$ whose eigenvalue equals ${\beta }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2$. Therefore, if (27) holds, then ${\alpha }_i^2$ remains the largest eigenvalue. Otherwise, when (28) holds, ${\beta }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2$ is the largest eigenvalue of $A_{i+1}^T{A}_{i+1}$. □

The restriction $i\ge n$ is due to the fact that if $i<n$, then the smallest eigenvalue of $A_i^T{A}_i$ is always zero. The extension of Theorem 5 to cover this case is achieved by setting zero instead of ${\beta }_n^2$. Similar results are obtained when ${\mathbf{a}}_{i+1}$ points to other eigenvectors of $A_i^T{A}_i$.

3. The Smallest Singular Value Anomaly

In this section we explore the behavior of the smallest singular values. We shall start by proving that the sequence ${\beta }_1,\dots ,{\beta }_n$ is descending. The proof uses the fact that for $i\le n$, the smallest eigenvalue of $A_i{A}_i^T$ is ${\beta }_i^2$.

Theorem 5.

For $i=1,\dots ,n-1$, we have the inequality

$${\beta }_i^2\ge {\beta }_{i+1}^2.$$

(30)

Proof. 

The matrix $A_i{A}_i^T$ is a principal submatrix of $A_{i+1}{A}_{i+1}^T$. Hence, (30) is a direct corollary of the Cauchy interlace theorem. □

Next, we show that the sequence ${\beta }_n,{\beta }_{n+1},\dots ,{\beta }_m$, is ascending.

Theorem 6.

For $i=n,n+1,\dots ,m-1$, we have the inequality

$${\beta }_{i+1}^2\ge {\beta }_i^2.$$

(31)

Proof. 

One way to conclude (31) is by using the fact that $A_i{A}_i^T$ is a principal submatrix of $A_{i+1}{A}_{i+1}^T$. Let

$${\lambda }_1\left(A_i{A}_i^T\right)\ge \cdots \ge {\lambda }_i\left(A_i{A}_i^T\right)\ge 0,$$

and

$${\lambda }_1\left(A_{i+1}{A}_{i+1}^T\right)\ge \cdots \ge {\lambda }_i\left(A_{i+1}{A}_{i+1}^T\right)\ge {\lambda }_{i+1}\left(A_{i+1}{A}_{i+1}^T\right)\ge 0,$$

denote the eigenvalues of these matrices. Then, since $i\ge n,{\beta }_i^2={\lambda }_n\left(A_i{A}_i^T\right)$, ${\beta }_{i+1}^2={\lambda }_n(A_{i+1},A_{i+1}^T)$, and (31) is a direct consequence of Cauchy interlace theorem.

As before, a second proof is obtained by comparing the matrices $A_i^T{A}_i$ and $A_{i+1}^T{A}_{i+1}$, and this approach provides us with useful inequalities. Let the unit vector ${\mathbf{v}}_i,i=n,\dots ,m$, denote an eigenvector of $A_i^T{A}_i$ that corresponds to ${\beta }_i^2$. Then

$$A_i^T{A}_i{\mathbf{v}}_i={\beta }_i^2{\mathbf{v}}_i$$

(32)

and ${\mathbf{v}}_i$ has the minimum property

$${\mathbf{v}}_i^T{A}_i^T{A}_i{\mathbf{v}}_i={\beta }_i^2=\min \{{\mathbf{x}}^T{A}_i^T{A}_i\mathbf{x}|\mathbf{x}\in {\mathbb{R}}^n{\mathrm{and}\parallel \mathbf{x}\parallel }_2=1\}.$$

(33)

The last property implies the inequality

$${\mathbf{v}}_{i+1}^T{A}_i^T{A}_i{\mathbf{v}}_{i+1}\ge {\mathbf{v}}_i^T{A}_i^T{A}_i{\mathbf{v}}_i={\beta }_i^2,$$

(34)

while a further use of (14) gives

$$\begin{array}{cc}\hfill {\beta }_{i+1}^2& ={\mathbf{v}}_{i+1}^T{A}_{i+1}^T{A}_{i+1}{\mathbf{v}}_{i+1}={\mathbf{v}}_{i+1}^T{A}_i^T{A}_i{\mathbf{v}}_{i+1}+{\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2\hfill \\ \hfill & \ge {\mathbf{v}}_i^T{A}_i^T{A}_i{\mathbf{v}}_i+{\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2\hfill \\ \hfill & ={\beta }_i^2+{\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2\ge {\beta }_i^2.\hfill \end{array}$$

(35)

□

The inequality (35) implies that the growing of ${\beta }_{i+1}^2$ depends on the size of the scalar product ${\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}$. Basically, it is difficult to estimate this product, but Lemma 1 and Theorem 5 give some insight. For example, if ${\mathbf{a}}_{i+1}$ is an eigenvector of $A_i^T{A}_i$ whose eigenvalue differs from ${\beta }_i^2$, then ${\beta }_{i+1}^2={\beta }_i^2$. If ${\mathbf{a}}_{i+1}$ is an eigenvector that corresponds to ${\beta }_i^2$, there are two possibilities to consider. If ${\beta }_i^2$ is a multiple eigenvalue, then, again, ${\beta }_{i+1}^2={\beta }_i^2$. Otherwise, when ${\beta }_i^2$ is a simple eigenvalue,

$${\beta }_{i+1}^2={\beta }_i^2+\min \{\delta ,\parallel {\mathbf{a}}_{i+1}{\parallel }_2^2\}$$

(36)

where $\delta >0$ is the difference between the two smallest eigenvalues of $A_i^T{A}_i$.

We have seen that the sequence ${\beta }_1,\dots ,{\beta }_n$ is descending, while the sequence ${\beta }_n,\dots ,{\beta }_m$ is ascending. This behavior is called the smallest singular value anomaly. The fact that ${\beta }_n$ is the smallest singular value in the whole sequence raises the question of what makes ${\beta }_n$ small. Clearly, ${\beta }_n$ is always smaller than

$$\min \{\parallel {\mathbf{a}}_1{\parallel }_2,\dots ,\parallel {\mathbf{a}}_n{\parallel }_2\}.$$

(37)

Thus, to obtain a meaningful answer, we make the simplifying assumption

$$\parallel {\mathbf{a}}_i{\parallel }_2=1\mathrm{for}i=1,\dots ,m,$$

(38)

which enables the following bounds.

Lemma 2.

Assume that (38) holds and define

$$\mu =\max \left\{\right|{\mathbf{a}}_i^T{\mathbf{a}}_j\left|\right|i=1,\dots ,n,j=1,\dots ,n,\mathrm{and}i\ne j\}.$$

(39)

Then,

$${\alpha }_n^2\ge 1+\mu$$

(40)

and

$${\beta }_n^2\le 1-\mu .$$

(41)

Proof. 

It is possible to assume that the above maximum is attained for the first two rows and that ${\mathbf{a}}_1^T{\mathbf{a}}_2=\mu >0$. In this case,

$$A_2{A}_2^T=\left(\begin{array}{cc}1& \mu \\ \mu & 1\end{array}\right),$$

(42)

and the eigenvalues of this matrix are ${\lambda }_1=1+\mu$ and ${\lambda }_2=1-\mu$. Therefore, since $A_2{A}_2^T$ is a principal submatrix of $A_n{A}_n^T$, the Cauchy interlace theorem implies (40) and (41). □

Usually, the bound (41) is a crude estimate of ${\beta }_n$. Yet, in some cases, it is the reason for a small value of ${\beta }_n$.

4. The Condition Number Anomaly

In this section we investigate the behavior of the sequence $k_i={\alpha }_i/{\beta }_i,i=1,\dots ,m$. The discussion is carried out under the assumption that ${\beta }_n>0$, which ensures that ${\beta }_i>0$ for $i=1,\dots ,m$. We have seen that the sequence ${\alpha }_1,\dots ,{\alpha }_n$ is ascending while the sequence ${\beta }_1,\dots ,{\beta }_n$ is descending. This proves that the sequence $k_1,\dots ,k_n$ is ascending. That is,

$$k_i\le k_{i+1}\mathrm{for}i=1,\dots ,n-1.$$

(43)

It is also known that the sequences ${\alpha }_n,\dots ,{\alpha }_m$ and ${\beta }_n,\dots ,{\beta }_m$ are ascending, but this does not provide decisive information about the behavior of the sequence $k_n,\dots ,k_m$. We shall start with examples that illustrate this point.

Example 1.

This example shows that $k_{i+1}$ can be larger than $k_i$. For this purpose, consider the case when ${\mathbf{a}}_{i+1}$ is a dominant eigenvector of $A_i^T{A}_i$. Then from Lemma 1 we see that ${\alpha }_{i+1}^2={\alpha }_i^2+{\parallel {\mathbf{a}}_{i+1}\parallel }_2^2$ but ${\beta }_{i+1}={\beta }_i$, which means that $k_{i+1}>k_i$.

Example 2.

A similar situation arises when A has the following property. Assume that as i grows, the sequence of rows directions ${\mathbf{a}}_i/{\parallel {\mathbf{a}}_i\parallel }_2$, $i=1,2,\dots$ converges rapidly toward some vector. In this case, the sequence ${\mathbf{u}}_i,i=1,2,\dots$ converges to the same vector, which brings us close to the situation of Example 1 (Tables 3 and 10 illustrate this possibility).

Example 3.

The third example shows that $k_{i+1}$ can be smaller than $k_i$. Consider the case described in Theorem 4, when (27) holds. Here ${\alpha }_{i+1}={\alpha }_i$, ${\beta }_{i+1}>{\beta }_i$ and $k_{i+1}<k_i$. More reasons that force decrease are given in Corollary 1 below.

Example 4.

The fourth example describes a situation in which the condition number behaves in a cyclic manner. Let $B\in {\mathbb{R}}^{l\times n}$ be a given matrix with $l\ge n$. Let the matrix A be obtained by duplicating B k times. That is, $m=k\times l$ and

$$A={[B^T,B^T,\dots ,B^T]}^T\in {\mathbb{R}}^{m\times n}.$$

Then when i takes the values $i=j\times l,j=1,\dots ,k$, the matrix $A_i^T{A}_i$ has the form

$$A_i^T{A}_i=j{B}^{T}B.$$

Hence, for these values of i we have ${\alpha }_i=j{\alpha }_l$, and ${\beta }_i=j{\beta }_l$, but $k_i=k_l$.

The situation in which the sequence $k_n,\dots ,k_m$ is descending,

$$k_i\ge k_{i+1}\mathrm{for}i=n,\dots ,m-1,$$

(44)

is called the condition number anomaly. The reasons behind this behavior are explained below.

Theorem 7.

Let the positive parameters ${\eta }_i$ and ${\nu }_i$ be defined by the equalities

$${\eta }_i^2={\left({\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2/{\alpha }_i^2$$

(45)

and

$${\nu }_i^2={\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2/{\beta }_i^2.$$

(46)

Then, for $i=n,\dots ,m-1$,

$$k_{i+1}^2\le k_i^2(1+{\eta }_i^2)/(1+{\nu }_i^2).$$

(47)

Proof. 

From (19), we see that

$${\alpha }_{i+1}^2\le {\alpha }_i^2+{\left({\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2={\alpha }_i^2(1+{\eta }_i^2).$$

(48)

Similarly from (35), we obtain

$${\beta }_{i+1}^2\ge {\beta }_i^2+{\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2={\beta }_i^2(1+{\nu }_i^2).$$

(49)

Hence, combining these inequalities gives (47). □

Corollary 1.

The inequality

$${\nu }_i^2\ge {\eta }_i^2$$

(50)

implies

$$k_i\ge k_{i+1}.$$

(51)

□

The last corollary is a key observation that indicates at which situations the condition number anomaly is likely to occur. Assume for a moment that the direction of ${\mathbf{a}}_{i+1}$ is chosen in some random way. Then, the scalar product terms ${\left({\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2$ and ${\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2$ are likely to be about the same size. However, since ${\beta }_i^2$ is (considerably) smaller than ${\alpha }_i^2$, the term ${\left({\mathbf{v}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2/{\beta }_i^2$ is expected to be larger than ${\left({\mathbf{u}}_{i+1}^T{\mathbf{a}}_{i+1}\right)}^2/{\alpha }_i^2$, which implies (50).

Summarizing the above discussion, we see that the condition number anomaly is likely to occur whenever the rows of the matrix satisfy two conditions: all the rows have about the same size, and the directions of the rows are scattered in some random way. This conclusion means that the phenomenon is shared by a wide range of matrices. The examples in Section 6 illustrate this point.

5. Iterations Anomaly

Let ${\mathbf{a}}_i,A_i,{\alpha }_i,{\beta }_i$ and $k_i,i=1,\dots ,m$, be as in the previous sections. Let $\mathbf{b}={(b_1,\dots ,b_m)}^T$ $\in {\mathbb{R}}^m$ be an arbitrary given vector, which is used to define the vectors ${\mathbf{b}}_i={(b_1,\dots ,b_i)}^T\in {\mathbb{R}}^i,i=1,\dots ,m$. In this section, we examine how the condition number anomaly affects the convergence of certain iterative methods for solving a linear system of the form

$$A_i\mathbf{x}={\mathbf{b}}_i.$$

(52)

We shall start by considering the Richardson method for solving the normal equations

$$A_i^T{A}_i\mathbf{x}=A_i^T{\mathbf{b}}_i,$$

(53)

e.g., [16,17,18]. Given ${\mathbf{x}}_k$ the k-th iteration, $k=1,2,\dots ,$ of Richardson method has the form

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k-w{A}_i^T(A_i{\mathbf{x}}_k-{\mathbf{b}}_i),$$

(54)

where $w>0$ is a pre-assigned relaxation parameter. Recall that $A_i^T(A_i{\mathbf{x}}_k-{\mathbf{b}}_i)$ is the gradient vector of the least-squares objective function

$$F\left(\mathbf{x}\right)=1/2\parallel A_i\mathbf{x}-{\mathbf{b}}_i{\parallel }_2^2$$

(55)

at the point ${\mathbf{x}}_k$. Hence, iteration (54) can be viewed as a steepest descent method for minimizing $F\left(\mathbf{x}\right)$ that uses a fixed step length. An equivalent way to write (54) is

$${\mathbf{x}}_{k+1}=(I-w{A}_i^T{A}_i){\mathbf{x}}_k+w{A}_i^T{\mathbf{b}}_i,$$

(56)

which shows that the rate of convergence of the method depends on the spectral radius of the iteration matrix

$$H_w=I-w{A}_i^T{A}_i.$$

(57)

Let $\rho \left(H_w\right)$ denote the spectral radius of $H_w$. Then the theory of iterative methods tells us that the method converges whenever

$$\rho \left(H_w\right)<1,$$

(58)

and the smaller $\rho \left(H_w\right)$ is, the faster the convergence; see, for example, Refs. [16,17,18]. Observe that the eigenvalues of $H_w$ lie in the interval $[1-w{\alpha }_i^2,1-w{\beta }_i^2]$. This shows that (58) holds for values of w that satisfy

$$0<w<2/{\alpha }_i^2.$$

(59)

Furthermore, let $w_{opt}$ denote the optimal value of w, for which $\rho \left(H_w\right)$ attains its smallest value. Then

$$w_{opt}=2/({\alpha }_i^2+{\beta }_i^2)$$

(60)

and

$$\begin{array}{cc}\hfill \rho (I-w_{opt}{A}_i^T{A}_i)& =-(1-w_{opt}{\alpha }_i^2)=2{\alpha }_i^2/({\alpha }_i^2+{\beta }_i^2)-1\hfill \\ \hfill & =({\alpha }_i^2-{\beta }_i^2)/({\alpha }_i^2+{\beta }_i^2)=(k_i^2-1)/(k_i^2+1).\hfill \end{array}$$

(61)

See [17] (pp. 22–23) and [18] (pp. 114–115) for detailed discussion of these results. Consequently, as $k_i$ increases, the spectral radius of the iteration matrix approaches 1, and the rate of convergence slows down. That is, the condition number anomaly results in a similar anomaly in the number of iterations. See Table 12.

Another useful iterative method for solving large sparse linear systems is the Cimmino method, e.g., [15,16,18,23]. Let the unit vectors

$${\tilde{\mathbf{a}}}_i={\mathbf{a}}_i/{\parallel {\mathbf{a}}_i\parallel }_2,i=1,\dots ,m,$$

(62)

be obtained by normalizing the rows of A. Let ${\tilde{A}}_i$ be an $i\times n$ matrix whose rows are ${\tilde{\mathbf{a}}}_j,j=1,\dots ,i$, and let $D_i$ denote the $i\times i$ diagonal matrix

$$D_i=diag\{{\mathbf{a}}_1^T{\mathbf{a}}_1,\dots ,{\mathbf{a}}_i^T{\mathbf{a}}_i\}.$$

(63)

Then

$${\tilde{A}}_i={[{\tilde{\mathbf{a}}}_1,\dots ,{\tilde{\mathbf{a}}}_i]}^T=D_i^{-1/2}{A}_i$$

(64)

for $i=1,\dots ,m$. Similarly, we define ${\tilde{b}}_i=b_i/{\parallel {\mathbf{a}}_i\parallel }_2$ for $i=1,\dots ,m$, and ${\tilde{\mathbf{b}}}_i=D_i^{-1/2}{\mathbf{b}}_i$. Then Cimmino method is aimed at solving the linear system

$${\tilde{A}}_i\mathbf{x}={\tilde{\mathbf{b}}}_i,$$

(65)

or the related normal equations

$${\tilde{A}}_i^T{\tilde{A}}_i\mathbf{x}={\tilde{A}}_i^T{\tilde{\mathbf{b}}}_i.$$

(66)

The kth iteration of the Cimmino method, $k=1,2,\dots ,$ has the form

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k-w\sum _{j=1}^i{\nu }_j{\tilde{\mathbf{a}}}_j({\tilde{\mathbf{a}}}_j^T{\mathbf{x}}_k-{\tilde{b}}_j),$$

(67)

where $w>0$ is a pre-assigned relaxation parameter, and ${\nu }_j,j=1,\dots ,i$ are weighting parameters that satisfy

$${\nu }_j>0\mathrm{for}j=1,\dots ,i\mathrm{and}{\nu }_1+\dots +{\nu }_i=1.$$

(68)

Observe that the point

$${\mathbf{x}}_k-{\tilde{\mathbf{a}}}_j({\tilde{\mathbf{a}}}_j^T{\mathbf{x}}_k-{\tilde{b}}_j)$$

(69)

is the projection of ${\mathbf{x}}_k$ on the hyperplane $\{\mathbf{x}\in {\mathbb{R}}^n|{\mathbf{a}}_j^T\mathbf{x}=b_j\}$, and the point

$${\mathbf{x}}_k-\sum _{j=1}^i{\nu }_j{\tilde{\mathbf{a}}}_j({\tilde{\mathbf{a}}}_j^T{\mathbf{x}}_k-{\tilde{b}}_j)$$

(70)

is a weighted average of these projections. The usual way to apply the Cimmino method is with equal weights. That is, ${\nu }_j=1/i$ for $j=1,\dots ,i$. This enables us to rewrite the Cimmino iteration in the form

$${\mathbf{x}}_{k+1}={\mathbf{x}}_k-w{\tilde{A}}_i^T({\tilde{A}}_i{\mathbf{x}}_k-{\tilde{\mathbf{b}}}_i),$$

(71)

which is the Richardson iteration for solving the normal equations (66). Therefore, from (61), we conclude that the optimal rate of convergence of the Cimmino method depends on the ratio

$$\rho (I-w_{opt}{\tilde{A}}_i^T{\tilde{A}}_i)=({\tilde{k}}_i^2-1)/({\tilde{k}}_i^2+1),$$

(72)

where ${\tilde{k}}_i$ is the condition number of ${\tilde{A}}_i$.

Another example is the Jacobi iteration for solving the equations

$$A_i{A}_i^T\mathbf{y}={\mathbf{b}}_i.$$

(73)

The basic iteration of this method has the form

$${\mathbf{y}}_{k+1}=(I-w{D}_i^{-1}{A}_i{A}_i^T){\mathbf{y}}_k+w{D}_i^{-1}{\mathbf{b}}_i,$$

(74)

where $D_i$ is the diagonal matrix (63) and $w>0$ is a pre-assigned relaxation parameter. Now the equalities

$$D_i^{1/2}(I-w{D}_i^{-1}{A}_i{A}_i^T)D_i^{-1/2}=I-w{D}_i^{-1/2}{A}_i{A}_i^T{D}_i^{-1/2}=I-w{\tilde{A}}_i{\tilde{A}}_i^T$$

(75)

indicate that the iteration matrix of the Jacobi method is similar to the matrix $I-w{\tilde{A}}_i{\tilde{A}}_i^T$. Hence, as before, the optimal rate of convergence depends on the ratio $({\tilde{k}}_i^2-1)/({\tilde{k}}_i^2+1)$. Thus, again, a condition number anomaly invites a similar anomaly in the number of iterations.

We shall finish this section by mentioning two further methods that share this behavior. The first one is the conjugate gradients algorithm for solving the normal Equations (53), whose rate of convergence slows down as the condition numbers of A increases. See, for example, Refs. [1] (pp. 312–314), [3] (pp. 299–300) and [18] (pp. 203–205). The second is Kaczmarz’s method, which is a popular “row-action” method; see Refs. [15,16,23,24,25]. The use of this method to solve $A\mathbf{x}=\mathbf{b}$ is equivalent to the SOR method for solving the system $A{A}^T\mathbf{y}=\mathbf{b}$, and both methods have the property that a small condition number results in fast convergence while a large condition number slows it [24,25].

6. Numerical Examples

In this section, we bring some examples that illustrate the actual behavior of the anomaly phenomena. The first examples consider small $m\times n$ matrices.

Table 1 describes the anomaly in a “two-ones” matrix. This matrix has $m=n(n-1)/2$ different rows. Each row has only two nonzero entries, and each nonzero entry has the value 1 (a matrix with n columns has at most $n(n-1)/2$ different rows of this type). This matrix exhibits a moderate anomaly, due to the fact that $A_n$ is well conditioned.

Table 1. The anomaly in small “two-ones” matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
2	10	1.732	1.000	1.732
4	10	2.236	1.000	2.236
6	10	2.646	1.000	2.646
8	10	3.000	1.000	3.000
10	10	3.171	0.377	8.416
15	10	3.286	0.568	5.781
20	10	3.464	0.874	3.963
30	10	3.785	1.294	2.924
40	10	4.067	2.048	1.986
45	10	4.243	2.828	1.500

Table 2 describes the anomaly in a small $m\times n$ segment of the Hilbert matrix. Here, the $(i,j)$ entry equals $1/(i+j-1)$. Consequently, the sequence of rows directions ${\mathbf{a}}_i/{\parallel {\mathbf{a}}_i\parallel }_2$, $i=1,2,\dots$, converges slowly toward the vector $\mathbf{e}/{\parallel \mathbf{e}\parallel }_2$, where $\mathbf{e}={(1,1,\dots ,1)}^T\in {\mathbb{R}}^n$. Hence, the decrease in the sequence $k_n,k_{n+1},\dots ,k_m$ is quite moderate.

Table 2. The anomaly in small Hilbert matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
2	5	1.394	1.129 × 10${}^{-1}$	1.234 × 10${}^1$
3	5	1.481	5.428 × 10${}^{-3}$	2.727 × 10${}^2$
4	5	1.532	1.711 × 10${}^{-4}$	8.956 × 10${}^3$
5	5	1.567	3.288 × 10${}^{-6}$	4.766 × 10${}^5$
6	5	1.592	6.400 × 10${}^{-6}$	2.488 × 10${}^5$
7	5	1.611	9.679 × 10${}^{-6}$	1.664 × 10${}^5$
10	5	1.648	1.928 × 10${}^{-5}$	8.549 × 10${}^4$
15	5	1.679	3.231 × 10${}^{-5}$	5.197 × 10${}^4$
20	5	1.696	4.182 × 10${}^{-5}$	4.056 × 10${}^4$
30	5	1.714	5.422 × 10${}^{-5}$	3.161 × 10${}^4$
50	5	1.723	6.689 × 10${}^{-5}$	2.585 × 10${}^4$
100	5	1.746	7.845 × 10${}^{-5}$	2.219 × 10${}^4$

In Table 3, we consider a small $m\times n$ segment of the Pascal matrix. Recall that the entries of this matrix are built in the following way: $a_{1j}=1$ for $j=1,\dots ,n$, and $a_{i1}=1$ for $i=1,\dots ,m$. The other entries are obtained from the rule

$$a_{ij}=a_{i,j-1}+a_{i-1,j}\mathrm{for}i=2,\dots ,m\mathrm{and}j=2,\dots ,n.$$

(76)

In this matrix, the norm of the rows grows very fast while the sequence of row directions ${\mathbf{a}}_i/{\parallel {\mathbf{a}}_i\parallel }_2,i=1,2,\dots$ converges rapidly toward the vector ${\mathbf{e}}_n={(0,0,\dots ,0,1)}^T\in {\mathbb{R}}^n$. Thus, as i becomes considerably larger than n, both ${\mathbf{a}}_i/{\parallel {\mathbf{a}}_i\parallel }_2$ and ${\mathbf{u}}_i$ approach ${\mathbf{e}}_n$, which causes $k_{i+1}$ to be larger than $k_i$.

Table 3. Failure of the anomaly in small Pascal matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
2	5	7.691 × 10${}^0$	9.194 × 10${}^{-1}$	8.366 × 10${}^0$
3	5	2.068 × 10${}^1$	3.495 × 10${}^{-1}$	5.915 × 10${}^1$
4	5	4.649 × 10${}^1$	8.258 × 10${}^{-2}$	5.630 × 10${}^2$
5	5	9.229 × 10${}^1$	1.084 × 10${}^{-2}$	8.517 × 10${}^3$
6	5	1.672 × 10${}^2$	2.623 × 10${}^{-2}$	6.376 × 10${}^3$
7	5	2.826 × 10${}^2$	4.751 × 10${}^{-2}$	5.948 × 10${}^3$
10	5	1.020 × 10${}^3$	1.391 × 10${}^{-1}$	7.331 × 10${}^3$
20	5	1.539 × 10${}^4$	5.132 × 10${}^{-1}$	2.998 × 10${}^4$

The random matrices that are tested in Table 4 and Table 5 provide nice examples of the anomaly phenomenon. In these matrices, each entry is a random number from interval $[-1,1]$. To generate these matrices, and the other random matrices, we used MATLAB’s command “rand”, whose random numbers generator is of uniform distribution. Similar results are obtained when “rand” is replaced with “randn”, which uses normal distribution.

Table 4. The anomaly in small random matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
4	16	2.772	1.492	1.858
8	16	3.194	1.006	3.189
12	16	3.817	0.286	13.337
16	16	4.068	0.030	136.876
20	16	4.258	0.295	14.439
30	16	4.683	1.179	3.971
50	16	5.956	2.273	2.621
80	16	6.891	3.338	2.064
120	16	7.960	4.412	1.804
160	16	8.875	5.291	1.677

Table 5. The anomaly in random matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
20	100	7.976	3.505	2.276
40	100	8.782	2.681	3.276
60	100	9.679	1.476	6.556
80	100	10.350	0.706	14.653
100	100	10.802	0.044	245.346
120	100	11.429	0.753	15.183
150	100	12.164	1.471	8.270
200	100	13.413	2.409	5.569
300	100	15.329	4.537	3.379
400	100	16.872	6.084	2.773
500	100	18.458	7.301	2.528
1000	100	23.266	12.593	1.879

The nonnegative random matrix that is tested in Table 6 is obtained by MATLAB’s command $A=rand(m,n)$. That is, here, each entry of A is a random number from the interval $[0,1]$. This yields a more ill-conditioned matrix and a sharper anomaly.

Table 6. The anomaly in nonnegative random matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
20	100	22.309	1.836	12.151
40	100	31.532	1.317	23.949
60	100	38.462	0.653	58.927
80	100	44.355	0.359	123.504
100	100	49.634	0.030	1662.0
120	100	54.511	0.357	152.782
150	100	61.162	0.734	83.301
200	100	70.735	1.211	58.391
300	100	86.710	2.269	38.213
400	100	100.217	3.102	32.306
500	100	112.018	3.649	30.702
1000	100	158.344	6.296	25.148

Table 7 and Table 8 consider a different type of random matrices. As its name says, the entries of the “$-1$ or 1” matrix are either $-1$ or 1, with equal probability. In practice, the $(i,j)$ entry, $a_{ij}$, is defined in the following way. First, sample a random number, r, from the interval $[-1,1]$. If $r>0$, then $a_{ij}=1$; otherwise $a_{ij}=-1$. The entries of the “0 or 1” matrix are defined in a similar manner: if $r>0$ then $a_{ij}=1$; otherwise $a_{ij}=0$. Both matrices display a strong anomaly. The “0 or 1” matrix is slightly more ill conditioned and, therefore, has a sharper anomaly.

Table 7. The anomaly in random “−1 or 1” matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
20	100	13.526	6.472	2.090
40	100	15.624	4.205	3.715
60	100	16.949	2.647	6.404
80	100	18.019	1.092	16.503
100	100	19.204	0.0355	541.70
120	100	20.095	1.106	18.169
150	100	21.600	2.601	8.305
200	100	23.172	4.052	5.718
300	100	26.536	7.337	3.617
400	100	29.177	9.943	2.934
500	100	31.626	12.834	2.464
1000	100	41.216	21.866	1.883

Table 8. The anomaly in random “0 or 1” matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
20	100	22.404	3.337	6.714
40	100	31.705	2.283	13.886
60	100	38.671	1.344	28.776
80	100	44.727	0.5560	80.443
100	100	50.027	0.0207	2416.8
120	100	54.963	0.4955	110.91
150	100	61.745	1.314	47.001
200	100	71.292	2.026	35.192
300	100	87.315	3.720	23.469
400	100	100.78	4.981	20.234
500	100	112.74	6.398	17.620
1000	100	159.07	10.981	14.486

The results of Table 9 and Table 10 are quite instructive. Both matrices are highly ill conditioned but display a different behavior. The “narrow range” matrix is a random matrix whose entries are sampled from the small interval [0.99, 1.01]. However, the directions of the rows are not converging, and the matrix displays a nice anomaly. The “converging rows” matrix is defined in a slightly different way. Here, the entries in the ith row, $i=1,\dots ,m$, are random numbers from the interval $[1-1/i,1+1/i]$. Hence, the related sequence of rows directions, ${\mathbf{a}}_i/{\parallel {\mathbf{a}}_i\parallel }_2,i=2,\dots$, converges toward the vector $\mathbf{e}/{\parallel \mathbf{e}\parallel }_2$, which is the situation described in Example 2. Consequently, when i becomes much larger than n, we see a moderate increase in the value of $k_i$.

Table 9. The anomaly in random “narrow range” matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
20	100	44.716	3.667 × 10${}^{-2}$	1.219 × 10${}^3$
40	100	63.240	2.621 × 10${}^{-2}$	2.413 × 10${}^3$
60	100	77.451	1.303 × 10${}^{-2}$	5.946 × 10${}^3$
80	100	89.432	7.184 × 10${}^{-3}$	1.245 × 10${}^4$
100	100	99.989	5.916 × 10${}^{-4}$	1.690 × 10${}^5$
120	100	109.54	7.901 × 10${}^{-3}$	1.545 × 10${}^4$
150	100	122.47	1.473 × 10${}^{-2}$	8.316 × 10${}^3$
200	100	141.43	2.423 × 10${}^{-2}$	5.836 × 10${}^3$
300	100	173.20	4.540 × 10${}^{-2}$	3.815 × 10${}^3$
400	100	200.00	6.204 × 10${}^{-2}$	3.224 × 10${}^3$
500	100	223.61	7.297 × 10${}^{-2}$	3.064 × 10${}^3$
1000	100	316.23	1.259 × 10${}^{-1}$	2.511 × 10${}^3$

Table 10. Failure of anomaly in matrix with “converging rows”.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
20	100	44.700	2.373 × 10${}^{-1}$	1.883 × 10${}^2$
40	100	63.219	8.897 × 10${}^{-2}$	7.106 × 10${}^2$
60	100	77.427	3.257 × 10${}^{-2}$	2.378 × 10${}^3$
80	100	89.409	1.486 × 10${}^{-2}$	6.017 × 10${}^3$
100	100	99.968	1.061 × 10${}^{-3}$	9.418 × 10${}^4$
120	100	109.51	1.018 × 10${}^{-2}$	1.076 × 10${}^4$
150	100	122.45	1.781 × 10${}^{-2}$	6.875 × 10${}^3$
200	100	141.40	2.536 × 10${}^{-2}$	5.576 × 10${}^3$
300	100	173.18	3.333 × 10${}^{-2}$	5.189 × 10${}^3$
400	100	199.98	3.724 × 10${}^{-2}$	5.369 × 10${}^3$
500	100	223.59	3.934 × 10${}^{-2}$	5.684 × 10${}^3$
1000	100	316.22	4.319 × 10${}^{-2}$	7.321 × 10${}^3$

Other matrices that possess the anomaly phenomena are large sparse matrices. The matrix in Table 11 is created by using MATLAB’s command $A=$ sprand$(m,n$, density) with $m=$100,000, $n=$ 10,000 and density $=100/n$. This way, each row of A has nearly 100 nonzero entries that have random values and random locations. Although not illustrated in this paper, our experience shows that the smaller the density, the sharper the anomaly.

Table 11. The anomaly in large sparse random matrix.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
2000	10,000	23.207	3.122	7.434
4000	10,000	32.275	2.085	15.479
6000	10,000	39.294	1.285	30.571
8000	10,000	45.201	0.605	74.710
9000	10,000	47.893	0.291	164.79
10,000	10,000	50.426	4.064 × 10${}^{-4}$	1.241 × 10${}^5$
11,000	10,000	52.854	0.279	189.50
12,000	10,000	55.161	0.5437	101.45
15,000	10,000	61.590	1.284	47.956
20,000	10,000	71.033	2.372	29.953
30,000	10,000	86.895	4.189	20.744
40,000	10,000	100.32	5.708	17.573
50,000	10,000	112.14	7.072	15.856

Table 12 illustrates the iterations anomaly phenomenon when using the methods of Richardson, Cimmino, and Jacobi. The first two methods were used to solve linear systems of the form

$$A_i\mathbf{x}={\mathbf{b}}_i,i=1,\dots ,m.$$

(77)

As before, each $A_i$ is an $i\times n$ submatrix that is composed from the first i rows of a given $m\times n$ matrix A. The construction of A is done in two steps. First, we generate a random matrix as in Table 4 and Table 5. Then the rows of the matrix are normalized to be unit vectors. The vector ${\mathbf{b}}_i$ is defined by the product

$${\mathbf{b}}_i=A_i\mathbf{e},$$

(78)

which ensures that $\mathbf{e}$ solves the linear system. Since $A_i$ has unit rows, Cimmino iteration (71) coincides with Richardson iteration (54). The value of w that we use is the optimal one,

$$w_{opt}=2/({\alpha }_i^2+{\beta }_i^2),$$

(79)

and the iterations start from the point ${\mathbf{x}}_0=\mathbf{0}$. The iterative process is terminated as soon as the residual vector satisfies

$$\parallel A_i{\mathbf{x}}_k-{\mathbf{b}}_i{\parallel }_2/{\parallel {\mathbf{b}}_i\parallel }_2\le {10}^{-10}.$$

(80)

The number of iterations which are required to satisfy this condition is displayed in the last column of Table 12.

Table 12. Iterations anomaly in the methods of Richardson, Cimmino, and Jacobi.

Number of Rows	Number of Columns	Largest Singular Value	Smallest Singular Value	Condition Number	Number of Iterations
$\mathit{i}$	$\mathit{n}$	${\mathbf{\alpha }}_{\mathit{i}}$	${\mathbf{\beta }}_{\mathit{i}}$	${\mathit{k}}_{\mathit{i}}$
10	50	1.372	0.6454	2.125	50
30	50	1.673	0.2265	7.385	580
40	50	1.808	0.1241	14.568	2192
45	50	1.861	9.303 × 10${}^{-2}$	20.008	4154
48	50	1.916	3.259 × 10${}^{-2}$	58.791	36,960
50	50	1.917	2.415 × 10${}^{-3}$	793.96	6,760,589
52	50	1.937	3.606 × 10${}^{-2}$	53.730	30,976
55	50	1.955	0.1078	18.133	3559
60	50	2.029	0.1468	13.825	2107
80	50	2.176	0.2744	7.929	681
100	50	2.337	0.4617	5.061	266
200	50	3.030	1.021	2.966	98
500	50	4.098	2.217	1.848	37

The Jacobi method was used to solve the linear systems (73), where $A_i$ and ${\mathbf{b}}_i$ are defined as above. Since $A_i$ has unit rows, $D_i$ is a unit matrix, and Jacobi iteration (74) is reduced to

$${\mathbf{y}}_{k+1}={\mathbf{y}}_k-w(A_i{A}_i^T{\mathbf{y}}_k-{\mathbf{b}}_i).$$

(81)

The last iteration uses the optimal value of w, given in (79). It starts from the point ${\mathbf{y}}_0=\mathbf{0}$ and terminates as soon as the residual vector satisfies

$$\parallel A_i{A}_i^T{\mathbf{y}}_k-{\mathbf{b}}_i{\parallel }_2/{\parallel {\mathbf{b}}_i\parallel }_2\le {10}^{-10}.$$

(82)

The number of required iterations is nearly identical to that of the Richardson method. This is not surprising since multiplying (81) by $A_i^T$ shows that the sequence

$${\mathbf{x}}_k=A_i^T{\mathbf{y}}_k,k=0,1,2,\dots ,$$

(83)

is generated by Richardson iteration (54) (there were only two minor exceptions: for $i=48$, the Jacobi method required 36,959 iterations instead of 36,960, while for $i=50$, the Jacobi required 6,772,151 iterations instead of 6,760,589. In all the other cases, the two methods required exactly the same number of iterations).

The figures in Table 12 demonstrate the close link between the condition number and the rate of convergence. As anticipated from (61), for large values of $k_i$ the spectral radius approaches 1 and the rate of convergence slows down. Thus, a large condition number results in a large number of iterations. Conversely, a small value of $k_i$ implies a small spectral radius and a small number of iterations. In other words, a condition number anomaly invites a similar anomaly in the number of iterations.

Usually, it is reasonable to assume that the computational effort in solving a linear system is proportional to the number of rows. That is, the more rows we have, the more computation time is needed. From this point of view, the iterations anomaly phenomenon is somewhat surprising, as solving a linear system with $i=10n$ rows needs considerably less time than solving a linear system with $i=n$ rows.

7. Concluding Remarks

As an old adage says, the whole is sometimes much more than the sum of its parts. The basic ascending (descending) properties of singular values are easily concluded from the Cauchy interlace theorem, while the inequalities that we derive enable us to see what causes a large, or small, increase. Combining these results gives a better overview of the whole situation. One consequence regards the anomalous behavior of the smallest singular values sequence ${\beta }_1,\dots ,{\beta }_m$, and the fact that ${\beta }_n$ is the smallest number in this sequence. The second observation is about the condition number anomaly. It is easy to conclude the increasing of the condition numbers sequence, $k_1,\dots ,k_n$, but Cauchy interlace theorem does not tell us how the rest of this sequence behaves. The answer is obtained by considering the expression for the ratio $k_{i+1}^2/k_i^2$. This expression explains the reasons behind the condition number anomaly and characterizes situations that invite (or exclude) such behavior. We see that the anomaly phenomenon is likely to occur in “random-like” matrices whose rows satisfy two conditions: all the rows have about the same size and the directions of the rows scatter in some random way. This suggests that the condition number anomaly phenomenon is common in several types of matrices, and the numerical examples illustrate this point.

The practical importance of the condition number anomaly lies in the use of iterative methods for solving large linear systems. As we have seen, several iterative solvers have the property that the rate of convergence depends on the condition number. Therefore, when solving “random-like” systems, a fast rate of convergence is expected in under-determined or over-determined systems, while a slower rate is expected in (nearly) square systems.