1. Introduction
One of the most powerful ideas in linear algebra is diagonalization, which renders many problems completely transparent. For example, if
is a symmetric matrix, the spectral theorem implies that there exists an orthogonal matrix
and a diagonal matrix
such that
(see, for instance, ([
1], Section 7.1)). To say that
V is orthogonal means that
, the
identity matrix, which implies that the columns
of
V form an orthonormal basis for
. If
D has diagonal entries
, then we can also write
If
is a general matrix (not assumed to be symmetric or even square), the singular value decomposition (SVD) allows us to diagonalize the matrix, at the cost of using two different orthonormal bases. There exist orthogonal matrices
and
and a diagonal matrix
such that
and
, with
(see ([
2], Lectures 4–5) or ([
1], Chapter 8)). This decomposition follows immediately from the spectral theorem for symmetric matrices; in fact,
is the spectral decomposition of the symmetric matrix
.
Among its other virtues, the SVD reveals the rank of
A and bases for the four fundamental subspaces associated with
A. If
A has rank
r, then exactly
r of the singular values
are positive, and we can write the SVD in the reduced form
where
,
are the columns of
U,
V, respectively,
,
, and
. The four fundamental subspaces are represented by orthonormal bases as follows:
It is well known that spectral theory is considerably more complicated for linear operators defined on infinite-dimensional spaces. However, for compact operators, the finite-dimensional theory carries through almost unchanged. Throughout the rest of this paper, X and Y will denote real Hilbert spaces. If is linear, then T is called compact if and only if has a convergent subsequence in Y for every bounded sequence in X (this is equivalent to T’s being continuous when the weak topology is imposed on X and the norm topology on Y).
The spectral theorem for self-adjoint compact operators (see ([
3], Section 4.3) or ([
4], Section 8.2)) says that if
is compact and self-adjoint, then there exists an orthonormal sequence
in
X and a sequence
of nonzero real numbers such that
where the outer product
is the bounded linear operator defined by
For ease of exposition, we will assume that the sequences
and
are infinite sequences; in the contrary case,
T is a finite-rank operator and the infinite series becomes a finite sum.
For a general (that is, not necessarily self-adjoint) compact operator
, we can derive the singular value expansion (SVE) of
T by applying the spectral theorem for self-adjoint compact operators to
(see [
3], Section 4.4). The result is
where
is an orthonormal sequence in
X,
is an orthonormal sequence in
Y, and
are positive numbers converging to zero. Moreover,
is a complete orthonormal set for
and
is a complete orthonormal set for
.
The SVE of a compact operator has many applications, particularly in analyzing linear operator equations of the form (that is, given , find or estimate satisfying ). When T is compact and not of finite rank, this equation is ill-posed in the sense that the solution x (if it exists) does not depend continuously on the data y; in this case, the equation is often referred to as an inverse problem. The singular value expansion of T is useful in analyzing approaches to regularizing , that is, to computing a stable (approximate) solution to the equation in the presence of noisy data.
When is not necessarily compact, there still exists a singular value expansion in the following form.
Theorem 1. Let X and Y be real Hilbert spaces and let be a bounded linear operator. Then there exist a Borel space with a second-countable topology , isometries , , and an essentially bounded measurable function such thatwhere is the generalized inverse of V and is the multiplication operator defined by σ:Moreover, a.e. By Borel space, we mean a measure space such that there is a topology defined on the set M and (the collection of measurable subsets of M) is the -algebra of Borel sets of . As noted in the theorem, the topology is guaranteed to be second-countable, that is, to have a countable base. Furthermore, note that for an isometry V, .
We will call the representation of Theorem 1 the SVE of
T. Pietsch ([
5], Section D.3) outlines a short proof of Theorem 1 based on the polar decomposition. We give a direct proof in the next section that is analogous to the derivation of the SVD of a matrix
A from the spectral decomposition of
. In
Section 3, we derive some basic results about this form of the SVE, including its relationship to the classical SVE of a compact operator and how to recognize from the SVE when
fails to be closed. We also include a brief discussion of the relationship of the SVE to notions of
s-numbers (the generalization of singular values) that have appeared in the literature. In
Section 4, we analyze the inverse problem
, including methods for regularizing the equation, using Theorem 1. The results in
Section 4 are not new, but we hope to convince the reader that the analysis based on Theorem 1 is particularly convenient. We conclude with a brief discussion in
Section 5.
2. The SVE of a Bounded Linear Operator
As noted above, the SVD of a matrix can be derived from the spectral decomposition of the symmetric matrix . In the same way, the SVE described by Theorem 1 can be derived from the following spectral theorem for a bounded, self-adjoint linear operator.
Theorem 2. Let T be a bounded and self-adjoint linear operator mapping a real Hilbert space X into itself. Then there exists a Borel space with a second-countable topology , a unitary operator , and an essentially bounded measurable function such thatwhere is the multiplication operator defined by θ. This version of the spectral theorem is usually stated in terms of a complex Hilbert space
X and complex
(see, for instance, [
6] for an accessible exposition), but it can be verified that the same proof yields this representation when
X is a real Hilbert space and
denotes the space of real-valued square-integrable functions.
To derive Theorem 1 from Theorem 2, we require the following preliminary result.
Lemma 1. ([7], Lemma 2.1) Let be a measure space and let be a measurable function that is positive a.e. Define Then S is dense in .
We can now prove a special case of Theorem 1.
Theorem 3. Let X and Y be real Hilbert spaces and let be a bounded linear operator with . Then there exist a Borel space with a second-countable topology , a unitary operator , an isometry , and an essentially bounded measurable function such that Moreover, a.e. and .
Proof. By Theorem 2, there exist a measure space
, a unitary operator
, and a bounded measurable function
such that
We first show that
a.e., which obviously follows if we prove that
However,
and hence, for any
,
(using the fact that
because
V is unitary). Therefore,
a.e., as desired, and we can assume that
everywhere.
Now define
. If
, then
in
, where
is the characteristic function of the set
E; this implies that
in
X and hence that
(since
is trivial). However,
because
on
E and
on
. This contradiction shows that
must be zero, that is,
a.e. in M.
Therefore, if we define
and
then Lemma 1 applies and we see that
S is dense in
. We define
by
. Since
for all
,
U is well-defined, and we see that it is linear and densely defined. We now show that
for all
. We have
However, now we see that U is bounded and densely defined and hence it extends to a bounded operator defined on all of . We will use U to denote this extension as well (therefore satisfies ). By continuity, we have for all and hence U is an isometry.
Next, we show that
. For each
,
because
. However, then we see that, for each
,
Therefore, , as desired.
It remains only to show that
. Let
. Then there exists
such that
, that is,
. It follows that
is a Cauchy sequence and hence, because
U is an isometry,
is a Cauchy sequence in
. Suppose
. Then
which shows that
. Since
by definition of
U, it follows that
. This completes the proof. □
Theorem 1 is an immediate corollary of Theorem 3.
Proof of Theorem 1. If we apply Theorem 3 to
, we obtain
where
is an isometry and
is unitary. We claim that
where
is defined by
for all
. Since
V is obviously an isometry, proving that (
4) holds will complete the proof. By definition,
is the minimum-norm least-squares solution of
. Moreover, since
is trivial,
is the unique least-squares solution of
, which is defined by
This proves (
4), which completes the proof. □
4. The SVE and Tikhonov Regularization
We believe that Theorem 1 will prove to be useful in a variety of applications. Here we show that it can be used to give transparent proofs of convergence theorems in the theory of Tikhonov regularization, the most popular method for addressing inverse problems.
We consider an equation of the form . We are given and wish to compute or estimate satisfying the equation. The problem is well-posed if there exists a unique solution x for each , where x depends continuously on y. Existence fails, at least for some , if is a proper subspace of Y. However, in that case, it is common to settle for a least-squares solution of the equation, that is, an that minimizes . Uniqueness fails to hold if is nontrivial, but we can select a unique (least-squares) solution by choosing the unique solution lying in , which is equivalent to choosing the minimum-norm least-squares solution. The interesting case occurs when fails to be closed. In that case,
Least-squares solutions exists only for y in the dense subspace of Y;
For each , there exists a unique minimum-norm least-squares solution , but x does not depend continuously on y.
The generalized inverse , where , is defined by the condition that is the minimum-norm least-squares solution of . It follows from the above discussion that, when fails to be closed, then is a densely defined unbounded linear operator. In this case, the problem , even when interpreted as asking for the minimum-norm least-squares solution, is ill-posed in that the solution does not depend continuously on the data. In this case, we call a (linear) inverse problem.
Many regularization techniques for solving
approximate
by a family
of bounded operators. Here
is called the regularization parameter and it is required that
pointwise as
. The most popular regularization method is Tikhonov regularization, in which
. This operator arises from solving
We first show that for all . For convenience, we will write .
By definition,
. Furthermore,
is defined by the equation
which implies that
We will write , and notice that since . It follows that . Moreover, since the least-squares solutions of are precisely the solutions of , it also follows that . These two facts (that and that , , can be defined by in place of y) make it convenient to use the singular value expansion as expressed in Theorem 3 (as opposed to the version of Theorem 1).
Suppose that, using the notation of Theorem 3,
and recall that
a.e. in
M. Since
satisfies
,
Furthermore,
(where we write
as
when it is convenient to do so), and hence
To show that
as
, it suffices to show that
Moreover, since
(as opposed to merely belonging to
—this follows from the fact that
), it follows that
. However, then, since
is bounded on
M and goes to 0 pointwise as
, it follows that
by the dominated convergence theorem. This shows that
as
. Henceforth, we will write
.
We will prove two other results to demonstrate the usefulness of the singular value expansion. The result that we just defined shows that, for each
,
as
. However, the result says nothing about the rate of convergence and, in fact, the convergence can be arbitrarily slow depending on the data
(or, equivalently, on the solution
). For certain
, though, we can bound the rate of convergence. We will not attempt to prove the most general theorem, but rather just consider what turns out to be the optimal rate of convergence. We will show that if
, then
From above, we have
and
. Therefore,
If we now assume that
, say
for some
, then we obtain
Since V is an isometry and is bounded by 1 (in general, ), it follows that , as desired.
We can also prove the following converse result, namely, that if
and
, then
. We will use the fact, easily verified, that
if and only if
, that is, if and only if
belongs to the domain of the densely defined operator
. Let us write
; then we must show that
We have
which implies that
Since
by assumption, there exists a constant
such that
that is,
Since
converges monotonically to
a.e. in
M as
, it follows from the monotone convergence theorem that
as desired.
5. Discussion
The proofs of the last section are offered as an illustration of the power of the singular value expansion. The reader can compare these proofs to other treatments of the same results that can be found in the literature on inverse problems. In Groetsch’s monograph [
14], the analysis is restricted to compact operators and Theorem 2.1.1, Corollary 3.1.2, and Theorem 3.2.2 correspond to our results; Groetsch’s proofs use the singular value expansion (
2) for compact operators. The reader will see that our proofs are direct generalizations of the derivations given there, and also that there is no difficulty in extending his other conclusions to general bounded linear operators. Groetsch does present his theory in greater generality, with much of the analysis applying to a certain family of regularization operators
, as opposed to just the Tikhonov approach. We restricted our presentation to Tikhonov regularization simply for convenience of exposition; there would be no difficulty in reproducing his results in the same level of generality.
To extend the results of [
14] to general operators, the standard approach is to use the spectral representation of
in the form
where
is the spectral resolution of
, and apply the so-called functional calculus, which allows the representation of functions of
via
It can be shown, for example, that
A good reference for this approach is the book [
15] by Engl, Hanke, and Neubauer, which (among other things) extends the results of [
14] to general bounded linear operators. There is no intrinsic difficulty in doing so, but it may be argued that the arguments are less intuitive and therefore harder to follow. For instance, it is necessary to work with integrals of the following types:
As Halmos stated in his popular expository article [
6] on the spectral theorem (one of the most-downloaded articles from the American Mathematical Monthly),
The result (namely, the spectral theorem for Hermitian matrices, when expressed using a resolution of the identity) is not intuitive in any language; neither Stieltjes integrals with unorthodox multiplicative properties, nor bounded operator representations of function algebras, are in the daily toolbox of every working mathematician. In contrast, the formulation of the spectral theorem given below uses only the relatively elementary concepts of measure theory.
We believe that the singular value expansion for general bounded linear operators, as described above, offers a similarly intuitive tool that can replace the standard use of the functional calculus in many contexts.