Next Article in Journal
A Solution Procedure Combining Analytical and Numerical Approaches to Investigate a Two-Degree-of-Freedom Vibro-Impact Oscillator
Previous Article in Journal
An Operator-Based Scheme for the Numerical Integration of FDEs
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Error Estimations for Total Variation Type Regularization

1
School of Cyberspace Security, Dongguan University of Technology, Dongguan 523808, China
2
Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen 518060, China
3
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China
4
Department of Mathematics, National University of Defense Technology, Changsha 410073, China
*
Author to whom correspondence should be addressed.
Mathematics 2021, 9(12), 1373; https://doi.org/10.3390/math9121373
Submission received: 1 June 2021 / Revised: 8 June 2021 / Accepted: 11 June 2021 / Published: 13 June 2021

Abstract

:
This paper provides several error estimations for total variation (TV) type regularization, which arises in a series of areas, for instance, signal and imaging processing, machine learning, etc. In this paper, some basic properties of the minimizer for the TV regularization problem such as stability, consistency and convergence rate are fully investigated. Both a priori and a posteriori rules are considered in this paper. Furthermore, an improved convergence rate is given based on the sparsity assumption. The problem under the condition of non-sparsity, which is common in practice, is also discussed; the results of the corresponding convergence rate are also presented under certain mild conditions.

1. Introduction

Compressed sensing [1,2] has gained increasing attention in recent years; it plays an important role in signal processing [3,4], imaging science [5,6] and machine learning [7]. Compressed sensing focusses on signals with sparse presentation. Let H 1 be a Hilbert space, and { e i H 1 | i N } be the orthonormal basis of H 1 . For any x H 1 , let x i : = x , e i . Given some operators K satisfy certain conditions, it is possible to recover a sparse x C n signal with length n by Basis Pursuit (BP) [8], i.e.,
min | | x | | 1 s . t . y = K x ,
from the samples y = K x , even K is ill-posed [2,9,10]. However, in most cases, noise is inevitable. The literatures has turned to studying the noised BP model
min | | x | | 1 s . t . K x y 2 δ ,
where δ is the allowed error. Actually, the unconstrained form of the noised BP model, i.e., sparse regularization which is the focus in [11,12,13,14,15,16] is more attractive. While the success of compressed sensing greatly inspired the development of sparse regularization, it is interesting to see that sparse regularization appeared much earlier than compressed sensing [11,12]. As an inverse problem, the error theory of sparse regularization is well studied in the literature [17,18,19].
In practical terms, a large crowd of signals is not sparse unless being transformed by some operators (maybe ill posed). Thus, many studies have been proposed to analyze the regularized optimization problem [20]. A typical example of them is signal with a sparse gradient which arises frequently from imaging processing (nature images are usually piece-wise constant, i.e., they have a sparse gradient). The Total Variation (TV) has been used extensively in the literature for decades in imaging sciences and a series of techniques have been dedicated to researching its choice of regularization parameter [21,22,23,24,25,26,27,28,29,30,31]; others [32,33] are developed based on this observation. Similar to [34], Total Variation can also smooth the signal of interest. Let H 2 be another Hilbert space. For any x H 1 , define that T : H 1 H 1 satisfies
( T x ) i : = x i x i + 1 .
Under the above definition, T is an ill-posed linear operator. Given a linear map K : H 1 H 2 and y δ H 2 , the total variation regularization problem can be represented as
Ψ α ( x ) = 1 2 K x y δ 2 2 + α i | ( T x ) i | ,
where α > 0 is the regularization parameter. The regularization term i | ( T x ) i | is the right total variation (TV) of x. The TV type regularization has a similar form to the sparse regularization. However, the perfect reconstruction result established in sparse regularization can not be applied to theTV type directly, especially when T is ill-posed (T has a nontrivial null space).
So in this paper, firstly, we discuss the stability and consistency of the minimizers of Ψ α . Besides basic properties, we are also interested in the convergence rate to solve the TV problem. Then, under the source conditions [19,35,36], convergence rates get obtained for both a priori and a posteriori parameter choice rules. However, the linear convergence rate requires K to be injective, which is strict usually. In the latter part, the linear convergence rate can also be derived under the sparsity assumption on T x and some suitable conditions for K. This requirement of deduction does not depend on the injectivity of K. Meanwhile, this paper also considers the case when the sparsity assumption on T x fails. Last, based on some recent works [37,38,39], which also assume the T x is not sparse, a convergence rate is also given in this case.
The rest of this paper is organized as follows. Section 2 provides a brief summary of the notations. Section 3 presents some basic properties and gives the convergence rate of the minimizer. Section 4 proves the improved convergence rate. Finally, Section 5 concludes the whole paper.

2. Notation

The notations described in this section are adopted throughout this paper. Let H 1 , H 2 be two Hilbert spaces and { e i H 1 | i N } , { ξ i H 2 | i N } be the orthonormal basis of H 1 and H 2 , respectively. For any x H 1 and y H 2 , x i : = x , e i and y i : = y , ξ i . The 1 and 2 norms of x and y are denoted by x 1 : = i | x i | , x 2 : = ( i | x i | 2 ) 1 2 and y 1 : = i | y i | , y 2 : = ( i | y i | 2 ) 1 2 , respectively. In this paper, if not specified, for any x H 1 and y H 2 , we assume that x , y L 2 , i.e., x 2 < + and y 2 < + . x n x means that x n converges weakly to x, while x n x means x n converges strongly to x. The operator norm of the linear operator K : H 1 H 2 is defined as
K : = max x 2 = 1 K x 2 .
Through the paper, x means the signal of interest; y : = K x are the measurements. y δ denotes an element in H 2 satisfying y y δ 2 δ . Under these notations, the TV regularization can be expressed as
Ψ α ( x ) = 1 2 K x y δ 2 2 + α T x 1 .
Denote that x α δ is one of the minimizers of Ψ α .
Remark 1.
Considering the set L = { x n } n = 1 , 2 , H 1 , where
x i n : = 1 / n i f i n , 0 i f i > n . .
Obviously, for any n, x n L 2 and T ( x n ) = 1 / n . As n + , x n 2 = 1 and T ( x n ) 0 . That means T is ill posed.
Remark 2.
Let D = T I d , where I d is the identical operator over H 1 . Then, ( D x ) i = x i + 1 for any i N . It is easy to verify that D is continuous. Then, T is continuous over H 1 and
T I d = D 1 .
In practice, The ill condition of T brings trouble to the analysis. To overcome this problem, we consider a condition which plays an important role in the deduction.
Condition 1.
There exist two constants c , m > 0 such that
c K x 2 + m T x 2 x 2
for any x H 1 .
We present a finite-dimensional understanding of this condition. Let d i m ( H 1 ) = M and d i m ( H 2 ) = N . Then, K R M × N satisfies n u l l ( K ) 0 and T R ( N 1 ) × N . In the finite dimension case, T has the form
1 1 1 1 1 1 ( N 1 ) × N .
The definition of T gives that n u l l ( T ) = s p a n ( 1 ) . If K 1 0 . Then, n u l l ( K ) n u l l ( T ) = 0 , we have that n u l l c ^ K m ^ T = 0 , where c ^ , m ^ > 0 . Hence, for any x R n and some ι > 0 , ι c ^ K m ^ T x 2 x 2 . Note that c ^ K m ^ T x 2 c ^ K x 2 + m ^ T x 2 ; we then have ι c ^ K x 2 + ι m ^ T x 2 x 2 .

3. Basic Error Estimations

The properties of TV type regularization are investigated in this section. First, a lemma is introduced which is used in this section frequently.
Lemma 1.
Let y δ be bounded, α be fixed and { x n } n = 1 , 2 , be a sequence. Assume that Condition 1 holds and { Ψ α ( x n ) } n = 1 , 2 , is bounded. Then, { x n } n = 1 , 2 , is also bounded.
Proof. 
It is trivial to prove { K x n y δ 2 } n = 1 , 2 , and { T x n 1 } n = 1 , 2 , are bounded. Note that
T x n 2 T x n 1 and K x n 2 K x n y δ 2 + y δ 2 ,
which implies { K x n 2 } n = 1 , 2 , and { T x n 2 } n = 1 , 2 , are bounded. From Condition 1, we derive that
x n 2 c K x n 2 + m T x n 2 .
implies the boundedness of { x n } n = 1 , 2 , . □

3.1. Stability

In this subsection, we investigate the performance of x α ^ δ as α ^ α , when y δ is fixed. A lemma is introduced which arises in convex optimization.
Lemma 2
([40,41]). Let χ * be the solution set of the convex minimization problem
min x Ψ α ( x ) .
Then, K x and T x 1 is constant over χ * .
Theorem 1.
Assume that K , T satisfies Condition 1. For any fixed α > 0 and y δ H 2 , we have
lim α n α K x α n δ = K x α δ .
Proof. 
The minimizing property of x α n δ gives that 1 2 K x α n δ y δ 2 2 + α T x α n δ 1 Ψ α n ( 0 ) . Then, Lemma 1 indicates that there exists a subsequence of { x α n δ } converging weakly to some x * 2 . For simplicity, we also denote this subsequence as { x α n δ } . By the weak lower continuity of the norms, we have
K x * y δ 2 lim inf n K x α n δ y δ 2 and T x * 1 lim inf n T x α n δ 1 .
Therefore, we have that
Ψ α ( x * ) = 1 2 K x * y δ 2 2 + α T x * 1 lim inf n { 1 2 K x α n δ y δ 2 2 + α n T x α n δ 1 } = lim inf n Ψ α n ( x α n δ ) .
On the other hand, by the minimizing property of x α n δ ,
lim sup n Ψ α n ( x α n δ ) lim sup n Ψ α n ( x α δ ) = lim n Ψ α n ( x α δ ) = Ψ α ( x α δ ) .
Obviously, it holds that
lim sup n Ψ α n ( x α n δ ) Ψ α ( x α δ ) Ψ α ( x * ) lim inf n Ψ α n ( x α n δ ) .
That means x * minimizes Ψ α ( x ) . From Lemma 2, K x * = K x α δ and T x * 1 = T x α δ 1 . Consequently, we have K x α n δ K x α δ , Ψ α n ( x α n δ ) Ψ α ( x α δ ) and T x α δ 1 lim inf n T x α n δ 1 . In the following, we present the proof by the mean of contradiction. Assume that t : = lim sup n K x α n δ y δ 2 > K x α δ y δ 2 . We can obtain that
α T x α δ 1 lim inf n { α T x α n δ 1 } = lim inf n { Ψ α n ( x α n δ ) K x α n δ y δ 2 2 } = Ψ α ( x α δ ) lim sup n K x α n δ y δ 2 2 = α T x α δ 1 + ( K x α δ y δ 2 t ) < α T x α δ 1 .
This is a contradiction. Then, we have
lim sup n K x α n δ y δ 2 K x α δ y δ 2 .
From relations (3), we can obtain that K x α n δ K x α δ . □
If K is injective, we can further have that lim α n α x α n δ = x α δ . The theorem above indicates that Ψ α ( x α δ ) and T x α δ 1 are continuous at α . In fact, we can obtain a stronger result; the value function is differentiable at α .
Theorem 2.
Let F ( α ) : = Ψ α ( x α δ ) ; then, F ( α ) is differentiable with respect to α, and F ( α ) = T x α δ 1 .
Proof. 
For α > α ^ , we have
F ( α ) F ( α ^ ) = 1 2 K x α δ y δ 2 2 + α T x α δ 1 1 2 K x α ^ δ y δ 2 2 α T x α ^ δ 1 + ( α α ^ ) T x α ^ δ 1 .
Due to that x α δ minimizing Ψ α , we have
1 2 K x α δ y δ 2 2 + α T x α δ 1 1 2 K x α ^ δ y δ 2 2 α T x α ^ δ 1 0 .
It follows that F ( α ) F ( α ^ ) ( α α ^ ) T x α ^ δ 1 . On the other hand, F ( α ) F ( α ^ ) can be written as
F ( α ) F ( α ^ ) = 1 2 K x α δ y δ 2 2 + α ^ T x α δ 1 1 2 K x α ^ δ y δ 2 2 α ^ T x α ^ δ 1 + ( α α ^ ) T x α δ 1 .
Similarly, we have F ( α ) F ( α ^ ) ( α α ^ ) T x α δ 1 . Combining the two inequalities above, we have
T x α δ 1 F ( α ) F ( α ^ ) α α ^ T x α ^ δ 1 .
When α < α ^ , similar results can be also obtained. The continuity of T x α δ 1 at α gives that d F ( α ) d α = T x α δ 1 . □

3.2. Consistency

The performance of x α δ is investigated under a prior parameter choice as δ 0 . In the analysis, we assume that the following conditions hold.
Condition 2.
For any x H 1 obeying K x = y , x satisfies that
T x 1 T x 1 .
The equality holds if and only if x = x .
Lemma 3.
Let { x n } n x * , K x n y 2 2 K x * y 2 2 and T x n 1 T x * 1 . Then, we have T ( x n x * ) 1 0 and K ( x n x * ) 2 0 .
Proof. 
We can obtain that
lim sup n T ( x n x * ) 1 = lim sup n ( ( T x n 1 + T x * 1 ) ( T x n 1 + T x * 1 ) + T ( x n x * ) 1 ) ) = 2 T x * 1 lim inf n ( ( T x n 1 + T x * 1 ) T ( x n x * ) 1 ) .
The triangle inequality gives that ( T x n 1 + T x * 1 ) T ( x n x * ) 1 0 . The Fatou’s lemma gives that
lim inf n ( ( T x n 1 + T x * 1 ) T ( x n x * ) 1 ) = lim inf n ( i | ( T x n ) i | + | ( T x * ) i | | ( T ( x n x * ) ) i | ) i lim inf n ( | ( T x n ) i | + | ( T x * ) i | | ( T ( x n x * ) ) i | ) .
Note that x n x * 0 ; then, T ( x n x * ) T 0 = 0 . Hence, [ T ( x n x * ) ] i 0 . Similarly, we can obtain ( T x n ) i ( T x * ) i . Therefore,
i lim inf n ( | ( T x n ) i | + | ( T x * ) i | | ( T ( x n x * ) ) i | ) = i 2 | ( T x * ) i | = 2 T x * 1 .
Thus, we have
lim sup n T ( x n x * ) 1 = 0 .
By the same method, we also can obtain that K ( x n x * ) 2 0 . □
Theorem 3.
Assume that K , T satisfies Condition 1 and Lemma 1. Let the parameters satisfy that
α ( δ ) , δ 2 α ( δ ) 0 a s δ 0 .
Then the sequence { x α δ } δ x .
Proof. 
By the definition of x α δ , we have
1 2 K x α δ y δ 2 2 + α T x α δ 1 1 2 K x y δ 2 2 + α T x 1 1 2 δ 2 + α T x 1 .
From the parameters’ choice rule of α and δ , we can see that { Ψ α ( x α δ ) } are bounded. Then, from Lemma 1, there exists a subsequence also denoted by { x α δ } δ and some point x * such that x α δ x * . We can have that
K x * y 2 2 lim inf δ K x α δ y 2 2 2 lim inf δ ( K x α δ y δ 2 2 + y δ y 2 2 ) lim inf δ ( δ 2 + 2 α ( δ ) T x 1 + δ 2 ) = 0 .
This means K x * = y . It is easy to see that lim δ K x α δ y 2 2 = 0 . On the other hand, we can obtain that
T x * 1 lim inf δ { T x α δ 1 + δ 2 2 α ( δ ) } = T x 1 .
Condition 2 gives that x * = x . From the inequality above, we see that lim δ T x α δ 1 = T x 1 . By Lemma 3, we have T ( x α δ x * ) 1 0 and K ( x α δ x * ) 2 0 . Consequently, from Condition 1, it holds that
lim δ x α δ x * 2 lim δ m T ( x α δ x * ) 2 + c K ( x α δ x * ) 2 lim δ m T ( x α δ x * ) 1 + c K ( x α δ x * ) 2 = 0 .
 □

3.3. Convergence Rate

This subsection concerns the convergence rate under different parameter choice rules (a priori and a posteriori). First, we discuss the a priori one. Like the classical Tikhonov regularization method [19,35,36], we introduce a source condition.
Condition 3.
Let x satisfy the source condition
w : K * w T * T x 1 .
Theorem 4.
If x satisfies the source condition, it holds that
K x α δ y δ 2 2 α w 2 + δ .
If K is injective, there exists γ > 0 such that
x α δ x 2 2 γ α w 2 + 2 γ δ .
Proof. 
The definition of x α δ gives that
1 2 K x α δ y δ 2 2 + α T x α δ 1 1 2 K x y δ 2 2 + α T x 1 .
Using the notation C ( x ) = T x 1 , we obtain that
1 2 K x α δ y δ 2 2 + α C ( x α δ ) 1 2 K x y δ 2 2 + α C ( x ) .
For any v C ( x ) , the convexity of C indicates C ( x α δ ) C ( x ) + v , x α δ x . Then, we have that
1 2 K x α δ y δ 2 2 + α C ( x ) + α v , x α δ x 1 2 K x α δ y δ 2 2 + α C ( x α δ ) 1 2 K x y δ 2 2 + α C ( x ) .
Choose v = K * w in the source condition; after simplification, we derive that
1 2 K x α δ y δ 2 2 + α w , K x α δ y δ 1 2 K x y δ 2 2 + α w , K x y δ .
By adding both sides with α 2 w 2 2 2 , we obtain that
K x α δ y δ + α w 2 K x y δ + α w 2 .
This means
K x α δ y δ 2 2 α w 2 + K x y δ 2 2 α w 2 + δ .
If K is injective, there exists γ > 0 such that x 2 γ K x 2 . Then, we derive that
x α δ x 2 γ K x α δ y 2 γ ( K x α δ y δ 2 + y δ y ) γ ( 2 α w 2 + 2 δ ) .
 □
Remark 3.
In fact, the first result in Theorem 4 has been proved by [42] for general convex regularization. The proof here is for the completeness.
The following part investigates the a posteriori parameter choice rule. The analysis is motivated by the work in [43,44]. For simplicity of presentation, the parameter α is chosen as
K x α δ y δ 2 = δ .
Theorem 5.
Assume that α is chosen as rule (5), and x satisfies Condition 2. It then holds that
lim δ 0 x α δ = x .
If K is injective, there exists θ > 0 such that
x α δ x * 2 2 θ δ .
Proof. 
It is trivial to prove that
1 2 K x α δ y δ 2 2 + α T x α δ 1 1 2 K x y δ 2 2 + α T x 1 .
Lemma 2 indicates that { x α δ } δ is bounded. Note that K x α δ y δ 2 = δ and K x y δ 2 δ . It then follows that
T x α δ 1 T x 1 .
Then, the sequence has a sub-sequence also denoted by { x α δ } δ converging weakly to some x * . We can easily see that
K x * y 2 lim inf δ 0 K x α δ y 2 lim inf δ 0 ( K x α δ y δ 2 + y δ y 2 ) lim inf δ 0 2 δ = 0 .
That is actually to say that K x * = y . Moreover, it is easy to see that lim δ K x α δ y 2 2 = 0 . Using relation (6), we have that
T x * 1 lim inf δ 0 T x α δ 1 T x 1 .
Condition 2 gives that x * = x ; hence, the whole sequence converges weakly to x and
T x 1 lim inf δ 0 ( T x α δ 1 ) lim sup δ 0 ( T x α δ 1 ) T x 1 .
Thus, we have T x α δ 1 T x 1 . From Lemma 3, we have T ( x α δ x * ) 1 0 and K ( x α δ x * ) 2 0 which leads to
x α δ x * 2 m T ( x α δ x * ) 2 + c K ( x α δ x * ) 2 m T ( x α δ x * ) 1 + c K ( x α δ x * ) 2 0 .
If K is injective, there exists θ > 0 such that x 2 θ K x 2 . Then, we derive that
x α δ x 2 θ K x α δ y 2 θ ( K x α δ y δ 2 + y δ y ) 2 θ δ .
 □

4. Improved Convergence Rate

In this section, we investigate the convergence rate when K may be not injective. The first part presents the analysis under the sparse assumption while the second one deals with the case when the sparsity assumption fails.

4.1. Performance under Sparsity Assumption

The analysis in this subsection assumes that T x is sparse. To prove the convergence rate we need the finite injectivity property [45].
Condition 4.
The operatorKsatisfies the uniformly finite injectivity property, i.e., for any finite subset S N , K | S is injective.
Remark 4.
In the finite dimension case, if s u p p ( S ) is small, it is easy to find that the finite injectivity property is actually the restrict isometry property [2,46].
Let z : = T x and z : = T x . Denote S as the set S : = { i N : | v i | > 1 2 } , where v z 1 satisfies the source condition. Let m = sup i S { | v i | } . Due to that v 2 , S is finite and it contains the support of z . Let P be the identical projection onto S and P be the one onto N \ S . From Condition 4, there exists some d > 0 such that
d K P z 2 P z 2 .
Lemma 4.
Assume that x satisfies the source condition and Condition 1 holds. If m d K < 1 , there exist c 1 > 0 and c 2 > 0 such that
T x 1 T x 1 c 1 x x 2 c 2 T ( x x ) 2 .
Proof. 
Assume the conditions in Lemma 4 are held. Then, we can obtain that
z z 2 P ( z z ) 2 + P ( z z ) 2 d K P ( z z ) 2 + P z 2 d K ( z z ) 2 + ( 1 + d K ) P z 2 .
Hence, we derive that
K ( z z ) 2 = K T ( x x ) 2 K ( x x ) 2 + K ( T I d ) ( x x ) 2 K ( x x ) 2 + K · T I d · x x 2 K ( x x ) 2 + K · x x 2 .
We now turn to estimating P z 2 . Let m = sup i S { | v i | } . Obviously, m 1 2 . We then have that
P z 2 i S | z i | 2 i S ( 1 m ) | z i | 2 i S ( | z i | v i z i ) 2 i S ( | z i | | z i | v i ( z i z i ) ) 2 ( z 1 z 1 v , z z ) = 2 ( T x 1 T x 1 v , z z ) .
The source condition gives that
v , z z = v , T x T x = T * v , x x = K * w , x x = w , K x K x w 2 · K x K x 2 .
Therefore, we have that
z z 2 d K · x x 2 + 2 ( 1 + d K ) ( T x 1 T x 1 ) + ( 2 w 2 + 2 d K w 2 + d ) K ( x x ) 2 .
From Condition 1, we have that
x x 2 = c K ( x x ) 2 + m T x T x 2 c K ( x x ) 2 + m z z 2 m d K · x x 2 + 2 ( m + m d K ) ( T x 1 T x 1 ) + ( 2 m w 2 + 2 d m K w 2 + m d + c ) K ( x x ) 2 .
Note that m d K < 1 ; let q = 1 1 m d K ; we have that
x x 2 2 q ( m + m d K ) ( T x 1 T x 1 ) + q ( 2 m w 2 + 2 m d K w 2 + m d + c ) K ( x x ) 2 .
 □
With the lemma above, we can obtain the following result. The proofs can be found in [44,47,48,49].
Theorem 6.
Let the regularization parameter be chosen a priori as α ( δ ) = O ( δ ) or a posteriori as α ( δ ) according to the strong discrepancy principle (5) Then we have the convergence rate
x α δ x 2 = O ( δ ) .

4.2. Performance if Sparsity Assumption Fails

In this subsection, we focus on the case where T x is not sparse. As presented in the last section, lemma 4 is critical for the convergence rate analysis. In this part, a similar lemma will be proposed. Then, the convergence rate will be proved. The first lemma is motivated by [37].
Lemma 5.
For any x H 1 and n N , it holds that
T ( x x ) 1 T x 1 + T x 1 2 ( k = n + 1 | ( T x ) k | + k = 1 n | ( T x ) k ( T x ) k | ) .
Proof. 
Denote the projection P n ( x ) = ( x 1 , x 2 , , x n , 0 , ) for any x H 1 . Hence, we have
T x 1 = P n T x 1 + ( I d P n ) T x 1 .
Algebra computation gives that
T ( x x ) 1 T x 1 + T x 1 = P n T ( x x ) 1 + ( I d P n ) T x 1 + ( I d P n ) ( T x T x ) 1 ( I d P n ) T x 1 + P n T x 1 P n T x 1 .
Note that
( I d P n ) ( T x T x ) 1 ( I d P n ) T x 1 + ( I d P n ) T x 1
and
P n T x 1 P n T ( x x ) 1 + P n T x 1 .
Combining the equations above, we obtain
T ( x x ) 1 T x 1 + T x 1 2 P n T ( x x ) 1 + 2 ( I d P n ) T x 1 .
 □
Condition 5.
For all k N there exists f k H 2 such that T * e k = K * f k and lim k f k 2 + .
Lemma 6.
Let φ ( t ) : = 2 inf n { k = n + 1 | ( T x ) k | + t k = 1 n f k } ; φ ( t ) is concave index function. Assume that x satisfies the source condition and Conditions 1 and 5 hold. If c K < 1 , it holds that
x x 2 c 1 T x 1 c 2 T x 1 + c 3 φ ( K ( x x ) 2 )
for some positive c 1 , c 2 , c 3 .
Proof. 
φ is concave and upper semi-continuous since it is an infimum of affine functions. For any t 0 , φ is finite and continuous. Note that φ ( 0 ) = 0 ; the upper semi-continuity at t = 0 gives the continuity of φ at t = 0 . We turn to the strict monotonicity of φ . Condition 5 means the infimum of φ ( t ) is attained at some n N . Considering 0 < t 1 < t 2 < + , we have
φ ( t 1 ) = 2 ( k = n 1 + 1 | ( T x ) k | + t 1 k = 1 n 1 f k 2 ) 2 ( k = n 2 + 1 | ( T x ) k | + t 1 k = 1 n 2 f k 2 ) < 2 ( k = n 2 + 1 | ( T x ) k | + t 2 k = 1 n 2 f k 2 ) = φ ( t 2 ) .
From Condition 5, we have that
k = 1 n | ( T x T x ) k | = k = 1 n | T x T x , e k | = k = 1 n | x x , T * e k | = k = 1 n | x x , K * f k | K ( x x ) 2 k = 1 n f k 2 .
Therefore, we obtain that
T ( x x ) 2 T x 1 T x 1 + 2 φ ( K ( x x ) 2 ) .
From Condition 1, we have that
x x 2 c K ( x x ) 2 + m T ( x x ) 2 c K x x 2 + m T x 1 m T x 1 + 2 m φ ( K ( x x ) 2 ) .
Let q = 1 1 c K , we obtain that
x x 2 q m T x 1 q m T x 1 + 2 q m φ ( K ( x x ) 2 ) .
 □
Theorem 7.
Let the regularization parameter be chosen a priori as α ( δ ) = O ( δ 2 φ ( δ ) ) or a posteriori as α ( δ ) according to the strong discrepancy principle (5). Then we have the convergence rate
x α δ x 2 = O ( φ ( δ ) ) .

5. Conclusions

In this paper, we study some problems in total variation type regularization. While owning a familiar form as the sparse regularization, the TV type is hard to investigate for the ill condition of T. A group of regularization conditions has been given in this paper. Under these conditions, we study several theoretical properties such as stability, consistency and convergence rates of the minimizer of the TV type regularization. These analyses are deepened for the convergence rate under the assumption of sparsity. In the non-sparse case, we also present a conservative result based on some recent works. Now, the regularizers learned from the data are all the rage in research. So, in future work, we will make the error estimations for this type of regularization problem.

Author Contributions

Conceptualization, K.L. and Z.Y.; methodology, C.H.; validation, K.L., C.H. and Z.Y.; formal analysis, K.L.; writing—original draft preparation, K.L. and Z.Y.; writing—review and editing, K.L. and Z.Y.; supervision, C.H.; project administration, Z.Y.; funding acquisition, K.L. and C.H. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Key Research and Development Program of China under Grant 2020YFA0709803, 173 Program under Grant 2020-JCJQ-ZD-029, the Science Challenge Project under Grant TZ2016002, and Dongguan Science and Technology of Social Development Program under Grant 2020507140146.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Donoho, D.L. Compressed sensing. IEEE T. Inform. Theory 2006, 52, 1289–1306. [Google Scholar] [CrossRef]
  2. Candes, E.J.; Tao, T. Decoding by linear programming. IEEE T. Inform. Theory 2005, 51, 4203–4215. [Google Scholar] [CrossRef] [Green Version]
  3. Lustig, M.; Donoho, D.L.; Santos, J.M.; Pauly, J.M. Compressed Sensing MRI. IEEE Signal Proc. Mag. 2008, 25, 72–82. [Google Scholar] [CrossRef]
  4. Haupt, J.; Bajwa, W.U.; Rabbat, M.; Nowak, R. Compressed Sensing for Networked Data. IEEE Signal Proc. Mag. 2008, 25, 92–101. [Google Scholar] [CrossRef]
  5. Yin, W.; Osher, S.; Goldfarb, D.; Darbon, J. Bregman iterative algorithms for 1-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 2008, 1, 143–168. [Google Scholar] [CrossRef] [Green Version]
  6. Cai, J.F.; Osher, S.; Shen, Z. Split Bregman methods and frame based image restoration. Multiscale Model. Sim. 2009, 8, 337–369. [Google Scholar] [CrossRef]
  7. Van den Berg, E.; Friedlander, M.P. Probing the Pareto Frontier for Basis Pursuit Solutions. SIAM J. Sci. Comput. 2008, 31, 890–912. [Google Scholar] [CrossRef] [Green Version]
  8. Chen, S.S.; Donoho, D.L.; Saunders, M.A. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 1998, 20, 33–61. [Google Scholar] [CrossRef]
  9. Candès, E.J.; Romberg, J.; Tao, T. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 2006, 52, 489–509. [Google Scholar] [CrossRef] [Green Version]
  10. Candès, E.J. The restricted isometry property and its implications for compressed sensing. CR Math. 2008, 346, 589–592. [Google Scholar]
  11. Tibshirani, R. Regression Shrinkage and Selection via the Lasso. J. R. Stat. Soc. B 1996, 58, 267–288. [Google Scholar] [CrossRef]
  12. Daubechies, I.; Defrise, M.; Mol, C.D. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 2004, 57, 1413–1457. [Google Scholar] [CrossRef] [Green Version]
  13. Daubechies, I.; Devore, R.; Fornasier, M.; Güntürk, C.S. Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 2010, 38, 1–38. [Google Scholar] [CrossRef] [Green Version]
  14. Bot, R.I.; Hofmann, B. The impact of a curious type of smoothness conditions on convergence rates in 1-regularization. Eur. J. Math. Comput. Appl. 2013, 1, 29–40. [Google Scholar]
  15. Jin, B.; Maass, P. Sparsity regularization for parameter identification problems. Inverse Probl. 2012, 28, 123001. [Google Scholar] [CrossRef]
  16. Grasmair, M.; Haltmeier, M.; Scherzer, O. Sparse regularization with q penalty term. Inverse Probl. 2008, 24, 055020. [Google Scholar] [CrossRef] [Green Version]
  17. Hans, E.; Raasch, T. Global convergence of damped semismooth Newton methods for 1 Tikhonov regularization. Inverse Probl. 2015, 31, 025005. [Google Scholar] [CrossRef]
  18. Lorenz, D.A.; Schiffler, S.; Trede, D. Beyond convergence rates: Exact recovery with the Tikhonov regularization with sparsity constraints. Inverse Probl. 2011, 27, 085009. [Google Scholar] [CrossRef]
  19. Lorenz, D.A. Convergence rates and source conditions for Tikhonov regularization with sparsity constraints. J. Inverse III-Pose P 2008, 16, 463–478. [Google Scholar] [CrossRef] [Green Version]
  20. Rubinov, A.M.; Yang, X.Q.; Bagirov, A.M. Penalty functions with a small penalty parameter. Optim. Methods Softw. 2002, 17, 931–964. [Google Scholar] [CrossRef]
  21. Cai, J.F.; Dong, B.; Osher, S.; Shen, A.Z. Image Restoration: Total Variation, Wavelet Frames and Beyond. J. Am. Math. Soc. 2012, 25, 1033–1089. [Google Scholar] [CrossRef] [Green Version]
  22. Rudin, L.I.; Osher, S.; Fatemi, E. Nonlinear total variation based noise removal algorithms. Phys. D 1992, 60, 259–268. [Google Scholar] [CrossRef]
  23. Jun, L.; Huang, T.Z.; Gang, L.; Wang, S.; Lv, X.-G. Total variation with overlapping group sparsity for speckle noise reduction. Neurocomputing 2019, 216, 502–513. [Google Scholar]
  24. Van den Berg, P.M.; Kleinman, R.E. A total variation enhanced modified gradient algorithm for profile reconstruction. Inverse Probl. 1995, 11, L5. [Google Scholar] [CrossRef]
  25. Clason, C.; Jin, B.; Kunisch, K. A Duality-Based Splitting Method for 1-TV Image Restoration with Automatic Regularization Parameter Choice. SIAM J. Sci. Comput. 2010, 32, 1484–1505. [Google Scholar] [CrossRef]
  26. Cai, J.F.; Xu, W.Y. Guarantees of total variation minimization for signal recovery. Inform. Infer. 2015, 4, 328–353. [Google Scholar] [CrossRef] [Green Version]
  27. Needell, D.; Ward, R. Stable Image Reconstruction Using Total Variation Minimization. SIAM J. Imaging Sci. 2012, 6, 1035–1058. [Google Scholar] [CrossRef] [Green Version]
  28. Sun, T.; Yin, P.; Cheng, L.; Jiang, H. Alternating direction method of multipliers with difference of convex functions. Adv. Comput. Math. 2018, 44, 723–744. [Google Scholar] [CrossRef]
  29. Sun, T.; Jiang, H.; Cheng, L.; Zhu, W. Iteratively Linearized Reweighted Alternating Direction Method of Multipliers for a Class of Nonconvex Problems. IEEE Trans. Signal Proc. 2018, 66, 5380–5391. [Google Scholar] [CrossRef] [Green Version]
  30. Leonov, A.S. On the total-variation convergence of regularizing algorithms for ill-posed problems. Comput. Math. Math. Phys. 2007, 47, 732–747. [Google Scholar] [CrossRef]
  31. Tikhonov, A.; Leonov, A.; Yagola, A. Nonlinear Ill-Posed Problems; De Gruyter: Berlin, Germany, 2011; pp. 505–512. [Google Scholar] [CrossRef]
  32. Hào, D.N.; Quyen, T.N.T. Convergence rates for total variation regularization of coefficient identification problems in elliptic equations I. Inverse Probl. 2011, 27, 075008. [Google Scholar] [CrossRef]
  33. Hào, D.N.; Quyen, T.N.T. Convergence rates for total variation regularization of coefficient identification problems in elliptic equations II. J. Math. Anal. Appl. 2012, 388, 593–616. [Google Scholar] [CrossRef] [Green Version]
  34. Ciegis, R.; Sev, A.J. Nonlinear Diffusion Problems In Image Smoothing. Math. Model. Anal. 2005, 1, 381–388. [Google Scholar]
  35. Grasmair, M.; Scherzer, O.; Haltmeier, M. Necessary and sufficient conditions for linear convergence of 1-regularization. Commun. Pure Appl. Math. 2011, 64, 161–182. [Google Scholar] [CrossRef]
  36. Burger, M.; Osher, S. Convergence rates of convex variational regularization. Inverse Probl. 2004, 20, 1411. [Google Scholar] [CrossRef]
  37. Burger, M.; Flemming, J.; Hofmann, B. Convergence rates in 1-regularization if the sparsity assumption fails. Inverse Probl. 2013, 29, 025013. [Google Scholar] [CrossRef] [Green Version]
  38. Flemming, J.; Hofmann, B.; Veselić, I. A unified approach to convergence rates for 1-regularization and lacking sparsity. J. Inverse III-Pose P 2015, 24, 139–148. [Google Scholar] [CrossRef]
  39. Anzengruber, S.W.; Hofmann, B.; Mathé, P. Regularization properties of the sequential discrepancy principle for Tikhonov regularization in Banach spaces. Appl. Anal. 2014, 93, 1382–1400. [Google Scholar] [CrossRef]
  40. Luo, Z.Q.; Tseng, P. On the convergence of the coordinate descent method for convex differentiable minimization. J. Optim. Theory Appl. 1992, 72, 7–35. [Google Scholar] [CrossRef] [Green Version]
  41. Luo, Z.Q.; Tseng, P. Error bound and convergence analysis of matrix splitting algorithms for the affine variational inequality problem. SIAM J. Optim. 1992, 2, 43–54. [Google Scholar] [CrossRef] [Green Version]
  42. Jin, B.; Lorenz, D.A. Heuristic parameter-choice rules for convex variational regularization based on error estimates. SIAM J. Numer. Anal. 2010, 48, 1208–1229. [Google Scholar] [CrossRef] [Green Version]
  43. Jin, B.; Zou, J. Iterative parameter choice by discrepancy principle. IMA J. Numer. Anal. 2012, 32, 1714–1732. [Google Scholar] [CrossRef]
  44. Jin, B.; Lorenz, D.A.; Schiffler, S. Elastic-Net Regularization: Error estimates and Active Set Methods. Inverse Probl. 2009, 25, 115022. [Google Scholar] [CrossRef] [Green Version]
  45. Bredies, K.; Lorenz, D.A. Linear convergence of iterative soft-thresholding. J. Fourier Anal. Appl. 2008, 14, 813–837. [Google Scholar] [CrossRef] [Green Version]
  46. Foucart, S.; Rauhut, H. A Mathematical Introduction to Compressive Sensing; Springer: New York, NY, USA, 2013. [Google Scholar]
  47. Anzengruber, S.W.; Ramlau, R. Convergence rates for Morozov’s discrepancy principle using variational inequalities. Inverse Probl. 2011, 27, 105007–105024. [Google Scholar] [CrossRef] [Green Version]
  48. Flemming, J. Generalized Tikhonov Regularization and Modern Convergence Rate Theory in Banach Spaces; Shaker Verlag GmbH: Düren, Germany, 2012. [Google Scholar]
  49. Hofmann, B.; Mathé, P. Parameter choice in Banach space regularization under variational inequalities. Inverse Probl. 2012, 6, 1035–1058. [Google Scholar] [CrossRef] [Green Version]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Li, K.; Huang, C.; Yuan, Z. Error Estimations for Total Variation Type Regularization. Mathematics 2021, 9, 1373. https://doi.org/10.3390/math9121373

AMA Style

Li K, Huang C, Yuan Z. Error Estimations for Total Variation Type Regularization. Mathematics. 2021; 9(12):1373. https://doi.org/10.3390/math9121373

Chicago/Turabian Style

Li, Kuan, Chun Huang, and Ziyang Yuan. 2021. "Error Estimations for Total Variation Type Regularization" Mathematics 9, no. 12: 1373. https://doi.org/10.3390/math9121373

APA Style

Li, K., Huang, C., & Yuan, Z. (2021). Error Estimations for Total Variation Type Regularization. Mathematics, 9(12), 1373. https://doi.org/10.3390/math9121373

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop