A New Approach for Proximal Split Minimization Problems

Abstract

We provide an alternative formulation of proximal split minimization problems, a very recently developed and appealing strategy that relies on an infimal post-composition approach. Then, forward–backward and Douglas–Rachford splitting algorithms will guide both the design and analysis of some split numerical methods. We provide evidence of globally weak convergence and the fact that these algorithms can be equipped with relaxed and/or inertial steps, leading to improved convergence guarantees.

1. Introduction

Convex optimization has emerged as an effective and powerful tool for modeling a wide class of problems arising in various branches of social, physical, engineering, and pure and applied sciences in united and general frameworks. In recent years, much attention has been given to developing efficient and implementable numerical methods including the projection method and its variant forms, the auxiliary problem principle, the proximal point algorithm, and the descent framework, for solving variational inequalities and related optimization problems. It is well known that the projection method and its variant forms cannot be used to suggest and analyze iterative methods for solving variational inequalities due to the presence of the nonlinear term. This fact motivated us to develop another technique, which involves the use of proximal mappings associated with proper convex and lower semicontinuous functions, the origin of which can be traced back to Martinet [1]. The resulting method, namely the proximal point algorithm, has been extended and generalized in different directions by using novel and innovative techniques and ideas, both for their own sake and for their applications relying, for example, on Bregman distance. Since, in general, it is difficult to evaluate proximal mapping, one alternative is to decompose the given function into the sum of two (or more) functions whose proximal mappings are easier to evaluate than the proximal mapping of the original one to yield an easily implementable algorithm. Such methods are known as splitting methods. This can lead to the development of very efficient methods, since one can treat each part of the original function independently. The splitting methods and related techniques have been analyzed and studied by many authors; for an excellent account of splitting methods, see [2] and the references therein. It is worth mentioning that the acceleration of the convergence of these algorithms is an important issue for applications and is based on either an analysis of the second order in space [3] or on an approach, initiated by Nesterov [4], which is based on an analysis of the second order in time, and gives rise to the so-called FISTA method [5].

In this note, we will focus on the split feasibility problem, which has received much attention due to its applications in signal processing and image reconstruction [6], with particular progress in intensity-modulated therapy [7]. Our purpose here is to study the more general case of proximal split minimization problems and to investigate the convergence properties of the associated numerical solutions. More precisely, we are interested in finding a solution $\overline{x}\in H_1$ of the following problem:

$$\underset{x\in H_1}{min}f\left(x\right)+g_{\lambda }\left(Ax\right),$$

(1)

where $H_1$, $H_2$ are two real Hilbert spaces, $f:H_1\to IR\cup \{+\infty \}$ and $g:H_2\to IR\cup \{+\infty \}$ are two proper convex lower semicontinuous functions, $A:H_1\to H_2$ is a bounded linear operator, and $g_{\lambda }\left(y\right)={min}_{u\in H_2}\{g\left(u\right)+{\displaystyle \frac{1}{2\lambda }}{\parallel u-y\parallel }^2\}$ is the Moreau envelope of the function g of parameter $\lambda$.

The first class of approaches includes primal algorithms such as the forward–backward algorithm (FB) and Douglas–Rachford algorithm (DR), whose use depends essentially on the Lipschitz differentiability or proximability of the functions. In our context, $g_{\lambda }$ is differentiable with a Lipschitz-continuous gradient. More precisely, $\nabla g_{\lambda }={\displaystyle \frac{I-pro{x}_{\lambda g}}{\lambda }}$, and an application of the FB algorithm leads to

$$x_{k+1}=pro{x}_{{\mu }_{k}f}(x_k-{\mu }_k{A}^{*}(I-pro{x}_{\lambda g})\left(A{x}_k\right),$$

(2)

The determination of the step size depends on the operator norm, whose computation (or at least estimation) is not an easy task. An approach to avoid this problem was proposed in [8].

To solve Problem (1) in Hilbert spaces, in this paper, we consider an approach based on the notion of the infimal post-composition of a convex function by a linear operator. We provide first an equivalent formulation involving the infimal post-composition, and we propose weakly convergent algorithms by means of both the FB and DR algorithms. These algorithms require the computation of the proximity operator of the infimal post-composition considered in [9]; see also [10] or even [11]. Observe that in the context of image restoration, an explicit formula of this proximity operator was given in [9]; see also [12] for applications in image super-resolution with examples in non-blind deblurring and interpolation as well as polyphase implementation for image super-resolution based on Sherman–Morrison–Woodbury identity or even [10,13] in the context of the preconditioned alternating direction method of multipliers.

In order to provide the essential information, we assume the reader has some basic knowledge of convex analysis, as can be found, for example, in [14,15].

2. New Formulation of Proximal Split Minimization Problems

In what follows, we will propose an alternative formulation of Problem (1) that involves infimal post-composition. To this end, let us first recall this notion; see, for example [9,14].

Definition 1.

Given a function $f:H_1\to IR\cup \{+\infty \}$ and a bounded linear operator $A:H_1\to H_2$, the infimal post-composition of f by A is defined by

$$A▹f:H_2\to IR\cup \{+\infty \}:u\mapsto \underset{x\in H_1;A\left(x\right)=u}{inf}f\left(x\right).$$

(3)

The infimal post-composition $A▹f$ is exact at $u\in H_2$, if $arg{min}_{x\in H_1;A\left(x\right)=u}f\left(x\right)\ne \varnothing$. Note that $dom(A▹f)\subset ran\left(A\right)$.

Remember also that $pro{x}_{\gamma f,A}$, which is a generalization of the classical proximal mapping, was defined according to

$$pro{x}_{\gamma f,A}\left(u\right)=\underset{x\in H_1}{inf}(f\left(x\right)+{\displaystyle \frac{1}{2\gamma }}{\parallel A\left(x\right)-u\parallel }^2).$$

It is worth mentioning that the latter reduces to the classical proximity operator when $A=I$, namely $pro{x}_{\gamma f,Id}=pro{x}_{\gamma f}:={min}_{x\in H_1}(f\left(x\right)+{\displaystyle \frac{1}{2\gamma }}{\parallel x-·\parallel }^2)$, and that it has a closed expression in several instances; see, for example [16]. It was formulated by Moreau [17] and possesses several attractive properties; see, for example [15].

Following the approach developed in [9], we obtain the formulation below.

Problem 1.

Let $f:H_1\to IR\cup \{+\infty \}$, $g:H_2\to IR\cup \{+\infty \}$ be two proper convex lower semicontinuous functions and let $A:H_1\to H_2$ be a bounded linear operator. We are interested in the following problems:

$$\underset{x\in H_1}{inf}f\left(x\right)+g_{\lambda }\left(Ax\right),$$

(4)

and

$$\underset{u\in H_2}{inf}{g}_{\lambda }\left(u\right)+(A▹f)\left(u\right).$$

(5)

Throughout this paper, the solution sets of $\left(4\right)$ and $\left(5\right)$ will be denoted by $S_1$ and $S_2$. By means of [9]’s Proposition 1, we directly infer the connection below between problems $\left(4\right)$ and $\left(5\right)$.

Proposition 1.

The following statements hold true:

(i) 

${inf}_{x\in H_1}f\left(x\right)+g_{\lambda }\left(A\left(x\right)\right)={inf}_{u\in H_2}{g}_{\lambda }\left(u\right)+(A▹f)\left(u\right)$.

(ii) 

Suppose that $\overline{x}\in S_1$, then $A\left(\overline{x}\right)\in S_2$.

(iii) 

Suppose that $\overline{u}\in S_2$, then $arg{min}_{x\in H_1;A\left(x\right)=\overline{u}}f\left(x\right)\subset S_1$.

Solving the alternative formulation of problem $\left(4\right)$ by means of proximal splitting algorithms requires the computation of the proximal mapping of infimal post-composition $A▹f$. This does not of course preclude the possibility of using other algorithms; see, for instance [18,19]. For this purpose, we recall first the following key proposition; see [9]’s Proposition 2.

Proposition 2.

If we assume, in addition to assumptions on $f,g$ and A, that $0\in sri(dom{f}^{*}-ran{A}^{*})$, then we obtain the following properties:

(i) 

The infimal post-composition $A▹f$ is a proper, convex and lower semicontinuous function and it is exact in $H_2$.

(ii) 

$pro{x}_{\gamma A▹f}\left(u\right)=Apro{x}_{\gamma f,A}\left(u\right)\forall \gamma >0$, a key property with $dom\left(pro{x}_{\gamma f,A}\right)=H_2$.

Here, $f^{*}\left(z\right)={sup}_{x\in H_1}(\langle x,z\rangle -f\left(x\right))$ stands for the conjugate of the function f, and $sri,ran$ denote the strong relative interior and the range, respectively. This was obtained thanks to the formula ${(f^{*}\circ A^{*})}^{*}=A▹f$.

The new Formulation (5), being an optimization problem involving the sum of two proper lower semicontinuous convex functions, can therefore be solved by using split proximal algorithms, if the proximity mapping of the function g is available.

Firstly, we can use the forward–backward algorithm, since the function $g_{\lambda }$ is differentiable. This, together with the expression of the proximity mapping of $A▹f$, therefore leads to an infimal post-composition FB algorithm by activating the proximity mappings of $A▹f$ and g, which can be described as follows:

FB algorithm: Let $z_0\in H_1$, fix $\lambda >0$, let $0<\gamma <2\lambda$, and compute for $n=0,1,···$

$$(FB-Algo)\left\{\begin{array}{ccc}\hfill u_n& =& pro{x}_{\lambda g}\left(z_n\right),\hfill \\ \hfill x_n& =& pro{x}_{\gamma f,A}\left(\left(1-{\displaystyle \frac{\gamma }{\lambda }}\right)z_n+{\displaystyle \frac{\gamma }{\lambda }}{u}_n\right),\hfill \\ \hfill z_{n+1}& =& A{x}_n.\hfill \end{array}\right.$$

We present below a convergence result of the FB algorithm.

Theorem 1.

Suppose that $S_1\ne \varnothing$, the qualification condition $0\in sri(dom{f}^{*}-ran{A}^{*})$ holds true, and let ${\left(z_n\right)}_{n\in IN}$ be a sequence generated by the FB-Inf-post-composition algorithm. Then, ${\left(z_n\right)}_{n\in IN}$ weakly converge to some solution $\overline{u}\in S_2$ provided that the parameters satisfy $0<\gamma <2\lambda$. Moreover, every $\overline{x}\in arg{min}_{x\in H_1;Ax=\overline{u}}f\left(x\right)$ solves the initial Problem (4).

Proof of Theorem 1. 

$S_1$ being assumed to be non-empty ensures that $S_2\ne \varnothing$ thanks to (ii)’s Proposition 1. In our setting, we also have that $A▹f$ is proper convex and lower semicontinuous and that $pro{x}_{\gamma (A▹f)}=Apro{x}_{\gamma f,A}$. On the other hand, the FB algorithm can be written as

$$z_{n+1}=pro{x}_{\gamma (A▹f)}(z_n-\gamma \nabla g_{\lambda }\left(z_n\right)).$$

(6)

Indeed,

$$z_{n+1}=Apro{x}_{\gamma f,A}\left((1-{\displaystyle \frac{\gamma }{\lambda }})z_n+{\displaystyle \frac{\gamma }{\lambda }}pro{x}_{\lambda g}\left(z_n\right)\right),$$

and equivalently,

$$z_{n+1}=Apro{x}_{\gamma f,A}\left(z_n-{\displaystyle \frac{\gamma }{\lambda }}(z_n-pro{x}_{\lambda g}\left(z_n\right))\right),$$

from which we infer that

$$z_{n+1}=pro{x}_{\gamma (A▹f)}(z_n-\gamma \nabla g_{\lambda }\left(z_n\right)),$$

since $\nabla g_{\lambda }\left(z_n\right)={\displaystyle \frac{z_n-pro{x}_{\lambda g}\left(z_n\right)}{\lambda }}$.

A direct application of [14]’s Corollary 27.9 to the forward–backward algorithm (6) implies that the sequence ${\left(z_n\right)}_{n\in IN}$ convergences weakly to a solution $\overline{u}$ of $S_2$, and the fact that $\overline{x}$ solves $S_1$ follows from the exactness property of $A▹f$ given by (i)’s Proposition 2 together with (iii)’s Proposition 1. □

Remark 1.

It is worth mentioning that relaxed and accelerated versions can be considered, for example, by replacing the third equation in the FB algorithm with $z_{n+1}=(1-{\alpha }_n)z_n+{\alpha }_{n}A{x}_n$. The convergence result is still valid for ${\left({\alpha }_n\right)}_{n\in IN}$, a sequence in $[0,\delta ]$ verifying ${\sum }_{n\in IN}{\alpha }_n(\delta -{\alpha }_n)=+\infty$ with $\delta =min\{1,{\displaystyle \frac{\lambda }{\gamma }}\}+{\displaystyle \frac{1}{2}}$. It can be also equipped with inertial steps, leading to improved convergence guarantees; see, for example [5].

Secondly, we also solve Problem (5) by using the Douglas–Rachford algorithm together with the expression of the proximity operator of $A▹f$. These lead to the following infimal post-composition DR algorithm:

$D{R}_1$ algorithm: Let $z_0\in H_1$, fix $\lambda >0$, let $\gamma >0$, and compute for $n=0,1,···$

$$(D{R}_1-Algo)\left\{\begin{array}{ccc}\hfill u_n& =& {\displaystyle \frac{\lambda }{\lambda +\gamma }}{z}_n+{\displaystyle \frac{\gamma }{\lambda +\gamma }}pro{x}_{(\lambda +\gamma )g}\left(z_n\right),\hfill \\ \hfill x_n& =& pro{x}_{\gamma f,A}(2u_n-z_n),\hfill \\ \hfill z_{n+1}& =& A{x}_n+z_n-u_n.\hfill \end{array}\right.$$

We present below a convergence result of the $D{R}_1$ algorithm.

Theorem 2.

Suppose that $S_1\ne \varnothing$, the qualification condition $0\in sri(dom{f}^{*}-ran{A}^{*})$ holds true, and let ${\left(x_n\right)}_{n\in IN}$ be a sequence generated by the DR-Inf-post-composition algorithm. Then, the sequences ${\left(u_n\right)}_{n\in IN}$ and $A{\left(x_n\right)}_{n\in IN}$ weakly converge to some solution $\overline{u}\in S_2$. Moreover, every $\overline{x}\in arg{min}_{x\in H_1;Ax=\overline{u}}f\left(x\right)$ solves the initial Problem (4).

Proof of Theorem 2. 

$S_1$ being assumed to be non-empty ensures that $S_2\ne \varnothing$ thanks to (ii)’s Proposition 1. In our setting, we also have that the function $A▹f$ is proper convex and lower semicontinuous and that $pro{x}_{\gamma (A▹f)}=Apro{x}_{\gamma f,A}$. Now, the $D{R}_1$ algorithm can be reformulated as

$$z_{n+1}=pro{x}_{\gamma (A▹f)}(2pro{x}_{\gamma g_{\lambda }}{z}_n-z_n)+z_n-pro{x}_{\gamma g_{\lambda }}{z}_n.$$

(7)

Indeed, the $D{R}_1$ algorithm can be written in the form

$$z_{n+1}=Apro{x}_{\gamma f,A}\left({\displaystyle \frac{\lambda }{\lambda +\gamma }}{z}_n-{\displaystyle \frac{\gamma }{\lambda +\gamma }}(z_n-2pro{x}_{(\lambda +\gamma )g}\left(z_n\right))\right)+{\displaystyle \frac{\gamma }{\lambda +\gamma }}(z_n-pro{x}_{(\lambda +\gamma )g}\left(z_n\right)).$$

Using the following key formula (see Formula (42) in [20]),

$$pro{x}_{\gamma g_{\lambda }}\left(z\right)={\displaystyle \frac{\lambda }{\lambda +\gamma }}z+{\displaystyle \frac{\gamma }{\lambda +\gamma }}pro{x}_{(\lambda +\gamma )g}\left(z\right),$$

(8)

we obtain

$$z_{n+1}=Apro{x}_{\gamma f,A}(2pro{x}_{\gamma g_{\lambda }}{z}_n-z_n)+z_n-pro{x}_{\gamma g_{\lambda }}{z}_n.$$

Therefore,

$$z_{n+1}=pro{x}_{\gamma (A▹f)}(2pro{x}_{\gamma g_{\lambda }}{z}_n-z_n)+z_n-pro{x}_{\gamma g_{\lambda }}{z}_n.$$

By virtue of [14]’s Corollary 27.4 applied to Algorithm (7), we obtain that the sequences $(u_n=pro{x}_{\gamma g_{\lambda }}{z}_n)$ and $(A{x}_n=Apro{x}_{\gamma f,A}(2u_n-z_n))$ converge weakly to a solution $\overline{u}$ of $S_2$, and the fact that $\overline{x}$ solves $S_1$ follows again from the exactness of the function $A▹f$ given by (i)’s Proposition 2 together with (iii)’s Proposition 1. □

Remark 2.

(i) 

By swapping the activation of the proximity operators of $A▹f$ and g, we can also design the following algorithm:

$D{R}_2$ algorithm:Let $z_0\in H_1$, fix $\lambda >0$, let $\gamma >0$, and compute for $n=0,1,···$

$$\left\{\begin{array}{ccc}\hfill x_n& =& pro{x}_{\gamma f,A}{z}_n,\hfill \\ \hfill u_n& =& pro{x}_{\gamma g_{\lambda }}(2A{x}_n-z_n)\hfill \\ \hfill z_{n+1}& =& u_n+z_n-A{x}_n.\hfill \end{array}\right.$$

We can of course use Formula (8) again to obtain the proximity operator of $g_{\lambda }$ in terms of the function g. Moreover, the convergence result in Theorem 2 is also valid for a sequence generated by the $D{R}_2$ algorithm by once again applying [14]’s Corollary 27.4.

(ii) 

It is worth mentioning that Formula (8) follows easily by writing ${(\nabla g_{\lambda })}_{\gamma }={(\partial g)}_{\lambda +\gamma }$.

3. Numerical Experiments

In this section, we conduct some numerical experiments on two different minimization problems, so as to compare the proposed algorithms with other classical and recent methods.

3.1. Application to a Simple Split Feasibility Problem

We first consider the following theoretical problem that has been investigated by Abbas et al. [21]:

$$x^{*}\in argmin\left(g\right)suchthatA{x}^{*}\in argmin\left(f\right),$$

(9)

where $f:={\parallel .\parallel }_2$ is the Euclidean norm while $g:={\sum }_{i=1}^{n}max\left\{\right|x_i|-1,0\}$ and $A=Id$.

It is obvious that 0 belongs to $argmin\left(g\right)\cap A^{-1}argmin\left(f\right)$ which is thus non-empty. Consequently, Problem 9 has the same set of solutions as the structured minimization problem

$$\underset{x\in {\mathbb{R}}^N}{min}F\left(x\right):=f\left(x\right)+g_{\lambda }\left(x\right),for\lambda >0.$$

(10)

We can therefore compare the convergence of the algorithm investigated in [21] with the FB algorithm and DR1 algorithm in this particular instance of Problem (4). As $A=Id$, the generalized proximal mapping $pro{x}_{\gamma f,A}$ reduces to the classical proximal mapping $pro{x}_{\gamma }f$, which has a closed form. Figure 1 and Figure 2 show a comparison between the three methods through the convergence to 0 of $F\left(x_n\right)$ and $\parallel x_{n+1}-x_n{\parallel }_2$. It appears that the algorithms converge, which is in accordance with the theoretical results. Moreover, as can be expected, a better convergence rate is reached for the DR1 algorithm.

3.2. Application to Image Restoration

We now consider the classical inverse problem of image denoising and deblurring. For simplicity, we consider the case of square grayscale $N\times N$ images, which can be represented as elements of ${\mathbb{R}}^{N^2}$. Given an altered image $b\in {\mathbb{R}}^{N^2}$, we aim to compute a restored image x by solving the following total variation denoising and deconvolution model (see, e.g., [9]):

$$\underset{x\in {\mathbb{R}}^{N^2}}{min}F\left(x\right):={\displaystyle \frac{1}{2}}{\parallel Lx-b\parallel }_2^2+{\parallel x\parallel }_2^2+\lambda {\parallel \nabla x\parallel }_1,$$

(11)

where $L:{\mathbb{R}}^{N^2}\to {\mathbb{R}}^{N^2}$ is a blurring operator, $\nabla \in \mathcal{L}({\mathbb{R}}^{N^2},{\mathbb{R}}^{2N^2})$ (set of linear functions from ${\mathbb{R}}^{N^2}$ to ${\mathbb{R}}^{2N^2}$) is the discrete gradient operator on ${\mathbb{R}}^{N^2}$, and $\lambda$ is a trade-off parameter between data fidelity and smoothness. ${\parallel \nabla (.)\parallel }_1$ is the classical total variation operator introduced by Ruding–Osher–Fatemi in their seminal paper [22]. Minimizing the total variation is known to reduce noise while preserving edges. This problem can be cast into Problem (4) by setting $f:={\displaystyle \frac{1}{2}}{\parallel Lx-b\parallel }_2^2+{\parallel x\parallel }_2^2$, $g:={\lambda \parallel .\parallel }_1$ and $Ax:=\nabla x$ for $x\in {\mathbb{R}}^{N^2}$. The explicit expression of the proximal mapping of the infimal post-composition (that is, $pro{x}_{\gamma A▹f}\left(u\right)=Apro{x}_{\gamma f,A}\left(u\right)$) required by both the FB algorithm and the DR1 algorithm) is given in [9]’s Proposition 5. For simplicity, we consider here the case of a pure denoising process (that is, $L=Id$) applied to an image altered by white Gaussian noise. The result is illustrated in Figure 3.

Figure 4 and Figure 5 illustrate the behavior of the sequences generated by the two proposed algorithms and other classical or recent methods (ADMM and the Chambolle–Pock Algorithm introduced in [18]). As in example 1, the sequences generated by all the considered algorithms converge, as predicted by the theoretical results. Here again, the DR1 algorithm outperforms the FB algorithm in terms of convergence rate.

4. Conclusions

In this paper, we propose a formulation for proximal split minimization problems relying on an infimal post-composition approach. This alternative formulation together with a recent implementation of the proximity operator of the infimal post-composition introduced in [9] allows the development of additional algorithms based on the forward–backward algorithm as well as others based on the Douglas–Rachford splitting method. The corresponding global weak convergence results are derived from classical convergence results. A relaxed version has been suggested, and we could do better with versions based on an inertial technique which has a remarkable acceleration effect. We expect that this note will become a further step towards developing other algorithms and applications. Indeed, in image restoration applications, PnP (Plug-and-Play) approaches can make use of a pre-trained CNN (Convolutional Neural Network) denoiser to evaluate $pro{x}_{\gamma f,A}$, which is a Gaussian-like denoising subproblem, by splitting it into a data recovery term and a feature expression term. FTT (Fast Fourier Transform) solves the data recovery term due to an analytical solution (see [12,23] and references therein) and the feature expression term, in this case, is nothing else than the proximity operator of f, which is very often available in a closed form. It is worth mentioning that PnP methods have recently made significant progress, particularly with the incorporation of learning-based denoisers. Moreover, Convolutional Neural Networks (CNNs) have shown good performances through end-to-end training for image denoising and non-blind deblurring. Finally, we would also like to stress that many algorithms and techniques in machine learning and data processing involve random projections, in general onto a lower-dimensional subspace. This naturally calls for the study of random isometric matrices $A^{p\times n},n\le p$ such that $A^{t}A=I_n$ (for instance, real Haar random matrices [24], Parseval frame [25] and Learned Stiefel matrices [26]). In this case, and for convex feasibility problems (i.e., $f={\delta }_C$ and $g={\delta }_Q$, the indicator functions of two non-empty closed convex sets C and D), we can easily check that the FB algorithm reduced to

Let $z_0\in C$, fix $\lambda >0$, let $0<\gamma <2\lambda$ and compute for $n=0,1,···$

$$z_{n+1}=APro{j}_Q\left(A^t\left(\left(1-{\displaystyle \frac{\gamma }{\lambda }}\right)z_n+{\displaystyle \frac{\gamma }{\lambda }}Pro{j}_C\left(z_n\right)\right)\right).$$

(12)

On the other hand, the $D{R}_1$ algorithm takes the following form:

Let $z_0\in C$, fix $\lambda >0$, let $\gamma >0$, and compute for $n=0,1,···$

$$\left\{\begin{array}{ccc}\hfill u_n& =& {\displaystyle \frac{\lambda }{\lambda +\gamma }}{z}_n+{\displaystyle \frac{\gamma }{\lambda +\gamma }}Pro{j}_Q\left(z_n\right),\hfill \\ \hfill z_{n+1}& =& APro{j}_C\left(A^t(2u_n-z_n)\right)+z_n-u_n.\hfill \end{array}\right.$$

(13)