Constrained Plug-and-Play Priors for Image Restoration

Abstract

The Plug-and-Play framework has demonstrated that a denoiser can implicitly serve as the image prior for model-based methods for solving various inverse problems such as image restoration tasks. This characteristic enables the integration of the flexibility of model-based methods with the effectiveness of learning-based denoisers. However, the regularization strength induced by denoisers in the traditional Plug-and-Play framework lacks a physical interpretation, necessitating demanding parameter tuning. This paper addresses this issue by introducing the Constrained Plug-and-Play (CPnP) method, which reformulates the traditional PnP as a constrained optimization problem. In this formulation, the regularization parameter directly corresponds to the amount of noise in the measurements. The solution to the constrained problem is obtained through the design of an efficient method based on the Alternating Direction Method of Multipliers (ADMM). Our experiments demonstrate that CPnP outperforms competing methods in terms of stability and robustness while also achieving competitive performance for image quality.

1. Introduction

The challenge of reconstructing a high-quality image $\mathbf{x}\in {\mathbb{R}}^n$ from its degraded measurement $\mathbf{b}\in {\mathbb{R}}^n$ is commonly formulated as a linear inverse problem. Such a problem has to be addressed in several imaging frameworks, such as in medicine [1,2,3,4], microscopy [5,6,7], and astronomy [8,9,10]. Although these are different and maybe distant topics, they share a common linear model [11] for the image acquisition process: namely,

$$\mathbf{b}=\mathbf{A}\mathbf{x}+\mathit{\eta },$$

(1)

where $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ is a known blur operator called the Point Spread Function (PSF) [12], and $\mathit{\eta }\in {\mathbb{R}}^n$ represents additive random noise with a standard deviation of ${\sigma }_{\mathit{\eta }}$. Linear inverse problems, due to the physics underlying the data acquisition process, often suffer from ill-posedness [12], necessitating the formulation of the solution ${\mathbf{x}}^{★}\in {\mathbb{R}}^n$ as a minimizer of a regularized objective function.

The standard model-based approach involves minimizing a regularized objective function of the form:

$${\mathbf{x}}^{*}\in \underset{\mathbf{x}\in {\mathbb{R}}^n}{\mathrm{argmin}}𝓁(\mathbf{x};\mathbf{A},\mathbf{b})+\mu \rho \left(\mathbf{x}\right),$$

(2)

where $𝓁(\mathbf{x};\mathbf{A},\mathbf{b})$ encodes data fidelity information, and $\rho \left(\mathbf{x}\right)$ is the regularization term. The choice of the function ℓ depends on the statistical noise perturbing the data. In the presence of additive Gaussian noise (AWGN), the natural choice is the least square function, whilst signal-dependent noise requires tailored functionals: for example, Poisson noise induces the employment of the Kullback–Leibler functional [13,14,15]. Defining the regularization parameter $\mu >0$ is crucial, and it is often selected by hand. The imaging community has developed several techniques for the automatic choosing of such a parameter. For example, well established methods such as the discrepancy principle, L-curve, or cross-validation [16] have been considered for Gaussian noise, while the discrepancy principle has been adapted also in the presence of Poisson noise [17,18]. Alternatively, another interesting approach consists of recasting the optimization problem (2) as a constrained one [19,20,21]: namely,

$${\mathbf{x}}^{*}\in \underset{\mathbf{x}\in {\mathbb{R}}^n}{\mathrm{argmin}}\rho \left(\mathbf{x}\right)\mathrm{s}.\mathrm{t}.𝓁(\mathbf{x};\mathbf{A},\mathbf{b})\le c.$$

(3)

The positive scalar c represents the strength of the constraints, and different from the parameter $\mu$ in (2), it has a physical meaning. For example, the value of c in (3) usually depends on the amount of noise.

Another grand challenge for the model-based approach is the design of an effective regularization functional $\rho \left(\mathbf{x}\right)$ capable of capturing intricate image features. Some examples include the well-known Tikhonov regularization [22], known as ridge regression in statistical contexts, and its variant that promotes diffuse components on the final reconstruction; the total variation functional, which aims to preserve sharp edges [2,23,24]; the ${𝓁}_p$-norms regularizers, with $0\le p\le 1$, which induce sparsity on the image and/or gradient domains [19,25,26]; and the elastic-net functional [4], which is a convex combination of ${𝓁}_1$ and ${𝓁}_2$ norms.

Nowadays, it has been recognized that among the possible strategies to solve imaging inverse problems, learning-based techniques represent the most efficient alternative. The advancements in deep learning for imaging problems were driven by the application of neural networks to learn an inverse mapping from measurements to the image space to obtain an approximate solution. In particular, a dataset is composed by considering several acquisitions and their corresponding ground truths, and a neural network is trained in order to minimize the empirical risk. This end-to-end approach is very interesting since the forward degradation model (1) is not needed. This property is extremely useful when the problem of interest is physically unknown or hard to express with an analytical expression. Another appealing characteristic of these inverse learning methods is their computational efficiency during the inference, as they outperform standard variational techniques. However, the major limitation of learning approaches regards the stability of the models [27]. The presence of noise in the measurement can produce various artifacts in the reconstruction obtained through a neural network [28]. Moreover, when a measurement falls outside the training set distribution, the model may produce hallucinated reconstructions containing misleading artifacts [29]. Such undesired behavior is particularly problematic in some applications. In addition, different from variational approaches, learned models have to be retrained whenever the acquisition model changes.

A popular technique to construct a model that is adaptable to various imaging tasks with loose dependence on the training data consists of decoupling the degradation model from the learning-based prior. The fast development of learning-based techniques in the field of imaging inverse problems has allowed the defining of data-driven regularizers that have largely outperformed handcrafted ones [30]. This approach is often referred to as Plug-and-Play (PnP) and represents a versatile and innovative paradigm to impose a statistically learned prior within a variational framework. The PnP prior framework [31,32,33,34,35] has emerged as a potent approach that leverages advanced denoisers as regularizers without explicitly defining $\rho \left(\mathbf{x}\right)$. However, the lack of an explicit objective function complicates theoretical analyses [35]. Regularization by denoising (RED) [36] addresses this by formulating an explicit regularization functional, but practical challenges persist, especially in ensuring denoisers align with the manifold of natural images.

The complexity of selecting an appropriate regularizer prompts exploration beyond traditional handcrafted terms. While model-based approaches often rely on handcrafted terms, this paper advocates for the PnP framework and demonstrates that closed-form regularizers are not always optimal for inducing prior information. The PnP approach, rooted in proximal algorithms, allows substituting the regularization term with off-the-shelf denoisers, diversifying prior information sources. The paper concludes with an overview of existing PnP studies that highlights the modular structure’s flexibility and the versatility of employing various proximal algorithms and denoisers.

The presented work is organized as follows. Section 2 is devoted to provide a brief introduction to the PnP method (Section 2.1), and then it presents the novel constrained approach (Section 2.2). Section 3 addresses the performance of the the proposed model. In particular, we define the implementation settings (Section 3.1); we discuss selection of the optimal denoiser, examining how different denoising priors influence the overall performance of the proposed model (Section 3.2); we show the robustness of the proposed method with respect to its parameters (Section 3.3 and Section 3.4); we provide comparisons from a quantitative and qualitative perspective with similar state-of-the-art algorithms (Section 3.5). Finally, Section 4 draws the conclusion and future perspectives.

Notations

Bold small letters refer to vectors, while bold capital letters refer to matrices. The operator ${\mathrm{prox}}_f\left(\mathbf{x}\right)$ stands for the proximity operator of f: ${\mathrm{prox}}_f\left(\mathbf{x}\right)=\underset{\mathbf{y}}{\mathrm{argmin}}f\left(\mathbf{y}\right)+\gamma /2{\parallel \mathbf{x}-\mathbf{y}\parallel }_2^2$. The projection on a set $\mathcal{A}$ is denoted with ${\mathrm{proj}}_A$. Square or rectangular images are vectorized: an image $\mathbf{x}\in {\mathbb{R}}^{p\times q}$ is seen as a vector belonging to ${\mathbb{R}}^n$, where $n=pq$ and the elements of $\mathbf{x}$ are stacked column-wise. The term ${\iota }_{\mathcal{A}}$ denotes the indicator function of the set $\mathcal{A}$. ${\mathbb{R}}^{+}$ denotes the set of positive real numbers.

2. Constrained PnP Model

This section briefly introduces the Plug-and-Play approach: showing its main idea and convergence properties. The second part is devoted to presenting the novel constrained strategy.

2.1. Plug-and-Play Models: A Brief Overview

The building block of Plug-and-Play methods is the established Alternating Direction of Multipliers Method (ADMM) used to solve Problem (2): this method introduces a novel variable that induces a further constraint; in this way, one is led to solve the following problem.

$$\underset{\mathbf{x}=\mathbf{v}}{\mathrm{argmin}}𝓁(\mathbf{x};\mathbf{A},\mathbf{b})+\mu \rho \left(\mathbf{v}\right).$$

(4)

The augmented Lagrangian function for (4) reads as

$$\mathcal{L}(\mathbf{x},\mathbf{v},\mu )=𝓁(\mathbf{x};\mathbf{A},\mathbf{b})+\mu \rho \left(\mathbf{v}\right)+⟨\mathit{\lambda },\mathbf{x}-\mathbf{v}⟩+\frac{\beta }{2}{\parallel \mathbf{x}-\mathbf{v}\parallel }_2^2,$$

where $\beta \in {\mathbb{R}}^{+}$ is a penalty parameter and $\mathit{\lambda }\in {\mathbb{R}}^n$ is the Lagrangian parameter relative to the constraint $\mathbf{x}=\mathbf{v}$. After minimal algebraic manipulations, the new unconstrained problem to be solved is

$$\underset{\mathbf{x},\mathbf{v}\in {\mathbb{R}}^n}{min}\underset{\mathit{\lambda }\in {\mathbb{R}}^n}{max}𝓁(\mathbf{x};\mathbf{A},\mathbf{b})+\mu \rho \left(\mathbf{v}\right)+\frac{\beta }{2}{∥\mathbf{x}-\mathbf{v}+\frac{\mathit{\lambda }}{\beta }∥}_2^2-\frac{1}{2}{∥\frac{\mathit{\lambda }}{\beta }∥}^2.$$

(5)

Such a problem is addressed by the Alternate Direction Method of Multipliers, which is shown in Algorithm 1. The astute reader will recognize that Steps 3 and 4 in Algorithm 1 are the proximity operators [37] of ℓ and $\mu \rho$ computed at ${\mathbf{v}}^k-{\mathit{\lambda }}^k/\beta$ and at ${\mathbf{x}}^{k+1}+{\mathit{\lambda }}^k/\beta$, respectively. For classical choices for ℓ, such as the least square or the Kullback–Leibler functionals, the proximity operators have explicit expressions (see [38] for a comprehensive list of proximity operators for several families of functions). The straightforward expression for the proximity operator of $\mu \rho$ is available for particular regularizations: such as, for example, ${𝓁}_1$, whose prox is the soft thresholding operator, and Tikhonov regularization, whose prox is simply a rescaling of the input [38].

Algorithm 1: Alternating Direction Method of Multipliers (ADMM).

1: Select parameters $\beta ,\mu \in {\mathbb{R}}^{+}$, set ${\mathbf{x}}^0,{\mathbf{v}}^0,{\mathit{\lambda }}^0$. 2: for  $k=0,1,...$do 3:     ${\mathbf{x}}^{k+1}=\underset{\mathbf{x}\in \mathbb{R}}{\mathrm{argmin}}𝓁({\mathbf{x}}^k;\mathbf{b})+\frac{\beta }{2}∥\mathbf{x}-{\mathbf{v}}^k+\frac{{\mathit{\lambda }}^k}{\beta }∥$ 4:     ${\mathbf{v}}^{k+1}=\underset{\mathbf{v}\in \mathbb{R}}{\mathrm{argmin}}\mu \rho \left(\mathbf{v}\right)+\frac{\beta }{2}∥{\mathbf{x}}^{k+1}-\mathbf{v}+\frac{{\mathit{\lambda }}^k}{\beta }∥$ 5:     ${\mathit{\lambda }}^{k+1}={\mathit{\lambda }}^k+\beta ({\mathbf{x}}^{k+1}-{\mathbf{v}}^{k+1})$ 6: end for

Under the framework presented in [31], Step 3 of Algorithm 1 is considered as a reconstruction step and provides the maximum a priori estimate given the data $\mathbf{b}$ and the operator $\mathbf{A}$. Step 4 is the denoising operator: this depends obviously on the design choice in (2), namely, in the choice for $\rho$. The strategy depicted in [31] instead suggests bypassing this designing and directly employing a denoising operator D in Step 3.

This is tantamount to still considering an objective function as in (2), but regularization functional $\rho$ is unknown. Algorithm 2 assumes that the operator D is the proximity operator of the unknown function $\mu \rho$ computed at $\mathbf{x}+\mathit{\lambda }/\beta$:

$$\mathbf{v}=D(\mathbf{x}+\mathit{\lambda }/\beta )={\mathrm{prox}}_{\mu \rho }(\mathbf{x}+\mathit{\lambda }/\beta ).$$

After its first presentation to the scientific community, several researcher showed that under a suitable hypothesis, Plug-and-Play methods converge to a solution of the original problem. In [39], the authors showed fixed-point convergence under the usage of denoisers belonging to the bounded denoiser class. The authors in [40] adopt an incremental version of PnP algorithm and prove its convergence under some explicit hypothesis on ℓ and on the chosen denoiser. Ref. [41] shows that under the hypotheses of D being averaged [41] (Definition 2.1) and ℓ being convex, Algorithm 2 converges and, moreover, that it can be proved that some denoisers are actually the prox of some particular functions (e.g., the non-local-mean filter is the prox of a quadratic convex function).

Algorithm 2: Plug-and-Play method (PnP).

1: Select parameters $\beta \in {\mathbb{R}}^{+}$, set ${\mathbf{x}}^0,{\mathbf{v}}^0,{\mathit{\lambda }}^0$, select a denoiser $D:{\mathbb{R}}^n\to {\mathbb{R}}^n$. 2: for  $k=0,1,...$do 3:     ${\mathbf{x}}^{k+1}=\underset{\mathbf{x}\in \mathbb{R}}{\mathrm{argmin}}𝓁({\mathbf{x}}^k;\mathbf{b})+\frac{\beta }{2}∥\mathbf{x}-{\mathbf{v}}^k+\frac{{\mathit{\lambda }}^k}{\beta }∥$ 4:     ${\mathbf{v}}^{k+1}=D\left({\mathbf{x}}^{k+1}\right)$ 5:     ${\mathit{\lambda }}^{k+1}={\mathit{\lambda }}^k+\beta ({\mathbf{x}}^{k+1}-{\mathbf{v}}^{k+1})$ 6: end for

2.2. The Proposed Constrained Model

Under the hypothesis of additive noise in (1) and following the constrained approach [42] shown in (3), Problem (2) is reformulated as

$$\underset{\mathbf{x}\in {\mathbb{R}}^n}{\mathrm{argmin}}\rho \left(\mathbf{x}\right)s.t.𝓁(\mathbf{x};\mathbf{A},\mathbf{b})\le \delta ,$$

where $\delta =\sqrt{n}\tau {\sigma }_{\mathit{\eta }}$, with $\tau \in [0,1]$ and ${\sigma }_{\mathit{\eta }}$ being the known noise level. Hereafter, we assume that an AWGN framework, i.e., additive Gaussian noise is perturbing the image, and hence, the choice of a fit-to-data functional consists of the least square:

$$𝓁(\mathbf{x};\mathbf{A},\mathbf{b})=\frac{1}{2}{∥\mathbf{A}\mathbf{x}-\mathbf{b}∥}_2^2.$$

The constrained formulation, hence, has the following form:

$$\underset{\mathbf{x}\in {\mathbb{R}}^n}{\mathrm{argmin}}\rho \left(\mathbf{x}\right)\mathrm{subject}\mathrm{to}\frac{1}{2}{\parallel \mathbf{A}\mathbf{x}-\mathbf{b}\parallel }_2^2\le \delta ,$$

(6)

We make the following assumption:

Assumption 1.

The function ρ is continuous.

The constraint set is compact; thus, under Assumption 1, Problem (6) has at least one solution by Weierstrass theorem. Problem (6) can be equivalently recast into the following form:

$$\begin{array}{cc}\hfill & \underset{\mathbf{x},\mathbf{v},\mathbf{r}\in {\mathbb{R}}^n}{\mathrm{argmin}}\rho \left(\mathbf{x}\right)+{\iota }_{B_{\delta }}\left(\mathbf{r}\right),\mathrm{s}.\mathrm{t}\mathbf{r}=\mathbf{A}\mathbf{x}-\mathbf{b},\mathbf{x}=\mathbf{v},\hfill \end{array}$$

(7)

where $B_{\delta }:=\{\mathbf{r}\in {\mathbb{R}}^n|{\parallel \mathbf{r}\parallel }_2^2\le \delta \}$ is a closed disk with a zero center and radius $\delta$. Adopting the approach of Section 2.1, the augmented Lagrangian function relative to Equation (7) reads as

$$\begin{array}{cc}\hfill \mathcal{L}(& \mathbf{x},\mathbf{r},\mathbf{v};{\mathit{\lambda }}_{\mathbf{v}},{\mathit{\lambda }}_{\mathbf{r}})=\rho \left(\mathbf{v}\right)+i_{B_{\mathbf{\delta }}}\left(\mathbf{r}\right)\hfill \\ \hfill & +\frac{{\beta }_{\mathbf{r}}}{2}{∥\mathbf{A}\mathbf{x}-\mathbf{b}-\mathbf{r}+\frac{{\mathit{\lambda }}_{\mathbf{r}}}{{\beta }_{\mathbf{r}}}∥}_2^2\hfill \\ \hfill & +\frac{{\beta }_{\mathbf{v}}}{2}{∥\mathbf{x}-\mathbf{v}+\frac{{\mathit{\lambda }}_{\mathbf{v}}}{{\beta }_{\mathbf{v}}}∥}_2^2-\frac{1}{2}{∥\frac{{\mathit{\lambda }}_{\mathbf{r}}}{{\beta }_{\mathbf{r}}}∥}_2^2-\frac{1}{2}{∥\frac{{\mathit{\lambda }}_{\mathbf{v}}}{{\beta }_{\mathbf{v}}}∥}_2^2,\hfill \end{array}$$

(8)

where ${\mathit{\lambda }}_{\mathbf{r}}$ and ${\mathit{\lambda }}_{\mathbf{v}}$ are the Lagrange multipliers, and ${\beta }_{\mathbf{r}}$ and ${\beta }_{\mathbf{v}}$ are the proximity penalties. The new task to be addressed is, hence,

$$\underset{\mathbf{x},\mathbf{v},\mathbf{r}}{min}\underset{{\lambda }_{\mathbf{r}},{\lambda }_{\mathbf{v}}}{max}\mathcal{L}(\mathbf{x},\mathbf{r},\mathbf{v};{\mathit{\lambda }}_{\mathbf{v}},{\mathit{\lambda }}_{\mathbf{r}})$$

and then the PnP-ADMM approach can be adopted to solve the above optimization problem. As previously shown, this amounts to substituting the proximity operator of $\rho$ computed at $\mathbf{x}+{\mathit{\lambda }}_{\mathbf{v}}/{\beta }_{\mathbf{v}}$ with a denoiser computed at the same point. Algorithm 3 shows the implementation of this strategy.

The noise level $\delta$ may be a priori known; on the other hand, when information only about the type of noise is available, one can find in the literature several methods to estimate the noise level (see for example [43]). The computation of ${\mathbf{x}}^{k+1}$ seems to pose some computational issues since it requires the inversion of a matrix, which could be cumbersome in terms of computational cost. Nonetheless, under a suitable hypothesis for the operator $\mathbf{A}$, which is practically satisfied in real-life applications, such an update can be easily pursued in Fourier spaces via FFTs.

Remark 1

(Convergence of the ADMM approach). We discuss some observations about the convergence of the presented scheme in Algorithm 3.

• A well-established result [44] states that ADMM converges even when more than two variables are considered in the formulation.
• The substitution of the proximity operator in Step 4 may hinder the convergence behavior of the whole algorithm. Coupling the result from [44] with [41], for example, assures the convergence for a suitable denoiser D.

Algorithm 3: Constrained Plug-and-Play approach (CPnP).

1: Set $\delta$, ${\mathbf{x}}^0,{\mathbf{v}}^0=0$, and ${\mathbf{r}}^0=\mathbf{b}-\mathbf{A}{\mathbf{x}}^0$, select ${\beta }_{\mathbf{u}},{\beta }_{\mathbf{r}}>0$, and initialize ${\mathit{\lambda }}_{\mathbf{r}}^0,{\mathit{\lambda }}_{\mathbf{v}}^0$. Choose a denoiser D. 2: for  $k=0,1,...$do 3:     ${\mathbf{x}}^{k+1/2}=\frac{{\beta }_{\mathbf{r}}}{{\beta }_{\mathbf{v}}}{\mathbf{A}}^{\top }(\mathbf{b}+{\mathbf{r}}^k-\frac{{\mathit{\lambda }}_{\mathbf{r}}^k}{{\beta }_{\mathbf{r}}})+({\mathbf{v}}^k-\frac{{\mathit{\lambda }}_{\mathbf{v}}^k}{{\beta }_{\mathbf{v}}})$ 4:     ${\mathbf{x}}^{k+1}\leftarrow {(\frac{{\beta }_{\mathbf{r}}}{{\beta }_{\mathbf{v}}}{\mathbf{A}}^{\top }\mathbf{A}+\mathbf{I})}^{-1}{\mathbf{x}}^{k+1/2}$ 5:     ${\mathbf{v}}^{k+1}=D({\mathbf{x}}^{k+1}+\frac{{\mathit{\lambda }}_{\mathbf{v}}^k}{{\beta }_{\mathbf{v}}})$ 6:     ${\mathbf{r}}^{k+1}={\mathrm{proj}}_{\mathcal{B}\delta }(\mathbf{A}{\mathbf{x}}^{k+1}-\mathbf{b}+\frac{{\mathit{\lambda }}_{\mathbf{r}}^k}{{\beta }_{\mathbf{r}}})$ 7:     ${\mathit{\lambda }}_{\mathbf{r}}^{k+1}\leftarrow {\mathit{\lambda }}_{\mathbf{r}}^k+{\beta }_{\mathbf{r}}({\mathbf{r}}^{k+1}-\mathbf{A}{\mathbf{x}}^{k+1}+\mathbf{b})$ 8:     ${\mathit{\lambda }}_{\mathbf{v}}^{k+1}\leftarrow {\mathit{\lambda }}_{\mathbf{v}}^k+{\beta }_{\mathbf{v}}({\mathbf{x}}^{k+1}-{\mathbf{v}}^{k+1})$ 9: end for

Remark 2.

We point out that the constrained approach has a remarkable positive outcome: it avoids selection of the regularization parameter in Problem (2).

3. Results

In this section, we delve into the outcomes of our study. The first subsection begins with an overview of the methodological setting. This encompasses the experimental setup, the metrics employed for the evaluation of the results, and a comprehensive presentation of the comparative methods. Subsequently, we focus on the analysis of the proposed method with respect to the choice of the ADMM penalty sequences. The final section offers a detailed examination, both qualitatively and quantitatively, of our approach in comparison to its competitors.

3.1. Settings, Evaluation Metrics, and Competing Baseline Methods

As a case study, we focus on the task of image deblurring with AWGN assumptions. Accordingly, in Equation (1), $\mathbf{A}$ represents a Gaussian blurring operator with a standard deviation of ${\sigma }_{\mathbf{A}}$, and $\mathit{\eta }$ denotes zero-mean Gaussian noise with a standard deviation of ${\sigma }_{\mathit{\eta }}$. We generate blurry and noisy data by applying the image formation model (1) to the images from Set5 [45] and Set24 [46], which are referred to as the ground truths (GTs).

Our method is compared with two baselines: (1) the original unconstrained Plug-and-Play model [47] solved via the half-quadratic splitting (HQS) algorithm and (2) the unconstrained RED formulation [36] solved via ADMM. The former is referred to as PnP and the latter is referred to as RED in the following.

We evaluate the quality of restored images using the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics: both assess the quality of the reconstruction. PSNR offers a numerical perspective by measuring the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. It quantifies the degree of distortion present in the restored image compared to the original and considers the full range of pixel values. Higher PSNR values typically indicate better-quality reconstruction as they suggest lower levels of distortion. SSIM evaluates the structural similarity between the restored image and the original from a perceptual standpoint. It considers factors such as luminance, contrast, and structure—mimicking human visual perception. SSIM scores closer to 1 indicate greater similarity between the restored and original images and reflect higher perceived quality. Additionally, from a theoretical standpoint, considering a ground truth image $\mathbf{x}$ and its blurred and noisy simulated data $\mathbf{b}$, we use ${\sigma }_{\mathbf{x}}:=\frac{{\parallel \mathbf{A}\mathbf{x}-\mathbf{b}\parallel }_2}{\sqrt{n-1}}$ as an unbiased estimator of ${\sigma }_{\mathit{\eta }}$. Thus, we compare the real noise standard deviation ${\sigma }_{\mathit{\eta }}$ with ${\sigma }_{{\mathbf{x}}^{*}}$, where ${\mathbf{x}}^{*}$ refers to the output of the algorithms.

We point out that, as one could expect, the quality of the restored images computed using PnP and RED strongly depends on the choice of the regularization parameters (since they consider unconstrained models). In the experiments, the regularization parameters are selected in order to maximize the PSNR metric. We stress that the proposed CPnP automatically selects the strength of the regularization, thus avoiding highly demanding parameter tuning.

Finally, as the stopping criterion, we choose the relative difference of the iterates within a tolerance of ${10}^{-4}$. The maximum number of iterations is set to 100 for all the methods.

All the experiments are conducted on a PC with an Intel(R) Core(TM) i7-8565U CPU @ 1.80 GHz 1.99 GHz (Intel Corporation, Santa Clara, CA, USA), running Windows 10 Pro and MATLAB 2023b. The codes of the proposed CPnP are available at https://github.com/AleBenfe/CPNP (accessed on 25 January 2024).

3.2. On the Choice of the Denoiser

In this section, we evaluate the CPnP method’s efficacy: specifically, by analyzing its performance when considering various state-of-the-art denoising techniques as denoising engines. Our investigation encompasses the Block-Matching and 3D Filtering (BM3D) [48], Non-Local Means (NLM) [49], and Deep Convolutional Neural Network (DnCNN) [47] methods.

BM3D [48] employs a multi-step process wherein it partitions the image into blocks, identifies similar blocks, collaboratively filters them to estimate clean signals, applies 3D transform filtering to reduce noise, and aggregates these filtered blocks to produce the final denoised image. In contrast, NLM [49] compares local image patches, averages similar patches to estimate clean pixels, and then outputs a denoised image. Lastly, DnCNN [47] utilizes a deep neural network to directly map noisy images to denoised counterparts, utilizing residual learning to boost its performance.

In this section, the focus is on images belonging to the Set5 dataset. We simulate blurred and noisy data by employing the linear image formation model defined in Equation (1) with parameters set to (${\sigma }_{\mathbf{A}}=0.8$, ${\sigma }_{\mathit{\eta }}=15$). Through this setup, we aim to assess how each of the aforementioned denoising methods impacts the quality of the restored images with respect to PSNR and SSIM metrics and visual quality.

In

Table 1. Mean values of PSNR and SSIM for the images in Set5 by varying the embedded denoiser: namely, NLM, BM3D, and DncNN. The best results are highlighted in bold. DnCNN outperforms both BM3D and NLM in terms of the considered metrics.

	Set5 (${\mathit{\sigma }}_{\mathit{A}}=0.8,{\mathit{\sigma }}_{\mathit{\eta }}=15$)
Metric	NLM	BM3D	DnCNN
PSNR	30.27	31.14	31.52
SSIM	0.90	0.91	0.92

, we report the mean values of PSNR and SSIM for images in the Set5 dataset for NLM, BM3D, and DnCNN. DnCNN consistently demonstrates superior performance compared to both BM3D and NLM across both PSNR and SSIM metrics. Specifically, this suggests that DnCNN induces better regularization compared to BM3D and NLM.

In Figure 1, we report the restored images using DnCNN, BM3D, and NLM alongside their respective ground truth and corrupted input data. DnCNN (Figure 1e) produces reconstructions that are noticeably more accurate and clearer (less blurred) and with fine details better preserved. In contrast, NLM (Figure 1c) tends to overly smooth out details, while the BM3D reconstruction (Figure 1d) appears out of focus.

These observations align with the findings from Table 1. The visual assessment further reinforces the superiority of DnCNN and highlights its ability to preserve image details and enhance overall image clarity when compared to traditional denoising methods like NLM and BM3D. Based on these findings, we solely consider DnCNN as the embedded denoiser in our CPnP framework for the subsequent sections.

3.3. On the Choice of the Penalty Sequence for the Proposed CPnP

In this section, we investigate the performance of the implemented CPnP method by systematically varying the ADMM penalty sequences. We define increasing penalty sequences according to the relations:

$${\beta }_{\mathbf{r}}^{k+1}=\gamma ·{\beta }_{\mathbf{r}}^k,$$

$${\beta }_{\mathbf{v}}^{k+1}=\gamma ·{\beta }_{\mathbf{v}}^k,$$

where $\gamma \ge 1$. Our experimental setup involves initializing the pair of parameters $({\beta }_{\mathbf{r}}^0,{\beta }_{\mathbf{t}}^0)$ from the set ${\{0.2,0.4,0.6,0.8,1\}}^2$ and testing different values of $\gamma$: specifically, $\gamma =1,1.01,1.05$.

We set $\tau =1$, and for the purposes of our experiments, we assume a known level of noise denoted by ${\sigma }_{\mathit{\eta }}$. Further discussion on the rationale behind choosing these parameters will be provided in the subsequent section.

As discussed in the previous section, we adopt the DnCNN introduced in [47] as the denoising prior due to its state-of-the-art performance in the field and its fast computation.

We consider the sole butterfly image from Set5. We simulate a degraded acquisition by setting ${\sigma }_{\mathbf{A}}=1.2$ and ${\sigma }_{\mathit{\eta }}=30$. We investigate the stability of CPnP with respect to the choice of $\gamma$, ${\beta }_{\mathbf{r}}^0$, and ${\beta }_{\mathbf{t}}^0$. We point out that in the experiments, we did not observe any significant difference when choosing different images.

The stability of the implemented CPnP is depicted in Figure 2a,b. The figures illustrate the distribution of the PSNR and SSIM metrics, respectively, while varying the starting points (${\beta }_{\mathbf{r}}^0$, ${\beta }_{\mathbf{t}}^0$) of the increasing ADMM penalty sequences for different values of $\gamma$. More specifically, for each ${\beta }_{\mathbf{r}}^0$, we consider the distribution of the considered metrics (average ± standard deviation) with respect to ${\beta }_{\mathbf{t}}^0$ for different values of $\gamma$. In-depth analysis reveals that CPnP consistently maintains high performance levels, as assessed by PSNR and SSIM metrics, even when varying the parameter $\gamma$ and employing different initializations for ${\beta }_{\mathbf{r}}^0$ and ${\beta }_{\mathbf{t}}^0$. This robustness across different settings underscores the reliability and effectiveness of the CPnP approach for deblurring and denoising tasks. For all subsequent sections, we set ${\beta }_{\mathbf{r}}^0=1$, ${\beta }_{\mathbf{t}}^0=1$, and $\gamma =1.01$ as fixed parameters.

3.4. On the Choice of the Constraint Parameters for the Proposed CPnP

In this section, we consider the images belonging to Set5. We generate blurred and noisy data by applying the linear image formation model (1) with two different degradation settings (${\sigma }_{\mathbf{A}}=1$, ${\sigma }_{\mathit{\eta }}=15$) and (${\sigma }_{\mathbf{A}}=1.3$, ${\sigma }_{\mathit{\eta }}=30$). We examine the impact on the restored images of $\delta =\tau \sqrt{n}{\sigma }_{\mathit{\eta }}$ with $\tau$ ranging in [0,1]. Two distinct scenarios are taken into account. In the first, which is referred to as the ideal scenario in the following, we assume that the magnitude of ${\sigma }_{\mathit{\eta }}$ is exactly known. In the second, which is referred to as the realistic scenario in the following, we assume that only an estimate $\overline{{\sigma }_{\mathit{\eta }}}$ of ${\sigma }_{\mathit{\eta }}$ is provided. The estimation is computed following the approach outlined in [43].

In Figure 3a–d, we illustrate the variations in ${\sigma }_{{\mathbf{x}}^{*}}$ and PSNR as a function of $\tau$ for the two different degradation levels considered. For each $\tau$, we display the distribution (average ± standard deviation) across all images in Set5 (depicted as shaded regions). The red and blue lines denote the means of these distributions for the idealized and realistic scenarios, respectively. Additionally, in Figure 3a,c, the yellow dashed line represents the true standard deviations (namely ${\sigma }_{\mathit{\eta }}=15$ and ${\sigma }_{\mathit{\eta }}=30$) of the Gaussian noise affecting all the data in Set5.

In Figure 3a,c, the idealized scenario (blue lines) demonstrates that the computed ${\sigma }_{{\mathbf{x}}^{*}}$ closely aligns with ${\sigma }_{\mathit{\eta }}$ when $\tau =1$ across all images in Set5 (thin shading). In contrast, within the realistic scenario, where only an estimate $\overline{{\sigma }_{\mathit{\eta }}}$ of the Gaussian noise level is provided, the optimal approximation of ${\sigma }_{\mathit{\eta }}$ is achieved at $\tau =0.98$ for both degradation levels. Multiple experiments have consistently revealed that this phenomenon stems from the algorithm in [43], which tends to overestimate the noise level in the simulated data $\mathbf{b}$.

Figure 3b,d illustrate the behavior of the PSNR metric with respect to $\tau$. Notably, comparable performance in terms of PSNR is observed for both scenarios. Additionally, Figure 3b,d emphasizes that the highest PSNR is achieved when setting $\tau <1$.

Lastly, Figure 3a indicates that smaller values of $\tau$ tend to underestimate, while larger values of $\tau$ tend to overestimate ${\sigma }_{\mathit{\eta }}$. Figure 3b shows that the PSNR metric is negatively affected by these under/overestimations of the noise level.

Figure 4 provides a visual inspection of the behavior of CPnP with respect to $\tau$ values. Small values produce several artifacts in the recovered images, while higher ones induce an oversmoothing effect on the final result. The most reliable results are obtained for $\tau$ close to $1^{-}$, i.e., for values close to 1 but still smaller than 1. This behavior is due to overestimation of the noise level given by the employed algorithm [43].

To enhance the reliability of our CPnP testing, we employ the noise level estimation method proposed by [43] to estimate ${\sigma }_{\mathit{\eta }}$, and consistently set $\tau =0.98$ for all subsequent experiments.

3.5. Comparisons with PnP and RED

In this section, we conduct a comparative analysis between our CPnP method and the other two competing methods: PnP and RED. The assessment of reconstruction metric performance involves utilizing images from Set24 under different degradation levels: (${\sigma }_{\mathbf{A}}=1.2,{\sigma }_{\eta }=25$) and (${\sigma }_{\mathbf{A}}=0.8,{\sigma }_{\eta }=35$).

Table 2. Mean values of PSNR and SSIM for the images in Set5 and Set24 when varying the degradation levels. The best results are highlighted in bold. Our CPnP outperforms both RED and PnP in terms of the considered metrics.

	Set24 (${\mathit{\sigma }}_{\mathit{A}}=1.2,{\mathit{\sigma }}_{\mathit{\eta }}=25$)			Set24 (${\mathit{\sigma }}_{\mathit{A}}=0.8,{\mathit{\sigma }}_{\mathit{\eta }}=35$)
Metric	RED	PnP	CPnP	RED	PnP	CPnP
PSNR	26.29	26.70	26.85	26.59	26.92	27.16
SSIM	0.75	0.77	0.78	0.76	0.77	0.79

presents the mean PSNR and SSIM values. For the competing methods, PnP and RED, the regularization parameters are estimated to achieve optimal performance in terms of the PSNR metric. The results clearly indicate that our CPnP method outperforms both RED and PnP in terms of both performance measures. Furthermore, the effectiveness of our CPnP approach is also assessed from a visual perspective. Figure 5 demonstrates the superiority of CPnP with respect to RED and PnP by providing two close-up views of the restored images. These close-ups highlight the superior reconstruction capabilities of CPnP in terms of clarity and noise reduction compared to both PnP and RED. This visual evidence confirms the outcomes of the quantitative results presented in Table 2 and emphasizes the enhanced performance and reliability of CPnP for deblurring and denoising tasks. Moreover, we finally remark that CPnP exhibits robustness regarding the choice of hyperparameters, unlike RED and PnP, as underlined in the previous sections. The physical interpretation of CPnP hyperparameters enhances the interpretability and practicality of CPnP compared to traditional methods such as RED and PnP.

4. Conclusions

In this paper, a novel constrained formulation of the well-established Plug-and-Play framework is presented and is denoted as CPnP. Within this model, the minimum of the regularization functional is compelled to adhere to a discrepancy-based threshold. The solution to the CPnP model is obtained within the ADMM framework and offers a straightforward yet effective approach for image restoration while allowing the usage of different denoising priors.

The determination of the threshold, serving as the regularization parameter, holds physical significance and involves estimating the standard deviation of the noise affecting the data. Efficient assessment of the noise level in the degraded data is achieved through the method outlined in [43] and eliminates the need for extensive parameter tuning as required by unconstrained models like PnP and RED.

In the experimental section, CPnP demonstrates stability and robustness concerning both the model and algorithm hyperparameters. Furthermore, it performs comparably, if not better, than both PnP and RED in terms of PSNR and SSIM metrics as well as visual inspection. The superior performance, coupled with its stability and robustness, position CPnP as a promising choice for various image restoration applications.