Self-Supervised Deep Hyperspectral Inpainting with Plug-and-Play and Deep Image Prior Models

Abstract

Hyperspectral images are typically composed of hundreds of narrow and contiguous spectral bands, each containing information regarding the material composition of the imaged scene. However, these images can be affected by various sources of noise, distortions, or data loss, which can significantly degrade their quality and usefulness. This paper introduces a convergent guaranteed algorithm, LRS-PnP-DIP(1-Lip), which successfully addresses the instability issue of DHP that has been reported before. The proposed algorithm extends the successful joint low-rank and sparse model to further exploit the underlying data structures beyond the conventional and sometimes restrictive unions of subspace models. A stability analysis guarantees the convergence of the proposed algorithm under mild assumptions, which is crucial for its application in real-world scenarios. Extensive experiments demonstrate that the proposed solution consistently delivers visually and quantitatively superior inpainting results, establishing state-of-the-art performance.

1. Introduction

Hyperspectral remote sensing has found widespread application in numerous fields such as astronomy, agriculture, environmental monitoring, and Earth observation. Hyperspectral images (HSIs) are often captured via satellite or airborne sensors, each presenting samples at different time slots, using push-broom strategies along the flying pathway [1]. The nature of the HSI acquisition system makes the HSI a high-resolution 3D data cube that covers hundreds or thousands of narrow spectral bands, conveying a wealth of spatio-spectral information. However, due to instrumental errors, imperfect navigation and atmospheric variations, practical HSIs can suffer from noise, and missing pixels or even entire lines of pixels (an example of this type of missing data can be found in the recently released Earth Surface Mineral Dust Source Investigation (EMIT) hyperspectral dataset, developed by NASA’s Jet Propulsion Laboratory and launched on 14 July 2022 [2]). These issues can severely impact the accuracy and reliability of subsequent analyses, making HSI inpainting a critical task in the field of remote sensing and Earth observation.

1.1. Hyperspectral Image Inpainting

Hyperspectral image inpainting refers to the restoration of missing or corrupted data in acquired HS images. Unlike RGB images, for which inpainting involves filling a single pixel value (red, green, and blue channels), HSI inpainting requires filling in a complex vector that contains extensive spectral information. This additional complexity makes the already challenging task of HSI inpainting even more difficult. The primary objective of HSI inpainting is to create visually convincing structures while ensuring that the texture of the missing regions is spectrally coherent. The inpainting of HS images typically involves two main steps: (a) estimating the missing or corrupted data by leveraging information from the neighboring spectral bands and spatially adjacent pixels and (b) refining the estimated data to ensure consistency with the overall spectral and spatial properties of the HS image. Traditional approaches, such as those proposed by [3], assume that the spectral vectors of the HSI always exist in some unknown low-dimensional subspaces, and the missing regions are estimated through projections. However, the performance of such techniques deteriorates significantly when large portions of the image are missing, and they may even fail when all spectral bands are absent. Another line of research involves the tensor completion process to restore incomplete observations [4,5,6]. These methods leverage the spatio-spectral correlations of the hyperspectral data cube, offering potential improvements over subspace-based techniques.

Over the past decade, the success of deep learning has brought new opportunities for solving HSI inpainting tasks. In [7,8], the authors proposed transformer-based frameworks to effectively capture both local and global contextual information in 3D images, demonstrating the potential of attention modules in exploring relationships between missing pixels and background pixels. In [9], the authors showed that missing pixels can be predicted using generative neural networks. However, this approach has two key limitations: (1) it requires training the networks on extensive datasets to achieve the desired performance, and (2) the inpainted areas often suffer from over-smoothing, resulting in the loss of critical information, such as abrupt changes in the surface materials. More recently, researchers discovered that the network’s structure itself may serve as a good inductive bias for regularizing image reconstruction tasks, such as image inpainting [10]. A deep image prior (DIP) enables the learning of priors directly from the image without the need for extensive training datasets, which was further developed and successfully applied to hyperspectral (HS) images [11]. Following [11], subsequent works [12,13,14,15,16,17,18] have shown that certain trained or untrained neural networks can be directly incorporated into iterative solvers to improve the reconstruction accuracy, i.e., as the regularizer. More recently, the authors of [19] introduced a self-supervised framework that leverages the powerful learning capabilities of diffusion models and sets new state-of-the-art benchmarks across several HSI reconstruction tasks. However, as it was reported in our preliminary report [20], as well as in a series of DIP-related works [13,21,22,23], the DIP may not always be the ideal solution, despite promising empirical results. On the practical side, this is because a DIP works by directly fitting a neural network to the corrupted HS image; in the absence of regularization or explicit denoising mechanisms, a DIP tends to overfit, which will eventually capture the noise in the image. On the theoretical side, a DIP lacks a strong theoretical foundations to explain its performance, and its convergence is highly sensitive to random initialization of the neural network. Different initializations could potentially lead to poor results. Although subsequent works [12,13,24] have attempted to mitigate this issue by introducing a plug-and-play (PnP) denoiser into the training of a DIP, they require either a suitable early stopping criterion or the manual control of the learning rate. As a result, there remains a need for a new HSI inpainting algorithm that combines the strengths of both PnP and DIP methods while offering convergence guarantees.

1.2. Sparsity and Low Rankness in HSIs

Despite the high dimensionality, HS images exhibit intrinsic sparse and low-rank structures due to the high correlation between spatial pixels and spectral channels. This property has made sparse representation—(SR) and low rank—(LR) based methods popular for the processing of HSIs [25,26,27]. SR relies on the key assumption that the spectral signatures of the pixels approximately lie in a low-dimensional subspace spanned by representative pixels from the same class. With a given dictionary, SR allows for sparse decomposition of HSIs into a linear combination of several atoms, thus exploiting the spatial similarity of HSIs (It is worth mentioning that, when the dictionary is not given, we can manually use the end-members of some pixels with pure spectra and build the dictionary, or learn the dictionary [28] in an unsupervised manner, using sparsity as a regularizer). This assumption about the underlying structures has been effectively applied in various HSI tasks, including HSI classification [29], denoising [30], and unmixing [31].

LR is another widely used regularizer in HSI processing, which assumes that pixels in the clean HSI image have high correlations in the spectral domain, thus capturing both the spectral similarity and global structure of HSIs. LR allows for the modeling of the HSI as a low-rank matrix; thus, the missing and corrupted information in the HSIs can be effectively reconstructed by leveraging the underlying low-rank structure. In many state-of-the-art HSI reconstruction algorithms, LR is often used in parallel with SR to achieve better reconstruction accuracy [30,32]. By combining the two priors together, algorithms can better handle missing data, noise, and corruption while maintaining the overall structure and spectral integrity of the image. For more detailed information on LR and SR of hyperspectral images, we refer readers to the review work in [33].

However, a notable limitation with such an LR or SR model is that the real HS data do not exactly follow the sparse and/or low-dimension assumptions.; e.g., there are long trails (numerous small but non-zero elements) in the descending sorted absolute sparse coefficients and singular values, where overly-restrictive assumptions on these coefficients could lead to under-fitting. In conclusion, both LR and SR rely on some predefined mathematical models that assume certain properties of the HS image, such as low-dimensional manifolds (low-rankness) or sparse representations in specific domains. While these models are useful, they struggle to capture more complex patterns in the real HS data, such as non-linear relationships and complex spatio-spectral dependencies. Moreover, when a significant percentage of the data are corrupted or entire spectral bands of pixels are missing, these methods often fail [3]. The question of how deep neural models can address this issue by exploiting more data-adaptive, low-dimensional models in a self-supervised learning framework is the main topic of this study.

In this paper, we start by giving a brief overview of the LRS-PnP-DIP algorithm [20]; we replaced the denoising/thresholding step with the PnP denoiser, and we used a DIP to replace the low-rank component of the formulation. Since there remains a lack of comprehensive theoretical analysis of the DIP and its extensions that can guarantee the convergence, we demonstrate both theoretically and empirically that small modifications can ensure the convergence of LRS-PnP and LRS-PnP-DIP, while the original algorithms in [20] may diverge. We propose a variant of the LRS-PnP-DIP algorithm dubbed LRS-PnP-DIP(1-Lip), which not only outperforms existing learning-based methods but also resolves the instability issue of a DIP in an iterative algorithm.

1.3. Contributions

This paper presents a novel HS inpainting algorithm that leverages the learning capability of deep networks while providing insights into the convergence behavior of such complex algorithms. Through rigorous mathematical analysis, we derive sufficient conditions for stability and fixed-point convergence and show that these conditions are satisfied by slightly modified algorithms. Our approach offers several key contributions, including the following:

• This paper explores the LRS-PnP and LRS-PnP-DIP algorithm in more depth by showing the following: (1) The sparsity and low-rank constraints are both important to the success of the methods in [20]. (2) DIP can better explore the low-rank subspace compared to conventional subspace models such as singular value thresholding (SVT).
• Under some mild assumptions, a fixed-point convergence proof is provided for the LRS-PnP-DIP algorithm (see Theorem 1). We introduce a variant to the LRS-PnP-DIP called LRS-PnP-DIP(1-Lip), which effectively resolves the instability issue of the algorithm by slightly modifying both the DIP and PnP denoiser.
• To the best of our knowledge, this is the first time the theoretical convergence of PnP with DIP replaced the low-rank prior being analyzed under an iterative framework.
• Extensive experiments were conducted on real-world data to validate the effectiveness of the enhanced LRS-PnP-DIP(1-Lip) algorithm with a convergence certificate. The results demonstrate the superiority of the proposed solution over existing learning-based methods in both stability and performance.

The remainder of this paper is organized as follows: Section 2 provides an introduction to our approach, and Section 3 describes our main convergence theory for the proposed LRS-PnP-DIP(1-Lip) algorithm. Section 4 provides the implementation details and discusses the experimental results. Section 5 concludes the paper with a brief discussion of potential future directions.

2. Proposed Methods

Mathematical Formulations

The task of HSI inpainting can be formulated as reconstructing the clean image, X, from a noisy and incomplete measurement, Y, where an additive noise, N, and a masking operator, M, are present:

$$Y=\mathcal{M}\left\{X\right\}+N$$

(1)

The clean image $X\in {\mathbb{R}}^q$ (where $q=n_r\times n_c\times n_b$), with $n_r$ and $n_c$ representing the spatial dimensions of the image, and $n_b$ represents the total number of spectral bands. The operator $\mathcal{M}\in {\mathbb{R}}^{q\times q}$ is a binary mask, where zero represents the missing pixels, and one represents the observed and valid pixels. Thus, the masking operator $\mathcal{M}$ can be represented as a diagonal square matrix. N is the additive Gaussian noise of appropriate size. Usually, M is provided, and the formulation (1) describes a linear system, which can be expressed as follows:

$$\mathit{y}=\mathrm{M}\mathit{x}+\mathit{n}$$

(2)

In this equation, $\mathit{x}$, $\mathit{y}$, and $\mathit{n}$ represent the vectorized forms of X, Y, and N, respectively, and $\mathrm{M}$ is a diagonal matrix. We obtain the recovered, i.e., inpainted, image, ${\mathit{x}}^{*}$, by applying sparse representation to each image patch, $P_i(\mathit{x})$. Here, the operator $P_i(·)$ extracts the i-th patch from image $\mathit{x}$, where each patch may consist of only valid pixels or a combination of valid and missing pixels, depending on the size of $P_i(·)$. The inpainted image ${\mathit{x}}^{*}$ can be obtained by solving the following optimization problem:

$$\begin{array}{c}\hfill ({\mathit{x}}^{*},{\mathit{\alpha }}^{*})=\underset{\mathit{x},\mathit{\alpha }}{argmin}{\gamma \parallel \mathit{y}-\mathrm{M}\mathit{x}\parallel }_2^2+w_{lr}{\parallel \mathit{x}\parallel }_{*}\\ \hfill +\parallel {\displaystyle \sum _i}(P_i(\mathit{x})-\mathsf{\Phi }{\mathit{\alpha }}_i){\parallel }_2^2+w_s{\displaystyle \sum _i}{\parallel {\mathit{\alpha }}_i\parallel }_1\end{array}$$

(3)

The objective in the expression above consists of three terms. The first term is the data fidelity term, weighted by the parameter $\gamma$. Since estimating $\mathit{x}$ from $\mathit{y}$ is inherently ill-posed, i.e., with more unknown variables than equations, the solution is non-unique. To address this, we introduce two additional “priors” to regularize the inpainting problem: low-rank and sparsity constraints. The second term penalizes solution $\mathit{x}$ to be low-rank, which is typically employed as a surrogate for rank minimization. The third term restricts the missing pixels to be generated from the subspace approximated by the valid pixels. These terms are weighted by parameters $w_{lr}$ and $w_s$, respectively. The sparse representation problem is solved using a given dictionary, $\mathsf{\Phi }$. Specifically, $\mathsf{\Phi }$ can be constructed either by using the end-members of some pixels with pure spectra or by learning from the noisy pixels in the observations. It is here exclusively learned from noisy pixels in the observations using online dictionary learning [28]. By adopting the augmented Lagrangian and introducing the auxiliary variable $\mathit{u}$ [34], problem (3) can be rewritten as follows:

$$\begin{array}{cc}\hfill ({\mathit{x}}^{*},{\mathit{\alpha }}^{*})=& \underset{\mathit{x},\mathit{\alpha }}{argmin}{\gamma \parallel \mathit{y}-\mathrm{M}\mathit{x}\parallel }_2^2+w_{lr}{\parallel \mathit{u}\parallel }_{*}+w_s{\displaystyle \sum _i}{\parallel {\mathit{\alpha }}_i\parallel }_1\hfill \\ \hfill & +{\displaystyle \frac{{\mathit{\mu }}_1}{2}}\parallel {\displaystyle \sum _i}(P_i(\mathit{x})-\mathsf{\Phi }{\mathit{\alpha }}_i)+{\displaystyle \frac{{\mathit{\lambda }}_1}{{\mathit{\mu }}_1}}{\parallel }_2^2\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathit{x}=\mathit{u}\hfill \end{array}$$

(4)

The Lagrangian multiplier and penalty terms are denoted as ${\mathit{\lambda }}_1$ and ${\mathit{\mu }}_1$, respectively. Using the alternating direction method of multipliers (ADMM), we can sequentially update the three variables $\mathit{\alpha }$, $\mathit{u}$, and $\mathit{x}$ to solve problem (4):

(1) Fixing $\mathit{u}$ and $\mathit{x}$, and updating α:

$$\begin{array}{c}\hfill {\mathit{\alpha }}^{k+1}=\underset{\mathit{\alpha }}{argmin}{\displaystyle \frac{{\mathit{\mu }}_1^k}{2}}{\displaystyle \sum _i}\parallel (P_i({\mathit{x}}^k)+{\displaystyle \frac{{\mathit{\lambda }}_1^k}{{\mathit{\mu }}_1^k}})-\mathsf{\Phi }{\mathit{\alpha }}_i{\parallel }_2^2\\ \hfill +w_s{\displaystyle \sum _i}{\parallel {\mathit{\alpha }}_i\parallel }_1\end{array}$$

(5)

The problem in Equation (5) is a patch-based, sparse coding problem that can be solved using iterative solvers. Let us denote the first term on the right-hand side of Equation (5) as $f(\mathit{\alpha })$ and the second term as $g(\mathit{\alpha })$ while dropping the subscript for simplicity. Problem (5) can be represented as a constrained optimization problem:

$$\begin{array}{cc}\hfill (\widehat{\mathit{\alpha }},& \widehat{\mathit{v}})=\underset{\mathit{\alpha },\mathit{v}}{argmin}f(\mathit{\alpha })+w_{s}g(\mathit{v})\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathit{\alpha }=\mathit{v}\hfill \end{array}$$

(6)

By introducing an auxiliary variable, $\mathit{u}$, and a multiplier, $\rho$, the augmented Lagrangian of (6) becomes the following:

$$\begin{array}{c}\hfill \mathcal{L}(\widehat{\mathit{\alpha }},\widehat{\mathit{v}},\widehat{\mathit{u}})=f(\mathit{\alpha })+w_{s}g(\mathit{v})+{\mathit{u}}^T(\mathit{\alpha }-\mathit{v})+{\displaystyle \frac{\rho }{2}}{\parallel \mathit{\alpha }-\mathit{v}\parallel }_2^2\end{array}$$

(7)

Then, ADMM finds solutions by breaking the constrained optimization problem (6) into several sub-problems and updating them separately in an iterative fashion. The pseudocode is provided in Algorithm 1.

The ${\mathit{v}}^{k+1}$ sub-problem can be solved by the proximal operator [35]. The authors in [36] realized that ${\mathit{\alpha }}^{k+1}+{\mathit{u}}^k$ can be treated as a “noisy” version of $\mathit{v}$, and $g(\mathit{v})$ can be seen as a regularization term. Then, the whole process resembles a denoising operation applying to the intermediate results, which can be simply replaced with the PnP denoisers such as BM3D and the non-local mean (NLM) denoiser. In our approach, we employed PnP-ISTA [36] as an alternative solution for solving problem (5). We denote the gradient of f as $\nabla f$ and I as an Identity matrix with an appropriate shape. The entire process can then be replaced with an off-the-shelf denoiser $\mathcal{D}$, operating on $\nabla f$ [36]. Each iteration takes the following form:

$$\begin{array}{c}\hfill {\mathit{\alpha }}^{k+1}=\mathcal{D}(\mathrm{I}-\nabla f)({\mathit{\alpha }}^k)\end{array}$$

(8)

PnP-ISTA can significantly improve the reconstruction quality over traditional ISTA [37] due to the following reasons: (1) it enables the integration of state-of-the-art denoisers to handle different noise patterns and different types of images, which is particularly flexible, as one can apply any off-the-shelf denoiser without massively changing the existing framework; and (2) it allows the use of denoisers tailored to different problems (e.g., a denoiser that promotes sparsity or smoothness, or even a denoising neural network that encodes more complex priors).

Algorithm 1 Pseudocode of ADMM for solving problem (6).

Initialization $w_s,\rho ,k\leftarrow 0,{\mathit{\alpha }}^k\leftarrow 0,{\mathit{u}}^k\leftarrow 0,{\mathit{v}}^k\leftarrow 0$. while Not Converged do     ${\mathit{\alpha }}^{k+1}\leftarrow {argmin}_{\mathit{\alpha }}f(\mathit{x})+{\displaystyle \frac{\rho }{2}}{\parallel \mathit{\alpha }-({\mathit{v}}^k-{\mathit{u}}^k)\parallel }_2^2$.     ${\mathit{v}}^{k+1}\leftarrow {argmin}_{\mathit{v}}{w}_{s}g(\mathit{v})+{\displaystyle \frac{\rho }{2}}{\parallel \mathit{v}-({\mathit{\alpha }}^{k+1}+{\mathit{u}}^k)\parallel }_2^2$.     ${\mathit{u}}^{k+1}\leftarrow {\displaystyle \frac{1}{\rho }}{\mathit{u}}^k+({\mathit{\alpha }}^{k+1}-{\mathit{v}}^{k+1})$. end while

(2) We can update $\mathit{u}$ in this setting by:

$$\begin{array}{c}\hfill {\mathit{u}}^{k+1}=\underset{\mathit{u}}{argmin}{w}_{lr}{\parallel \mathit{u}\parallel }_{*}+{\displaystyle \frac{{\mathit{\mu }}_{\mathit{2}}^k}{2}}{\parallel ({\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_{\mathit{2}}^k}{{\mathit{\mu }}_2^k}})-\mathit{u}\parallel }_2^2\end{array}$$

(9)

where ${\mathit{\lambda }}_2$ and ${\mathit{\mu }}_2$ are the corresponding Lagrangian multiplier and penalty term, respectively, which we use subscripts to distinguish between ${\mathit{\lambda }}_1$ and ${\mathit{\mu }}_1$ in step (4). The problem can be solved using the singular value thresholding (SVT) algorithm [38] as follows:

$$\begin{array}{c}\hfill {\mathit{u}}^{k+1}=\mathcal{SVT}({\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}})\end{array}$$

(10)

Specifically, denote ${\displaystyle \frac{w_{lr}}{{\mathit{\mu }}_2^k}}$ as $\tau$, matrix ${\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}}$ as $\mathrm{A}$, and the soft thresholding operator ${\mathcal{S}}_{\tau }$ as follows:

$$\begin{array}{c}\hfill {\mathcal{S}}_{\tau }(x)=\mathit{sign}(x)·\mathit{max}(|x-\tau |,0)\end{array}$$

(11)

where $sign(·)$ is an operator taking the positive part of its input. Applying (11) to the singular values of $\mathrm{A}$ promotes low-rankness:

$$\begin{array}{c}\hfill {\mathit{u}}^{k+1}=\mathcal{SVT}(\mathrm{A})={\mathrm{U}}_{\mathrm{A}}{\mathcal{S}}_{\tau }({\Sigma }_{\mathrm{A}}){\mathrm{V}}_{\mathrm{A}}^{\mathrm{T}}\end{array}$$

(12)

In the proposed LRS-PnP-DIP algorithm [20], this step is substituted with a deep image prior (DIP) $f_{\theta }(\mathit{z})$, where $\theta$ denotes the network parameters that need to be updated, and the input $\mathit{z}$ is set to ${\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}}$; i.e., the latent image from the previous iterations:

$$\begin{array}{c}\hfill {\mathit{u}}^{k+1}=f_{\theta }({\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}})\end{array}$$

(13)

(3) Fixing α and $\mathit{u}$, and updating $\mathit{x}$:

$$\begin{array}{c}\hfill {\mathit{x}}^{k+1}=\underset{\mathit{x}}{argmin}{\gamma \parallel \mathit{y}-\mathrm{M}\mathit{x}\parallel }_2^2+{\displaystyle \sum _i}\parallel (P_i(\mathit{x})+{\displaystyle \frac{{\mathit{\lambda }}_1^k}{{\mathit{\mu }}_1^k}})\\ \hfill -\mathsf{\Phi }{\mathit{\alpha }}_i^{k+1}{\parallel }_2^2+{\displaystyle \frac{{\mathit{\mu }}_2^k}{2}}{\parallel (\mathit{x}+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}})-{\mathit{u}}^{k+1}\parallel }_2^2\end{array}$$

(14)

A closed-form solution for $\mathit{x}$ exists as follows:

$$\begin{array}{cc}\hfill & {\mathit{x}}^{k+1}={(\gamma {\mathrm{M}}^T\mathrm{M}+{\mathit{\mu }}_1^k{\displaystyle \sum _i}{{\mathrm{P}}_i}^T{\mathrm{P}}_i+{\mathit{\mu }}_2^k\mathrm{I})}^{-1}\hfill \\ \hfill & (\gamma {\mathrm{M}}^T\mathit{y}+{\mathit{\mu }}_1^k{\displaystyle \sum _i}{\mathrm{P}}_i\mathsf{\Phi }{\mathit{\alpha }}_i^{k+1}+{\mathit{\mu }}_2^k{\mathit{u}}^{k+1}-{\displaystyle \sum _i}{\mathrm{P}}_i{\mathit{\lambda }}_1^k-{\mathit{\lambda }}_2^k\mathrm{I})\hfill \end{array}$$

(15)

In the expression above, the patch selection operator $P_i(·)$ is rewritten in its matrix form ${\mathrm{P}}_i$. To calculate ${\mathit{x}}^{k+1}$ in practice, we can leverage the fact that, for the HSI inpainting task, both ${\mathrm{M}}^T\mathrm{M}$ and ${{\mathrm{P}}_i}^T{\mathrm{P}}_i$ are diagonal matrices. As a results, the matrix inversion of ${(\gamma {\mathrm{M}}^T\mathrm{M}+{\mathit{\mu }}_1^k{\sum }_i{{\mathrm{P}}_i}^T{\mathrm{P}}_i+{\mathit{\mu }}_2^k\mathrm{I})}^{-1}$ can be simply implemented as an element-wise division. This trick significantly reduces the computational cost of the inversion, making the proposed algorithm more scalable for practical use.

(4) The Lagrangian and penalty terms are updated as follows:

$$\begin{array}{c}\hfill {\mathit{\lambda }}_1^{k+1}={\mathit{\lambda }}_1^k+{\mathit{\mu }}_1^k({\mathit{x}}^{k+1}-\mathsf{\Phi }{\mathit{\alpha }}^{k+1})\\ \hfill {\mathit{\lambda }}_2^{k+1}={\mathit{\lambda }}_2^k+{\mathit{\mu }}_2^k({\mathit{x}}^{k+1}-{\mathit{u}}^{k+1})\end{array}$$

(16)

$$\begin{array}{c}\hfill {\mathit{\mu }}_1^{k+1}={\rho }_1{\mathit{\mu }}_1^k\\ \hfill {\mathit{\mu }}_2^{k+1}={\rho }_2{\mathit{\mu }}_2^k\end{array}$$

(17)

The LRS-PnP-DIP algorithm proposed in [20] is summarized in Algorithm 2.

Algorithm 2 (LRS-PnP-DIP) algorithm.

Require: masking matrix: $\mathrm{M}$, noisy and incomplete HSI: $\mathit{y}$, learned dictionary: $\mathsf{\Phi }$. denoiser: $\mathcal{D}$. max iteration: $I{t}_{max}$. DIP: $f_{\theta }$ Output: inpainted HSI image $\mathit{x}$.  Initialization DIP parameters $\theta$, ${\mathit{\lambda }}_1,{\mathit{\lambda }}_2,{\mathit{\mu }}_1,{\mathit{\mu }}_2,{\rho }_1,{\rho }_2$.  while Not Converged do      for $i=1:I{t}_{max}$ do:    ${\mathit{\alpha }}^{k+1}=\mathcal{D}(\mathrm{I}-\nabla f)({\mathit{\alpha }}^k)$      update DIP parameters $\theta$, with target: $\mathit{y}$ and intermediate results: ${\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}}$ as input.      update $\mathit{x}$ by (15).      update Lagrangian parameters and penalty terms.  end while

• Remarks. In contrast to its original form, it is here proposed to replace the conventional PnP denoiser in (8) with an averaged NLM denoiser, as introduced in [39]. Additionally, we modify the $f_{\theta }(\mathit{z})$ in (13) to obtain Lipschitz continuity with Lipschitz constant 1. The averaged NLM denoiser is a variant of NLM denoiser whose weight is designed to be a doubly symmetric matrix, ensuring that both the columns and rows sum to 1. With this doubly symmetric property, the spectral norm of the weight matrix is bounded by 1, meaning the averaged NLM denoiser is by design, non-expansive. (We refer the reader to [39] for detailed structures of the averaged NLM denoiser).

The enhanced algorithm is denoted as LRS-PnP-DIP(1-Lip), which is shown in Algorithm 3, and its flow chart is provided in Figure 1.

Algorithm 3 LRS-PnP-DIP(1-Lip) algorithm.

Require: masking matrix: $\mathrm{M}$, noisy and incomplete HSI: $\mathit{y}$, learned dictionary: $\mathsf{\Phi }$. averaged NLM Denoiser: $\mathcal{D}$. max iteration: $I{t}_{max}$. 1-Lipschitz DIP: $f_{\theta }$ Output: inpainted HSI image $\mathit{x}$.  Initialization DIP parameters $\theta$, ${\mathit{\lambda }}_1,{\mathit{\lambda }}_2,{\mathit{\mu }}_1,{\mathit{\mu }}_2,{\rho }_1,{\rho }_2$.  while Not Converged do      for $i=1:I{t}_{max}$ do:    ${\mathit{\alpha }}^{k+1}=\mathcal{D}(\mathrm{I}-\nabla f)({\mathit{\alpha }}^k)$      update DIP parameters $\theta$, with target: $\mathit{y}$ and intermediate results: ${\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_2^k}{{\mathit{\mu }}_2^k}}$ as input.      update $\mathit{x}$ by (15).      update Lagrangian parameters and penalty terms.  end while

3. Convergence Analysis

In this section, we describe the fixed-point convergence of the proposed LRS-PnP-DIP(1-Lip) algorithm under mild assumptions. Fixed-point convergence refers to the type of convergence wherein the algorithm asymptotically enters a steady state. To establish our main theorem, we need the following results:

Definition 1

(Non-expansive operator). An operator, $T:{\mathbb{R}}^n\to {\mathbb{R}}^n$, is said to be non-expansive if, for any $x,y\in {\mathbb{R}}^n$, the following applies:

$$\begin{array}{cc}\hfill {\parallel (T(x)-T(y))\parallel }^2& \le {\parallel (x-y)\parallel }^2\hfill \end{array}$$

(18)

Definition 2

($\theta$-averaged). An operator, $T:{\mathbb{R}}^n\to {\mathbb{R}}^n$, is said to be θ-averaged with some $\theta \in (0,1)$ if there exists a non-expansive operator, R, such that we can write $T=(1-\theta )I+\theta R$.

Lemma 1.

Let $T:{\mathbb{R}}^n\to {\mathbb{R}}^n$ be θ-averaged for some $\theta \in (0,1)$. Then, for any $x,y\in {\mathbb{R}}^n$, the following applies:

$$\begin{array}{cc}\hfill {\parallel (T(x)-T(y))\parallel }^2& \le {\parallel (x-y)\parallel }^2\hfill \\ & -{\displaystyle \frac{1-\theta }{\theta }}{\parallel ((I-T)(x)-(I-T)(y)\parallel }^2\hfill \end{array}$$

(19)

Proof of this lemma can be found in [[40], Lemma 6.1].

Definition 3

(Fixed point). We say that $x^{*}\in {\mathbb{R}}^n$ is a fixed point of the operator $T:{\mathbb{R}}^n\to {\mathbb{R}}^n$ if $T(x^{*})=x^{*}$. We denote the set of fixed points as fix(T).

Definition 4

($\beta$-smoothed). Let $f:{\mathbb{R}}^n\to {\mathbb{R}}^n$ be differentiable; we say that f is β-smooth if there exists $\beta >0$, such that $\parallel \nabla f(x)-\nabla f(y)\parallel \le \beta \parallel x-y\parallel$ for any $x,y\in {\mathbb{R}}^n$.

Definition 5

(Strong convexity). A differentiable function, f, is said to be strongly convex with modulus $\rho >0$ if $f(x)-{\displaystyle \frac{\rho }{2}}{\parallel x\parallel }^2$ is convex.

Lemma 2

(Property of strong convexity). Let f be strongly convex with modulus $\rho >0$. Then, for any $x,y\in {\mathbb{R}}^n$, the following applies:

$$\begin{array}{cc}\hfill 〈\nabla f(x)-\nabla f(y),x-y〉& \ge {\rho \parallel (x-y)\parallel }^2\hfill \end{array}$$

(20)

Proof of this Lemma can be found in [41].

• We made the following assumptions:

Assumption 1.

We assume that (1) the denoiser $\mathcal{D}$ used in the sparse coding step is linear and θ-averaged for some $\theta \in (0,1)$, (2) the function $f(\mathit{\alpha })={\displaystyle \frac{{\mathit{\mu }}_1^k}{2}}\parallel ({\mathit{x}}^k+{\displaystyle \frac{{\mathit{\lambda }}_1^k}{{\mathit{\mu }}_1^k}})-\mathsf{\Phi }\mathit{\alpha }{\parallel }_2^2$ is β-smoothed, and (3) $\mathrm{fi}x(\mathcal{D}(\mathrm{I}-\nabla f))\ne \varnothing$.

• Remarks. The $\theta$-averaged property is a subset of the non-expansive operator that most of the existing PnP frameworks have worked with [39,42]. In fact, the non-expansive assumption is found to be easily violated for denoisers such as BM3D and NLM denoisers when used in practice. Nevertheless, the convergence of PnP methods using such denoisers can still be empirically verified. In the experiments, we adopted the modified NLM denoiser whose spectral norm is bounded by 1. Hence, it satisfies the $\theta$-averaged property by design [39].

Lemma 3

(Fixed-point convergence of PnP sparse coding). If Assumption 1 holds, then, for any α, and $\rho >\beta /2$, the sequence ${(\mathit{\alpha })}_{k\ge 0}^{\mathit{k}}$ generated via step (8) converges to some ${\mathit{\alpha }}^{*}\in \mathrm{fi}x(\mathcal{D}(I-\nabla f))$.

Proof of this Lemma can be found in [[40], Theorem 3.5].

Assumption 2.

We assume that the DIP function $f_{\theta }$ is L-Lipschitz-bounded:

$$\begin{array}{c}\hfill \parallel (f_{\theta }(x)-f_{\theta }(y)){\parallel }^2\le L{\parallel (x-y)\parallel }^2\end{array}$$

(21)

for any x, y, and $L\le 1$.

• Remarks. In the above assumption, the smallest constant L that makes the inequality hold is called the Lipschitz constant. The Lipschitz constant is expressed as the maximum ratio between the absolute change in the output and the input. It quantifies how much the function output changes with respect to the input perturbations or, roughly speaking, how robust the function $f_{\theta }$ is. In deep learning, it. Assumption 2 guarantees that the trained DIP has a Lipschitz constant, $L\le 1$, which is a desirable feature so that a network does not vary drastically in response to minuscule changes in its input. In the experiments, this can be achieved by constraining the spectral norm of each layer of the neural network during training.

We are now ready to state our main theorem.

Theorem 1

(Convergence of LRS-PnP-DIP(1-Lip) in the Lyapunov Sense). If both Assumptions 1 and 2 hold, with penalty μ, and with an L-Lipschitz constrained DIP ($L\le 1$), then there exists a non-increasing function, $H^k=2\parallel {\mathit{x}}^k-{\mathit{x}}^{*}{\parallel }^2+{\displaystyle \frac{1}{{\mathit{\mu }}^2}}\parallel {\mathit{\lambda }}_1^k-{\mathit{\lambda }}_1^{*}{\parallel }^2+{\displaystyle \frac{1}{{\mathit{\mu }}^2}}{\parallel {\mathit{\lambda }}_2^k-{\mathit{\lambda }}_2^{*}\parallel }^2$, such that all trajectories generated via LRS-PnP-DIP(1-Lip) are bounded, and that as $k\to \infty$, $\parallel {\mathit{x}}^k-{\mathit{x}}^{*}{\parallel }^2\to 0$, $\parallel {\mathit{\alpha }}^k-{\mathit{\alpha }}^{*}{\parallel }^2\to 0$, and $\parallel {\mathit{u}}^k-{\mathit{u}}^{*}{\parallel }^2\to 0$; i.e., even if the equilibrium states of $\mathit{x}$, α, and $\mathit{u}$ are perturbed, they will finally converge to ${\mathit{x}}^{*}$, ${\mathit{\alpha }}^{*}$, and ${\mathit{u}}^{*}$, respectively. The proposed LRS-PnP-DIP(1-Lip) algorithm is, thus, asymptotically stable.

• We placed the detailed proof of Theorem 1 in Appendix A.2.

4. Experimental Results

4.1. Implementation Details

We evaluate the proposed inpainting model on two publicly available hyperspectral datasets:

• The Chikusei airborne hyperspectral dataset [43] (the link to the Chikusei dataset can be found at https://naotoyokoya.com/Download.html, accessed on 4 January 2025); the test HS image consists of 192 channels, and it was cropped to 36 × 36 pixels in size.
• The Indian Pines dataset from AVIRIS sensor [44]; the test HS image consisted of 200 spectral bands, and it was cropped to 36 × 36 pixels size.

For each dataset, we trained the dictionary $\mathsf{\Phi }$ with a size of 1296 × 2000, using only noisy and incomplete HSI images (the size of the test HS image and the dictionary were chosen to match the practical use case where the proposed inpainting algorithms were to be mounted on each nano-satellite to process different small image tiles in parallel [45]), this is a necessary step if we do not have access to pure spectra. If a standard spectral dataset exists, learning can be performed in the form of a pre-training step. However, we used the input HS image for this task to demonstrate that the proposed algorithms operate in a self-supervised setting. Additionally, we introduced Gaussian noise with noise level $\sigma =0.12$ to the cropped HS images. Since the main focus of this paper is image inpainting, instead of denoising, $\sigma$ was kept fixed across all experiments. We applied mask $\mathrm{M}$ to all the spectral bands in the given region, which represents the most challenging case. For the choice of the PnP denoiser, we used both the BM3D and modified NLM denoisers [39]; the latter is an averaged denoiser, which enjoys a non-expansive property that is crucial to convergence analysis.

Our implementation of the deep image prior model follows the same structure as in the deep hyperspectral prior [11]. During the training of 1-Lipschitz DIP, we applied Lipschitz regularization to all layers, as in [46]. In each iteration, we fed the DIP with the latent image from the previous iteration, rather than the random signal, as suggested by [11]. We did not perturb the input to support our convergence analysis, which is different from the canonical DIP. To eliminate the need for manually selecting the total number of iterations for the DIP model, we adopted the early stopping criterion proposed in [47] with a fixed window size of 20 and a patience number of 100, which detects the near-peak MPSNR point using windowed moving variance (WMV). Note that, for the training of conventional DIP, the early-stopping strategy is often employed to prevent it from fitting to random noise after certain iterations. However, the proposed LRS-PnP-DIP(1-Lip) algorithm mitigates the over-fitting issue due to the imposed constraint. The early-stopping criterion is applied primarily to optimize the algorithm’s runtime so that one does not have to wait for the algorithm to reach the maximum number of iterations. In our formulation, the low-rank and sparsity constraints in the equation were weighted with parameters $w_{lr}$ and $w_s$, respectively, while the data fidelity term was weighted with $\gamma$. Initially, we set $w_s/w_{lr}$ to 1 and $\gamma$ to 0.5. Notably, the choice of $\gamma$ is highly dependent on the noise level of the observed image. If the noise level is low, the recovered image, X, should be similar to the noisy observation, Y, parameter $\gamma$ should be large, and vice versa. The Adam optimizer was used, and the learning rate was set to 0.1. Finally, we used three widely used metrics, namely the mean signal-to-noise ratio (MPSNR), mean structural similarity (MSSIM), and mean spectral angle mapper (MSAM), to evaluate the performance of the model in all experiments.

4.2. Low-Rank vs. Sparsity

In this section, we examine the effectiveness of both sparsity and low-rank constraints in our proposed LRS-PnP algorithm. Figure 2 shows the inpainting performance of LRS-PnP over different $\tau$ for different masks (in percentages), where $\tau$ is defined as the ratio of the weight of the sparsity constraint and the low-rank constraint. For example, $\tau =0$ means that the sparsity constraint is disabled, and mask = 50% means that 50% of the pixels are missing. We can conclude from the results that (1) the LRS-PnP algorithm works better with both low-rank and sparsity constraints, as using either an overwhelming sparsity constraint or a low-rank constraint alone tends to reduce inpainting performance, and (2) when the percentage of missing pixels increases, more weight should be placed on the sparsity constraint in order to achieve high-quality reconstruction. On the one hand, approximation errors may be introduced in the sparse coding step; i.e., step (5) in Algorithms 2 and 3. These errors can potentially be suppressed by the constraint of low rank. On the other hand, the sparsity constraint exploits the spectral correlations; i.e., the performance will be enhanced if spatial information is considered.

4.3. Low Rankness Due to DIP

In the original LRS-PnP-DIP Algorithm [20], we propose solving the low-rank minimization problem using DIP instead of the traditional SVT projection. A natural question arises: does the DIP (e.g., the architecture proposed in Figure 1) effectively capture the redundancies in the spectral domain? This is empirically justified in Figure 3, where we trained DIP on the corrupted HS image with varying numbers of spectral bands as input and output. The results show that the inpainting performance of DIP improved substantially when more spectral bands were utilized during training; the peak MPSNR was obtained when the neural network saw the entire 128 bands of the input HS images, rather than each single band individually. This finding is counterintuitive since DIP with 2D convolution is generally expected to handle only the spatial correlation of input HSIs. Similar observations have been reported in [10,48], where the authors confirmed that DIP with 2D convolution implicitly explored the channel correlations, possibly due to the upsampling layers in the decoder part that manipulates the HSI channels in the spectral domain.

In Figure 4, we sort the singular values of the inpainted image of LRS-PnP (the pink line), LRS-PnP-DIP [20] (the blue line), and PnP sparse coding without a low-rank constraint (green line), in descending order. Among these algorithms, LRS-PnP performs singular value projection to promote low rankness, while it is replaced with DIP in LRS-PnP-DIP. It is observed that using a well-trained DIP alone was sufficient to capture these low-rank details. Thus, we suggest replacing the conventional SVT process with DIP. Although there is no theoretical evidence on how DIP mimics the SVT, we show through experiments that the DIP can well preserve some small yet important singular values that would otherwise be discarded via SVT projection. Hence, it has the ability to better exploit the low-rank subspace in a data-driven fashion, possibly owing to its highly non-linear network structures and inductive bias.

4.4. Convergence

The use of the PnP denoiser and DIP in the context of HSIs has been recently investigated in [12,13]. However, theoretical evidence of the stability of the algorithm is still lacking. To bridge this gap, we verify the convergence of LRS-PnP-DIP(1-Lip). In Figure 5, the top left, top right, and bottom left figures represent the successive differences of each state on a log scale, and the bottom right figure represents the inpainting performance. For LRS-PnP-DIP, we used the BM3D denoiser and DIP without constraints. For LRS-PnP-DIP(1-Lip), we used the modified NLM denoiser and the 1-Lipschitz constraint DIP. The latter automatically satisfies Assumptions 1 and 2; thus, the fixed-point convergence can be guaranteed. Interestingly, we noticed that LRS-PnP-DIP with BM3D and conventional DIP still works pretty well in practice even though there are fluctuations in state $\mathit{x}$, as can be seen in the top right and bottom left plots in Figure 5. We deduced that such instability mainly comes from the process of solving DIP sub-problems, rather than the denoising sub-problems, as the primal variable ${\mathit{\lambda }}_1$ smoothly converges with BM3D denoisers as well.

From the bottom right plot, we observed that the best results are obtained with LRS-PnP-DIP in terms of reconstruction quality, and the LRS-PnP-DIP(1-Lip) is only slightly lower than its unconstrained counterpart. Our observations on the reduced performance of Lipschitz DIP are in agreement with existing works [46,49], where imposing the Lipschitz constraint during training does hurt the capacity and expressivity of the neural network.

4.5. Comparison with State of the Art

The proposed algorithms were compared with learning-based algorithms: DHP [11], GLON [4], R-DLRHyIn [16], DeepRED [22], PnP-DIP [24], DeepHyIn [50], and DDS2M [19]. To ensure the fairness of comparison, we employed the same U-net as the backbone for DHP, DeepRED, R-DLRHyIn, PnP-DIP, and LRS-PnP-DIP(1-Lip) and the attention-based U-net with the same complexity for DeepHyIn and DDS2M. For GLON, we replaced its original FFDNet with the one trained on the hyperspectral dataset [51] and kept the others intact. (The reason for making this adaption is that the neural network proposed in GlON [4] was trained on RGB/grayscale images. It was found to be beneficial to use an FFDNet, e.g., in [51], trained on the HSI dataset.) For DeepHyIn, we used the end members extracted from the clean image patch. For DDS2M, we fine-tuned the diffusion-related parameters, as suggested in [19]. The statistical results of each test sample are averaged over 20 experiments to account for the effect of random seeds during training.

In Figure 6, we plot the recovered spectrum of the center of the missing area of LRS-PnP-DIP(1-Lip) for each method. It can be seen that LRS-PnP-DIP(1-Lip) produces a more consistent and realistic spectrum in the missing region compared to methods such as GLON, DHP, DeepRED, R-DLRHyIn, DDS2M, and DeepHyIn. We present the visual inpainting results for each method on the Chikusei dataset and the Indian Pines dataset, as depicted in Figure 7 and Figure 8, respectively. The quantitative statistical results for each dataset, evaluated under various mask shapes, are summarized in

Table 1. Comparison between the proposed algorithm and other learning-based inpainting algorithms on the Chikusei dataset: the mean and variance over 20 samples are shown here. During the experiments, four different mask shapes were used, which are depicted in the first four columns of Figure 7. The best results are underlined for each type of mask. For the metrics MPSNR and MSSIM, higher values indicate better performance. Conversely, for the MSAM metric, the lower the better.

Mask Type	Methods	Input	GLON [4]	DHP [11]	R-DLRHyIn [16]	DeepRED [22]	DeepHyIn [50]	PnP-DIP [24]	DDS2M [19]	LRS-PnP-DIP(1-Lip)
Mask Type 1	MPSNR ↑	33.0740	$41.0324$	41.3956(±0.52)	41.5714(±0.31)	41.5765(±0.28)	41.6902(±0.57)	41.7595(±0.25)	$41.7728(\pm 0.35)$	$\underset{=}{41.8023(\pm 0.16)}$
	MSSIM ↑	0.2441	$0.9078$	0.9102(±0.03)	0.9135(±0.01)	0.9121(±0.02)	0.9180(±0.02)	0.9250(±0.02)	$0.9268(\pm 0.02)$	$\underset{=}{0.9275(\pm 0.01)}$
	MSAM ↓	0.7341	0.1342	0.1296(±0.01)	0.1250(±0.01)	0.1237(±0.01)	0.1221(±0.01)	$0.1203(\pm 0.01)$	$\underset{=}{0.1183(\pm 0.01)}$	$0.1185(\pm 0.01)$
Mask Type 2	MPSNR ↑	31.7151	38.4103	39.7510(±0.65)	40.1586(±0.84)	40.4119(±0.33)	40.6260(±0.40)	40.6661(±0.52)	40.7904(±0.74)	$\underset{=}{41.4910(\pm 0.23)}$
	MSSIM ↑	0.2338	0.8768	0.8961(±0.01)	0.9041(±0.01)	0.9049(±0.01)	0.9100(±0.01)	0.9121(±0.01)	0.9130(±0.01)	$\underset{=}{0.9189(\pm 0.01)}$
	MSAM ↓	0.8296	0.1518	0.1436(±0.02)	0.1373(±0.01)	0.1320(±0.01)	0.1315(±0.01)	0.1298(±0.01)	$0.1264(\pm 0.01)$	$\underset{=}{0.1203(\pm 0.01)}$
Mask Type 3	MPSNR ↑	30.3258	35.6927	38.1440(±0.90)	38.9580(±0.79)	39.6952(±0.99)	39.8608(±0.68)	39.8150(±0.40)	40.0779(±0.82)	$\underset{=}{40.8400(\pm 0.49)}$
	MSSIM ↑	0.2018	0.8558	0.8805(±0.01)	0.8871(±0.01)	0.8902(±0.01)	08950(±0.01)	0.8958(±0.01)	0.9003(±0.01)	$\underset{=}{0.9088(\pm 0.01)}$
	MSAM ↓	0.8441	0.1670	0.1563(±0.01)	0.1507(±0.01)	0.1495(±0.01)	0.1480(±0.01)	0.1476(±0.01)	$0.1455(\pm 0.01)$	$\underset{=}{0.1392(\pm 0.01)}$
Mask Type 4	MPSNR ↑	27.9802	34.5011	36.6018(±1.15)	37.6504(±0.89)	37.9080(±0.59)	38.1961(±0.77)	38.3442(±0.73)	38.7529(±0.88)	$\underset{=}{39.4068(\pm 0.54)}$
	MSSIM ↑	0.1700	0.8267	0.8693(±0.01)	0.8740(±0.01)	0.8772(±0.01)	0.8769(±0.01)	0.8820(±0.01)	0.8849(±0.01)	$\underset{=}{0.8927(\pm 0.01)}$
	MSAM ↓	0.8847	0.1803	0.1714(±0.03)	0.1636(±0.02)	0.1602(±0.01)	0.1580(±0.01)	0.1557(±0.01)	$0.1532(\pm 0.01)$	$\underset{=}{0.1449(\pm 0.01)}$

for the Chikusei dataset and

Table 2. Comparison between the proposed algorithm and other learning-based inpainting algorithms on the Indian Pines dataset: The mean and variance over 20 samples are shown here. During the experiments, four different mask shapes were used, which are depicted in the first four columns of Figure 7. The best results are underlined for each type of mask.

Mask Type	Methods	Input	GLON [4]	DHP [11]	R-DLRHyIn [16]	DeepRED [22]	DeepHyIn [50]	PnP-DIP [24]	DDS2M [19]	LRS-PnP-DIP(1-Lip)
Mask Type 1	MPSNR ↑	31.6903	$39.7676$	40.1917(±0.33)	40.5490(±0.41)	40.7982(±0.32)	40.9309(±0.60)	40.8897(±0.34)	$41.1084(\pm 0.29)$	$\underset{=}{41.3423(\pm 0.24)}$
	MSSIM ↑	0.2268	$0.8884$	0.8917(±0.02)	0.8930(±0.02)	0.8932(±0.01)	0.9028(±0.01)	0.9050(±0.01)	$0.9092(\pm 0.01)$	$\underset{=}{0.9139(\pm 0.01)}$
	MSAM ↓	0.7723	0.1594	0.1580(±0.01)	0.1369(±0.01)	0.1368(±0.01)	0.1320(±0.01)	$0.1255(\pm 0.01)$	$0.1240(\pm 0.01)$	$\underset{=}{0.1256(\pm 0.01)}$
Mask Type 2	MPSNR ↑	29.6815	36.9092	38.1331(±0.22)	38.7072(±0.25)	38.5630(±0.19)	38.7976(±0.40)	39.0021(±0.32)	39.9861(±0.40)	$\underset{=}{40.3010(\pm 0.23)}$
	MSSIM ↑	0.2098	$0.8457$	0.8849(±0.01)	0.8902(±0.01)	0.8864(±0.01)	0.8919(±0.01)	0.8972(±0.01)	$0.9011(\pm 0.01)$	$\underset{=}{0.9048(\pm 0.01)}$
	MSAM ↓	0.8723	0.1514	0.1509(±0.01)	0.1469(±0.01)	0.1450(±0.01)	0.1444(±0.01)	$0.1420(\pm 0.01)$	$0.1380(\pm 0.01)$	$\underset{=}{0.1310(\pm 0.01)}$
Mask Type 3	MPSNR ↑	28.6195	36.7447	37.0714(±0.28)	37.4790(±0.20)	37.3976(±0.39)	37.4555(±0.39)	37.7821(±0.22)	37.8070(±0.52)	$\underset{=}{38.0599(\pm 0.33)}$
	MSSIM ↑	0.1864	$0.8395$	0.8787(±0.01)	0.8870(±0.01)	0.8869(±0.01)	0.8893(±0.01)	0.8921(±0.01)	$0.8945(\pm 0.01)$	$\underset{=}{0.8982(\pm 0.01)}$
	MSAM ↓	0.8823	0.1584	0.1579(±0.01)	0.1539(±0.01)	0.1521(±0.01)	0.1490(±0.01)	$0.1488(\pm 0.01)$	$0.1470(\pm 0.01)$	$\underset{=}{0.1458(\pm 0.01)}$
Mask Type 4	MPSNR ↑	24.6998	34.5630	35.8009(±0.30)	35.8292(±0.22)	35.8145(±0.32)	35.8905(±0.23)	36.1022(±0.18)	36.4938(±0.34)	$\underset{=}{36.5104(\pm 0.23)}$
	MSSIM ↑	0.1672	$0.8342$	0.8592(±0.01)	0.8651(±0.01)	0.8678(±0.01)	0.8698(±0.01)	0.8733(±0.01)	$0.8764(\pm 0.01)$	$\underset{=}{0.8779(\pm 0.01)}$
	MSAM ↓	0.8977	0.1644	0.1610(±0.02)	0.1611(±0.01)	0.1602(±0.01)	0.1588(±0.01)	$0.1580(\pm 0.01)$	$\underset{=}{0.1552(\pm 0.01)}$	$0.1554(\pm 0.01)$

for the Indian Pines dataset.

It is shown that LRS-PnP-DIP(1-Lip) can effectively suppress noise compared to methods such as R-DLRHyIn, DeepHyIn, and DDS2M. Moreover, it alleviates the over-smoothing phenomenon of the DeepRED and PnP-DIP algorithms by preserving more details in the non-missing region; this is clear for the last two test samples in Figure 7 and for the first four test samples in Figure 8. From Table 1 and Table 2, it can be seen that LRS-PnP-DIP(1-Lip) almost always demonstrates the highest inpainting performance on both the Chikusei and Indian Pines datasets compared to other methods. The second best is established via DDS2M, which is the closest method to ours in terms of inpainting quality. This is possibly due to its ability to leverage inductive bias from hierarchical diffusion structures. However, as is evident in Figure 7 and Figure 8, DDS2M and other DIP-based extensions occasionally exhibit limitations, such as producing noisy backgrounds or failing to effectively inpaint missing strips. It is important to note that these methods utilize conventional DIP as backbones; hence, they are susceptible to the instability issue of DIP where the algorithm may eventually fit into pure noise without proper constraints. In contrast, LRS-PnP-DIP(1-Lip) is superior, as it not only harvests the strong learning capability of DHP but also enjoys a stability guarantee. For the HSI inpainting task, we anticipate that some information, from the background/non-missing pixels to be carried to the deeper layers, there is a performance loss when removing the skip connections between the encoder and the decoder. Nevertheless, LRS-PnP-DIP(1-Lip) mutes these layers to trade performance for convergence, and it is empirically found that, even if LRS-PnP-DIP(1-Lip) is highly restricted due to layer-wise spectral normalization, it can preserve more textures and produce less noisy results than DDS2M. This observation leads to the conclusion that the proposed LRS-PnP-DIP(1-Lip) algorithm would be preferable for solving practical inpainting tasks compared to existing solutions. It has the ability to yield faithful reconstructions while ensuring stability and robustness.

5. Conclusions

The LRS-PnP algorithm, along with its variant LRS-PnP-DIP proposed in the previous work [20], are innovative hyperspectral inpainting methods designed to address the challenge of missing pixels in noisy and incomplete HS images. These techniques can better handle the challenging situations in practical HSI acquisition systems where the entire spectral band may be absent. Through experiments, we show that both sparsity and the low-rank priors can be effectively integrated as constraints during the training of DIP; the resulting LRS-PnP-DIP algorithm leverages the strong learning capability of the powerful generative DIP model and is able to exploit spectral and spatial redundancies inherent to HSIs. Because the proposed methods do not require any training data, they are particularly attractive for practical hyperspectral inpainting tasks, where only the observed corrupted image is available for reference. However, the LRS-PnP-DIP algorithm suffers from instability issues and may occasionally diverge due to the DIP. In this paper, this long-lasting issue was effectively addressed through the proposed LRS-PnP-DIP(1-Lip) algorithm. Experimental results on real datasets demonstrate that the LRS-PnP-DIP(1-Lip) algorithm not only resolves the instability issue but also delivers results competitive with the state-of-the-art HSI inpainting algorithms. Unlike previous works where the convergence of the algorithm is either missing or only empirically verified, such as [12,13,14,24,52], this work has provided theoretical evidence and established convergence guarantees under non-expansiveness and Lipschitz continuity assumptions on the PnP denoiser and the DIP neural network. However, the performance of DIP has been observed to be sensitive to the network structure [11] and to the training parameters, such as the number of iterations and the learning rate. When the proposed LRS-PnP-DIP(1-Lip) algorithm is given an insufficient number of iterations to learn, DIPs may not work as expected. In future work, we would like to explore the inductive bias of DHPs, as well as a broader variety of deep learning models, such as transformers (the transformer has recently been explored in the remote sensing community [53]) to exploit their potential in solving HSI inpainting problems. Additionally, the extension of the proposed methods to other HSI processing tasks, such as self-supervised HSI denoising, unmixing, and deblurring, is left to a future investigation.