Variational Low-Rank Matrix Factorization with Multi-Patch Collaborative Learning for Hyperspectral Imagery Mixed Denoising

Abstract

In this study, multi-patch collaborative learning is introduced into variational low-rank matrix factorization to suppress mixed noise in hyperspectral images (HSIs). Firstly, based on the spatial consistency and nonlocal self-similarities, the HSI is partitioned into overlapping patches with a full band. The similarity metric with fusing features is exploited to select the most similar patches and construct the corresponding collaborative patches. Secondly, considering that the latent clean HSI holds the low-rank property across the spectra, whereas the noise component does not, variational low-rank matrix factorization is proposed in the Bayesian framework for each collaborative patch. Using Gaussian distribution adaptively adjusted by a gamma distribution, the noise-free data can be learned by exploring low-rank properties of collaborative patches in the spatial/spectral domain. Additionally, the Dirichlet process Gaussian mixture model is utilized to approximate the statistical characteristics of mixed noises, which is constructed by exploiting the Gaussian distribution, the inverse Wishart distribution, and the Dirichlet process. Finally, variational inference is utilized to estimate all variables and solve the proposed model using closed-form equations. Widely used datasets with different settings are adopted to conduct experiments. The quantitative and qualitative results indicate the effectiveness and superiority of the proposed method in reducing mixed noises in HSIs.

1. Introduction

Hyperspectral images (HSIs) are acquired by hyperspectral sensors, represented as a 3D data-cube containing both rich spectral and spatial information. Due to the limitations of the acquisition and transmission process, HSIs unavoidably suffer from various degradations, such as noise contamination, stripe corruption, missing data relating to the voxels in the data-cube or entire spectral bands [1,2,3,4,5]. These degradations severely limit the quality of the images and influence the precision of the subsequent processing, including unmixing, target detection, and classification [6,7,8,9]. Therefore, image restoration is of critical importance and challenging in the preprocessing stage of HSI analysis.

Previously, the traditional 2D or 1D denoising models have been applied for reducing noises in HSIs pixel-by-pixel [10] or band-by-band [11]; however, these methods ignore the correlations between different spectral bands or adjacent pixels and often result in relatively low-quality results. To further enhance the denoising performance, more efficient methods have been proposed, of which the key point is to elaborately encode the prior knowledge on the structure underlying a natural HSI, especially the characteristic across the spatial and spectral dimensionality of the image.

Othman and Qian [12] made an initial attempt to resolve this issue by designing a hybrid spatial–spectral derivative-domain wavelet shrinkage model, which was constructed by exploring the dissimilarity of the signal regularity existing along the space and spectrum of a natural HSI. Fu et al. [13] proposed an effective restoration model by considering the underlying sparsity across the spatial–spectral domain, high correlation across spectra, and non-local self-similarity over space. Meanwhile, a series of methods expanding the wavelet-based method from 2D to 3D has been proposed, for example, the so-called “non-local means” filtering approach has become popular in image processing and extensions have been developed in order to denoise structural 3D images [14]. Letexier et al. [15] proposed a generalized multi-dimensional Wiener filter for denoising hyperspectral images. Similarly, Chen et al. [16] extended Sendur and Selesnick’s bivariate wavelet thresholding from 2D image denoising to 3D data cube denoising. To get better denoising results, as an extension of the BM3D [11] method, Maggioni et al. presented a BM4D model [17]. Utilizing highly correlated spectral information and highly similar spatial information, the spectral-spatial adaptive sparse representation model was proposed for reducing the noise in HSIs [18]. By explicitly treating HSI data as a 3D cube, denoising models based on tensor decomposition have appeared. In Reference [19], a novel coupled spectral–spatial tensor representation framework was proposed for noise reduction of hyperspectral images. Chen et al. proposed a low-rank tensor decomposition model for HSI restoration [20]; however, most of the above-mentioned approaches are limited due to their insufficient usage of the correlations in the spectral domain, which results in suboptimal performance while suppressing mixed noises.

By efficiently exploring the latent knowledge across spectral bands for HSIs, low-rank models have been proposed and widely utilized to restore the pure datasets from the degraded images, with competitive performances [21,22,23,24,25,26]. The classical low-rank matrix factorization (LRMF) model was presented by K. Mitra et al. and T. Okatani et al. [24]. Subsequently, using the low-rank matrix recovery (LRMR) framework, an HSI restoration technique was explored to simultaneously remove various noises in an HSI [25]. The global and non-local low-rank factorization (GLF) was proposed to suppress the noises in HSIs by utilizing the low dimensional sub-spaces and the self-similarity of the real HSI [26]. These approaches obtained satisfactory results by effectively exploiting the spectral information.

To sufficiently enhance the denoising performance, it is necessary to well integrate the spatial characteristics of HSIs into the low-rank-based models [27,28,29,30,31,32]. Wang et al. [29] proposed a novel low-rank constraint and spatial–spectral total variation regularization model by jointly utilizing global low-rank and local spatial–spectral smooth properties of HSIs. Wang et al. [30] developed a total variation regularized low-rank tensor decomposition (LRTDTV) method, in which HSI was regarded as a third-order tensor rather than being divided into the patches or being unfolded into the matrix. In [31], a novel robust principal component analysis approach was introduced into the spatial–spectral low-rank model for mixed noise removal by fully identifying the intrinsic structures of the mixed noise and clean HSIs. Based on the global correlation along the spectra and nonlocal self-similarity across space, a low-rank tensor dictionary learning (LTDL) approach was explored with satisfactory performance in [32]. In the spatial domain, HSI has latent consistency. Using this, patch learning has been widely applied to depict spatial information and has achieved good performance [33,34,35,36,37]. When the HSIs are heavily polluted by noise, the patches with little effective information are usually not directly used to recover the noise-free data. Therefore, these methods could not efficiently learn and represent the intrinsic spatial consistency and nonlocal similarities of HSIs and thus limit their denoising performance.

Deep learning has also been widely used for HSIs [38,39,40,41,42,43]. Their success suggests its effectiveness for learning and depicting latent features when denoising the HSI. Additionally, hyperspectral images are usually polluted by various noises, which often have different statistical features [44,45], such as the noises depending on the signal, the noises depending on the space domain, or spectral domain noise and mixed noise. Therefore, it is necessary to construct a model for suppressing complex mixed noises in order to deal with real HSI scenarios.

To alleviate the above limitations, a variational low-rank matrix factorization model, combined with multi-patch collaborative learning (VLRMFmcl), is proposed in the Bayesian framework to suppress various noises in HSIs. The main contributions of this work are summarized as follows.

(1) Multi-patch collaborative learning is exploited to effectively depict and learn the spatial consistency and the nonlocal self-similarity in the HSI. The pixels within the same collaborative patches share similar spatial–spectral characteristics, which are utilized to effectively improve the performance of denoising the patches polluted by heavy noises.

(2) Variational low-rank matrix factorization is proposed to learn and characterize the collaborative patch data by exploring latent characteristics across the spatial–spectral domain, in which the latent clean image in degraded HSI has the property of low rank and the mixed noises do not. The Gaussian distribution with zero mean and the variance adjusted by gamma distribution is explored to represent the latent clean image. The Dirichlet process Gaussian mixture is exploited to depict the inherent statistical features of different noises in the HSI, which are adapted and learned by exploring the Gaussian distribution, the inverse Wishart distribution, and the Dirichlet process. Through this process, the underlying mixed noise of the HSI can be fit adaptively without needing to know specific noise types or intensity.

(3) Considering the uncertainty of information about latent variables, the posteriors of the latent clean image and the mixed noises are both explicitly parameterized and update in a closed form by utilizing the variational inference. The feasibility and validity of the VLRMFmcl method are evaluated under different experimental conditions. Compared with several popular denoising methods, VLRMFmcl can reduce the noises of the hyperspectral images while preserving the structural information.

The paper is organized as follows. Section 2 gives a detailed description of the proposed restoration model, which is performed using variational inference. In Section 3, several experimental results are presented by utilizing the real-world HSI datasets. The conclusions are given in Section 4.

2. Proposed Model

To effectively suppress the various noises in HSIs, multi-patch collaborative learning is explored to represent the intrinsic spatial consistency and non-local self-similarity of HSIs. Then, the learned patches are input into the variational low-rank matrix factorization model, which is developed to suppress the mixed noises of each patch in the Bayesian framework. Figure 1 presents the framework of the proposed VLRMFmcl.

2.1. Multi-Patch Collaborative Learning

In HSIs, the adjacent pixels have high consistency in the space domain [26,27,28,29]. Based on this fact, they are often divided into overlapping three-dimensional patches for the HSI analysis. The effective information of one patch is very scarce when most of the pixels within this patch are polluted by a large amount of noise. Therefore, it is very difficult to recover the noiseless data by directly exploiting this image patch. To solve these problems, it becomes important to effectively utilize the patches in HSIs.

Figure 2 presents some patches from the Pavia Centre data (presented in Section 3.2), in which the area marked by the red box represents a test patch, and the ones marked by three green boxes represent its neighboring patches. The similarities between the patches marked by the red box and the green boxes are very different. Inspired by this basic characteristic in the HSI, the heavily polluted patches could be restored by the patches with high similarities with them. Additionally, it has been argued that the “collaborative” nature of the approximation can improve classification accuracy [46]. Considering that the HSI denoising aims to facilitate subsequent applications (e.g., classification), a similar “collaborative” [46] nature is introduced. Based on these, a multi-patch collaborative learning strategy is proposed by exploring the similarities between different image patches to effectively learn about the HSI.

Supposing the hyperspectral image X, it is segmented into overlapped three-dimensional patches with the size of $d\times d\times \lambda$, where $d$ represent the spatial size of patches and $\lambda$ is the total number of bands in the HSI. For a pixel $x_i$ from X, the collection $\mathrm{N}(x_i),\hspace{0.17em}\hspace{0.17em}1\le i\le d^2$ consists of all pixels of a patch centered at the sample $x_i$. The pixels in $\mathrm{N}(x_i)$ can be considered to contain similar characteristics. The $y_i$ is formed by stacking all pixels of $\mathrm{N}(x_i)$ into a vector, which can be regarded as the fusing feature of $x_i$. The similarities between the different patch data can be formulated as

$$SimilarIndex\left(y_i,y_j\right)=\exp (-\left(\frac{1}{d^2\lambda }\left|I_{1\times d^2\lambda }{y}_i-I_{1\times d^2\lambda }{y}_j\right|\right)),$$

(1)

where $I_{1\times d^2\lambda }$ is the row vector with the dimension of $d^2\lambda$, of which the elements are equal to one. Obviously, the larger the value of the $SimilarIndex\left(y_i,y_j\right)$ is, the higher the similarities that are observed between $\mathrm{N}(x_i)$ and $\mathrm{N}(x_j)$. According to Equation (1), we can select the most similar (P–1) patches to the patch centered at $x_i$, and construct the non-local patch data Y_i. When $d$ is large enough, it can be considered as all the most similar data to be searched in the whole hyperspectral image.

2.2. Variational Low-Rank Matrix Decomposition

In the existing literature, many computer vision, machine learning, and statistical problems can be approached by solving and learning a low-dimensional linear model. In this case, the low-rank matrix decomposition has been widely concerned and applied in many fields [3,4,5,6], which can effectively explore the low-dimensional properties of the observed data. Assuming X = [y₁,...,y_H]$\in R^{M\times H}$ represents the observed data, $M$ and $H$ represent the spatial size of X, the general low-rank matrix decomposition model is formulated as

$$X=U{V}^T+n,$$

(2)

where $U$ = [u₁,...,u_L]$\in R^{M\times L}$ and V = [v₁,...,v_H]${}^T\in R^{H\times L}$ represent the decomposition matrix, $L\ll min(M,\hspace{0.17em}H)$; $n$ is the noise, which is depicted by Gaussian distribution, Laplace distribution, or polynomial distribution.

Obviously, the pixels of the same collaborative patch have similar characteristics in the spatial and spectral domains. In other words, these pixels have the low-rank property and can be effectively learned and expressed by the low-rank matrix decomposition. Additionally, the hyperspectral images are usually polluted by various noises with different statistical properties. The Gaussians mixture model can effectively learn and depict the different noises, including Gaussian noise, sparse noise, and so on. Above all, the noise model is explored to depict the complex noises in real HSIs, in which the Dirichlet process is utilized to adaptively achieve the selection of Gaussian distribution and the determination of the number of Gaussian distributions. The symbol $Y={\left\{y_i\right\}}_{i=1}^P$ represents the collaborative patch data, and $M=d^2\lambda$ represents the dimensions of the sample $y_i$. According to (2), the proposed Bayesian low-rank matrix decomposition model for denoising the collaborative patch data can be written as follows:

$$Y=U{V}^T+n.$$

(3)

The first term is the low-rank decomposition term, in which $u_i\in R^M$ and $v_j\in R^L$ are defined as $u_i~N(0,{\tau }_{ui}^{-1}I)$ and $v_j~N(0,{\tau }_{vj}^{-1}I)$ separately. That is, $u_i$ and $v_j$ are drawn from the Gaussian distribution with zero mean and variances of ${\tau }_{ui}$ and ${\tau }_{vj}$, individually. In order to improve the model robustness and reduce the sensitivity of parameters, the gamma distribution is introduced to adaptively adjust the parameter ${\tau }_{ui}$ and ${\tau }_{vj}$. The first term can be formulated as:

$$\begin{array}{l}{u}_i~N(0,{\tau }_{ui}^{-1}I),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{v}_j~N(0,{\tau }_{vj}^{-1}I)\\ {\tau }_{ui}^{}~\Gamma (a_0,b_0),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\tau }_{vj}^{}~\Gamma (c_0,d_0)\end{array}$$

(4)

where $I$ is the column vector, of which the entries are all equal to one; $a_0$, $b_0$, $c_0$ and $d_0$ represent the hyper-parameters of the gamma distribution.

The second term in Equation (3) represents the mixed noises in the real HSI. Considering the complex statistical properties of the noises in the HSI, the Gaussians mixture model is utilized to depict the different noises, which are displayed as

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{n}_i~{\displaystyle {\prod }_{k=1}^{\infty }N{({\mu }_k,{\Sigma }_k)}^{z_{ijk}}}\\ {\mu }_k~N({\mu }_0,{\Sigma }_0)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\Sigma }_k~iWishart(e_0,f_0)\\ z_{ij}~Mult(\pi )\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\pi }_t~v_t{\displaystyle \prod _{j=1}^{t-1}\left(1-v_j\right)}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{v}_t~beta(1,\beta )\end{array}$$

(5)

In Equation (5), ${\mu }_k$ and ${\Sigma }_k$ are the mean and variance of the k-th Gaussian distribution, which are learned and represented by the Gaussian distribution and the inverse Wishart distribution. The Gaussian distribution and the inverse Wishart distribution are conjugate. ${\mu }_0$ and ${\Sigma }_0$ represent the mean and variance of the parameter ${\mu }_k$; and $e_0$ and $f_0$ are the freedom degree and scale matrix of the inverse Wishart distribution. To effectively depict the various noises in a data-driven way, the indicator variable $z_{ij}\in {\left\{0,1\right\}}^K,\hspace{0.17em}K\to \infty$ is introduced to determine and learn the number and mode of the Gaussian distribution, and $\sum _k{z}_{ijk}=1$. $z_{ij}$ is drawn from the polynomial distribution with the parameter $\pi$, which is learned through the Dirichlet process. Figure 3 shows the graphical representation of the Bayesian low-rank matrix decomposition model.

Additionally, $Y=\Delta \circ Y$ is introduced into Equation (3) when recovering the missing pixels. $\Delta ={\left\{0,1\right\}}^{M\times P}$ is the sampling matrix whose elements are equal to 0 or 1. Therefore, ${\Sigma }_{f i}=0$ ($f=1,\cdots ,M$) represents the loss of the f-th element in $y_i$ when acquiring; ${\Sigma }_{f i}=1$($f=1,\cdots ,M$) means that the f-th element in $y_i$ is effectively collected.

2.3. Variational Bayesian Inference

According to Equations (3) and (4) and Figure 3, it can be observed that all the variances in the proposed Bayesian low-rank matrix decomposition model satisfy the conjugation. Therefore, variational Bayesian inference can be used to solve the model. Assuming the symbol $\Psi =\{u_i,v_i,{\mu }_k,{\Sigma }_k,z_i,{\tau }_{ui},{\tau }_{vj},v_t\}$ represents the variables of the proposed model; the symbol $\Theta =\left\{a_0,b_0,c_0,d_0,{\mu }_0,{\Sigma }_0,e_0,f_0,\beta \right\}$ is the corresponding hyperparameter in the model. The variational Bayesian inference can be achieved by estimating the posterior distribution of the latent variable $\Psi$ with the observation data Y and the hyperparameter $\Theta$ given. To solve the proposed model, the real posterior distribution $\mathrm{p}(\Psi |Y,\Theta )$ of the latent variables $\Psi$ is approximated based on the distribution $q(\Psi )$. Then, we can obtain

$$\begin{array}{l}\ln \mathrm{p}(Y|\Theta )\hspace{0.17em}=\hspace{0.17em}\ln {\displaystyle \int \mathrm{p}(Y,\Psi |\Theta )}\mathrm{d}\Psi \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\ln {\displaystyle \int \frac{\mathrm{p}(Y,\Psi |\Theta )q(\Psi )}{q(\Psi )}}\mathrm{d}\Psi \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\ge {\displaystyle \int \ln \frac{\mathrm{p}(Y,\Psi |\Theta )q(\Psi )}{q(\Psi )}}\mathrm{d}\Psi \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\ln \mathrm{p}(Y|\Theta )-\mathrm{KL}(q(\Psi )|\mathrm{p}(\Psi |Y,\Theta ))\end{array}$$

(6)

where $\mathrm{KL}(q(\Psi )|\mathrm{p}(\Psi |Y,\Theta ))$ is utilized to represent the KL divergence distance between the variational approximation $q\left(\mathsf{\Psi }\right)$ and the true joint probability distribution $\mathrm{p}(\Psi |Y,\Theta )$. It can be easily seen that the expression $\ln p\left(Y|\Theta \right)$ has a strict lower bound because $\mathrm{KL}\left(q\left(\mathsf{\Psi }\right)|p\left(\Psi |\mathrm{Y},\Theta \right)\right)$≥0. Therefore, the optimal solution of the proposed model can be calculated by minimizing $\mathrm{KL}(q(\Psi )|\mathrm{p}(\Psi |Y,\Theta ))$. Algorithm 1 presents the pseudocode of the VLRMFmcl method.

Algorithm 1. The VLRMFmcl Method

Input: the noisy HSI image X; the spatial size $d$ of patches; the total number $\lambda$ of bands; the hyperparameter $\Theta =\left\{a_0,b_0,c_0,d_0,{\mu }_0,{\Sigma }_0,e_0,f_0,\beta \right\}$;

Output: the denoised image $Y$;

Process Source-Code of multi-patch collaborative learning:

Divide X into the overlapping patches with the size of $d\times d\times \lambda$;

for each pixel $x_i$ in X do

Obtain the collection $N(x_i)$ and $y_i$, where $1\le i\le d^2\lambda$

Calculate the similarities $SimailarIndex(y_i,y_j)$ between $N(x_i)$ and $N(x_j)$;

Select the most similar (P–1) patches to the patch centered at $x_i$, and construct the collaborative patch data $Y_i$;

Process Source-Code of variational low-rank matrix factorization:

Obtain the variables $\Psi =\{u_i,v_i,{\mu }_k,{\Sigma }_k,z_i,{\tau }_{ui},{\tau }_{vj},v_t\}$;

Calculate $\mathrm{p}(\Psi \mid Y,\Theta )$ by $q(\Psi )$; update $\Psi$ by minimizing $\mathrm{KL}(q(\Psi )\mid \mathrm{p}(\Psi \mid \mathrm{H},\Theta ))$;

return denoised image $Y$;

The updating equations of the model variances are listed as follows.

(1) updating $z_{ij}$ and $v_t$:

The posterior probability of $v_t$ is still drawn from the beta process. Supposing $v_t~beta\left(g_t,h_t\right)$, $g_t$ and $h_t$ can be calculated by

$$g_{tk}=1+{\Sigma }_i{\Sigma }_j{q}_{ijk}(t),h_{tk}=\beta +{\Sigma }_{j-t+1}^T{\Sigma }_i{q}_{ijk}(t).$$

(7)

For the variance $z_{ij}$,

$$q(z_{ij})={\displaystyle {\prod }_k{q}_{ijk}{}^{z_{ijk}}},$$

(8)

where

$$\begin{array}{l}{q}_{ijk}(t)=\frac{{\rho }_{ijk}(t)}{{\displaystyle {\sum }_{l=1}^T{\rho }_{ijk}(l)}}\\ {\rho }_{ijk}(l)=\exp ({\gamma }_{l,1}^{ik}+{\gamma }_{l,2}^{ik})\end{array}$$

(9)

Supposing $\mathsf{\Phi }$ is the digamma function, ${\gamma }_{l,1}^{ik}$ and ${\gamma }_{l,2}^{ik}$ are expressed as:

$$\begin{array}{l}{\gamma }_{l,1}^{ik}=\Phi (g_{lk})+\Phi (h_{lk})\\ {\gamma }_{l,2}^{ik}=-0.5tr\{\langle {\Sigma }_l^{-1}\rangle [\langle {\mu }_l{\mu }_l^T\rangle +y_{ij}{y}_{ij}^T+\langle v_i^{}{v}_i^T\rangle \langle U_{}^T{U}_{}\rangle \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2\langle U\rangle \langle v_i^{}\rangle {(y_{ij}-{\mu }_l)}^T-2y_{ij}\langle {\mu }_l^T\rangle ]+\ln \left|{\Sigma }_t^{}\right|\}\end{array}$$

(10)

(2) updating ${\mu }_k$:

The posterior probability of ${\mu }_k$ is still drawn from the Gaussian distribution, which satisfies the following conditions:

$$\begin{array}{l}\langle {\mu }_i\rangle ={[{\Sigma }_0^{-1}+{\displaystyle {\sum }_{i,k}{q}_{ijk}}(t)\langle {\Sigma }_t^{-1}\rangle ]}^{-1}[{\Sigma }_0^{-1}{\mu }_0\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\displaystyle {\sum }_{i,k}{q}_{ijk}}(t)\langle {\Sigma }_t^{-1}\rangle (y_{ij}-\langle U\rangle \langle v_i\rangle )]\\ \langle {\mu }_i{\mu }_i^T\rangle ={[{\Sigma }_0^{-1}+{\displaystyle {\sum }_{i,k}{q}_{ijk}}(t)\langle {\Sigma }_i^{-1}\rangle ]}^{-1}+\langle {\mu }_i\rangle {\langle {\mu }_i\rangle }^{\mathrm{T}}\end{array}$$

(11)

(3) updating ${\Sigma }_k$:

The posterior probability of ${\Sigma }_k$ is still drawn from the inverse-Wishart distribution, which satisfies the following conditions:

$$\begin{array}{l}{e}_t=\hspace{0.17em}{e}_0+{\varsigma }_t\\ f_t=f_0+0.5{\displaystyle {\sum }_i{\displaystyle {\sum }_j{q}_{ijk}(t)(\langle {\mu }_t{\mu }_t^T\rangle +y_{ij}{y}_{ij}^T+\langle u_i{u}_i^T\rangle \langle v_j^T{v}_j^{}\rangle }}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(y_{ij}-{\mu }_t)\langle U\rangle \langle v_j^{}\rangle -2\langle {\mu }_t^T\rangle y_{ij})\end{array}$$

(12)

From Equation (12), we can obtain the following expression:

$$\begin{array}{l}\langle {\Sigma }_t^{-1}\rangle =e_t{(f_t)}^{-1}\\ \langle \ln \left|{\Sigma }_t^{}\right|\rangle =\frac{1}{\Phi (0.5e_t)+d^2\lambda \ln 2+\ln \left|f_t^{-1}\right|}\end{array}$$

(13)

(4) updating $u_i$:

The posterior probability of $u_i$ is still drawn from the Gaussian distribution, which is formulated as

$$u_i~N({\mu }_{u_i},{\Omega }_{u_i}^{-1}),$$

(14)

where the mean ${\mu }_{u_i}$ and the variance ${\Omega }_{u_i}$ are shown as

$$\begin{array}{l}{\Omega }_{u_i}=\langle {\tau }_{ui}\rangle \mathrm{I}+{\displaystyle {\sum }_k{\Sigma }_k^{-1}}{\displaystyle {\sum }_j\langle z_{ijk}^{}\rangle \langle v_j^T{v}_j\rangle }\\ {\mu }_{u_i}={\Omega }_{u_i}^{-1}{\displaystyle {\sum }_k{\Sigma }_k^{-1}}{\displaystyle {\sum }_j\langle z_{ijk}^{}\rangle (}{y}_{ij}-\langle {\mu }_k^{-1}\rangle )\langle v_j\rangle \end{array}$$

(15)

(5) updating ${\tau }_{ui}$:

The posterior probability of ${\tau }_{ui}$ is still drawn from the gamma distribution, which satisfies the following conditions:

$${\tau }_{ui}~\Gamma \left(a,b_i\right),$$

(16)

where the parameters a and b_i are shown as

$$a=a_0+0.5LM,\hspace{0.17em}\hspace{0.17em}{b}_i=b_0+0.5\langle u_i{u}_i^T\rangle .$$

(17)

(6) updating $v_i$:

The posterior probability of $v_i$ is still drawn from the Gaussian distribution, which can be written as follows:

$$v_j~N({\mu }_{v_j},{\Omega }_{v_j}^{-1}),$$

(18)

where the mean ${\mu }_{v_j}$ and the variance ${\Omega }_{v_j}$ are:

$$\begin{array}{l}{\Omega }_{v_j}=\langle {\tau }_{vj}\rangle I+{\displaystyle {\sum }_k{\Sigma }_k^{-1}}{\displaystyle {\sum }_i\langle z_{ijk}^{}\rangle }\langle u_i{u}_i^T\rangle \\ {\mu }_{v_j}={\Omega }_{v_j}^{-1}{\displaystyle {\sum }_k{\Sigma }_k^{-1}}{\displaystyle {\sum }_j\langle z_{ijk}^{}\rangle (}{y}_{ij}-\langle {\mu }_k^{-1}\rangle )\langle u_i\rangle \end{array}$$

(19)

(7) updating ${\tau }_{vj}$:

The posterior probability of ${\tau }_{ui}$ is still drawn from the gamma distribution, which satisfies the following conditions:

$${\tau }_{vj}^{}~\Gamma (c,d_j),$$

(20)

where the parameters c and d_j are shown as

$$c=c_0+0.5LP,\hspace{0.17em}\hspace{0.17em}{d}_j=d_0+0.5\langle v_j^T{v}_j\rangle .$$

(21)

3. Experiments

To validate the effectiveness of the proposed VLRMFmcl model, three popular hyperspectral images were chosen as the experimental datasets, the Beads, Pavia Centre, and Urban datasets. In addition, BM3D [11], ANLM3D [14], BM4D [17], LRMR [25], GLF [26], LRTDTV [30], LTDL [32], DnCNN [42], and HSID-CNN [43] were chosen as the compared methods. The necessary parameters in the BM3D, ANLM3D, and LRTDTV methods, were automatically/manually adjusted to generate the optimal denoising results, as the references suggested. In BM4D, the noise variance was selected from the set {0.01, 0.03, 0.04, 0.05, 0.07, 0.09, 1.1}. In LRMR, the rank of the noiseless matrix was chosen from {3, 5, 6, 7, 9, 11}, and the cardinality of the sparse term was chosen from the set {0, 500, 1000, 1500, 2000, 3000, 4000, 5000}. In GLF, the number of subspaces was chosen from the set {5, 8, 9, 11, 13}. In LTDL, the noise variance was selected from the set {0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4}. In the DnCNN and HSID-CNN methods, the pre-trained weights and the related settings were utilized to conduct the experiments as the references suggested.

In addition, five metrics were chosen to numerically evaluate the denoising performance of the different algorithms—the peak signal-to-noise ratio (PSNR), feature similarity (FSIM) [47], the mean spectral angle (MSA), noise reduction (NR) [48,49], and the mean relative deviation (MRD) [48,49]. At the same time, the visual effect was utilized as an intuitive way to determine the denoising performance. Suppose ${\mathrm{I}}^{den}$ and ${\mathrm{I}}^{ref}$ represent the denoised and reference images, respectively; ${\mathrm{I}}_1$ and ${\mathrm{I}}_2$ represent the spatial size of the image.

(a) The greater the value of PSNR is, the better the denoising image quality is. PSNR (in dB) is formulated as:

$$\mathrm{PSNR}=10{\log }_{10}(\frac{{255}^2\ast {\mathrm{I}}_1{\mathrm{I}}_2}{{\Vert {\mathrm{I}}^{ref}-{\mathrm{I}}^{den}\Vert }^2})$$

(22)

(b) The greater the value of FSIM is, the better the denoising image quality is. FSIM is formulated as:

$$\mathrm{FSIM}=\frac{{\displaystyle {\sum }_{x=\Omega }{S}_l(x)\ast P{C}_m(x)}}{{\displaystyle {\sum }_{x=\Omega }P{C}_m(x)}},$$

(23)

where $S_l(x)$ is derived from the phase congruency and the image gradient magnitude of ${\mathrm{I}}^{den}$ and ${\mathrm{I}}^{ref}$; $P{C}_m(x)$ is the maximum phase congruency of $P{C}_{den}$ (for ${\mathrm{I}}^{den}$) and $P{C}_{ref}$ (for ${\mathrm{I}}^{ref}$); and Ω represents the entire airspace of the image.

(c) MSA was used to estimate spectral fidelity between the denoising images and reference images in the spectral domain. The smaller the value of MSA is, the better the spectral fidelity of the restored algorithms. The MSA is calculated by:

$$\mathrm{MSA}=\frac{1}{{\mathrm{I}}_1{\mathrm{I}}_2}{\displaystyle \sum _{i=1}^{{\mathrm{I}}_1}{\displaystyle \sum _{j=1}^{{\mathrm{I}}_2}{\cos }^{-1}}}\left(\frac{I_{ij}^{den}\cdot I_{ij}^{ref}}{\Vert I_{ij}^{den}\Vert \cdot \Vert I_{ij}^{ref}\Vert }\right)\hspace{0.17em}\hspace{0.17em}$$

(24)

(d) NR was used to evaluate the noise reduction of different restored methods in the frequency domain. The greater the value of NR is, the better the performance of the denoising algorithms. NR is formulated as

$$\mathrm{NR}=N_0/N_1,$$

(25)

where ${\mathrm{N}}_1$ is the power of the frequency components generated by stripes in the restored image and ${\mathrm{N}}_0$ is for the reference image. ${\mathrm{N}}_1$ and ${\mathrm{N}}_0$ can be obtained by

$$N_c={\displaystyle \sum _{\wp }{P}_c(D)},$$

(26)

where $P_c(D)$ is the averaged power spectrum down the columns of an image with D being the distance from its reference image in Fourier space, and $\wp$ is the stripe noise region of the spectrum.

(e) MRD was utilized to compare the degree of distortion between the selected noiseless region of the restored images and reference images. The smaller the value of MRD is, the smaller the image distortion. In the experiment, a 10 $\times$ 10 window was selected to calculate the MRD value. MRD is formulated as:

$$\mathrm{MRD}=\frac{1}{{\mathrm{I}}_1{\mathrm{I}}_2}{\displaystyle \sum _{i=1}^{{\mathrm{I}}_1}{\displaystyle \sum _{j=1}^{{\mathrm{I}}_2}\frac{\left|I_{ij}^{den}-I_{ij}^{ref}\right|}{I_{ij}^{ref}}}}\times 100\%.$$

(27)

3.1. Experiment on the Beads Data Set

The Beads data set, acquired from Columbia University, has a spectral resolution of 10 nm and a spectral range from 400 nm to 700 nm. The data set has a total number of 31 consecutive spectral bands. The spatial resolution in each band is 512 $\times$ 512.

Three kinds of noises were considered in the simulation experiment. The detailed descriptions are listed as follows.

(1) Gaussian white noise with the mean 0 and fixed variance.

(2) Poisson noise is added by adjusting the ratio between the maximum brightness and the initial image brightness, which can be written as $X_{poission}=X\cdot peak$. $X_{poission}$ represents the image polluted by Poisson noise; $X$ is the initial image data; $peak$ refers to the intensity of the Poisson noise. To reduce the passion noise, we utilized settings similar to those in [50]. The variance-stabilizing transformation (VST) was utilized to convert Poisson noise into Gaussian noise before implementing a denoising approach. The final denoising images were obtained by the inverse variance stability transformation.

(3) Sparse noise is added to the randomly selected pixels by utilizing uniform distribution with the interval [−10, 10].

Mixed noise, consisting of zero-mean Gaussian noise with variance σ = 0.1 and Poisson noise with $peak=\left\{5,10,20,30,50,70,100,130,160\right\}$, was added to the Beads data. Then the nine compared methods and the proposed one were utilized to restore the noisy Beads data. The performance curves of the simulation experiments are shown in Figure 4, where the vertical coordinates represent the values of PSNR, FSIM, and MSA, respectively. The horizontal coordinates represent the value of parameter $peak$. Comparing the curves of PSNR, FSIM, and MSA, it can be clearly observed that both the PSNR and FSIM values of the VLRMFmcl method were higher than those of BM3D, ANLM3D, BM4D, LRMR, GLF, LRTDTV, DnCNN, and HSID-CNN methods. At the same time, the MSA of VLRMFmcl was lower than these eight compared algorithms. Compared with LTDL, the proposed model is superior in PSNR. For FSIM and MSA, it showed better values than LTDL in most cases. These facts indicate that VLRMFmcl can effectively improve the quality of the noisy HSI by better maintaining the image feature information and restoring the spectral information in the HSI. In addition, the performance curve of the VLRMFmcl method is smoother than the nine comparison algorithms, which means VLRMFmcl is more stable when denoising the HSI.

Figure 5 shows the restored images of band 27 obtained by different models, which are polluted by Gaussian noise, sparse noise, and missing pixels. Compared with the noisy image in Figure 5b, the quality of Figure 5c–l is significantly improved. According to Figure 5, it can be seen that VLRMFmcl can effectively reduce the various forms of noise in the HSI with a large difference in brightness. The denoising results can preserve the structural information and the edges of the homogeneity region. Using patch-matching three-dimensional filtering, BM3D smoothed out some feature structures and blurred the visual effect while suppressing the different noises and restoring the missing pixels. ANLM3D showed better visual performance than BM3D and DnCNN; however, ANLM3D had a weaker ability to recover the detailed information in the HSI. The restored images obtained by BM4D were too smooth and lost some information. As shown in Figure 5f, the results of the LRMR method were significantly better than BM3D, ANLM3D, and DnCNN, but the results of LRMR still showed obvious sparse noise. Utilizing low-rank factorization of tensors constructed by nonlocal similar 3D patches, GLF was able to recover the basic shapes of the Beads dataset, but its result lacked sharpness. As shown in Figure 5, the results of LRTDTV, LTDL, and VLRMFmcl were much better than those of BM3D, ANLM3D, BM4D, LRMR, GLF, DnCNN, and HSID-CNN. In general, the VLRMFmcl method can remove the mixed noises and restore the missing pixels of the Beads data.

To make a more intuitive comparison of different algorithms, Figure 6 shows the pseudo-color images of the restored images (R: 3, G: 12, B: 25). As can be seen in Figure 6, the denoising results of VLRMFmcl were better than those of BM3D, ANL3D, BM4D, LRMR, GLF, LRTDTV, LTDL, DnCNN, and HSID-CNN. Additionally, the restored results of VLRMFmcl were very similar to those of the reference images, which can be easily observed by comparing Figure 5a,l.

3.2. Experiment on the Pavia Centre Dataset

The Pavia Centre dataset was acquired from the Reflective Optics System Imaging Spectrometer. It contains 115 bands, and each band consists of 1096 × 715 pixels. The spectral range of the Pavia Centre dataset is from 0.43 micrometers to 0.86 micrometers. After removing 13 noisy bands, the remaining 102 bands were used in the following analysis. In the experiment, a subset of the Pavia Centre with the size of 400 × 400 × 102 was used.

Three kinds of noises were considered for the Pavia Centre dataset: (1) Gaussian white noise with the mean 0 and the noise variance $\sigma =0.1$. (2) Sparse noise was added to the randomly selected pixels by utilizing the uniform distribution with the interval [−5, 5]. (3) Deadlines were added to the same position as the selected bands in the HSI. Their width varied from one line to three lines.

Table 1. Quantitative evaluation results for the Pavia Centre data set.

	Band 109	BM3D	ANLM3D	BM4D	LRMR	GLF	LRTDTV	LTDL	DnCNN	HSID-CNN	VLRMFmcl
PSNR(dB)	13.97	17.92	30.25	33.65	34.57	34.12	34.91	35.06	22.79	35.57	35.63
FSIM	0.8458	0.7903	0.9117	0.9739	0.9835	0.9702	0.9832	0.9551	0.8125	0.9907	0.9901
MSA	0.3616	0.3019	0.1101	0.1014	0.1181	0.1067	0.1025	0.0991	0.2216	0.0983	0.0976

shows the evaluation results for PSNR, FSIM, and MSA calculated by different approaches. All the bold numbers in Table 1 indicate the best results. By utilizing the nonlocal self-similarity and adaptively learning the noises in the HSI, the VLRMFmcl method shown the best PSNR/MSA values and the suboptimal FSIM value than the compared methods. Compared with the noisy image, the PSNR and FSIM values obtained by VLRMFmcl were increased by 21.66 and 0.1443, respectively, and MSA was reduced by 0.264. HSID-CNN simultaneously assigned the spatial information and adjacent correlated bands to the network, where multiscale feature extraction was employed to capture both the multiscale spatial feature and spectral feature. Its FSIM value was optimal. Instead of learning the noise variance, BM3D and DnCNN denoised the HSI with the predefined fixed noise variance band by band, which could not efficiently utilize the spectral correlations of the HSI. As shown in Table 1, the PSNR, FSIM, and MSA values of BM3D and DnCNN were significantly lower than those of the other methods while reducing the mixed noises. Noticing that, ANLM3D denoised the HSI by using the high nonlocal self-similarity and making a balance between the smoothing and details preservation; BM4D adopted the 3D nonlocal self-similarity data cube to exploit the local correlations between some neighboring bands; and GLF reduced the mixed noise by utilizing low-rank factorization of tensors constructed by nonlocal similar 3D patches. In Table 1, the ANLM3D, BM4D, and GLF methods presented relatively good results by exploring the spatial–spectral information of the HSI. LRMR, LRTDTV, and LTDL took advantage of the low-rank property in HSI, and their PSNR and FSIM values were better than those of BM3D, ANLM3D, GLF, and DnCNN.

Figure 7 shows the results of band 90 obtained by different denoising approaches. To make a better visual evaluation, Figure 8 shows the comparison of the pseudo-color images (R: 60, G: 30, B 2). It can be seen that the image qualities of the BM3D, ANLM3D, BM4D, LRMR, GLF, LRTDTV, TDTL, DnCNN, HSID-CNN, and VLRMFmcl methods were significantly improved compared to the noise images as shown in Figure 7b and Figure 8b. As can be seen in Figure 7c and Figure 8c, the denoising results obtained by the BM3D method were relatively fuzzy, and the method could not effectively inhibit the strip noise. ANLM3D, BM4D, LRMR, GLF, LRTDTV, LTDL, DnCNN, and HSID-CNN could only suppress some of the noise. It can be easily seen in Figure 7l and Figure 8l that the proposed VLRMFmcl model was able to effectively suppress Gaussian noise, sparse noise, and deadlines, and its results were better than those of the compared methods.

3.3. Experiment on the Urban Dataset

Urban data, with a size of 307×307×210, was acquired from the HYDICE sensor. Due to the detector-to-detector difference, it has different strips and mixed noises versus bands. The image contains 210 bands, and each band consists of 307 × 307 pixels. Its spectral range is from 0.4 to 2.5 micrometers.

Table 2. Quantitative evaluation of noise reduction (NR) and mean relative deviation (MRD) for band 109.

	Band 109	BM3D	ANLM3D	BM4D	LRMR	GLF	LRTDTV	LTDL	DnCNN	HSID-CNN	VLRMFmcl
NR	1	1.8614	2.1051	2.4648	2.5173	2.7051	2.5936	2.6759	2.4973	2.6993	2.8386
MRD	0	3.2201	3.9113	3.5576	4.2165	3.3927	3.5261	3.6017	3.5162	3.3965	3.1976

gives the NR and MRD values of band 109 for the Urban dataset, in which all the bold numbers indicate the best results. In Table 2, it can be seen that the proposed approach was able to effectively reduce the noise of the Urban data, and could retain the detailed information well, which means that VLRMFmcl can effectively reduce noises with low resolution rates.

Figure 9 and Figure 10 show the denoising results of band 109 and band 151 for the Urban dataset, respectively. As shown in Figure 9a and Figure 10a, these two images were heavily polluted with stripes and mixed noise. As shown in the blue rectangles in Figure 9, obvious stripes can be observed in the results obtained by BM3D, ANLM3D, BM4D, LRMR, LRTDTV, and LTDL. Their structure and edge information were also fuzzy. This fact indicates that BM3D, ANLM3D, BM4D, LRMR, LRTDTV, and LTDL showed weaker performance in denoising the severely polluted bands for the Urban dataset. The LRMR method performed better in the target and detail recovery, but its denoising results still showed obvious stripes and mixed noises. As shown in Figure 9i, the result of DnCNN smoothed out some structures and blurred the visual effect. As shown in Figure 9 and Figure 10, GLF, HSID-CNN, and VLRMFmcl could effectively restore the edges and textures of the image, while suppressing the mixed noise.

To facilitate the visual comparison, Figure 11 presents the pseudo-color images of the restored results calculated by different approaches (R: 55, G: 103, B: 207). Comparing the white oval regions, it can be easily seen that the proposed VLRMFmcl method was able to effectively suppress the noises in the smooth area. Meanwhile, it could effectively restore the edge and structure information. Therefore, VLRMFmcl was superior to BM3D, ANLM3D, BM4D, LRMR, GLF, LRTDTV, LTDL, DnCNN, and HSID-CNN when denoising the Urban data.

4. Conclusions

Introducing multi-patch collaborative learning into low-rank matrix decomposition, a variational model was proposed under the Bayesian framework to achieve the reduction of three kinds of noise in HSIs. The non-local self-similarity of HSIs was explored by developing multi-patch collaborative learning. Through this process, the pixels from edges and heterogeneous regions could be effectively depicted. Then, the variational low-rank matrix decomposition model was constructed to separate the latent noise-free data and mixed noises for collaborative patches. Gaussian distribution with the zero mean and variance adaptively regulated by a gamma distribution was exploited to learn and represent the low-rank property of collaborative patches in the spatial–spectral domain and obtain the related clean data. To sufficiently suppress the mixed noise, their statistical characteristics were effectively depicted by the Dirichlet process Gaussian mixture model, which was constructed using the Gaussian distribution, the inverse Wishart distribution, and the Dirichlet process. Variational Bayesian inference was used to solve the model, having the advantages of simple calculation and high stability. Simulation experiments with different combinations of Gaussian noise, Poisson noise, deadlines, and stripe noise demonstrated the effectiveness of the proposed method. Compared with the BM3D, ANLM3D, BM4D, LRMR, GLF, LRTDTV, LTDL, DnCNN, and HSID-CNN methods, the proposed VLRMFmcl method showed superior performance in both the quantitative and qualitative evaluations.