Hyperspectral Pansharpening Based on Homomorphic Filtering and Weighted Tensor Matrix

Abstract

Hyperspectral pansharpening is an effective technique to obtain a high spatial resolution hyperspectral (HS) image. In this paper, a new hyperspectral pansharpening algorithm based on homomorphic filtering and weighted tensor matrix (HFWT) is proposed. In the proposed HFWT method, open-closing morphological operation is utilized to remove the noise of the HS image, and homomorphic filtering is introduced to extract the spatial details of each band in the denoised HS image. More importantly, a weighted root mean squared error-based method is proposed to obtain the total spatial information of the HS image, and an optimized weighted tensor matrix based strategy is presented to integrate spatial information of the HS image with spatial information of the panchromatic (PAN) image. With the appropriate integrated spatial details injection, the fused HS image is generated by constructing the suitable gain matrix. Experimental results over both simulated and real datasets demonstrate that the proposed HFWT method effectively generates the fused HS image with high spatial resolution while maintaining the spectral information of the original low spatial resolution HS image.

1. Introduction

Depending on the number of acquired bands, remote sensing imaging technology has developed from collecting panchromatic (PAN) and color images to multispectral (MS) images, and it can now capture hyperspectral (HS) images with dozens of hundreds of bands. A PAN image with very high spatial resolution is a single-band grayscale image acquired in the visible range. It is able to obtain the shape feature of objects, but cannot distinguish colors. A color image consists of three bands which are red, green and blue, and displays the colors of objects. However, it is difficult to distinguish the features in similar colors. An MS image not only obtains spatial features, but also obtains spectral information in several bands, which is more capable of distinguishing categories of different features. However, the rough spectral resolution of MS images may not meet the requirements in some applications, and it is hard to realize fine feature detection [1]. An HS image with a higher spectral resolution on the order of nanometers can provide finer classification [2], which has been applied to many fields [3,4,5,6,7] and some practical applications, such as vegetation study [8], precision agriculture [8], regional geological mapping [9], mineral exploration [10], and environment monitoring [11]. Due to technical limitations, the spatial resolution of an HS image is low.

As both high spatial and spectral resolutions are important in practical applications, obtaining a high spatial resolution HS (HRHS) image is crucial. One effective way is to perform hyperspectral pansharpening, which fuses a high spatial resolution PAN (HRPAN) image with a low spatial resolution HS (LRHS) image. Figure 1 shows the concept of hyperspectral pansharpening.

Many hyperspectral pansharpening algorithms were developed, among which hyperspectral pansharpening methods using Bayesian and matrix factorization have been proposed in recent years. The Bayesian-based approaches include Bayesian naive Gaussian prior [12], Bayesian sparsity promoted Gaussian prior [13], and HySure [14]. These algorithms utilize the posterior distribution, and are based on maximum a posteriori estimation to fuse LRHS and HRPAN images [15]. The matrix factorization approach generates a fused HRHS image by using the nonnegative matrix factorization (NMF) under some constraints to estimate endmember and abundance matrices [16]. The matrix factorization approach is well represented by the nonnegative sparse coding (NNSC) [17] and constrained nonnegative matrix factorization (CNMF) [18] methods. The main challenge in hyperspectral pansharpening is to effectively improve the spatial resolution while preserving the original spectral information. The Bayesian and matrix factorization approaches are able to achieve good results on this challenge, but have a high computational cost.

Component substitution (CS) and multi-resolution analysis (MRA) approaches are two classical hyperspectral pansharpening approaches which have simple and fast implementation. For the CS class, intensity-hue-saturation (IHS) transform [19,20], principal component analysis (PCA) transform [21,22], Gram–Schmidt (GS) [23], and adaptive GS (GSA) [24] are the most representative methods. The CS class extracts spatial details of the HS image, and replaces the extracted spatial details with the HRPAN image. Regardless of superior spatial performance, the CS class suffers from serious spectral distortion [25]. The typical algorithms of the MRA technique are smoothing filter based intensity modulation (SFIM) [26], Laplacian pyramid [27], modulation transfer function generalized Laplacian pyramid (MTF-GLP) [28], and MTF-GLP with high pass modulation (MTF-GLP-HPM) [29]. The MRA methods generally utilize a multi-resolution decomposition to extract spatial details which are imported into the HS image. Compared with the CS methods, the MRA methods generate less spectral distortion, but usually have a larger computational burden [30]. Recently, several algorithms based on the CS and MRA approaches have been proposed, such as the Sentinel-2A CS and MRA based sharpening algorithm [31], the multiband Filter estimation (MBFE) algorithm [32], and the guided filter PCA (GFPCA) algorithm [33]. Moreover, several intelligent processing-based methods have also been proposed, and examples include deep two-branches convolutional neural network (Two-CNN-Fu) [34], Bidirectional Pyramid Network [35], and 3D-convolutional neural network (3D-CNN) [36].

The CS and MRA approaches mostly extract the spatial information of the HRPAN image and inject it into the LRHS image, but without considering the spatial information of the LRHS image. Due to the incomplete spatial information injection, the CS and MRA approaches may result in distortion. To address this problem, we propose a novel hyperspectral pansharpening method by combining homomorphic filtering with a weighted tensor matrix. An optimized weighted tensor matrix-based method which considers the structure information of the LRHS and HRPAN images is proposed to generate more comprehensive spatial information. In addition, to extract the spatial structure information of the LRHS images, open-closing morphological operation is first used for noise removal, and homomorphic filtering is then introduced to extract the spatial details of each band. Finally, a weighted root mean squared error based method is proposed to obtain the total spatial component of the LRHS image from extracted spatial details of each band, and the Laplacian pyramid networks super-resolution algorithm is adopted to enhance the spatial resolution of the obtained spatial component. Comparative analysis was used to demonstrate the applicability and superiority of the proposed method in both spectral and spatial qualities.

As stated above, a new hyperspectral pansharpening method based on homomorphic filtering and weighted tensor matrix is proposed in this paper. The main novelties of the proposed hyperspectral pansharpening method are concluded in the following aspects.

• A novel HS image spatial component extraction strategy is proposed. Open-closing morphological operation and homomorphic filtering are first introduced to remove the noise and extract the spatial details of each band of the HS image, respectively. Then, a weighted root mean squared error-based method is proposed to obtain the total spatial component of the HS image.
• An optimized weighted tensor matrix-based method is proposed to integrate the spatial component of the HS image with the spatial component of the PAN image. The weighted structure tensor matrix that represents the structural information of multiple images is applied to hyperspectral pansharpening for the first time. The classical methods which mostly extract the spatial information of the PAN image inject the incomplete spatial information, and may lead to distortion. Unlike there classical methods, the proposed optimized weighted tensor matrix-based method generates the spatial information not only from the PAN image but also from the HS image, and can reduce the distortion caused by the insufficient spatial information.

The remainder of this paper is organized as follows. Section 2 describes the weighted structure tensor matrix and homomorphic filtering. In Section 3, the proposed homomorphic filtering and weighted tensor matrix-based hyperspectral pansharpening algorithm is presented. Experimental results and discussion are provided in Section 4, and conclusions are drawn in Section 5.

2. Related Work

2.1. Weighted Structure Tensor Matrix

For an image $I$, the structure tensor matrix $M$ can be decomposed as:

$$M=\nabla I·\nabla I^{\mathrm{T}}=\left[\begin{array}{cc}{I}_x^2& I_x{I}_y\\ I_x{I}_y& I_y^2\end{array}\right]=\left[\begin{array}{cc}{e}_1& e_2\end{array}\right]\left[\begin{array}{cc}{v}_1& 0\\ 0& v_2\end{array}\right]{\left[\begin{array}{cc}{e}_1& e_2\end{array}\right]}^{\mathrm{T}}$$

(1)

where $I_x=\partial I/\partial x$ and $I_y=\partial I/\partial y$ are the horizontal and vertical partial derivatives of the image, $\nabla I={[\begin{array}{cc}{I}_x& I_y\end{array}]}^{\mathrm{T}}$, ${()}^{\mathrm{T}}$ is the transpose operation, $v_1$, $v_2$, $e_1$ and $e_2$ are the two eigenvectors and the corresponding eigenvalues, respectively. As shown in Equation (1), the tensor matrix $M$ which is a symmetric and semi-definite positive matrix has eigen-decomposition, and it has been exploited in some fields, such as texture synthesis [37], image regularization [38], denoising [39], and recognition systems [40]. The eigenvalues obtained by decomposing the tensor matrix are utilized to describe the structure information of the image. For the multiple images $[I_1,I_2,\dots ,I_n]$, the structural information is coupled from all images by employing the linear combination:

$$M_w=\frac{1}{n}{\displaystyle {\sum }_{l=1}^n{M}_l}=\frac{1}{n}\left[\begin{array}{cc}{\displaystyle {\sum }_{l=1}^n{(\partial I_l/\partial x)}^2}& {\displaystyle {\sum }_{l=1}^n((\partial I_l/\partial x)(\partial I_l/\partial y))}\\ {\displaystyle {\sum }_{l=1}^n((\partial I_l/\partial x)(\partial I_l/\partial y))}& {\displaystyle {\sum }_{l=1}^n{(\partial I_l/\partial y)}^2}\end{array}\right]$$

(2)

where $M_w$ is the weighted structure tensor matrix, $M_l$ is the structure tensor matrix of the $l\mathrm{th}$ image, $\partial I_l/\partial x$ and $\partial I_l/\partial y$ are the horizontal and vertical partial derivatives of the $l\mathrm{th}$ image, respectively. The weighted tensor matrix $M_w$ is also a symmetric and semi-definite positive matrix, and can be proceeded the eigen-decomposition.

2.2. Homomorphic Filtering

Homomorphic filtering which is a type of frequency domain filtering can compress the image brightness range and enhance the image contrast. Homomorphic filtering has been applied to some image processing problems [41,42,43] and is based on an image imaging model:

$$f=f_{\mathrm{H}}·f_{\mathrm{L}}$$

(3)

where $f$ represents an image, $f_{\mathrm{H}}$ represents the high frequency reflectance component, and $f_{\mathrm{L}}$ represents the low frequency illumination component. Homomorphic filtering aims to reduce the low frequency component of an image. Logarithmic transformation is utilized to separate the two components:

$$\ln (f)=\ln (f_{\mathrm{H}})+\ln (f_{\mathrm{L}})$$

(4)

After applying the Fourier transform:

$$F=F_{\mathrm{H}}+F_{\mathrm{L}}$$

(5)

where $F$, $F_{\mathrm{H}}$ and $F_{\mathrm{L}}$ denote the Fourier transform of $\ln (f)$, $\ln (f_{\mathrm{H}})$ and $\ln (f_{\mathrm{L}})$, respectively. Then, the high-pass filter $H$ is applied to Equation (5) as

$$S=F·H=F_{\mathrm{H}}·H+F_{\mathrm{L}}·H$$

(6)

where $S$ is the filtered result. The final image is obtained by the inverse Fourier transform and the exponential operation:

$$f_{hf}=\exp (s)=\exp ({\Im }^{-1}(S))$$

(7)

where $f_{hf}$ denotes the homomorphic filtered image, $s$ denotes the inverse Fourier transform of $S$, and ${\Im }^{-1}$ denotes the inverse Fourier transform.

3. Proposed Method

3.1. Hyperspectral Image Preprocessing

Figure 2 shows the schematic of the proposed homomorphic filtering and weighted tensor (HFWT) matrix-based hyperspectral pansharpening algorithm. Let the LRHS image be represented by $X_{\mathrm{LR}}^{\mathrm{HS}}\in {\mathrm{R}}^{m\times n\times B}$, and the HRPAN image be denoted by $X^{\mathrm{PAN}}\in {\mathrm{R}}^{M\times N\times 1}$, where $m\times n$ and $M\times N$ are the size of the LRHS and the HRPAN images, respectively, and $B$ is the number of the LRHS image bands. The fused HRHS image is represented by $X_{\mathrm{HR}}^{\mathrm{HS}}\in {\mathrm{R}}^{M\times N\times B}$.

The open-closing operation which belongs to the mathematical morphology operation is an effective denoising processing operation [44,45]. The effects of denoising using the open operation and closed operation alone are usually not very good, since they may cause amplitude deflection. By contrast, the open-closing operation has the better denoising effect. The open operation is first applied on the image, and the selected structure element is larger than the noise size to remove the background noise. Then, the closed operation is utilized to remove the noise of the image obtained in the previous step. The open-closing denoising operation is suitable for the images which have less small details. Since the LRHS image has the low spatial resolution, the fine spatial details are few. The open-closing operation is applicable to removing noise with high interference in the LRHS image. The open-closing morphological operation is applied as:

$${(X_{\mathrm{LR}}^{\mathrm{RNH}})}^k=({(X_{\mathrm{LR}}^{\mathrm{HS}})}^k\circ S_1)•S_2$$

(8)

for $k=1,2,\dots ,B$, where $X_{\mathrm{LR}}^{\mathrm{RNH}}$ denotes the denoised LRHS image, ${(X_{\mathrm{LR}}^{\mathrm{HS}})}^k$ and ${(X_{\mathrm{LR}}^{\mathrm{RNH}})}^k$ denote the $k\mathrm{th}$ band of the LRHS image and the denoised LRHS image, respectively, and $S_1$ and $S_2$ are the structure elements. Here, $\circ$ represents the opening operation which first uses the erosion operation and then the dilation operation, and $•$ denotes the closing operation which does in reverse. The erosion and dilation operations obtain the local minimum and maximum of the image, respectively. Equation (8) can be expressed in detail as:

$${(X_{\mathrm{LR}}^{\mathrm{OHS}})}^k={(X_{\mathrm{LR}}^{\mathrm{HS}})}^k\circ S_1=({(X_{\mathrm{LR}}^{\mathrm{HS}})}^k\mathsf{\Theta }{S}_1)\oplus S_1=\underset{j\in S_1}{\max }\{[\underset{j\in S_1}{\min }({(X_{\mathrm{LR}}^{\mathrm{HS}})}^k(i+j)-S_1(j))]+S_1(j)\}$$

(9)

$$({(X_{\mathrm{LR}}^{\mathrm{HS}})}^k\circ S_1))•S_2={(X_{\mathrm{LR}}^{\mathrm{OHS}})}^k•S_2=\underset{j\in S_2}{\min }\{[\underset{j\in S_2}{\max }({(X_{\mathrm{LR}}^{\mathrm{OHS}})}^k(i+j)-S_2(j))]+S_2(j)\}$$

(10)

for $k=1,2,\dots ,B$, where $\mathsf{\Theta }$ and $\oplus$ denote the erosion and dilation operations, respectively.

3.2. Hyperspectral Image Spatial Information Extraction

Homomorphic filtering is a filtering method that transforms the nonlinear problem into a linear problem. It transforms the nonlinear multiplicative mixed problem into an additive model by the logarithmic transformation, and then uses linear filtering to process it. The homomorphic filtering suppresses the low frequency illumination component and enhances the high frequency reflectance component. For an HS image, the high frequency component of each band is considered as the spatial component for each band. To obtain the spatial information for each band, we apply homomorphic filtering to each band of the denoised LRHS image. Through the use of homomorphic filtering, the low frequency component of each band of the denoised LRHS image is suppressed, and the high frequency component is extracted. Therefore, in this research, homomorphic filtering is applied to each band of the denoised LRHS image to extract the spatial component of each band. The homomorphic filtering processing is based on the following image imaging model:

$${(X_{\mathrm{LR}}^{\mathrm{RNH}})}^k={(X_{\mathrm{LR}\_\mathrm{H}}^{\mathrm{RNH}})}^k·{(X_{\mathrm{LR}\_\mathrm{L}}^{\mathrm{RNH}})}^k$$

(11)

for $k=1,2,\dots ,B$, where $X_{\mathrm{LR}\_\mathrm{H}}^{\mathrm{RNH}}$ represents the high frequency component of the denoised LRHS image, $X_{\mathrm{LR}\_\mathrm{L}}^{\mathrm{RNH}}$ represents the low frequency component, and ${(X_{\mathrm{LR}\_\mathrm{H}}^{\mathrm{RNH}})}^k$ and ${(X_{\mathrm{LR}\_\mathrm{L}}^{\mathrm{RNH}})}^k$ represent the $k\mathrm{th}$ band of $X_{\mathrm{LR}\_\mathrm{H}}^{\mathrm{RNH}}$ and $X_{\mathrm{LR}\_\mathrm{L}}^{\mathrm{RNH}}$, respectively. Based on Equation (4)-(6), Logarithmic transformation, Fourier transform, and high-pass filtering operations are applied to Equation (11):

$${(S_{\mathrm{LR}})}^k=[\Im (\ln {(X_{\mathrm{LR}\_\mathrm{H}}^{\mathrm{RNH}})}^k)]·H+\Im [\ln {(X_{\mathrm{LR}\_\mathrm{L}}^{\mathrm{RNH}})}^k]·H$$

(12)

for $k=1,2,\dots ,B$, where $S_{\mathrm{LR}}$ is the high-pass filtered imageh, ${(S_{\mathrm{LR}})}^k$ is the $k\mathrm{th}$ band of $S_{\mathrm{LR}}$, $\Im$ represents Fourier transform, and $H$ is the high-pass filter, defined as:

$$H(x,y)=({\beta }_{\mathrm{H}}-{\beta }_{\mathrm{L}})[1-\exp (-(D^2(x,y)/D_0^2)]+{\beta }_{\mathrm{L}}$$

(13)

where $D_0$ is the cut-off frequency, $D$ is the distance between $(x,y)$ and the center, ${\beta }_{\mathrm{H}}$ and ${\beta }_{\mathrm{L}}$ are the high and low frequency gains. Figure 3 shows the 3-D mesh of the high-pass filter. Since homomorphic filtering aims to reduce the low frequency component and extract the high frequency component, ${\beta }_{\mathrm{H}}$ is greater than 1 and ${\beta }_{\mathrm{L}}$ is smaller than 1. By adjusting the value of the cut-off frequency $D_0$, the sharpness of the transition between ${\beta }_{\mathrm{L}}$ and ${\beta }_{\mathrm{H}}$ can be controlled. In practice, the values of these parameters are generally determined empirically. In this paper, empirically, ${\beta }_{\mathrm{H}}$, ${\beta }_{\mathrm{L}}$, and $D_0$ are set to 2, 0.25, 40, respectively. $S_{\mathrm{LR}}$ is the high-pass filtered image in which the low frequency component has been weakened. Then, the spatial component of each band is obtained by applying the inverse Fourier transform and the exponential operation to $S_{\mathrm{LR}}$.

$${(X_{\mathrm{LR}}^{\mathrm{I}})}^k=\exp [{\Im }^{-1}({(S_{\mathrm{LR}})}^k)]$$

(14)

for $k=1,2,\dots ,B$, where $X_{\mathrm{LR}}^{\mathrm{I}}$ denotes the spatial component of each band of the denoised LRHS image, ${(X_{\mathrm{LR}}^{\mathrm{I}})}^k$ denotes the $k\mathrm{th}$ band of $X_{\mathrm{LR}}^{\mathrm{I}}$, and ${\Im }^{-1}$ denotes the inverse Fourier transform.

After introducing homomorphic filtering to obtain the spatial component of each band of the denoised LRHS image, a weighted root mean squared error (RMSE)-based method is presented to extract the spatial intensity information of the HS image. Let $I_{\mathrm{LR}}={\displaystyle {\sum }_{k=1}^B{\lambda }_k}{(X_{\mathrm{LR}}^{\mathrm{I}})}^k$ denote the total spatial information of the LRHS image, where $[{\lambda }_1,{\lambda }_2,\dots ,{\lambda }_B]$ is the weighted vector. To determine the values of the weighted vector, we utilize the RMSE index to measure the deviation of two images. A smaller value of RMSE indicates a better result, and the optimal value is 0. In the RMSE-based method, the spatial information of the PAN image is considered, and the RMSE value between the total spatial information $I_{\mathrm{LR}}$ and the PAN image $X^{\mathrm{PAN}}$ is calculated. The smallest value of RMSE is computed to obtain the optimal values of the weights $[{\lambda }_1,{\lambda }_2,\dots ,{\lambda }_B]$:

$$\min (\sqrt{\frac{1}{T}{\displaystyle \sum _{i=1}^T{[{({\displaystyle {\sum }_{k=1}^B{\lambda }_k}{(X_{\mathrm{LR}}^{\mathrm{I}})}^k)}_i-{(\downarrow X^{\mathrm{PAN}})}_i]}^2}}$$

(15)

where $T=m\times n$ represents the total pixel number of one band of the LRHS image, $\downarrow$ represents down-sampling operation, $\downarrow X^{\mathrm{PAN}}$ represents that the PAN image is down-sampled to the size of the LRHS image, ${(\downarrow X^{\mathrm{PAN}})}_i$ and ${({\displaystyle {\sum }_{k=1}^B{\lambda }_k}{(X_{\mathrm{LR}}^{\mathrm{I}})}^k)}_i$ represent the values of the $i\mathrm{th}$ pixel in $\downarrow X^{\mathrm{PAN}}$ and ${\displaystyle {\sum }_{k=1}^B{\lambda }_k}{(X_{\mathrm{LR}}^{\mathrm{I}})}^k$, respectively. The laplacian pyramid networks (LapSRN) [46] super-resolution method can effectively improve the spatial resolution of an image, and has the advantages of parameter sharing, local skip connections, and multi-scale training. So it is adopted to super-resolve the spatial information of the LRHS image $I_{\mathrm{LR}}$ for an $I_{\mathrm{HR}}$ with super-resolution spatial information.

3.3. Panchromatic Image Preprocessing and Total Spatial Information Acquisition

To make the spatial information of the PAN image clearer, the Laplacian of Gaussian (LOG) [47] image enhancement algorithm is applied to the PAN image, which uses a Gaussian filter to reduce noise followed by a Laplace operator for enhancement. Let $I_{\mathrm{s}}^{\mathrm{PAN}}$ represent the enhanced PAN image.

The HS and PAN images contain the different and complementary information for a scene. To acquire the total spatial information, we should consider simultaneously the spatial structure details of these two images, and then propose an optimized weighted tensor matrix-based method. $I_{\mathrm{HR}}$ and $I_{\mathrm{s}}^{\mathrm{PAN}}$ include the spatial structure information of the HS and PAN images, respectively. Based on Equation (2), for the multiple images $[I_{\mathrm{HR}},I_{\mathrm{s}}^{\mathrm{PAN}}]$, the weighted structure tensor matrix at pixel $p$ is given by:

$$M_{w,p}^{\mathrm{HP}}=\frac{1}{2}\left[\begin{array}{cc}{(\partial (I_{\mathrm{HR},p})/\partial x)}^2+{(\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial x)}^2& (\partial (I_{\mathrm{HR},p})/\partial x)(\partial (I_{\mathrm{HR},p})/\partial y)+(\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial x)(\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial y)\\ (\partial (I_{\mathrm{HR},p})/\partial x)(\partial (I_{\mathrm{HR},p})/\partial y)+(\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial x)(\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial y)& {(\partial (I_{\mathrm{HR},p})/\partial y)}^2+{(\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial y)}^2\end{array}\right]$$

(16)

where $M_w^{\mathrm{HP}}$ denotes the obtained weighted tensor matrix, $M_{w,p}^{\mathrm{HP}}$ denotes $M_w^{\mathrm{HP}}$ at pixel $p$, $\partial (I_{\mathrm{HR},p})/\partial x$, $\partial (I_{\mathrm{HR},p})/\partial y$, $\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial x$, and $\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial y$ are the $x$ and $y$ partial derivatives of $I_{\mathrm{HR}}$ and $I_{\mathrm{s}}^{\mathrm{PAN}}$ at pixel $p$, respectively. The weighted tensor matrix $M_{w,p}^{\mathrm{HP}}$ is semi-definite, and it can be decomposed as:

$$\begin{array}{cc}\hfill M_{w,p}^{\mathrm{HP}}& =\left[\begin{array}{cc}{e}_{w11,p}& e_{w21,p}\\ e_{w12,p}& e_{w22,p}\end{array}\right]\left[\begin{array}{cc}{v}_{w1,p}& 0\\ 0& v_{w2,p}\end{array}\right]{\left[\begin{array}{cc}{e}_{w11,p}& e_{w21,p}\\ e_{w12,p}& e_{w22,p}\end{array}\right]}^{\mathrm{T}}\hfill \\ & =v_{w1,p}{e}_{w1,p}{(e_{w1,p})}^{\mathrm{T}}+v_{w2,p}{e}_{w2,p}{(e_{w2,p})}^{\mathrm{T}}\hfill \end{array}$$

(17)

where ${()}^{\mathrm{T}}$ is the transpose operation, $v_{w1}$ and $v_{w2}$ are the two eigenvalues, $v_{w1,p}$ and $v_{w2,p}$ are the two eigenvalues at pixel $p$, $e_{w1,p}={[\begin{array}{cc}{e}_{w11,p}& e_{w12,p}\end{array}]}^{\mathrm{T}}$ and $e_{w2,p}={[\begin{array}{cc}{e}_{w21,p}& e_{w22,p}\end{array}]}^{\mathrm{T}}$ are the eigenvectors corresponding to the two eigenvalues at pixel $p$, respectively.

The two eigenvalues generally have a larger value and a smaller value. We assume that $v_{w1}$ is the larger eigenvalue. When $v_{w1}\approx v_{w2}\approx 0$, $v_{w1}>v_{w2}\approx 0$, and $v_{w1}>v_{w2}>0$, the structure region of this pixel are flat area, edge area, and corner, respectively. We test on some images to study the eigenvalues of weighted tensor matrix. Figure 4 shows the two eigenvalues at each pixel of the weighted tensor matrix for the Salinas scene data. It can be seen that for many pixels, the smaller eigenvalues shown in Figure 4d are approximately ${10}^{-5}$, and are very small. By experimenting on other numerous images, we have also discovered that the smaller eigenvalues are mostly very small. Thus, the approximation of $M_{w,p}^{\mathrm{HP}}$ is expressed as:

$${\tilde{M}}_{w,p}^{\mathrm{HP}}=\left[\begin{array}{cc}{e}_{w11,p}& e_{w21,p}\\ e_{w12,p}& e_{w22,p}\end{array}\right]\left[\begin{array}{cc}{v}_{w1,p}& 0\\ 0& 0\end{array}\right]{\left[\begin{array}{cc}{e}_{w11,p}& e_{w21,p}\\ e_{w12,p}& e_{w22,p}\end{array}\right]}^{\mathrm{T}}=v_{w1,p}{e}_{w1,p}{(e_{w1,p})}^{\mathrm{T}}$$

(18)

where ${\tilde{M}}_{w,p}^{\mathrm{HP}}$ is the approximation of $M_{w,p}^{\mathrm{HP}}$. Based on Equation (1), the structure tensor matrix satisfies that $M=\nabla I·\nabla I^{\mathrm{T}}$. The weighted gradient $G_w$ at pixel $p$ satisfies that ${\tilde{M}}_{w,p}^{\mathrm{HP}}=G_{w,p}·{(G_{w,p})}^{\mathrm{T}}$$=v_{w1,p}{e}_{w1,p}{(e_{w1,p})}^{\mathrm{T}}$. So, $G_{w,p}$ is deduced as $G_{w,p}=\sqrt{v_{w1,p}}·e_{w1,p}$. Since the direction of the eigenvector corresponding to $v_{w1}$ is not unique, the direction of the weighted gradient $G_{w,p}$ is also not unique. We specify the direction of the weighted gradient $G_{w,p}$ as the gradients average of the individual multiple source images $[I_{\mathrm{HR}},I_{\mathrm{s}}^{\mathrm{PAN}}]$:

$$G_{w,p}=\sqrt{v_{w1,p}}·e_{w1,p}·\mathrm{sign}\langle e_{w1,p},\frac{1}{2}(\nabla I_{\mathrm{HR},p}+\nabla I_{\mathrm{s},p}^{\mathrm{PAN}})\rangle$$

(19)

where $\nabla I_{\mathrm{HR},p}=[\begin{array}{cc}\partial (I_{\mathrm{HR},p})/\partial x& \partial (I_{\mathrm{HR},p})/\partial y\end{array}]$ and $\nabla I_{\mathrm{s},p}^{\mathrm{PAN}}=[\begin{array}{cc}\partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial x& \partial (I_{\mathrm{s},p}^{\mathrm{PAN}})/\partial y\end{array}]$, $\langle \cdot ,\cdot \rangle$ represents the inner product of two vectors, and $\mathrm{sign}(\cdot )$ represents the sign function. Once the weighted gradient $G_{w,p}$ is acquired from the multiple images $[I_{\mathrm{HR}},I_{\mathrm{s}}^{\mathrm{PAN}}]$, an optimization model is proposed to obtain the total spatial information $I_{\mathrm{T}}^{\mathrm{HP}}$ as:

$$\min {\Vert (\nabla I_{\mathrm{T}}^{\mathrm{HP}})-G_w\Vert }^2$$

(20)

where $\nabla I_{\mathrm{T}}^{\mathrm{HP}}=[\begin{array}{cc}\partial I_{\mathrm{T}}^{\mathrm{HP}}/\partial x& \partial I_{\mathrm{T}}^{\mathrm{HP}}/\partial y\end{array}]$, $\partial I_{\mathrm{T}}^{\mathrm{HP}}/\partial x$, and $\partial I_{\mathrm{T}}^{\mathrm{HP}}/\partial y$ denote the $x$ and $y$ partial derivatives of $I_{\mathrm{T}}^{\mathrm{HP}}$. Equation (20) is an unconstrained optimization problem, and we solve it by the conjugate gradient method. Equation (20) can effectively ensure that the total spatial information $I_{\mathrm{T}}^{\mathrm{HP}}$ contains the spatial structure details of both the HS and PAN images.

3.4. Fused High Spatial Resolution Hyperspectral Image Generation

The LRHS image $X_{\mathrm{LR}}^{\mathrm{HS}}$ is interpolated to the scale of the HRPAN image. By constructing a suitable gain matrix $R$, the total spatial information $I_{\mathrm{T}}^{\mathrm{HP}}$ is injected into the interpolated HS image to generate the fused HRHS image $X_{\mathrm{HR}}^{\mathrm{HS}}$. For the gain matrix $R$, it is beneficial to preserve the ratio of each HS pair band unchanged to reduce the spectral distortion. Thus, $R$ should satisfy that $R^k\propto \frac{{(X_{\mathrm{IN}}^{\mathrm{HS}})}^k}{(1/B){\displaystyle {\sum }_{k=1}^B{(X_{\mathrm{IN}}^{\mathrm{HS}})}^k}}$, where $X_{\mathrm{IN}}^{\mathrm{HS}}$ is the interpolated HS image, ${(X_{\mathrm{HR}}^{\mathrm{HS}})}^k$ and $R^k$ are the $k\mathrm{th}$ band of $X_{\mathrm{IN}}^{\mathrm{HS}}$ and $R$, respectively. Then, a tradeoff parameter $\epsilon$ is defined to regulate the amount of the injected details to reduce the spatial distortion. This process can be expressed as:

$${(X_{\mathrm{HR}}^{\mathrm{HS}})}^k={(X_{\mathrm{IN}}^{\mathrm{HS}})}^k+R^k·I_{\mathrm{T}}^{\mathrm{HP}}={(X_{\mathrm{IN}}^{\mathrm{HS}})}^k+\epsilon ·\frac{{(X_{\mathrm{IN}}^{\mathrm{HS}})}^k}{(1/B){\displaystyle {\sum }_{k=1}^B{(X_{\mathrm{IN}}^{\mathrm{HS}})}^k}}·I_{\mathrm{T}}^{\mathrm{HP}}$$

(21)

for $k=1,2,\dots ,B$, where $X_{\mathrm{HR}}^{\mathrm{HS}}$ is the fused HRHS image, and ${(X_{\mathrm{HR}}^{\mathrm{HS}})}^k$ is the $k\mathrm{th}$ band of $X_{\mathrm{HR}}^{\mathrm{HS}}$.

4. Experimental Results and Discussion

4.1. Datasets and Experimental Setup

In order to evaluate the effectiveness of the proposed HFWT hyperspectral pansharpening method (named as HFWT), experiments were performed on two simulated hyperspectral datasets which were a Washington DC and a Salinas scene, and one real hyperspectral dataset, the Hyperion dataset. The Salinas scene hyperspectral dataset was collected by Airborne Visible Infrared Imaging Spectrometer (AVIRIS) [48], and the Washington DC dataset was acquired by the Spectral Information Technology Application Center of Virginia. The used real dataset is provided by the EO-1 spacecraft. The EO-1 spacecraft has a Hyperion instrument which provides the real LRSH images and an Advanced Land Imager (ALI) instrument acquires the HRPAN images [48].

Table 1. Characteristic of the datasets.

Dataset	Reference HS Size	Simulated Size	Reference HS Spatial Resolution (SR)	Simulated SR	Band Number	Spectral Range
Washington DC	200 × 200	PAN 200 × 200	3m	PAN 3m	191	0.4–2.5 $\mathsf{\mu }\mathrm{m}$
		HS 50 × 50		HS 12m
Salinas scene	200 × 200	PAN 200 × 200	3.7 m	PAN 3.7 m	204	0.4–2.4 $\mathsf{\mu }\mathrm{m}$
		HS 50 × 50		HS 18.5 m
Dataset	Real HS Size	Real PAN size	Real HS SR	Real PAN SR	Band Number	Spectral Range
Hyperion	100 × 100	300 × 300	30 m	10 m	174	0.4–2.5 $\mathsf{\mu }\mathrm{m}$

lists the characteristics of each dataset.

The proposed HFWT method is compared with several state-of-the-art hyperspectral pansharpening methods: Gram-Schmidt (GS) [23], guided filter principal component analysis (GFPCA) [33], coupled nonnegative matrix factorization (CNMF) [18], Bayesian sparsity promoted Gaussian prior (Bayesian) [13], and HySure [14]. Four typical quantitative evaluation indexes are adopted: cross correlation (CC) [49], spectral angle mapper (SAM) [50], root mean squared error (RMSE), and erreur relative globale adimensionnelle desynthèse (ERGAS) [51]. The CC and SAM measure the spectral and spatial distortion, respectively. The larger value of CC and the smaller value of SAM indicate the better fusion result. The RMSE and ERGAS are the global indexes that measure both the spatial and spectral performance, and their value ranges are all (0,1), with 0 being the optimal value.

In order to perform the objective fusion evaluation of the simulated hyperspectral datasets, the available HS image is used as the reference HS image. The simulated LRHS image and HRPAN image are generated according to Wald’s protocol [52,53]. The reference HS image is blurred and down sampled 4 times to obtain the simulated HS image. The simulated PAN image is obtained by averaging the visible light band of the reference HS image.

For the real datasets, the real LRHS and HRPAN images are available, and the reference high resolution HS image is not available. In order to test the objective quality of the real hyperspectral images, the real LRHS image is served as the reference image. The real LRHS and HRPAN images available are degraded, and the two degraded images are fused to obtain a fused image. This fused image is compared to the real LRHS image to evaluate the objective quality.

In the proposed HFWT method, we define a tradeoff parameter $\epsilon$ which regulates the amount of the injected details and ensures spatial performance. In practice, the optimal value is determined based on experience. By adjusting different values of $\epsilon$, the optimal value can be determined by the fusion result. In this paper, by experience, the values of tradeoff parameter $\epsilon$ are set as 0.25, 0.05 and 0.2 for the Washington DC, Salinas scene and Hyperion dataset, respectively.

4.2. Validity Discussion of the Open-Closing Denoising Operation

To verify the effectiveness of the open-closing HS image denoising operation, the proposed HFWT method was conducted on the Washington DC dataset with different denoising processing. The compared denoising algorithms contain average filtering, Gaussian filtering, open operation, closed operation, and open-closing operation.

Table 2. Performance of the HFWT method with different HS image denoising processing. (The best values of each index are marked in bold.)

Index	No Denoising	Average	Gaussian	Open	Closed	Open-Closing
RMSE	0.0125	0.0119	0.0123	0.0121	0.0123	0.0112
SAM	6.8506	6.8507	6.8506	6.8507	6.8506	6.8506
CC	0.9156	0.9176	0.9176	0.9175	0.9176	0.9176
ERGAS	25.5071	25.4812	25.4822	25.5052	25.4880	25.4804

shows the fusion performance of different image denoising processing. As outlined in Table 2, the HFWT method without HS image denoising had the worst fusion results. The proposed methods with each HS image denoising preprocessing have the better fusion results compared with the proposed method without HS image denoising, which demonstrates that the HS image denoising preprocessing is significative and effective. By contrast, the proposed HFWT method with the open-closing operation achieves the best fusion performance, and it demonstrates that the open-closing operation is an effective HS image denoising preprocessing.

4.3. Experiments on Simulated Hyperspectral Datasets

Figure 5 shows the fusion experimental results for the Washington DC dataset, where Figure 5(a1) shows the reference HS image, and the subjective fused HS images of each method are displayed in Figure 5(b1–g1). Moreover, Figure 5(a2) shows the enlarged subareas of the reference HS image, and the two enlarged subareas of each fused image are shown in Figure 5(b2–g2). The reference error image is shown in Figure 5(a3), and the error images between each fused HS image and the reference HS image are reported in Figure 5(b3–g3). Except for the first column, each column in Figure 5 shows the experimental results corresponding to each method. By visually comparing the fused HS images with the reference HS image, the fused result of the GS method suffers from serious spectral distortion. For example, the two enlarged subareas of the GS method are distorted seriously. The GFPCA approach generates fuzzy spatial details in some regions, such as the enlarged subareas shown in Figure 5(c2). This is because the spatial information of the GFPCA approach is injected insufficiently. As depicted in Figure 5(d1,d2), the spatial information of the fused image is well enhanced using the CNMF method, but some slight spectral distortion is appeared in the roof of the buildings. A closer inspection revealed that the HySure method seems to generate some distortion in the circular building in the upper left corner. By contrast, the fused HS images obtained by the Bayesian and HFWT methods achieve superior performance in terms of both spectral and spatial aspects. In order to further compare the performance of each fusion method, the third row of Figure 5 shows the error images of different methods. The error image is the difference (absolute value) of pixel values between the fused HS images and the reference HS image. We can see that, the GS, GFPCA and CNMF methods have larger differences, the HySure and Bayesian approaches generated relatively smaller differences, and the proposed HFWT approach shows the smallest differences in most areas, which demonstrates the excellent fusion capacity of the proposed method.

Similar to the previous experiments, for the Salinas scene dataset, the fused results are shown in Figure 6. Figure 6(a1–a3) show the reference HS image, the enlarged subareas of the reference HS image, and the reference SAM image, respectively. Figure 6(b1–g1) in the first row show the subjective pansharpened results of each algorithm, and Figure 6(b2–g2) in the second row display each enlarged subarea. The SAM images of each approach are shown in Figure 6(b3–g3). The reference SAM image of enlarged subarea is shown in Figure 6(a4), and the SAM images of enlarged subarea obtained by each method are shown in Figure 6(b4–g4). The spectral distortion caused by the GS method is very obvious, and the degree of spatial enhancement is also not acceptable for the GS method, as depicted on Figure 6(b1,b2). Compared with the GS approach, the GFPCA method performs better in terms of the spectral quality. However, the fused HS image obtained by the GFPCA approach shows an indistinct area in the left region of Figure 6(c1). Despite having a preeminent spatial quality, the CNMF method generates significant spectral distortion in the triangle region in the lower half of Figure 6(d1). From the visual analysis, the HySure, Bayesian and HFWT methods effectively improve spatial performance while maintaining spectral information, and the HFWT method shows better spectral quality compared with the HySure and Bayesian methods in some regions, such as the upper area of the enlarged subarea. The SAM images and the SAM images of the enlarged subareas of different approaches are shown in the third and fourth rows of Figure 6, to further verify the fusion performance of the proposed method. It can be seen that the proposed HFWT method yields the lowest SAM values for most regions. These results demonstrate that the proposed HFWT algorithm performs well in both the spatial and spectral aspects.

In addition to visual inspection, the performance of each algorithm for the Washington DC and Salinas scene datasets is analyzed quantitatively in

Table 3. Objective quality evaluation of each method for the simulated datasets. (The best values of each index are marked in bold.)

Dataset	Index	Method
		GS	GFPCA	CNMF	HySure	Bayesian	HFWT
Washington DC	RMSE	0.0114	0.0139	0.0116	0.0097	0.0117	0.0112
	SAM	7.1069	9.8467	7.6438	6.8601	7.2586	6.8506
	CC	0.8856	0.8179	0.8879	0.9018	0.8954	0.9176
	ERGAS	36.0732	37.7303	34.3126	26.5678	29.3497	25.4804
Salinas scene	RMSE	0.0426	0.2224	0.0162	0.0163	0.0167	0.0138
	SAM	3.7807	2.9814	1.7586	1.7039	1.8015	1.5460
	CC	0.8542	0.9429	0.9544	0.9583	0.9515	0.9625
	ERGAS	4.3301	3.0843	2.6346	2.4451	2.9079	2.5312

, where the best results for each quantitative index are marked in bold. As can be seen from Table 3, the objective quantitative results are roughly consistent with the subjective qualitative effects. Same as the subjective results, the GS and GFPCA algorithms produce worse objective performance compared with other algorithms. The HySure approach obtains the best RMSE value for the Washington DC dataset and the optimal ERGAS value for the Salinas scene dataset. Most of the quality indexes generated by applying the proposed HFWT method are the best, in which the SAM, CC and ERGAS values are the best for Washington DC, and the RMSE, SAM and CC indexes are ranked first for the Salinas scene.

4.4. Experiments on Real Hyperspectral Datasets

Figure 7 shows the pansharpened images of each method for the Hyperion dataset to confirm the fusion performance of the proposed HFWT method in the real dataset. Figure 7a–c show the real HS, real PAN, and interpolated HS images, respectively. The GS method shown in Figure 7d generates obvious spectral distortion, especially in the wharf area. In spite of good spatial improvement, the spatial details shown in the fused images obtained by using the GS and HySure approaches are too sharp. Spectral quality of the GFPCA and Bayesian methods seems be acceptable, but the GFPCA and Bayesian methods perform poorly from the spatial aspect. By contrast, the subjective effects of the CNMF and HFWT approaches are the best, and the HFWT method yields better spatial capacity compared to the CNMF method. The objective quality evaluation for the Hyperion dataset are presented in

Table 4. Objective quality evaluation of each method for the real dataset. (The best values of each index are marked in bold.)

Index	Method
	GS	GFPCA	CNMF	HySure	Bayesian	HFWT
RMSE	0.0426	0.0451	0.0368	0.0398	0.0387	0.0358
SAM	11.1037	15.6849	12.2755	12.8308	12.9605	9.1809
CC	0.9296	0.9241	0.9631	0.9443	0.9504	0.9821
ERGAS	15.3536	16.3011	11.1116	12.3441	12.2353	11.1109

. As reported in Table 4, the HFWT method provides the best quantitative evaluation results in terms of the RMSE, SAM, CC, and ERGAS indices, which indicates that the HFWT method successfully maintains the spectral information of the original LRHS image and improves the spatial resolution.

4.5. Computational Complexity Analysis and Time Comparisons

The proposed HFWT algorithm contains simple sequential statements, several loop statements without nesting, and a two-layer loop statements with nesting (In the proposed HFWT algorithm, the program statement of Equation (19) which are applied on each pixel is a two-layer loop statements). The simple sequential statement is naturally $O(1)$ time, and the loop statement without nesting is $O(n)$ time. The two-layer loop statement is $O(n^2)$. According to the summation rule of algorithm complexity, the total algorithm complexity is $O(n^2+n+1)=O(n^2)$. The proposed HFWT algorithm which is $O(n^2)$ time belongs to the polynomial time, and is considered a fast algorithm. The computing time (in seconds) of each method for three datasets is shown in

Table 5. Computing time (seconds) of each method.

Dataset	Method
	GS	GFPCA	CNMF	HySure	Bayesian	HFWT
Washington DC	1.1764	2.3455	8.8369	43.3926	70.5347	6.9068
Salinas scene	2.3953	4.8471	8.9498	55.4224	71.4972	7.1413
Hyperion	2.6618	7.5118	23.3787	117.1729	158.7151	10.6448

. The experiments in this paper were all performed using MATLAB R2015b, and tested on a PC with an Intel Core i5-7300HQ CPU @ 2.50 GHz and 8 GB of memory. The GS and GFPCA methods are very efficient, but the fusion performance of the GS and GFPCA methods is unsatisfactory. The proposed HFWT method is faster than the CNMF algorithm, and takes much less computing time than the HySure and Bayesian algorithms. The time cost of the proposed HFWT is acceptable.

5. Conclusions

This paper presents a novel hyperspectral pansharpening method based on the merger of the homomorphic filtering and weighted tensor matrix. The proposed HFWT algorithm introduces the open-closing morphological operation and homomorphic filtering to remove noise and extract spatial information of each band of an HhS image, respectively. Moreover, we propose a weighted RMSE-based method to obtain the total spatial information of the HS image. In order to generate the adequate spatial information from both the HS and the corresponding PAN images, an optimized weighted tensor matrix based method is proposed. Specifically, the weighted tensor matrix, eigenvalues and eigenvectors are deduced and analyzed to obtain the weighted gradient, and an optimization model is presented to acquire the integrated spatial information. Compared with the state-of-the-art methods, experiments performed on the Washington DC, Salinas scene and Hyperion datasets demonstrate the proposed method performs superiorly in terms of both subjective and objective assessment.