Deep Pansharpening via 3D Spectral Super-Resolution Network and Discrepancy-Based Gradient Transfer

Abstract

High-resolution (HR) multispectral (MS) images contain sharper detail and structure compared to the ground truth high-resolution hyperspectral (HS) images. In this paper, we propose a novel supervised learning method, which considers pansharpening as the spectral super-resolution of high-resolution multispectral images and generates high-resolution hyperspectral images. The proposed method learns the spectral mapping between high-resolution multispectral images and the ground truth high-resolution hyperspectral images. To consider the spectral correlation between bands, we build a three-dimensional (3D) convolution neural network (CNN). The network consists of three parts using an encoder–decoder framework: spatial/spectral feature extraction from high-resolution multispectral images/low-resolution (LR) hyperspectral images, feature transform, and image reconstruction to generate the results. In the image reconstruction network, we design the spatial–spectral fusion (SSF) blocks to reuse the extracted spatial and spectral features in the reconstructed feature layer. Then, we develop the discrepancy-based deep hybrid gradient (DDHG) losses with the spatial–spectral gradient (SSG) loss and deep gradient transfer (DGT) loss. The spatial–spectral gradient loss and deep gradient transfer loss are developed to preserve the spatial and spectral gradients from the ground truth high-resolution hyperspectral images and high-resolution multispectral images. To overcome the spectral and spatial discrepancy between two images, we design a spectral downsampling (SD) network and a gradient consistency estimation (GCE) network for hybrid gradient losses. In the experiments, it is seen that the proposed method outperforms the state-of-the-art methods in the subjective and objective experiments in terms of the structure and spectral preservation of high-resolution hyperspectral images.

1. Introduction

A hyperspectral (HS) image is a spatial–spectral data cube, which consists of spatial and spectral information. The large amount of spectral information in HS images improves the performance of target detection [1], denoising [2], and image classification [3,4] compared to multispectral (MS) images (such as RGB images). Thus, high-resolution (HR) HS images are required in a vast amount of remote sensing applications. However, capturing the HR HS images is still a challenging task because of technical limitations. One of the methods is the single-image super-resolution method, which generates the HR HS images from low-resolution (LR) HS images. Because of the technical developments, multi-sensor fusion has attracted more attention in recent years. MS images mainly focus on spatial resolution with a few bands, while HS images provide a large number of bands with low resolution. Thus, pansharpening (i.e., the fusion of HR MS images and LR HS images) emerges in the remote sensing area. Compared to single-image super-resolution, pansharpening provides more accurate HR HS images with auxiliary HR MS images.

2. Related Work

So far, traditional pansharpening methods can be mainly divided into four parts: component substitution (CS), multiresolution analysis (MRA), Bayesian methods, and variational methods. CS-based methods split the HS and MS images into spatial and spectral domains. The spatial information of MS images was utilized to enhance the spatial resolution of HS images [5]. Yang et al. proposed HS computational imaging using collaborative Tucker3 tensor decomposition [6]. The Tucker3 tensor decomposition was developed to model the low rankness and similarities of nonlocal patches. The spatial factor matrices and the core tensor of panchromatic images by Tucker3 tensor decomposition are utilized to constrain the spatial structure of HS images. MRA methods first employed multiscale filters to decompose the MS and HS images. They transferred the structures of MS images to HS images in each sub-band. Several decomposition methods were utilized such as high-pass filters, generalized Laplacian pyramids, and undecimated discrete wavelet transform [7,8,9]. Bayesian methods model a posterior probability of HR HS images and utilize Bayesian theory to predict the estimated HR images [10]. The variational approach was derived from Bayesian methods. Researchers introduced some suitable prior knowledge to probability terms and then transformed the Bayesian problem into an optimization problem, e.g., a dynamic sparsity regularizer [11] and multi-order gradient regularization [12].

In recent years, with the rise of the machine learning methods, some learning-based methods have been introduced to the pansharpening problem. Among them, more and more researchers employ deep learning to improve the performance of pansharpening. Masi et al. borrowed the idea from image super-resolution and built a three-layer convolutional network for pansharpening [13]. Shao et al. [14] considered the pansharpening problem as image fusion. They proposed two branch network architectures, which were used to extract the features from LR HS images and HR MS images, respectively. Then, two features were fused to generate the high-resolution HS results. Some researchers employed deep learning to learn the prior of HR HS images and then imported the learned prior into the optimization problem. Li et al. [15] proposed the detail-based deep Laplacian pansharpening method to improve the spatial resolution of HS images. The spatial and spectral details were learned by deep learning. Dian et al. utilized the residual learning to learn the image prior [16], and Xie et al. built the elaborated neural network to learn the high frequency of HR HS images [17]. Wang et al. integrated learned deep priors into the objective function derived from the degradation model [18]. The deep priors were learned using a two-stream fusion network. The regularization parameter was automatically selected using a golden section search strategy. Other researchers focused on the design of the loss function. Luo et al. [19] proposed an unsupervised convolutional neural network architecture with a set of specific design losses for spectral and spatial constraints between MS and HS images. Zhou et al. proposed the perceptual loss for an unsupervised pansharpening method [20]. The perceptual loss consisted of the pixel level loss, feature level loss, and GAN loss. Finally, some researchers considered pansharpening as the spectral super-resolution. Ozcelik et al. employed the colorization concept for HS pansharpening and proposed the panchromatic image colorization based on a generative adversarial network (GAN) [21]. However, they only considered one to three spectral channels’ mapping and utilized a 2D CNN without considering spectral correlations.

In this paper, we propose a deep HS pansharpening method based on the spectral super-resolution. The network is based on a 3D encoder–decoder framework, which consists of spatial/spectral feature extraction, feature transform, and image reconstruction. A 3D CNN is employed to consider the spectral correlation with thespectral super-resolution. Moreover, we propose the discrepancy-based deep hybrid gradient (DDHG) losses, which transfer the HR HS and HR MS image gradients considering the discrepancy. Finally, it is shown by experiments that the proposed method achieves the best performance in structure and spectral preservation from the HR HS images compared to the state-of-the-art methods. The main contributions of this paper are summarized as follows:

• We propose a supervised 3D-CNN-based spectral super-resolution network of HR MS images for pansharpening. The 3D CNN is employed to consider the spatial and spectral correlation simultaneously with thespectral super-resolution. Compared to the state-of-the-art methods, extensive experiments show that the proposed method achieves the best performance in spectral and spatial preservation from HR HS images.
• The 3D spectral super-resolution network constructs the encoder–decoder framework. In the decoder part, we design the image reconstruction network with a set of skip connections and spatial–spectral fusion (SSF) blocks, which fuse the spatial and spectral features efficiently.
• We define the discrepancy-based deep hybrid gradient (DDHG) losses, which contain the spatial–spectral gradient (SSG) loss and deep gradient transfer (DGT) losses. The losses are developed to constrain the spatial and spectral consistency from the ground truth HS images and HR MS images. To overcome the spatial and spectral discrepancy between two images, we design the spectral downsampling (SD) network and gradient consistency estimation (GCE) network in the DDHG losses.

The remaining part of this paper is organized as follows. Section 2 presents the proposed method in detail. Section 3 introduces the experimental results and discusses the performance of the proposed method. Finally, Section 4 provides the conclusion.

3. Proposed Method

Given LR HS images $I\in {\mathbf{R}}^{w\times h\times L}$ and HR MS images $I_M\in {\mathbf{R}}^{W\times H\times l}$, the objective of pansharpening problem is to generate the HR HS images $I^{\prime }\in {\mathbf{R}}^{H\times W\times L}$. $(w,h)$ and $(W,H)$ are the width and height of LR HS images and HR MS images. The relationship of the width and height between HS and MS images is $W=s\times w$ and $H=s\times h$, where s is the scale factor. L and l are the channels of the HS and MS images. Thus, the main target is to generate the resulting images with high spatial and spectral resolution. In this paper, we considered the pansharpening problem as the spectral super-resolution, which enhances the spectral channels of HR MS images. We built the 3D-CNN-based encoder–decoder neural network, which extracts the spatial and spectral features from the input HR MS and LR HS images and then reuses them in image reconstruction. The proposed method considers a more general case with arbitrary spectral channels’ enhancement and employs 3D CNN to take the spectral correlation into account. Moreover, we define the DDHG loss function to minimize the spatial and spectral distortion considering the spectral and spatial discrepancy.

3.1. Motivation

Figure 1 shows HR MS images and HR HS images at bands 3 and 27 for test image paints and chart and stuffed toy in the CAVE dataset [22]. It is shown that HR MS images provide a sharper structure and detail compared to the ground truth HS images (see the red blocks in the first row of Figure 1). Moreover, we provide the discrete entropy (DE) measurement of the CAVE and Chikusei datasets [23]. DE measures the detail and structure sharpness of a one-channel image [24], and thus, we developed average DE (ADE) scores on all channels of MS and HS images as follows:

$$H\left(p\right)=-{\displaystyle \frac{1}{c}}\sum _{i=1}^{n-1}\sum _{j=0}^{c-1}p(i,j){log}_{2}p(i,j)$$

(1)

where $p(i,j)$ is the probability density function of the histogram at the pixel intensity i and the channel j. i is the pixel intensity from 0 to n. j is the image channel from 1 to c. n is the maximum pixel value, and c is the channel number of HS/MS images.

Table 1. DE evaluation of the CAVE and Chikusei datasets.

Dataset	CAVE	Chikusei
HR MS images	6.41	7.07
HR HS images	5.21	4.58

evaluates the average detail and structure sharpness of HS and MS images on two datasets. It is seen that HR MS images contain sharper detail and structure than HR HS images.

However, HS and MS images also contain a large spatial (gradient) discrepancy (see the red blocks in the second row of Figure 1). A large spatial discrepancy leads to the gradient distortion in the generated HS images. HS and MS images contain different numbers of spectral channels, which cannot transfer the gradient of the MS images to the generated HS images. Therefore, we propose a novel pansharpening via the spectral super-resolution of HR MS images considering the spectral and spatial discrepancy between two images.

3.2. Spectral Super-Resolution Network

The architecture of the proposed network is shown in Figure 2, which depends on a modified and expanded version of U-Net [25]. With the input of LR HS images I and HR MS images $I_M$, the proposed spectral super-resolution network (SSRN) G generates the resulting HR HS images $I^{\prime }$ ($I^{\prime }=SSRN(I,I_M;{\theta }_G)$, where ${\theta }_G$ is the weight and bias parameters in the network G. These parameters are utilized while training and testing the network G. The LR HS images are upsampled using bicubic interpolation to have the same spatial size as the HR MS images. The HR MS images are expanded using the duplication operation, and the expanded HR HS images contain the same spectral channels as the HR HS images. The proposed neural network employs a 3D CNN considering spatial and spectral correlation simultaneously with thespectral super-resolution of the HR MS images. We constructed the 3D encoder–decoder framework, which encodes the spatial and spectral feature and decodes the fused spatial–spectral feature to reconstruct the result HS images. The SSR network consists of three parts: spatial/spectral feature extraction, feature transform, and image reconstruction. The spatial/spectral feature extraction (SFE) network introduces the 3D-Conv blocks to extract the spatial and spectral features from the HR MS and LR HS images. The feature transform (FT) network employs 3D-residual blocks in the middle network and transforms the features to prepare for pansharpened image reconstruction. The image reconstruction (IR) network mainly generates the spectrally enhanced images from the extracted spatial, spectral, and reconstruction features. It consists of two architectures: spatial and spectral fusion (SSF) blocks and 3D upsampling blocks. The former one is developed to fuse the extracted spatial feature $S{a}_i$ and spectral features $S{e}_i$ with the reconstruction features $c_i$ in each layer. The latter one is employed to reconstruct the resulting images with enhanced spatial and spectral resolutions. We summarize the sub-neural network block of the proposed network as follows (see the Figure 3):

• The 3D-Conv blocks: In the SFE network, we employed the 3D-Conv blocks with the kernel size of $(3,3,3)$ and stride of $(2,2,2)$. Then, the leaky rectified linear unit (LReLU) activation is used. The 3D-Conv block is introduced to extract the spatial–spectral feature $S{a}_i$ and $S{e}_i$ from HR MS image $I_M$ and LR HS image I.
• The 3D residual blocks: In the FT network, 3D residual blocks are used to learn the extracted feature more efficiently in deep layers via residual learning [26]. In each block, we applied the 3D-Conv blocks with a kernel size of $(3,3,3)$ and stride of $(1,1,1)$. The ReLU layers are utilized as the output layer. The addition layer is employed to concatenate the input feature and the learned residual feature.
• The spatial–spectral fusion (SSF) blocks: In the image reconstruction network, the SSF blocks with a set of skip connections are utilized to fuse the reconstructed feature $c_i$ with the extracted spatial feature $S{a}_i$ and the spectral feature $S{e}_j$ efficiently. It can overcome the extracted feature distortion in the feature transform network. We utilized the concatenate operation and addition operation to fuse the spatial feature and the spectral feature respectively. We employed the 3D-Conv layer (kernel size: $(3,3,3)$ and stride: $(1,1,1)$) with the ReLU activation to transform the fused feature.
• The 3D upsample blocks: After the SSF blocks, we introduced the 3D upsample blocks to increase the spatial and spectral resolution. In each block, the 3D upsample layer with the nearest neighbor interpolation is first applied to generate the upsampled feature with enhanced spatial and spectral resolution. Then, the 3D-Conv layer (kernel size: $(3,3,3)$ and stride: $(1,1,1)$) with the ReLU activation is employed to generate the reconstructed feature $c_i$. The 3D upsample blocks are applied to provide the spatial and spectral super-resolution of the reconstructed feature $c_i$ and obtain the resulting images $I^{\prime }$.

3.3. Loss Function

Designing a suitable loss function is most important to improve the performance of the proposed neural network. The proposed loss function was developed to preserve the spatial and spectral content from the ground truth HS images. The sharp structure of HR MS images can be transferred to the generated HS images considering the spatial and spectral discrepancy between two images. We define the loss function, which consists of three parts, as follows:

$$L_{loss}={\alpha }_1{l}_p+{\alpha }_2{l}_{SSG}+{\alpha }_3{l}_{DGT}$$

(2)

(1)

Pixel loss: The pixel loss function $l_p$ enforces the pixel intensity consistency (content loss), which is defined as the $l_1$-norm between the reconstructed image $I^{\prime }$ and the ground truth $I^{gt}$:

$$l_p=\frac{1}{WHL}\sum _{i=1}^W\sum _{j=1}^H\sum _{k=1}^L{|I_{i,j,k}^{\prime }-I_{i,j,k}^{gt}|}_1$$

(3)

where i, j, and k are the pixel location in the image. The $l_1$-norm has been widely used in super-resolution with less blurring results compared to the $l_2$-norm [27].

(2)

Spatial–spectral gradient (SSG) loss: To enforce the gradient consistency in terms of the spatial and spectral domains from the ground truth, we designed the 3D spatial–spectral gradient loss using the $l_1$-norm as follows:

$$\begin{array}{ccc}\hfill l_{SSG}& =& \frac{1}{(W-1)(H-1)(L-1)}\hfill \\ \hfill & ·& \sum _{i=1}^{W-1}\sum _{j=1}^{H-1}\sum _{k=1}^{L-1}(1-W_{i,j,k}^x)\times {|G_x{\left(I^{\prime }\right)}_{i,j,k}-G_x{\left(I^{gt}\right)}_{i,j,k}|}_1\hfill \\ \hfill & +& (1-W_{i,j,k}^y)\times {|G_y{\left(I^{\prime }\right)}_{i,j,k}-G_y{\left(I^{gt}\right)}_{i,j,k}|}_1\hfill \\ \hfill & +& |G_z{\left(I^{\prime }\right)}_{i,j,k}-G_z{\left(I^{gt}\right)}_{i,j,k}{|}_1)\hfill \end{array}$$

(4)

where $G_x$, $G_y$, and $G_z$ are the gradient operators (i.e., forward difference operator) to obtain the x-, y-, and z-direction gradients of images. $I^{gt}$ is the ground truth HS image. This loss can preserve the spatial and spectral gradients from the ground truth HS images. $W_{i,j,k}^x$ and $W_{i,j,k}^y$ measure the spatial gradient consistency estimation between the reconstructed HS images and HR MS images (we mention them in the next section).

(3)

Deep gradient transfer (DGT) loss: deep gradient transfer loss is designed to transfer the structure of HR MS images to the reconstructed HS images. However, HS and MS images have a large discrepancy in the spatial gradient and spectral channels. To overcome the spatial and spectral discrepancy, the GCE net and SD net are utilized while transferring the structure of the HR MS images. The DGT loss is designed as follows:

$$\begin{array}{ccc}\hfill & l_{DGT}& =\frac{1}{(W-1)(H-1)}\hfill \\ \hfill & ·& \sum _{i=1}^{W-1}\sum _{j=1}^{H-1}(W_{i,j,k}^x\times {|G_x{\left(I_d^{\prime }\right)}_{i,j,k}-G_x{\left(I_M\right)}_{i,j,k}|}_1\hfill \\ \hfill & +& W_{i,j,k}^y\times |G_y{\left(I_d^{\prime }\right)}_{i,j,k}-G_y{\left(I_M\right)}_{i,j,k}{|}_1)\hfill \end{array}$$

(5)

where $I_d^{\prime }$ is the spectral downsampled version of the reconstructed HS images. It is generated by $SDN(·)$, which is the spectral downsample network (SD network) (see Figure 4) with several 3D-Conv blocks (see the left subfigure in Figure 3) (kernel size: (3, 3, 3) and stride: (1, 1, 2)). The output images have the same spectral channels as the HR MS images. To tackle the spatial discrepancy, the deep spatial gradient consistency (i.e., $W_{i,j,k}^x$ and $W_{i,j,k}^y$) between the spectral downsampled HS images and HR MS images is estimated as follows:

$$\begin{array}{ccc}\hfill W_{i,j,k}^x& =& \alpha \times GCEN(G_x{\left(I_d^{\prime }\right)}_{i,j,k},G_x{\left(I_M\right)}_{i,j,k})\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill W_{i,j,k}^y& =& \alpha \times GCEN(G_y{\left(I_d^{\prime }\right)}_{i,j,k},G_y{\left(I_M\right)}_{i,j,k})\hfill \end{array}$$

(7)

where $GCEN(·)$ is the gradient consistency estimation (GCE) network with one multiplication operation, one $l_1$ absolute norm operation, and several 3D-Conv blocks (see the right subfigure in Figure 4) (kernel size: (3, 3, 3) and the stride: (1, 1, 1)). The multiplication operation is utilized to obtain the consistency gradient of the downsampled HS images and the HR MS images. Then, the $l_1$ absolute norm of the consistency gradient is obtained. Finally, the convolution symbol is included in the GCE network, which learns the consistency structure from the multiplication of the two image gradients. $\alpha$ is the weight of the output of GCE network and is set to 10 in the proposed method.

The GCE network was developed to learn the gradient consistency between two images, and then, the HR MS image gradient is selectively transferred based on the gradient consistency. A large gradient consistency value (i.e., $W_{i,j,k}^x$ and $W_{i,j,k}^y$) means two images have the consistency gradient, and thus, the proposed method provides the strong HR MS image gradient transfer to the reconstructed HS images. On the contrary, a small spatial gradient consistency value enforces the spatial gradient preservation from the ground truth HS images. Thus, we assign $W_{i,j,k}^{x,y}$ and $1-W_{i,j,k}^{x,y}$ as the DGT loss and SSG loss. Finally, the 2D spatial gradient losses ($G_x$ and $G_y$ are the same as those in Equation (4)) are applied to constrain the spatial gradient consistency between the HR MS images and the reconstructed HS images.

4. Experiments

In this section, we conduct the experiments on the public hyperspectral datasets to evaluate the superior performance of the proposed method. The compared methods are coupled nonnegative matrix factorization unmixing (CNMF) for pansharpening [28], RFuse [29], detail injection-based deep convolutional neural networks for pansharpening (DI-DCNN) [30], and MS/HS fusion net (MHF net) [31]. The former two methods are traditional methods, and the latter two methods are deep learning methods. We implemented the proposed method in tensorflow 2.0 and trained it on the Geforce RTX 2080Ti. In our experiments, we trained the network by using the Adam optimizer with the initial learning rate of $0.0001$. The iteration number was 200,000, and the batch size was 10. The slope of the leakyReLU activation was $0.2$ in all activation layers. To obtain the best performance of the proposed method, we set ${\alpha }_1=1.0$, ${\alpha }_2=2.0$, ${\alpha }_3=1.0$ for the CAVE dataset, set ${\alpha }_1=1.0$, ${\alpha }_2=3.0$, ${\alpha }_3=1.0$ for the Chikusei dataset, and set ${\alpha }_1=1.0$, ${\alpha }_2=2.0$, ${\alpha }_3=1.0$ for the WV2 dataset. The three datasets are mentioned in the Section 4.1.

4.1. Dataset and Evaluation Metrics

In the experiments, we used three hyperspectral datasets. These are the CAVE Multispectral Image dataset [22], Chikusei dataset [23], and World View-2 dataset (https://www.harrisgeospatial.com/DataImagery/SatelliteImagery/HighResolution/WorldView-2.aspx, accessed on 13 August 2021):

(1)

CAVE dataset: The CAVE dataset consists of 32 images with a spatial size of $512\times 512$ and the total spectral channels of 31 bands. The band range is from 400 nm to 700 nm. The size of the HR MS image is $512\times 512\times 3$. The first 20 images were set as the training data, and we randomly cropped the $64\times 64\times 24$ patches from each HS image as the ground truth, i.e., HR HS images. The LR HS images were generated by downsampling the ground truth HS images by a factor of 4. The average operation over $4\times 4$ was used in the downsampling operation, which refers to [31]. Thus, the training HR HS, HR MS, and LR MS images had a size of $64\times 64\times 24$, $64\times 64\times 3$, and $16\times 16\times 24$. We utilized the remaining 12 images as the test data. The LR HS images and the HR HS images were utilized as the input and the ground truth.

(2)

Chikusei dataset: The Chikusei dataset (http://naotoyokoya.com/Download.html, accessed on 20 August 2020) is airborne HS images captured over Chikusei on 29 July 2014 [23]. The size of the HR HS image is $2517\times 2335\times 128$ and the size of the HR MS image is $2517\times 2335\times 3$. The band range is from 363 nm to 1018 nm. We selected the top-left portion with a size of $500\times 2000$ as the training data, and the remaining parts were treated as the test data. The training and test datasets were generated like the CAVE dataset. The training HR HS, HR MS, and LR HS images had a size of $64\times 64\times 96$, $64\times 64\times 3$, and $16\times 16\times 96$. During the test procedure, we extracted 16,320 $\times 320$-spatial-size patches from the test data of HR HS and HR MS images. The HR HS patches were employed as the ground truth, and the LR HS patches were generated as the input.

(3)

World View-2 (WV-2) dataset: The World View-2 dataset is a real dataset consisting of LR multispectral images with a size of $1336\times 1121\times 4$ and HR panchromatic images with a size of $5344\times 4484\times 1$. We employed the Wald protocol [32] to prepare the training samples for the real data. We treated the LR multispectral images as the HR HS images and downsampled the HR panchromatic images by a factor of 4. The downsampled HR panchromatic images were considered as the HR MS images. We utilized the top portion (size: $338\times 1121$) of the HR HS and HR MS images as the training data and the remaining part as the test data. As with the CAVE dataset, the extracted training HR HS, HR MS, and LR HS patches are $64\times 64\times 4$, $64\times 64\times 1$, and $16\times 16\times 4$. In the test data, we extracted $96\times 160$-spatial-size patches from the test data of the HR HS and HR MS images. The HR HS patches were employed as the ground truth, and the LR HS patches were generated as the input.

The network structure of the three datasets is illustrated in

Table 2. Network architecture of the proposed spectral super-resolution on the three datasets.

Datasets	CAVE	Chikusei	WV2
SFE Net	3 × 3D-Conv	5 × 3D-Conv	2 × 3D-Conv
FT Net	8 × 3D-Residual	8 × 3D-Residual	8 × 3D-Residual
IR Net	3 × SSF + 3 × 3D Upsample	5 × SSF + 5 × 3D Upsample	2 × SSF + 2 × 3D Upsample
SD Net	3 × 3D-Conv	5 × 3D-Conv	2 × 3D-Conv
GCE Net	2 × 3D-Conv	2 × 3D-Conv	2 × 3D-Conv

. Because the three datasets had different spectral numbers in the images, we utilized the same network structure with different layer numbers. To better extract the spatial and spectral features, we provided the deeper layers to the images with more spectral channels. In Figure 3, the kernel size and stride of the 3D-Conv layers are $(3,3,3)$ and $(2,2,2)$ in 3D-Conv blocks. The kernel size and stride of the 3D-Conv layers are $(3,3,3)$ and $(1,1,1)$ in the 3D residual blocks, 3D upsample blocks, and SSF blocks.

To evaluate the performance of the compared methods, in addition to the subjective evaluation, five evaluation indexes were employed. These are the peak-signal-to-noise ratio (PSNR), structural similarity (SSIM) [33], spectral angle mapper (SAM) [34], Erreur Relative Globale Adimensionnelle de Synthése (ERGAS) [35], and Q2N [36]. The PSNR metric is the traditional image quality index, which estimates the spatial quality of the reconstructed image using the mean-squared error. The SSIM metric evaluates the structural similarity between the ground truth and the reconstructed image. The SAM assesses the spectral distortion by calculating the average angle between the spectral vectors of the generated HS images and the ground truth. The SAM is formulated as follows:

$$SAM(I^{\prime },I^{gt})=\frac{1}{WH}\sum _{i=1}^W\sum _{j=1}^{H}arccos\frac{I_{i,j}^{\prime T}{I}_{i,j}^{gt}}{\parallel I_{i,j}^{\prime }{\parallel }_2{\parallel I_{i,j}^{gt}\parallel }_2}$$

(8)

where arccos is the arc-cosine function. The ERGAS measures the overall fused image quality based on the downsampling ratio, which is calculated as

$$\begin{array}{c}\hfill ERGAS(I^{\prime },I^{gt})=\frac{100}{d}\sqrt{\frac{1}{L}\sum _{k=1}^L\frac{MSE(I_k^{\prime },I_k^{gt})}{{\mu }^2\left(I_k^{gt}\right)}}\end{array}$$

(9)

where d is the spatial downsampling factor. ${\mu }^2\left(I_k^{gt}\right)$ is the mean value of $I^{gt}$ in the band k. $MSE(I_k^{\prime },I_k^{gt})$ is the mean-squared difference between the result $I_k^{\prime }$ and the ground truth $I_k^{gt}$. The Q2N metric [36] reflects the image quality with the computation of the hypercomplex correlation coefficient between the result and the ground truth in the spectral and spatial domains. Smaller SAM and ERGAS values mean better spectral preservation of the HS images. Larger PSNR values report better image fidelity in the spatial domain. Larger SSIM values mean more structure similarity between the result and the ground truth. Q2N values reflect better spectral and spatial correlation between the ground truth and the reconstruction results.

4.2. Comparison to State-of-the-Art

Figure 5 and Figure 6 show the results of five compared methods on the CAVE dataset. In terms of subjective results, the proposed, MHF, and DI-DCNN methods generated visually similar results to the ground truth (see the first row in Figure 5 and Figure 6c,e,f). From the absolute difference map (see the second row of the figures), the proposed method produced results with the least difference compared to MHF and DI-DCNN. Figure 7 and Figure 8 show the results on the Chikusei and WV2 datasets. As for the CAVE datasets, the proposed method generated results with the least difference among the five compared methods (see the second row in the figures). Thus, the proposed method achieves the best performance in subjective results.

Table 3. Quantitative measurement of the experimental results on the CAVE dataset. The best scores are highlighted in bold font.

Methods	SSIM	SAM	ERGAS	PSNR	Q2N
RFuse	0.840	17.233	10.528	30.947	0.853
MHF	0.973	8.741	3.485	40.590	0.939
CNMF	0.903	16.106	9.606	31.703	0.844
DI-DCNN	0.975	8.585	3.906	42.648	0.946
Proposed	0.989	4.416	2.158	43.034	0.940

,

Table 4. Quantitative measurement of the experimental results on the Chikusei dataset. The best scores are highlighted in bold font.

Methods	SSIM	SAM	ERGAS	PSNR	Q2N
RFuse	0.794	7.568	7.300	29.015	0.770
MHF	0.834	3.006	4.087	35.629	0.932
CNMF	0.677	4.882	7.238	29.417	0.772
DI-DCNN	0.778	4.144	5.704	31.450	0.872
Proposed	0.884	2.416	3.319	37.152	0.951

and

Table 5. Quantitative measurement of the experimental results on the WV-2 dataset. The best scores are highlighted in bold font.

Methods	SSIM	SAM	ERGAS	PSNR	Q2N
RFuse	0.949	1.082	0.949	32.933	0.870
MHF	0.955	0.958	1.021	32.195	0.777
CNMF	0.951	1.015	0.869	33.714	0.883
DI-DCNN	0.871	1.850	1.853	30.660	0.718
Proposed	0.968	0.780	0.684	35.847	0.920

show the objective evaluation of the five compared methods on the CAVE, Chikusei, and WV2 datasets. The proposed method achieved the best performance in the SSIM, SAM, ERGAS, and PSNR among the three datasets. In the Q2N metric, the proposed method obtained the best performance on the Chikusei and WV2 datasets, but obtained slightly worse performance compared to DI-DCNN on the CAVE dataset. It is shown that the proposed method generates the results with good performance in structure and spectral preservation from the ground truth.

4.3. Model Analysis

4.3.1. Loss Function

Table 6. Average objective results of different losses on the three datasets. $l_p+l_{SSG}$ and $l_p+l_{SSG}+l_{DGT}$ do not consider the spatial discrepancy (i.e., $W_{i,j,k}^{x,y}$) by GCE net in Figure 4. $L_{loss}$ includes all losses with GCE net. The best scores are highlighted in bold font.

Dataset	Loss	SSIM	SAM	ERGAS	PSNR	Q2N
CAVE	$l_p$	0.980	6.200	2.935	40.310	0.920
	$l_p+l_{SSG}$	0.984	5.840	2.608	41.595	0.926
	$l_p+l_{SSG}+l_{DGT}$	0.981	5.774	2.837	40.685	0.924
	$L_{loss}$	0.989	4.416	2.158	43.034	0.940
Chikusei	$l_p$	0.716	6.031	6.828	31.604	0.868
	$l_p+l_{SSG}$	0.876	2.538	3.991	36.814	0.946
	$l_p+l_{SSG}+l_{DGT}$	0.830	3.636	4.291	35.234	0.925
	$L_{loss}$	0.884	2.416	3.319	37.152	0.951
WV-2	$l_p$	0.962	0.853	0.748	35.167	0.908
	$l_p+l_{SSG}$	0.967	0.797	0.687	35.821	0.919
	$l_p+l_{SSG}+l_{DGT}$	0.967	0.792	0.694	35.746	0.918
	$L_{loss}$	0.968	0.780	0.684	35.847	0.920

reports the objective evaluation of the proposed method with different loss functions. $l_p+l_{SSG}$ and $l_p+l_{SSG}+l_{DGT}$ do not consider the spatial discrepancy (i.e., we removed the weight $W_{i,j,k}^{x,y}$ of Equations (4) and (5)). The loss functions are rewritten as Equations (10) and (11).

$$\begin{array}{ccc}\hfill l_{SSG}& =& \frac{1}{(W-1)(H-1)(L-1)}\hfill \\ \hfill & ·& \sum _{i=1}^{W-1}\sum _{j=1}^{H-1}\sum _{k=1}^{L-1}{|G_x{\left(I^{\prime }\right)}_{i,j,k}-G_x{\left(I^{gt}\right)}_{i,j,k}|}_1\hfill \\ \hfill & +& |G_y{\left(I^{\prime }\right)}_{i,j,k}-G_y{\left(I^{gt}\right)}_{i,j,k}{|}_1\hfill \\ \hfill & +& |G_z{\left(I^{\prime }\right)}_{i,j,k}-G_z{\left(I^{gt}\right)}_{i,j,k}{|}_1)\hfill \end{array}$$

(10)

$l_{SSG}$ enforces the spatial and spectral structure consistency between the results and the ground truth. Thus, $l_p+l_{SSG}$ had better performance compared to $l_p$ in the objective assessment. With additional employment of the DGT loss, $l_p+l_{SSG}+l_{DGT}$ achieved a slightly worse performance because a large spatial discrepancy leads to the gradient distortion in the reconstructed HS images (see the red blocks in Figure 9). Therefore, considering the spatial discrepancy (GCE net in Figure 4), the proposed method achieved the best performance among the four loss functions.

$$\begin{array}{ccc}\hfill & l_{DGT}& =\frac{1}{(W-1)(H-1)}\hfill \\ \hfill & ·& \sum _{i=1}^{W-1}\sum _{j=1}^{H-1}(W_{i,j,k}^x\times {|G_x{\left(I_d^{\prime }\right)}_{i,j,k}-G_x{\left(I_M\right)}_{i,j,k}|}_1\hfill \\ \hfill & +& W_{i,j,k}^y\times |G_y{\left(I_d^{\prime }\right)}_{i,j,k}-G_y{\left(I_M\right)}_{i,j,k}{|}_1)\hfill \end{array}$$

(11)

4.3.2. Component Analysis

This section analyzes the effect of the spatial feature fusion and spectral feature fusion in the SSF blocks. We utilized different network structures of the spatial–spectral fusion (SSF) block for this experiment. The experiments analyzed the effect of spatial feature fusion, spectral feature fusion, and both features’ fusion. The different network structures are shown in Figure 10 and the rightmost figure of Figure 4. These networks should be retrained.

Table 7. Average objective results of different model structures on the three datasets. The best scores are highlighted in bold font. Comp. represents component.

Dataset	Comp.	SSIM	SAM	ERGAS	PSNR	Q2N
CAVE	Spa	0.734	31.449	13.767	27.374	0.831
	Spe	0.978	5.262	3.123	39.700	0.915
	Spa + Spe	0.989	4.416	2.158	43.034	0.940
Chikusei	Spa	0.555	11.204	13.444	26.855	0.708
	Spe	0.716	3.483	6.533	31.059	0.861
	Spa + Spe	0.884	2.416	3.319	37.152	0.951
WV2	Spa	0.888	2.763	1.737	29.013	0.752
	Spe	0.964	0.803	0.720	35.412	0.912
	Spa + Spe	0.968	0.780	0.684	35.847	0.920

shows the objective evaluation of the results generated by the spatial feature fusion, spectral feature fusion, and both features’ fusion. Figure 11 shows the visual results by the spatial fusion, spectral fusion, and both features’ fusion. It is shown that the proposed method with both feature’s fusion preserves more spatial and spectral information from the ground truth (see Figure 11c,d). Table 7 also validates that the proposed method with both features’ fusion achieves the best performance in the objective evaluation.

4.4. Compared to 2D Convolutional Neural Network

This section compares the performance of the 2D CNN and 3D CNN in the proposed method. Figure 12 shows that the results of the 2D CNN and 3D CNN. It is seen that the 3D CNN considers the spectral correlation and generates a results with spectral information more similar to the ground truth (see the similar colors in Figure 12b,c). It is shown that the the proposed method by the 3D CNN provides better performance compared to the 2D CNN.

4.5. Extension to Different Downsampling Factors

In the previous section, we conducted the experiments on the corresponding LR HS images with a downsampling factor of four. Here, we provide the objective experiments of the compared methods on different downsampling factors.

Table 8. Average objective evaluation of the five compared methods on the three datasets with a $1/16$ downsampling factor. The best scores are highlighted in bold font.

Dataset	Methods	SSIM	SAM	ERGAS	PSNR	Q2N
CAVE	RFuse	0.585	28.925	4.070	24.030	0.818
	MHF	0.791	13.337	2.744	30.219	0.793
	CNMF	0.878	10.797	6.194	31.223	0.799
	DI-DCNN	0.852	11.843	3.900	27.547	0.809
	Proposed	0.961	8.314	1.268	35.701	0.893
Chikusei	RFuse	0.737	12.297	10.740	26.354	0.391
	MHF	0.945	3.863	4.980	32.563	0.991
	CNMF	0.784	5.936	8.153	28.516	0.350
	DI-DCNN	0.885	6.126	9.923	27.676	0.974
	Proposed	0.951	3.649	4.870	32.770	0.994
WV-2	RFuse	0.911	2.005	1.381	30.152	0.791
	MHF	0.955	1.458	1.021	32.990	0.777
	CNMF	0.937	1.468	1.061	32.287	0.842
	DI-DCNN	0.940	1.578	1.172	31.763	0.823
	Proposed	0.970	1.311	0.978	33.324	0.845

and

Table 9. Average objective evaluation of the five compared methods on the three datasets with a $1/32$ downsampling factor. The best scores are highlighted in bold font.

Dataset	Methods	SSIM	SAM	ERGAS	PSNR	Q2N
CAVE	RFuse	0.578	28.694	4.673	23.584	0.800
	MHF	0.791	13.971	3.810	29.800	0.776
	CNMF	0.877	9.860	6.277	30.247	0.802
	DI-DCNN	0.788	15.918	5.298	24.614	0.770
	Proposed	0.953	9.122	1.516	33.968	0.885
Chikusei	RFuse	0.726	14.794	11.687	25.778	0.391
	MHF	0.923	4.648	5.461	30.819	0.988
	CNMF	0.778	6.720	8.861	27.894	0.343
	DI-DCNN	0.862	8.593	11.242	25.510	0.979
	Proposed	0.942	4.503	5.394	31.288	0.996
WV-2	RFuse	0.878	2.668	1.681	28.772	0.772
	MHF	0.935	1.667	1.150	31.963	0.810
	CNMF	0.926	1.798	1.211	31.469	0.821
	DI-DCNN	0.923	2.003	1.523	30.337	0.792
	Proposed	0.961	1.660	1.131	32.240	0.821

represent the average objective evaluation of the five compared methods with downsampling factors of $1/16$ and $1/32$. We utilized the five objective evaluations of the SSIM, SAM, ERGAS, PSNR, and Q2N metrics for the experiment. With the increase of the downsampling factors, it is shown that the proposed method achieved the best performance among the compared methods in terms of the five objective metrics.

5. Conclusions

In this paper, we proposed a novel supervised pansharpening network, which was designed as the spectral super-resolution framework. The network consists of three stages: spectral/spatial feature extraction, feature transformation, and image reconstruction. We designed the SSF blocks, which reuse the extracted feature in the image reconstruction. The DDHG losses were developed to preserve the spatial and spectral features from the HR MS and HR HS images considering the spatial and spectral discrepancy between two images. The experimental results showed that the proposed method achieved the best performance in the subjective and objective evaluation of the spectral and spatial preservation among state-of-the-art methods.