Feedback Refined Local-Global Network for Super-Resolution of Hyperspectral Imagery

Abstract

Powered by advanced deep-learning technology, multi-spectral image super-resolution methods based on convolutional neural networks have recently achieved great progress. However, the single hyperspectral image super-resolution remains a challenging problem due to the high-dimensional and complex spectral characteristics of hyperspectral data, which make it difficult for general 2D convolutional neural networks to simultaneously capture spatial and spectral prior information. To deal with this issue, we propose a novel Feedback Refined Local-Global Network (FRLGN) for the super-resolution of hyperspectral image. To be specific, we develop a new Feedback Structure and a Local-Global Spectral block to alleviate the difficulty in spatial and spectral feature extraction. The Feedback Structure can transfer the high-level information to guide the generation process of low-level features, which is achieved by a recurrent structure with finite unfoldings. Furthermore, in order to effectively use the high-level information passed back, a Local-Global Spectral block is constructed to handle the feedback connections. The Local-Global Spectral block utilizes the feedback high-level information to correct the low-level feature from local spectral bands and generates powerful high-level representations among global spectral bands. By incorporating the Feedback Structure and Local-Global Spectral block, the FRLGN can fully exploit spatial-spectral correlations among spectral bands and gradually reconstruct high-resolution hyperspectral images. Experimental results indicate that FRLGN presents advantages on three public hyperspectral datasets.

1. Introduction

Hyperspectral imaging sensors collect and process information across different bands of the entire electromagnetic spectrum. Compared with multi-spectral image, the resulting hyperspectral image (HSI) contains richer spectral information and has been applied to resource management, target detection and land cover detection [1,2,3,4,5], etc. However, because of the limitation of imagery system, it is difficult to acquire an HSI with high spatial resolution. Therefore, how to obtain a reliable high-resolution HSI is still a very challenging problem.

Recently, HSI super-resolution approaches have been intensively studied in remote sensing [6]. Based on the number of input images, the HSI super-resolution methods can be roughly divided into fusion-based HSI super-resolution [7,8,9] and single HSI super-resolution [10,11,12]. The fusion-based HSI super-resolution methods improve the spatial resolution by combining the observed low-resolution HSI with high-resolution multispectral images or panchromatic. For example, Wei et al. [13] introduced a variational-based approach to merge a high-resolution multispectral image with a low-resolution HSI. By considering the HSI as a 3D tensor, Wan et al. [14] designed a nonlocal 4-D tensor dictionary learning-based fusion approach. More recently, deep learning-based fusion methods have achieved excellent performance with the powerful representation capability of convolution neural network. Wei et al. [15] suggested using the deep neural network to capture plenty of HSI statistics and then putting these priors to regularize the super-resolution procedure of HSIs. Wei et al. [16] recently further designed a deep recursive residual network to probe the deep statistical prior information. Most fusion-based methods assume that the high-resolution auxiliary image is well co-registered with the low-resolution HSI. In real applications, it is difficult to obtain these co-registered auxiliary images, which hinders the progress of such technique.

By contrast, the single HSI super-resolution approaches do not need any auxiliary information and have a better feasibility in practice, which only reconstruct the high-resolution HSI from a low-resolution HSI. In order to explore the spatial-spectral prior information of HSIs, some single HSI super-resolution methods based on dictionary learning, sparse representation and low-rank approximation have been proposed. For instance, Huang et al. [17] designed a noise-insensitive super-resolution mapping method based on multi-dictionary sparse representation. Wang et al. [18] introduced a new tensor-based approach to solve the HSI super-resolution problem by modeling three intrinsic characteristics of hyperspectral data. However, the hand-crafted priors can only reflect one aspect of the hyperspectral data, which make the reconstruction effect obvious only for the specified HSIs. In recent years, due to the success of deep learning technology in many fields [19,20,21], it has been applied to the single hyperspectral super-resolution task, and achieved satisfying super-resolution results. For example, Hu et al. [22] designed a spectral difference convectional network to alleviate spectral distortion. Besides, Mei et al. [23] constructed a 3D super-resolution network to extract the prior information. Although the spectral correlation can be well exploited by 3D convolution operator, the amount of computation required by the model is very large. To solve the problem of high computation of 3D convolution, Jiang et al. [24] introduced a group convolution to explore the spatial information and the correlation among the spectral bands. Wang et al. [25] further designed a recurrent structure to investigate the spectral correlation among groups. Nonetheless, due to the high dimension and complex spectral patterns of hyperspectral data, it is still hard to simultaneously explore the joint spatial and spectral information between continuous bands.

In this paper, in order to alleviate the difficulty of extracting spatial-spectral prior information from hyperspectral data, we propose a novel network for the single HSI super-resolution task, namely Feedback Refined Local-Global Network (FRLGN). FRLGN is motivated by the feedback mechanism [26], which can make the network transmit high-level semantic information back to the previous layers and refine these low-level feature representations. Recently, some researchers have adopted this feedback mechanism to design the network architecture for various vision tasks [27,28,29]. For instance, Li et al. [29] proposed a recurrent neural network with the feedback manner for the multi-spectral image super-resolution task. Han et al. [30] designed a two-state recurrent neural network, which makes the information exchange between two hidden states in both directions. We design a Feedback Structure (FS) and a Local-Global Spectral Block (LGSB) in FRLGN by using feedback mechanism to enhance the super-resolution results of HSIs. To be specific, the Feedback Structure allows to use the feedback high-level information to correct the low-level representations through feedback connections. The FS is achieved by a recurrent structure with finite unfoldings. Furthermore, we construct a Local-Global Spectral Block to take full advantage of the feedback high-level information. The LGSB is composed of local and global spectral feature extraction layers, which can adjust the local spectral low-level representation input using the feedback high-level global spectral information and create a powerful high-level global spectral representation. The FRLGN is essentially a recurrent neural network with a Local-Global Spectral Block, which is specifically designed to explore the spatial and spectral prior of hyperspectral data. Experimental results indicate that LGSB is more suitable for HSI super-resolution task. Besides, we aggregate the losses of each iteration to optimize the network model and make the feedback high-level feature contain high-resolution HSI information. In summary, the principle of the feedback mechanism of our model is to generate a better super-resolution HSI with coarse reconstructed HSI facilitates.

The main contributions of our work can be summarized as follows:

• A novel Feedback Refined Local-Global Network is proposed for the single HSI super-resolution task, which can effectively explore spatial-spectral priors between spectral bands.
• We construct a new Feedback Structure to correct the low-level representation using feedback high-level semantic information.
• We design a Local-Global Spectral Block to refine the local spectral low-level representations using the feedback information, and then generate a more powerful global spectral high-level representation.

2. Proposed Method

In this section, the detail description of FRLGN will be presented, and then the proposed Feedback Structure and Local-Global Spectral Block are introduced. At last, we interpret the loss function used in FRLGN approach.

2.1. Network Architecture

As shown in Figure 1, the FRLGN is unfolded to T iterations, where the order of each iteration t is from 1 to T. We link the losses of each iteration together so that the hidden state in FRLGN can contain the information of the high-resolution HSI. We describe the specific details of the loss function in the next Loss Function part. The sub-network in each iteration t consists of three Blocks: the Embedding Block, the Local-Global Spectral Block and the Reconstruction Block. Each iteration shares the weights of each block. For each iteration t, we also design a global skip connection that transmits an up-sampled HSI to the final output. Therefore, each iteration t of the sub-network is used to recover a residual image when a low-resolution HSI is input.

2.1.1. The Embedding Block

Different from the previous method of treating the HSI as a whole or multiple single-channel images, we divide the entire input low-resolution HSI into several groups. With this strategy, we can not only explore the correlation between adjacent spectral bands of the input HSI more easily, but also reduce the spectral dimension of the HSI. Specifically, the input low-resolution HSI $I_{LR}$ is divided into G spectral groups without overlapping. We pad the last spectral group with zero bands to ensure that each group has the same number of bands. More details are discussed in the experiment section. As shown in Figure 1, for each group $I_{LR}^g$, we use one convolution operation to extract its shallow feature $F_{EB}^g$. The mathematical formulations of the Embedding Block are as follows:

$$I_{LR}=[I_{LR}^1,I_{LR}^2,I_{LR}^g,···,I_{LR}^G]$$

(1)

$$F_{EB}^g=f_{EB}\left(I_{LR}^g\right)$$

(2)

$$F_{EB}=[F_{EB}^1,F_{EB}^2,F_{EB}^g,···,F_{EB}^G]$$

(3)

where the $f_{EB}$ denotes the operations of Embedding Block, eg., feature extraction layer for all groups. The $\left[\right]$ represents a concatenating operation. After that, $F_{EB}$ is used as input to the Local-Global Spectral Block.

2.1.2. The Local-Global Spectral Block

For t-th iteration, Local-Global Spectral Block receives the the shallow feature $F_{EB}$ and hidden state from past iteration $F_{LGSB}^{t-1}$ through a feedback connection. $F_{LGSB}^t$ denotes the result of LGSB. The mathematical formula of LGSB is as follows:

$$F_{LGSB}^t=f_{LGSB}(F_{LGSB}^{t-1},F_{EB})$$

(4)

where $f_{LGSB}$ denotes the operations of the LGSB. More details of the LGSB can be found in Local-Global Spectral Block part.

2.1.3. The Reconstruction Block

The reconstruction block firstly uses PixelShuffle [31] to upscale the feature $F_{LGSB}^t$ to high-resolution one, and then a 3 × 3 convolution operation is applied to create the residual image $I_{Res}^t$. The formula for reconstruction block is defined as:

$$I_{Res}^t=f_{RB}\left(F_{LGSB}^t\right)$$

(5)

where $f_{RB}$ is the operation of the reconstruction block.

For the t-th iteration, the output super-resolution image $I_{SR}^t$ is obtained by:

$$I_{SR}^t=I_{Res}^t+f_{UP}\left(I_{LR}\right)$$

(6)

where $f_{UP}$ represents an upsampling operation. The choice of upsampling method is arbitrary. In this paper, we apply a Bicubic upsample approach. After T iterations, we will generate T super-resolution images $(I_{SR}^1,I_{SR}^2,···,I_{SR}^T)$.

2.2. Feedback Structure

In HSI super-resolution task, some researchers [16,25,32] have made an effort to introduce the recurrent structure to improve super resolution results. However, in their network frameworks, the information flow from the low-resolution HSI to final super-resolution HSI is still feed-forward. As can be seen from Figure 2b, the recurrent structure adopted by these methods can be abstracted into a single-state recurrent network. These methods improve the feature representation of the model by running recursively on a specially designed network structure.

In this work, we design a Feedback Structure to reroute the output of the HSI super-resolution system to correct the input in each iteration. Figure 2a illustrates the Feedback Structure of FRLGN. Specifically, the Local-Global Spectral Block receives the information of input low-resolution HSI and feedback high-level information from last iteration, then generates coarse super-resolution result and high-level semantic guidance information for next iteration. The Feedback Structure can be characterized by:

$$I_{SR}^t=f_{FS}(I_{LR},I_{SR}^{t-1})$$

(7)

where the $f_{FS}$ denotes the function of Feedback Structure.

2.3. Local-Global Spectral Block

As an ill-posed problem, image super-resolution requires additional prior knowledge to regularize the reconstruction process. Traditional super-resolution methods usually make an effort to construct the regular terms of the super-resolution model [33,34], such as low-rank [35,36], nonlocal self-similarity [37,38] and sparsity [39,40]. Whether the designed prior knowledge can characterize the observed HSI data directly determines the performance of the super-resolution method. Therefore, for the HSI super-resolution task, it is also essential to study the inherent characteristics of hyperspectral data, e.g., the spatial non-local self-similarity and the high-correlation among spectral bands [41]. However, the manually designed constraints are not enough to achieve accurate restoration of HSIs.

In this work, a novel Local-Global Spectral Block is introduced to exploit the spatial-spectral prior with the help of feedback high-level semantic information from hidden state. As can be seen in Figure 3a, for iteration t, the LGSB inputs the feedback global spectral high-level information $F_{LGSB}^{t-1}$ to correct the G groups local spectral low-level representations, $F_{EB}=[F_{EB}^1,F_{EB}^2,F_{EB}^g,···,F_{EB}^G]$, and then creates more effective high-level feature $F_{LGSB}^t$ for the next iteration and the reconstruction block. The LGSB contains G groups local spectral feature extraction layers and one global spectral feature extraction layer. For simplicity, we use $Conv\left(k\right)$ and $Deconv\left(k\right)$ to denote a convolution operation and a deconvolutional operation, where k represents the size of the convolution kernel.

At the beginning of the LGSB, the downsampled $F_{LGSB}^{t-1}$ and each group $F_{EB}^g$ are concatenated and compressed by one $Conv\left(1\right)$ operation to refine the input each group feature $F_{EB}^g$ by feedback information $F_{LGSB}^{t-1}$, producing the refined group feature $F_g^t$.

$$F_g^t=f_{Com}\left([f_{Down}\left(F_{LGSB}^{t-1}\right),F_{EB}^g]\right)$$

(8)

where $f_{Down}$ refers to downsample operation using average pooling with a kernel size of 2 and a stride size of 2. The $[f_{Down}\left(F_{LGSB}^{t-1}\right),F_{EB}^g]$ refers to the concatenation of $f_{Down}\left(F_{LGSB}^{t-1}\right)$ and $F_{EB}^g$. The $f_{Com}$ denotes the initial compression operation.

After obtaining the refined group feature $F_g^t$, we add a local spectral feature extraction layer to explore the local spectral correlation, which consists of two residual blocks as shown in Figure 3b. Let $L_g^t$ be the g-th group local spectral LR feature map. $L_g^t$ can be obtained by:

$$L_g^t=f_{Local}\left(F_g^t\right)$$

(9)

where the $f_{Local}$ denotes local spectral feature extraction layer.

After that, we pass all the local spectral LR feature maps to the global spectral feature extraction layer, which contains one upsample $Deconv\left(2\right)$ operation and two residual blocks. Note that we propose a new feedback strategy to enhance the high-level semantic information passed back. Specifically, we enable the transfer of high-level information from high-resolution space to low-resolution space by adding an upsampling operation in the global spectral feature extraction layer. The effectiveness of this strategy is analyzed in the experimental section. At last, the global spectral high-level feature $F_{LGSB}^t$ can be obtained by:

$$F_{LGSB}^t=f_{Global}\left([L_1^t,L_2^t,···,L_G^t]\right)$$

(10)

where the $f_{Global}$ denotes the the global spectral feature extraction layer.

2.4. Loss Function

To measure the HSI reconstruction performance, several loss functions have been investigated to make super-resolution results approximate to the ground truth (high-resolution HSI), such as $l_1$, $l_2$ and perceptual losses, etc. Since the $l_1$ loss function can effectively penalize small errors and have better convergence during training stage, we choose the $l_1$ loss to measure the HSI super-resolution performance. Finally, in order to make the feedback high-level feature contain high-resolution HSI information, the output result of FRLGN is the weighted average of all intermediate super-resolution results. Thus, we have

$$I_{SR}=\frac{1}{T}\sum _{t=1}^T{I}_{Res}^t+f_{UP}\left(I_{LR}\right)$$

(11)

The loss function of FRLGN is determined by:

$$L(\Theta )={∥I_{HR}-I_{SR}∥}_1$$

(12)

where the $\Theta$ represents the parameters of our proposed FRLGN and the $I_{HR}$ is the corresponding target high-resolution HSI. The training procedure of FRLGN is shown in Algorithm 1.    

Algorithm 1: Training Process of FRLGN

3. Experiments and Results

3.1. Datasets

3.1.1. CAVE Dataset

The CAVE dataset [42] is a HSI dataset of real-world materials and objects, which is captured by a Cooled CCD camera. The hyperspectral camera collects information from the 400 nm–700 nm spectral range in 10 nm steps. This dataset consists of 32 HSIs with a size of $512\times 512\times 31$ pixels, which are further divided into 5 groups, namely food and drinks, skin and hair, paints, real and fake, and stuff (e.g., feathers, flowers, superballs, etc.). Each band is stored in 16-bit PNG format.

3.1.2. Harvard Dataset

The Harvard dataset [43] contains 77 HSIs of $1040\times 1392\times 31$ size from outdoor and indoor scenes. These HSIs are captured by a commercial hyperspectral camera, which collects the spectral data in 10 nm steps over the wavelength range of 400 nm to 700 nm. Different from the CAVE dataset, this dataset is stored as a .mat file.

3.1.3. Chikusei Dataset

The Chikusei dataset [44] consists of $2517\times 2335$ pixels with a spatial resolution of 2.5 m. The dataset is taken by an airborne hyperspectral imaging sensor in the agricultural and urban areas of Chikusai, Japan. This dataset captures 128 spectral bands from the 363 nm to 1018 nm. Besides, this dataset is also stored as a .mat file. Since the lack of edge information, we first cut the original HSI to generate an image of 2304 × 2048 × 128 pixels and then the generated image is further split into a training set and a test set. In particular, we extract the top region of the generated image to create the test set, which consists of four HSIs with a pixel size of $512\times 512\times 128$ that do not overlap each other. And the remaining region of the generated image is used as training data.

3.2. Implementation Details

Since HSIs are collected by different hyperspectral imaging sensors, HSI datasets tend to have different numbers of spectral channels. Therefore, we need to learn a super-resolution HSI model separately for each HSI dataset. In the next experiments, 80% of samples in the dataset are used to train the super-resolution models and the remaining samples are utilized for testing.

During training, 12 randomly selected patches are fed to the FRLGN network. In order to obtain low-resolution HSIs, we down-sample these patches to $32\times 32\times L$ pixels based on the scale factor s. L represents the number of spectral bands of HSI. Similar to [24], we set the scale factor s to 4 and 8 in the next experiments. Furthermore, we use the bicubic interpolation function to down-sample these patches. In our network, the convolution operators with a kernel size of 3 adopt a zero-padding strategy to ensure that the intermediate features have the same spatial size. The base number of filters is set to 256. We up-sample the resulting features by a factor of 2 using a deconvolution with a kernel size of 2 and a stride size of 2 in LGSB. The ADAM [45] with an initial learning rate of $2\times {10}^{-4}$ is used to optimize the FRLGN network.

At the testing stage, in order to improve testing efficiency, we use only the $512\times 512$ area in the upper left corner of test HSIs for evaluation. In this work, the Pytorch library is used to implement and train our proposed FRLGN network.

3.3. Evaluation Metrics

In this section, we choose six commonly used quantitative metrics to evaluate the performance of FRLGN, i.e., cross correlation (CC) [46], spectral angle mapper (SAM) [47], root mean squared error (RMSE), the erreur relative globale adimensionnelle de synthese (ERGAS) [48], peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) [49]. As the CC, RMSE, PSNR and SSIM are widely used quantitative metrics in HSI super-resolution tasks, we omit their detailed description here. In addition, ERGAS performs a global statistical measure on the reconstructed HSIs, which is calculated by

$$ERGAS(I_{HR},I_{SR})=100s\sqrt{\frac{1}{L}\sum _{l=1}^L{\left(\frac{RMS{E}_l}{{\mu }_l}\right)}^2}$$

(13)

in which $RMS{E}_l=(\parallel I_{SR}^l-I_{HR}^l{\parallel }_F/\sqrt{n})$. Here, n and ${\mu }_l$ represent the number of spatial pixels and mean of the lth band from the ground truth $I_{HR}$, respectively. The $I_{SR}^l$ and $I_{HR}^l$ denote the lth band of $I_{SR}$ and $I_{HR}$, respectively. SAM is used to evaluate the preservation of spectral band information for each spatial location of the HSI. SAM is obtained by calculating the angle between two spectral vectors from the same spatial position of $I_{SR}$ and $I_{HR}$. The formula of SAM is presented as

$$SAM(\mathbf{x},\widehat{\mathbf{x}})=arccos\left(\frac{\langle \mathbf{x},\widehat{\mathbf{x}}\rangle }{{\parallel \mathbf{x}\parallel }_2{\parallel \widehat{\mathbf{x}}\parallel }_2}\right)$$

(14)

in which $\widehat{\mathbf{x}}$ and $\mathbf{x}$ denote the two spectral vectors from $I_{SR}$ and $I_{HR}$, respectively. And the $\langle ·,·\rangle$ is the dot product of two vectors, ${\parallel \mathbf{x}\parallel }_2$ represent the $l_2$ regularization operation of a vector. For PSNR and SSIM, we present the average metric values of all spectral bands. The best values for CC, SAM, RMSE, ERGAS, PSNR, SSIM are 1, 0, 0, 0, $+\infty$, and 1, respectively.

3.4. Study of T and G

In this part, we discuss the effect of iterations (denoted as T) and local spectral groups (denoted as G) in the Local-Global Spectral Block with a scale factor of 4 on the CAVE dataset. By fixing G to 8, we first explore the influence of T on HSI reconstruction.

Table 1. Convergence analysis of T when G = 8.

T	1	2	3	4	5	6
PSNR	37.0815	37.5073	37.7574	37.8131	37.8639	37.8905
SAM	3.7272	3.5754	3.5256	3.4789	3.4585	3.4332

shows that the super-resolution performance is improved with the help of feedback connections compared to the network without feedback connections (T = 1). Moreover, the quality of reconstruction has been further improved as the increasing iteration T. On the other hand, it also indicates that our proposed Local-Global Spectral Block would certainly benefit from cross-time feedback information. It is worth noting that with the increase of T, the performance of FRLGN will be gradually improved, but the calculation amount of the model is also constantly expanding. After that, we also discuss the influence of G by fixing the T to 6. From

Table 2. Convergence analysis of G when T = 6.

G	1	2	4	8
PSNR	37.7251	37.7329	37.8201	37.8905
SAM	3.5977	3.5276	3.4792	3.4332

, we can observe that with the help of local-spectral grouping strategy, the spectrum reconstruction performance is enhanced compared to the network without the grouping strategy (G = 1). In addition, with the increase of G, the spectral representation of FRLGN becomes more powerful and the spatial-spectral reconstruction quality is improved. However, it will also greatly increase the computational overhead. Because the more branches of FRLGN, the more calculations are required. Therefore, in order to achieve a balance between the calculation amount and objective results, we set T = 6, G = 8 for CAVE dataset and Harvard dataset, and T = 6, G = 12 for Chikusei dataset unless otherwise stated.

3.5. Study of Feedback Structure

In this subsection, we analyze the Feedback Structure from two aspects. On the one hand, we compare the HSI reconstruction performance of the LGSB-based feedback structure (LGSB + FS) and the LGSB-based traditional recursive structure (LGSB + TRS). Specifically, we construct a traditional recurrent super-resolution network with LGSB as a recursive module, i.e. only the last iteration of LGSB is input into the reconstruction block to generate the final super-resolution result.

Table 3. Quantitative analysis of feedback structure and traditional recursive structure.

	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑	SSIM↑
LGSB + TRS	0.9930	3.5393	0.0151	5.1665	37.9050	0.9734
LGSB + FS	0.9930	3.4332	0.0152	5.1599	37.8905	0.9737

shows the quantitative analysis of LGSB + FS and LGSB + TRS on the CAVE dataset with a scale factor of 4. From Table 3, we can observe that LGSB + FS performs better on 4 out of 6 metrics. In terms of spectral reconstruction (SAM), LGSB + FS is significantly better than LGSB + TRS, indicating that the feedback structure can better capture the spectral prior information of HSI. However, the spatial reconstruction effect of LGSB + FS and LGSB + TRS is not much different. On the other hand, we study the feedback strategy proposed in LGSB, which transfers high-level information from high-resolution (HR) space to low-resolution (LR) space by adding an upsampling operation (HR to LR). To show the advantages of this strategy, we construct two other feedback strategies for comparison, namely LR to LR and LR, HR to LR. For the LR to LR feedback strategy, we directly remove the upsampling operation in LGSB to realize the information transfer from LR space to LR space. As for the LR, HR to LR feedback strategy, we pass the output of local spectral feature extraction layer and global spectral feature extraction layer to guide the low-level feature extraction.

Table 4. Quantitative analysis of three Feedback strategies.

Strategies	s	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑	SSIM↑
LR to LR	4	0.9930	3.4430	0.0153	5.1929	37.9194	0.9728
LR, HR to LR	4	0.9929	3.4416	0.0154	5.2385	37.8239	0.9731
HR to LR	4	0.9930	3.4332	0.0152	5.1599	37.8905	0.9737
LR to LR	8	0.9713	5.2059	0.0325	10.3697	31.3910	0.9139
LR, HR to LR	8	0.9706	5.1854	0.0329	10.4487	31.3267	0.9146
HR to LR	8	0.9712	5.0550	0.0323	10.3982	31.4007	0.9159

presents the quantitative performance of three feedback strategies on the CAVE dataset. From Table 4, it can be seen that HR to LR achieves better performance than LR to LR on most quantitative metrics, which indicates that high-level information from HR space can guide the extraction of low-level features more effectively than high-level information from LR space. The reason is that the output from the HR space contains more high-resolution HSI information. Moreover, compared with HR to LR, the LR, HR to LR gets worse results, which indicates that directly transferring high-level information from HR space and LR space back to the shallow layers of the network reduces the feedback effect. In conclusion, the HR to LR strategy is more suitable for HSI super-resolution task than the other two feedback strategies.

3.6. Comparisons with the State-of-the-Art Methods

In this section, we evaluate the single image super-resolution effect of FRLGN in detail on three benchmarks, namely CAVE dataset [42], Harvard dataset [43] and Chikusei dataset [44]. Specifically, we compare the FRLGN with five existing super-resolution approaches, including two advanced deep multispectral image super-resolution methods, VDSR [50], RCAN [51] and three representative HSI super-resolution methods, 3DCNN [23], GDRRN [32] and SSPSR [24]. In addition, we carefully tune the hyper-parameters of these super-resolution methods to obtain a good performance. Moreover, the bicubic interpolation is used as our baseline model.

Table 5. Quantitative analysis of seven different comparison methods on CAVE test dataset involving six metrics.

	s	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑	SSIM↑
Bicubic	4	0.9846	5.1832	0.0224	7.7384	34.5069	0.9472
VDSR [50]	4	0.9896	4.3622	0.0188	6.3067	36.1348	0.9612
RCAN [51]	4	0.9913	4.3058	0.0172	5.7796	36.7979	0.9657
3DCNN [23]	4	0.9862	4.2297	0.0212	7.3182	34.9853	0.9549
GDRRN [32]	4	0.9891	4.2970	0.0192	6.5087	35.8465	0.9594
SSPSR [24]	4	0.9915	3.7384	0.0168	5.7527	37.0479	0.9682
FRLGN	4	0.9930	3.4332	0.0152	5.1599	37.8905	0.9737
Bicubic	8	0.9564	7.3210	0.0385	12.8323	29.5763	0.8741
VDSR [50]	8	0.9615	5.8692	0.0369	12.0527	30.0080	0.8999
RCAN [51]	8	0.9671	5.9008	0.0340	11.1373	30.7372	0.9061
3DCNN [23]	8	0.9594	5.6079	0.0370	12.3341	29.8880	0.8961
GDRRN [32]	8	0.9611	5.8864	0.0368	12.0684	30.0042	0.8966
SSPSR [24]	8	0.9675	5.6617	0.0341	11.0506	30.7976	0.9098
FRLGN	8	0.9712	5.0550	0.0323	10.3982	31.4007	0.9159

,

Table 6. Quantitative analysis of seven different comparison methods on Harvard test dataset involving six metrics.

	s	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑	SSIM↑
Bicubic	4	0.9606	2.5671	0.0101	3.0957	43.9037	0.9582
VDSR [50]	4	0.9640	2.5709	0.0090	2.8602	44.6486	0.9634
RCAN [51]	4	0.9671	2.4097	0.0086	2.7537	45.1204	0.9663
3DCNN [23]	4	0.9614	2.3917	0.0098	3.0324	44.1815	0.9600
GDRRN [32]	4	0.9630	2.4924	0.0093	2.9276	44.4577	0.9620
SSPSR [24]	4	0.9704	2.2766	0.0082	2.5893	45.5460	0.9684
FRLGN	4	0.9722	2.2496	0.0074	2.4463	46.1866	0.9730
Bicubic	8	0.9098	3.0165	0.0179	5.0694	39.6681	0.9131
VDSR [50]	8	0.9185	3.0093	0.0165	4.7369	40.2490	0.9223
RCAN [51]	8	0.9312	2.7808	0.0150	4.3438	40.9853	0.9313
3DCNN [23]	8	0.9128	2.7853	0.0172	4.9422	39.9615	0.9175
GDRRN [32]	8	0.9175	2.8669	0.0166	4.7946	40.1831	0.9214
SSPSR [24]	8	0.9338	2.6202	0.0149	4.2458	41.1869	0.9313
FRLGN	8	0.9373	2.7665	0.0139	4.0316	41.6320	0.9374

and

Table 7. Quantitative analysis of seven different comparison methods on Chikusei test dataset involving six metrics.

	s	CC↑	SAM↓	RMSE↓	ERGAS↓	PSNR↑	SSIM↑
Bicubic	4	0.8987	3.7666	0.0176	7.6532	36.5603	0.8882
VDSR [50]	4	0.9176	3.1003	0.0155	6.9534	37.5648	0.9113
RCAN [51]	4	0.9142	3.0936	0.0156	7.1099	37.4313	0.9104
3DCNN [23]	4	0.9047	3.4808	0.0169	7.3419	36.9090	0.8931
GDRRN [32]	4	0.9144	3.2178	0.0159	7.0426	37.3754	0.9060
SSPSR [24]	4	0.9250	2.8281	0.0148	6.6082	37.9698	0.9193
FRLGN	4	0.9283	2.7580	0.0143	6.4953	38.2085	0.9240
Bicubic	8	0.7546	5.9617	0.0274	11.9665	32.7047	0.7829
VDSR [50]	8	0.7840	5.3103	0.0250	10.9097	33.4964	0.8069
RCAN [51]	8	0.7630	6.5447	0.0258	11.9078	33.0475	0.7946
3DCNN [23]	8	0.7723	5.5506	0.0257	11.0971	33.3107	0.7955
GDRRN [32]	8	0.7842	5.3033	0.0249	10.9107	33.5236	0.8062
SSPSR [24]	8	0.7880	5.2415	0.0247	10.7863	33.6194	0.8106
FRLGN	8	0.7887	5.2122	0.0246	10.8033	33.6332	0.8145

depict the quantitative performance of all super-resolution algorithms over testing images on three datasets, where bold indicates the best results.

Table 5 shows that our FRLGN method outperforms other comparative methods in all objective assessment metrics. Specifically, the baseline approach has the worst performance among these compared algorithms. As the competitive multispectral image super-resolution methods, VDSR and RCAN can generate very satisfactory results. Nonetheless, in comparison with those HSI super-resolution methods, i.e., 3DCNN [23], GDRRN [32] and SSPSR [24], their spectral reconstruction effect (SAM) is relatively poor due to the lack of a network structure dedicated to exploring high-dimensional HSI data. This indicates that the multispectral super-resolution approaches cannot effectively explore the spectral prior information from the hyperspectral data. The 3DCNN [23] and GDRRN [32] employ 3D convolution operations and recursive residual blocks to extract spatial-spectral prior information, respectively, and obtain good super-resolution results. Similar to our work, SSPSR [24] also adopts a group strategy but neglects the continuous relationship among band groups. Therefore, it achieves the suboptimal results. Compared with other comparison SR methods, our proposed FRLGN can obtain better performance in spectral and spatial dimensions. In term of PSNR, the FRLGN was 0.8 and 0.6 higher than the suboptimal method for upsampling factors s of 4 and 8, respectively. The Table 6 and Table 7 show the similar results. In conclusion, FRLGN has presented advantages on three datasets compared to some SR methods, especially for PSNR and SSIM.

In order to further prove the effectiveness of FRLGN, Figure 4 and Figure 5 display the mean absolute error maps across all spectral bands of two HSIs with the scale factor $\times 4$ from the CAVE testing dataset and Harvard testing dataset, respectively. Principally, more bluer the color of the error map, more better the reconstructed HSI. From Figure 4 and Figure 5, we can easily discover that the FRLGN method can obtain better reconstruction fidelity when restoring the spatial information of the original HSI. Specifically, in contrast to with the suboptimal SSPSR method, FRLGN performs better in reconstructing textures such as edges and structures. Besides, we also display two reconstructed high-resolution HSIs from Chikusei test dataset with a downsampling factor of 4 in Figure 6 and Figure 7. As can be seen from Figure 6 and Figure 7, our FRLGN can restore finer texture details than other comparison methods.

In addition, to prove our advantage in reconstructing spectral information, Figure 8, Figure 9 and Figure 10 show the average absolute difference of all comparison methods along the spectral dimension. The average spectral error curve has a better visualization effect than displaying the spectral reflectance of multiple locations. As shown in Figure 8, Figure 9 and Figure 10, our method has the lowest average spectral error curve, which indicates that FRLGN has better spectral reconstruction ability. This can be attributed to the guidance of the global spectral feedback information to the local spectral band group. Moreover, as iterations increase, the local spectral group information gradually accumulates, leading to better spectral reconstruction performance.

4. Conclusions

Considering the difficulties of simultaneously exploring the spatial and spectral information of hyperspectral data, a new approach for the single HSI super-resolution task has been proposed, called Feedback Refined Local-Global Network. FRLGN can produce a clear high-resolution HSI by introducing a Feedback Structure and a Local-Global Spectral Block. In particular, a recurrent neural network with feedback connection has been built, and it is able to refine low-level feature representations by using feedback global spectral high-level semantic information. Furthermore, taking advantage of the feedback high-level semantic information, a Local-Global Spectral Block is designed to guide the extraction process of low-level representations between local spectral bands using the feedback information, and then generate a more powerful high-level feature among global spectral bands. With the increasing number of iterations, the spatial-spectral prior gradually accumulates, leading to better HSI reconstruction performance. The comprehensive experimental results and visual data analysis show the effectiveness of the proposed FRLGN.

Although FRLGN has achieved good results, it is still difficult to apply in practical applications due to the large amount of computation brought by the recursive structure. However, the main contribution of this work is that the way in which global spectral high-level information guides local spectral feature extraction can improve the feature extraction capability of high-dimensional data in convolutional neural networks. Therefore, in the future, we will introduce the lightweight network structure to reduce the heavy computational overhead.