1. Introduction
Hyperspectral (HS) images provide abundant and fine spectral information, which can be widely used in many applications, including forest mapping [
1], urban environment monitoring [
2], ecophysiology assessment [
3], and quality control of agriculture [
4]. However, the optical remote sensing systems are limited by the amount of incident energy, onboard storage, and bandwidth transmission [
5,
6]. The spectral and spatial resolution of remote sensing images cannot increase at the same time. Therefore, the HS images generally have lower spatial resolution than other remote sensing images. For this reason, the HS image pansharpening technique is developed to generate a high-resolution HS image by combining a panchromatic (PAN) image and a HS image from the given area.
In HS image pansharpening, to protect the spectral information in a HS image and to further generate the accurate fine spectrums at higher spatial resolution are important problems. If the spectral information of the HS images is distorted in pansharpening, various applications of the pansharpened images are directly affected, including lithological mapping [
7], coral reef inversion [
8], and discrimination of forest types [
9]. The accuracy of interpretation and quantitative inversion based on the HS images will be significantly reduced. Therefore, a preferred HS pansharpening algorithm should be implemented to enhance the spatial detail information in HS images, to protect the spectral information in HS images, and to generate accurate spectral information in high resolution.
Numerous HS pansharpening methods have been proposed over recent decades. On the basis of the underlying principles, current pansharpening methods can be broadly categorized into component substitution (CS), multi-resolution analysis (MRA), matrix factorization, and Bayesian approaches [
10,
11]. CS approaches separate the spatial and spectral information of the HS images by different transformations. The component with spatial details is replaced by the PAN image, and the components are inversely transformed to reconstruct the pansharpened image. Typical CS approaches include Gram–Schmidt adaptive (GSA) [
12], guided filter principal component analysis (GFPCA) [
13], etc. These algorithms provide good spatial information [
12]. However, due to the wavelength ranges of PAN and HS images are different, radiometric distortion is increased [
10]. MRA extract the spatial details by taking the difference between the original and low-pass filtered PAN image. The obtained spatial details are used to enhance the HS image. MRA involves many methods, such as modulation transfer function generalized Laplacian pyramid (MTF_GLP) [
14], smoothing filter-based intensity modulation (SFIM) [
15]. These methods perform well in robustness and efficiency [
16,
17], but suffer from aliasing effect and spatial details distortion [
18]. Matrix factorization decomposes the images based on the linear spectral mixture model. Then, the endmember matrix of the HS image and the abundance matrix of the multispectral image are multiplied to recover the pansharpened image. Matrix factorization provides good spectral qualities, but only applied in the fusion of multispectral and HS data [
19], which is different from pansharpening. Bayesian approaches first define appropriate prior distributions of the images, and obtain the posterior distribution following Bayes’ theorem. Then, the Bayesian estimators are computed for fusing the images. Well-known Bayesian approaches include wavelet-based Bayesian fusion [
20], maximum a posteriori (MAP) estimation fusion [
21], and variational approach [
22]. These methods do not limit the number of bands [
23], but suffer from high computational cost and large spectral distortion.
Variational approach is one of the promising pansharpening methods. The first variational approach is based on three assumptions: the first one describes that the HS and PAN images contain same geometry, the second one represents the relation of the PAN image to each HS bands, and the third one denotes that the low-resolution pixels equal to the convolution and downsampling of high-resolution pixels [
22]. Based on the assumptions, the constraint and regularize terms are proposed to construct an energy function, whose optimal solution is the pansharpened image. Following the idea, Moeller et al. [
24] extended the variational pansharpening to HS images by building a spectral correlation preserving term based on the definition of spectral angle mapper. Zhang et al. [
23] improved the HS variational method by assuming that fused bands should retain the correlation with the upsampled multi-spectral bands. Subsequently, Duran et al. proposed a band-decoupled variational method (NLVD) [
25], which uses a constraint to preserve the radiometric ratio between each spectral bands and the PAN image. On the other hand, to effectively preserve spatial information, Chen et al. [
26] introduced a dynamic gradient sparsity penalty, which exploits sharp boundaries from PAN images. Liu et al. [
27] proposed an optimizing method by integrating spatial fractional-order geometry and spectral–spatial low-rank priors. As a result, this method strengthens both the geometric and spectral preserving constraints. The aforementioned methods can achieve promising results. However, the spectral distortion may be caused by two reasons: (1) the spectral information may be changed in pansharpening process; (2) the spectral information of pansharpened images is much finer than that of the original HS images because of an increase in spatial resolution. Thus, the spectral preservation of variational methods should consider both the spectral feature preserving and the change in spatial resolution. Meanwhile, upsampled HS images often have blocky or blurry artifacts, which may lead to a low correlation between pansharpened and the actual observed images.
The present paper is extended from our previous work [
28], designing a variational pansharpening method for HS imagery constrained by spectral shape feature and image correlation. This work focuses on the spectral distortion caused by spatial resolution enhancement and the low correlation between the pansharpened and the actual observed images. Specifically, the two main contributions of this paper are:
We assume that the change in spatial resolution may lead to spectral information differences between pansharpened and low-resolution HS images. Based on this assumption, the new spectral fidelity term firstly uses the spectral shape feature to characterize the spectral information in the HS image. Then, the pixels in the neighboring window are assigned with different weights. The weights and the spectral shape features of the neighboring pixels are integrated to generate the spectrum at a high spatial resolution.
A new correlation fidelity term based on GSA method is designed to maintain the correlation between the actual observed high spatial resolution image and the pansharpened image. The result of GSA is used as our correlation constraint, which is obtained by inversely transforming the transformed bands of the HS image and the simulated intensity image.
Afterward, the proposed fidelity terms are combined with the spatial fidelity term in the classic variational pansharpening to construct a new energy function. The optimal solution of this function is the pansharpened image, which is computed by gradient descent method.
The remaining parts of this paper are organized as follows. We briefly introduce the classic variational pansharpening method in
Section 2. Our assumptions and the novel variational method is described in
Section 3. The evaluation method and the experimental results are presented in
Section 4, together with the comparison among the advanced pansharpening methods. The conclusions are provided in
Section 5.
2. Classic Variational Framework
The classic variational pansharpening framework [
22] is briefly introduced in this section. Let
be an open bounded image domain. We refer to the PAN image as
.
represents the original HS image, and
represents the pansharpened image, where
i represents the
ith band of the images.
n denotes the number of the bands of the HS image. On the basis of the definition of image pansharpening, three hypotheses are proposed to describe the relation between the test images and the pansharpened images. Then, the pansharpening problem can be regularized by optimizing an energy function based on the hypotheses.
The first hypothesis states that the geometry of the bands in pansharpened images is contained in the topographic map of corresponding PAN images [
25]. Since the pansharpened images and the PAN images contain the same area, the surface objects and features in the two images are nearly the same. Therefore, the geometry information, such as textures and edges, in the PAN images should be in consistent with that in the pansharpened images. The geometry information can be represented by the topographic map, which is acquired by the vector field. Then, the vector field of the PAN image should equal to that of each band in the pansharpened image. The spatial fidelity term is built as follows:
where
is the gradient of the pansharpened image
u,
is the normal vector field of the PAN image.
The second assumption expresses that the PAN image is a linear combination of the bands in the HS image with mixing coefficients
[
29]. Due to the bandwidth of the PAN image is wider than that of the HS image, the PAN image can be treated as a weighted sum of the narrow bands in the HS image. The weight of each band in the HS image is distributed according to the spectral response of the corresponding channel. On the basis of this idea, the PAN image can be synthesized from the bands in the pansharpened image. Then, the image relation preserving term is derived as follows:
The third assumption considers that a low-resolution pixel can be formed from the high-resolution one by low-pass filtering followed by subsampling [
25]. Compared with the high spatial resolution image with the same area, the low spatial resolution image consists of less pixels, while the area of each pixel is larger. The low-pass filter makes the pixels contain neighboring information. The subsampling helps reduce the number of the pixels in an image. In this way, the low spatial resolution HS image can be simulated from the pansharpened image. This relation is expressed as follows:
where
is convolution kernel of the low-pass filter, and
is a down-sampling process using grid
S.
Then, the three terms are used to construct the energy function as follows:
where
are the weighting coefficients of the corresponding terms. The Equation (4) constraints the properties of the pansharpened image to be consistent with that of the PAN and HS images. Thus, the low energy of this function indicates that the pansharpened image satisfies the desired properties. Many methods have been used to find the minimization and obtain the resulting image, such as gradient descent method and a parabolic equation.
5. Conclusions
In this paper, we proposed a novel variational pansharpening method for HS imagery constrained by spectral shape feature and image correlation. First, we propose the constraints of spectral shape feature and the correlation preservation. Second, the spectral and correlation fidelity terms are constructed. Finally, we combine these fidelity terms to build the new energy function, and obtain the resulting image based on the optimal solution. Our main contributions are that the new spectral fidelity term and the new correlation fidelity term are designed in the variational framework. The spectral fidelity term uses the difference between bands and the mean value of images to correct the spectral distortion. We assumed that the spatial resolution change may lead to spectral information difference. The neighboring pixels are also considered by distributing the weights in this term. The correlation fidelity term preserves the correlation of the pansharpened image based on the GSA method.
Experiments on three datasets from HJ-1A and EO-1 satellites show that the novel method performs well in terms of spatial and spectral fidelity. The overall performance of the proposed method is better than the comparative methods, including GSA, GFPCA, MTFGLP, SFIM, the classic variational pansharpening and NLVD. In the quantitative evaluation, the mean values of SAM, ERGAS, RMSE, CC and UIQI (in
Table 7) of the proposed method are 3.2789, 4.0983, 228.6753, 0.9367 and 0.8781, respectively, which have been both improved from 3.9795, 5.6598, 309.6987, 0.9040 and 0.8127 of the classic variation method. The mean computation time of our method are 16.84 s. The algorithm efficiency meets the practical requirements in applications.
However, the performance of the proposed method is limited by two problems. First, the enhancement of the spatial details may lead to the distortion in spectral information. The mutual promotion between the spatial and spectral qualities is difficult to be realized by the current constraints. Second, the proposed spectral fidelity terms may produce large errors when processing mixed pixels. In the future work, we will focus on developing new constraint, which could integrate the spatial and spectral information to improve the comprehensive quality of pansharpened images. In addition, Automatic parameter optimization method will be further considered in our future studies.