A Two-Stage Pansharpening Method for the Fusion of Remote-Sensing Images

Abstract

The pansharpening (PS) of remote-sensing images aims to fuse a high-resolution panchromatic image with several low-resolution multispectral images for obtaining a high-resolution multispectral image. In this work, a two-stage PS model is proposed by integrating the ideas of component replacement and the variational method. The global sparse gradient of the panchromatic image is extracted by variational method, and the weight function is constructed by combining the gradient of multispectral image in which the global sparse gradient can provide more robust gradient information. Furthermore, we refine the results in order to reduce spatial and spectral distortions. Experimental results show that our method had high generalization ability for QuickBird, Gaofen-1, and WorldView-4 satellite data. Experimental results evaluated by seven metrics demonstrate that the proposed two-stage method enhanced spatial details subjective visual effects better than other state-of-the-art methods do. At the same time, in the process of quantitative evaluation, the method in this paper had high improvement compared with that other methods, and some of them can reach a maximal improvement of 60%.

1. Introduction

To overcome the trade-off between the spatial and spectral resolutions of remote-sensing images, pansharpening (PS) fuses geometric details of a panchromatic (PAN) image with the spectral information of a multispectral (MS) image to obtain a high-resolution MS image where the PAN image is the high-resolution image and the MS image is the low-resolution image. PS methods mainly include the following categories: component substitution (CS), multiresolution analysis (MRA), variational optimization (VO), and deep learning (DL).

The class of CS methods first projects the MS image into a new space on the basis of a spectral transformation and then substitutes the matching spatial part by the PAN image, and finally obtains the fused MS image through the inverse projection. The representative works of CS methods include the intensity hue saturation (IHS) method, principal component analysis (PCA) [1], and the Gram–Schmidt adaptive (GSA) [2] method. The main idea of the IHS method, as a classical work in CS methods, is to first perform IHS transform on an upsampled MS image, replace the I intensity part in the IHS space using the histogram-matched PAN image, and perform inverse IHS transform to obtain a fusion result. The IHS method runs with fast efficiency and low computational complexity. There are many improved methods based on IHS, such as the generalized IHS (GIHS) [3], matting model [4], adaptive IHS (AIHS) [5], and improved adaptive IHS (IAIHS) [6], evolutionary optimization IHS (EIHS) [7], and multiobjective IHS (MIHS) [8] methods, in addition to the band-dependent spatial detail (BDSD) [9,10] method, the adaptive fusion method based on component replacement [11], clustering method based on mixed pixels [12], and the combination of IHS and PCA [13]. The CS method has the advantage of fast computational efficiency, but problems such as spectral distortion usually arise because of the difference in PAN images and the inclusion of spatially detailed parts.

The class of MRA methods is to keep composite approaches that use multiscale decomposition. The classical methods are wavelet-transform-based [14], high pass filter (HPF) fusion [15], generalized Laplacian pyramids (GLP) based [16], GLP with robust regression [17], morphological operator-based fusion (MF) [18], and smoothing filter based intensity modulation (SFIM) [19] methods. The HPF method mainly uses box filter and additive injection image for fusion, but different filter sizes produce serious spectral distortion. The GLP method mainly relies on MTF filter using MS sensor. The MF method mainly uses nonlinear method for image fusion. The SFIM method adopts the same box filter as the HPF method. The difference is that the weight coefficient is the ratio of the number of bands of MS image to the pan image after downsampling. Due to the redundancy of the spatial details extracted by MRA methods, there are problems such as spatial degradation in the fusion process.

The VO method mainly uses the prior knowledge of the image to show target energy function and solves it by using an optimization algorithm. Some classical methods are proposed for PS [20,21,22], these methods are based on the linear relationship between the input image and the fused image. Others build models based on a variety of prior information [22,23]. In addition, there are some methods based on tensor decomposition [24,25], super-resolution methods [26,27] and other variational methods [28,29,30,31,32]. Since the VO method uses an optimization algorithm to get the optimal solution, it can effectively reduce the spectral distortion to a certain extent, but to select parameters in the optimization process easily leads to a non-global optimal solution, which greatly impacts the quality of the fused image.

DL-based studies were applied to the PS method. The PS method employs deep neural networks (DNNs) [33], which are mainly used it to the depth networks utilizing image blocks from the input images. There are also the neural network method (PNN) [34] and adaptive CNN-based pansharpening (A-PNN) [35] dedicated to PS. The residual-network-based PS method [36] is sampled from the input image without other training images, and the method uses the trained network to reconstruct the fused image. The progressive cascade deep residual network (PCDRN) model [37] is trained on the basis of image patches and uses residual-learning to optimize the network. The unsupervised generative adversarial network (PAN-GAN) method [38], this process does not depend on the so-called ground truth images in the network training, but uses the new images generated by the GAN for PS. A new deep detail network architecture based on packet multiscale extended convolution was proposed in [39]; the structure uses an end-to-end network to directly fuse MS and PAN images to produce fusion results. Deep convolutional neural networks (DCNNs) [40] mainly combine a neural network with CS and MRA fusion schemes and use these two algorithms to estimate the fused image for the nonlinear injection model. A Laplacian pyramid PS network structure was proposed in [41] in which the input image is first decomposed into pyramids, and the fusion convolution neural network is used to fuse the results of pyramid decomposition.A deep spatial–spectral global reasoning network is proposed to consider the local and global information of the image is proposed in [42]. PS with spatial and spectral gradient difference-induced nonconvex sparsity priors (PSSGDNSP) [43] uses the eigenband correlation of MS images to process MS images as third-order tensors. In addition, there are some fusion methods based on DL [44,45,46,47,48,49,50,51,52,53,54]. The DL method’s main disadvantages are the lack of ideal PS samples for training, it relies on generating reference samples from unlabeled real data (such as MS images).

For an image, the gradient can best describe the shape and portray the edges of the target. Commonly used gradients are total variation [55], higher-order, and global gradients, where the global gradient is mainly considered to fully use the overall information of the image, such as the nonlocal TV model [56] and global sparse gradient (GSG) [57].

In addition, according to existing research, the global extraction of information from input images is unstable. The main contributions of this paper can be summarized as follows.

(1)

We developed a two-stage PS method based on the CS and variational models, namely, the global sparse gradient-based improved adaptive IHS (GIAIHS) method, and reduced the instability of fused image global information.

(2)

We used the GSG information of the image to construct the weight function. GSG is a better representation of the accuracy and robustness gradient information of an image, and we used variational ideas to obtain the optimal solution for the GSG information of the image.

(3)

As all existing methods currently use a one-stage direct fusion method to obtain fusion results, loss of information during the fusion process is not considered. In this paper, a two-stage PS fusion algorithm was designed on this basis, which further refines the image for direct fusion, greatly improving the null spectral information of the image. In addition, the method can meet different satellite data needs and maintain a balance between spatial enhancement and spectral fidelity.

The remainder of this paper is organized as follows. Section 2 presents the basic knowledge needed to frame this paper. Section 3 describes our proposed framework in detail. Section 4 shows both the qualitative and the quantitative analyses through experimental results. Lastly, this paper draws conclusions and discusses future work in Section 5.

2. Related Works

The CS method is popular because of its fast operation efficiency. GIHS [3] can process multichannel images, but it needs many calculations during processing. At the same time, this method only considers the influence of partial information of an MS image on the fused image, so it is not only inefficient but also prone to spectral distortion. AIHS [5] mainly uses the gradient information of PAN image to constrain the spatial details. After that, IAIHS [6] considers the gradient information of PAN and MS images, builds a new weighting matrix, and obtains better spatial information fusion capability than the AIHS method. The EIHS [7] method considers the relationship between fused and given images by objective function, and obtains the best control parameters for rebuilding the high-resolution MS image according to the optimization algorithm. MIHS [8] transforms the PS problem into a multiobjective optimization problem by showing an objective function. Although CS is efficient, it produces serious spectral distortion in the process of fusion.

Most VO methods are from model construction to model solution. The P + XS method [20] uses image gradient features to characterize the spatial information of PAN image, build an optimization model, and solve it with variational ideas. In contrast to the spatial information used in the aforementioned methods, the reduced-rank (RR) method [21] considers the spectral low-rank relationship between PAN and fused images by using $l_2$ norm to construct regularization terms, and the consistency of PAN and fusion images is simulated. In order to maintain spectral similarity while eliminating blur, prior knowledge of the original image is introduced using total variation (TV) [22], sparse representation [58], and nonlocal prior [23]. There are some methods based on tensor decomposition mainly using the representation of MS image as three-dimensional tensors, and using spectral dictionaries to approximate the fusion results, such as image fusion methods based on coupled sparse tensor decomposition [24] and low-rank tensor-based methods [25]. A special class of variational based super-resolution methods [26,27] has also been widely studied in recent years. In addition, the LGC [28] method mainly uses the local gradient information of the image to transform it into a convex optimization problem. The method proposed in [29] mainly uses Toeplitz to express the correlation between adjacent bands. The method proposed in [30] was mainly based on the assumption of the sparsity of the fused image under B-spline frame transformation, and uses the linear correlation between the fused image and MS image to fuse. The method proposed in [31] decomposes the image into a sparse kernel tensor based on tensor decomposition, and then multiplies it with the three modal dictionary to get the fusion result. The method proposed in [32] obtains the fusion result based on the super Laplace error distribution between the upsampled MS image and the fused image in the gradient domain. However, the above variational methods are based on model construction and finally get the fused image. Due to different constraint structures, different fusion results and different degradation levels are produced.

Combined with the above analysis, to obtain better fusion results and avoid degradation due to constraints, we transformed the model-based variational into data-based variational, that is, the GSG obtained through the variational idea further characterizes the image. At the same time, to speed up efficiency, the CS method was selected for fusion.

A two-stage remote-sensing image PS fusion method (GIAIHS) was designed here. First, GSG information about PAN images is solved, and the weighted matrix is defined by the gradient information on MS image, which is substituted into the fusion framework to obtain the first stage fusion result. Second, GSG information of the first-stage fusion image is solved, and a new weighting matrix is defined by the GSG information on PAN image, which is substituted into the fusion framework to obtain the second-stage fusion results. Algorithm 1 show the algorithm pseudocode and of the method proposed in this paper. The code is available at the following GitHub link: https://github.com/Yazhen1/GIAIHS.

Algorithm 1: GIAIHS algorithm. Proposed algorithm for two-stage restoration.

Input: MS image: M, PAN image: P. Output: Fusion image: $M_k^z$.   $Q_P\left(x\right)$←PAN image;   ${\omega }_P$$\leftarrow Q_P\left(x\right)$;   ${\omega }_{M_k^{one}}$← MS image gradient;   ${\omega }_k^{one}\leftarrow {\omega }_{M_k^{one}}$ and ${\omega }_P$; $M_k^P\leftarrow M_k+{\omega }_k^{one}(P-I)$;   $Q_M\left(x\right)$←$M_k^P$   ${\omega }_{M_k^{two}}$$\leftarrow Q_M\left(x\right)$;   ${\omega }_k^{two}\leftarrow {\omega }_{M_k^{two}}$ and ${\omega }_P$; $M_k^z\leftarrow M_k^P+{\omega }_k^{two}(P-I)$.

3. Proposed Method

3.1. GIAIHS Fusion Model

The GIAIHS method can be expressed as

$$M_k^P=M_k+{\omega }_k(P-I),k=1,2,\cdots ,N.$$

(1)

where $M_k^P$ is the k band of the fused image, $M_k$ is the k band of the MS image after upsampling (in the process of upsampling, the MS image needs to be upsampled to the same size of the PAN image, such as a cubic spline interpolation algorithm), P is the PAN image, $I={\sum }_{k=1}^N{\beta }_k{M}_k$, N is the total number of bands in the image, ${\beta }_k$ is the weight parameter of the MS image band after upsampling. ${\omega }_k$ is the weighting matrix and is obtained by the equation.

$${\omega }_k=\frac{M_k}{\frac{1}{N}{\sum }_{k=1}^N{M}_k}(\alpha {\omega }_P+(1-\alpha ){\omega }_{M_k}),$$

(2)

where $\alpha$ is the weight parameter. ${\omega }_{M_k}$ and ${\omega }_P$ are weighted matrices guided by gradient information of MS image and PAN image.

$$\omega =exp(\frac{-\lambda }{{\left|Q\left(x\right)\right|}^4+ϵ}),$$

(3)

where $Q\left(x\right)$ represents the gradient. If $Q\left(x\right)$ represents the gradient of the input MS image, the result is ${\omega }_{M_k}$. If $Q\left(x\right)$ represents the gradient of the input PAN image, the result is ${\omega }_P$. $\lambda$ and $ϵ$ are nonzero normal numbers to avoid too-large molecules and zero denominators.

For input image $f\left(x\right)$, $x\in \mathrm{\Omega }\subset {\mathbb{R}}^2$, $\mathrm{\Omega }$ is a bounded region; for gradient $Q\left(x\right)$, the following equation is given by

$$Q^{*}\left(x\right)=\underset{Q\left(x\right)=(q_1\left(x\right),q_2\left(x\right))}{argmin}(D\left(Q\left(x\right)\right)+\lambda R\left(Q\right(x\left)\right)),$$

(4)

where $\lambda >0$ is the regularization parameters, $Q\left(x\right)$ is the image gradient, $D\left(Q\right(x\left)\right)$ is the fidelity item; the first-order Taylor expansion is used to construct and estimate the gradient with estimation points with the help of the points in the neighborhood of the point to be estimated. $R\left(Q\right(x\left)\right)$ is a regularization term.The above formula can also be simplified to this equation:

$$Q^{*}\left(x\right)=\underset{Q\left(x\right)=(q_1\left(x\right),q_2\left(x\right))}{argmin}\{\frac{1}{|\mathrm{\Omega }|}\int {\int }_{\mathrm{\Omega }\times \mathrm{\Omega }}{w}_{xy}^S{(f\left(x\right)-f\left(y\right)+Q\left(x\right)(y-x))}^{2}dydx+\lambda {\parallel Q\left(x\right)\parallel }_{L^1})\},$$

(5)

where $w_{xy}^S=exp(-\frac{{|y-x|}^2}{S^2})$ is the weight function that keeps local similarity, S is the control decay rate parameter of the weight function, and $Q\left(x\right)$ is the optimal solution. For the calculation of the optimal solution of the equation, readers are referred to [57].

3.2. Weight Function

The gradient had a good description of the edge of the target, we used the data from the IKONOS satellite for gradient comparison Figure 1. Figure 1a is the original PAN image, Figure 1b is the gradient map of Figure 1a, and Figure 1c is the GSG map of Figure 1a. The small red box in the figure is the local image, and the large red box is the local enlarged image.

The figure shows that the edge of the original image is very obvious, which shows that the edge description of the original image had to be relatively close to the PAN image to better reflect the edge details of the pan image. From the GSG and original PAN images in Figure 1a, it is obvious that the two images had good similarities on the global edge of the image, especially in the relatively unobvious places, which can also show good edges and reflect the details. However, there is a serious blur phenomenon in the gradient image of Figure 1b, which had low similarity compared with the PAN image and could show less edge information. The comparative observation of Figure 1b,c shows that the GSG graph in Figure 1c had a significant increase in the extraction of edge information relative to Figure 1b.

3.3. Results

Because the model proposed in this paper is a two-stage PS model, to verify its effectiveness, an ablation experiment was preliminarily carried out.

For the proposed GIAIHS model, the IKONOS satellite contained a high-resolution PAN image (1 m resolution) and a low-resolution MS image (4 m resolution). IKONOS satellite data were selected for preliminary fusion analysis, and results are shown in Figure 2. The PAN image size of IKONOS satellite data was 400 × 400 pixels, and the MS image size was 100 × 100 pixels. The used MS image included four channels: red, blue, green, and near-infrared bands. Figure 2 shows that, through the effective fusion in the first stage compared with the IAIHS method, the quality of the fused image was greatly improved, especially for details such as boundaries.

Analyzing the fusion results in Figure 2 showed that they were better in the first stage than those in the IAIHS method in terms of edge details; fusion results in the second stage were simultaneously better than those in the first stage in terms of edge details.

$D_s$, $D_{\lambda }$ and QNR are used for objective evaluation. $D_s$ can measure the spatial information of the fused image, and $D_{\lambda }$ can measure the spectral information of the fused image, but their description of the overall information of the image is poor. The lower its value is, the better the spatial and spectral information of the fused image is. QNR can better measure the overall information of the fused image, and the higher the value is, the better the fusion effect. Figure 3 shows the comparison results of the three above indicators, where lower the purple and blue columns indicate better results, and yellow columns indicate better results. Figure 3 shows that the evaluation indices in the second stage had better results than those in the first stage. At the same time, the AIHS and IAIHS methods in the same CS method were compared to test the effectiveness of the model. The comparison showed hat the fusion results of the second stage of the GIAIHS model were better than those of the two methods regarding visual effect and evaluation index. To verify the feasibility of the model on the remaining satellite data, experiments with the MRA, VO, and ML methods were carried out.

To better illustrate the model in this paper, the flow chart of the algorithm is given in Figure 4, where ${\omega }_k^{one}$ and ${\omega }_k^{two}$ are different weighting functions, as detailed in Algorithm 1.

4. Experiments and Analysis

4.1. Experimental Setup

All experiments were run on Windows 10 (64 bit) PC-Intel(R) Core (TM) i5-4210U CPU 2.40 GHz, 4 GB of RAM using MATLAB R2018b. We selected five different methods to compare with the GIAIHS method, namely, two CS methods (IAIHS and GSA), an MRA method (HPF), a VO method (RR), and a DL method (A-PNN). Due to the lack of corresponding high-spatial-resolution MS images in the experiment, the commonly used objective evaluation method is to downsample the fused image to the same size as the original MS image, and use the original MS image as the reference image for evaluation. In this paper, dimensionless global relative error of synthesis (ERGAS) [59], $Q4$ [60], relative average spectral error (RASE) [61], root mean squared error (RMSE), quality with no reference (QNR) [62], spectral distortion index $D_{\lambda }$, and spatial distortion index $D_s$ were used to evaluate the quality of the fused image.

The ERGAS index is a normalized dissimilarity index:

$$ERGAS=100\frac{d_h}{d_l}\sqrt{\frac{1}{N}\sum _{k=1}^N{(\frac{RMSE\left(k\right)}{\mu \left(k\right)})}^2},$$

(6)

where $\frac{d_h}{d_l}$ is the ratio between the pixel sizes of PAN and MS, N is the total number of bands in the image, and $\mu \left(k\right)$ refers to the mean value of K band. The lower the value of ERGAS was, the higher the correlation between fused and MS images, that is, the better the quality of the fused image.

The Q4 index was derived from $Q{2}^n$ index, and formula $Q{2}^n$ is as follows:

$$Q=\frac{{\sigma }_{z\tilde{z}}}{{\sigma }_z{\sigma }_{\tilde{z}}}·\frac{2{\sigma }_z{\sigma }_{\tilde{z}}}{{\sigma }_z^2+{\sigma }_{\tilde{z}}^2}·\frac{2\mid \overline{z}\mid \mid \overline{\tilde{z}}\mid }{\mid \overline{z}{\mid }^2\mid \overline{\tilde{z}}{\mid }^2},$$

(7)

where $z=z(m,n)$ and $\tilde{z}=\tilde{z}(m,n)$ represent the pixel spectral vector of MS and fused images, and $Q{2}^n$ refers to the average value of the whole image, which is usually divided into $N\times N$ block.

The RASE index is defined as

$$RASE=\frac{100}{M}\sqrt{\frac{1}{N}\sum _{k=1}^N{RMSE}^2(F_k,M{S}_k)},$$

(8)

where $F_k$ represents the k band of the fused image, and $M{S}_k$ represents the k band of the MS image, which is used to calculate the global spectral quality of the fused image. The smaller the value of RASE is, the better the quality of the fused image.

The RMSE index is defined as

$$RMSE(I,J)=\sqrt{{(I-J)}^2},$$

(9)

When I = J, RMSE reaches the ideal value of zero. It is used to calculate the average difference between fused and MS images. The smaller the value of RMSE is, the better the quality of the fused image.

The assumption of the QNR index is that the similarity between each MS band and PAN should remain unchanged before and after fusion. Before calculating QNR, spectral distortion index $D_{\lambda }$ and spatial distortion index $D_s$ need to be calculated.

Spectral distortion index $D_{\lambda }$ is estimated by

$$D_{\lambda }=\sqrt[P]{\frac{1}{N(N-1)}\sum _{i=1}^N\sum _{j=1,j\ne i}^N|Q(\widetilde{M{S}_i},M{S}_j)|-|Q({\widetilde{MS}}_i,{\widetilde{MS}}_j){|}^P},$$

(10)

the closer the value of $D_{\lambda }$ is to 0, the greater the spectral distortion of the fused image and the better the quality of the fused image are.

Spatial distortion index $D_s$ is estimated by

$$D_s=\sqrt[P]{\frac{1}{N}\sum _{i=1}^N|Q(\widetilde{M{S}_i},P)|-|Q(M{S}_i,P_L){|}^P},$$

(11)

where $P_L$ is the image with the same size from the PAN image downsampling to MS image. The closer the value of $D_s$ is to 0, the greater the spatial distortion of the fused image and the better the quality of the fused image are.

The QNR index is estimated by

$$QNR={(1-D_{\lambda })}^{\alpha }{(1-D_s)}^{\beta }.$$

(12)

it is obtained by weighting $D_{\lambda }$ and $D_s$ by $\alpha$ and $\beta$; the higher the QNR value is, the better the quality of the fused image. When both $D_{\lambda }$ and $D_s$ are 0, QNR theoretically reaches the optimal value of 1.

4.2. Datasets

To further verify the effectiveness of the proposed model, experimental comparison and analysis were carried out on QuickBird, Gaofen-1, and Worldview-4 satellite data. The QuickBird satellite contains a high-resolution PAN image (0.7 m resolution) and a low-resolution MS image (2.8-m resolution). The Gaofen-1 satellite contains a high-resolution PAN image (2-m resolution) and a low-resolution MS image (8-m resolution). The Worldview-4 satellite contains a high-resolution PAN image (0.31-m resolution) and a low-resolution MS image (1.24-m resolution). The PAN image size used by QuickBird satellite data is 800 × 800 pixels, and the MS image size is 200 × 200 pixels. The PAN image size of Gaofen-1 satellite data is 1024 × 1024 pixels, and the MS image size is 256 × 256 pixels. In worldview-4 satellite data, the PAN image size is 1024 × 1024 pixels, and the MS image size is 256 × 256 pixels. The MS images used include four channels: red, blue, green, and near-infrared. The image was corrected for radiation and sensor distortion, and the acquisition effect was eliminated. In addition, collected satellite images were corrected for viewing angle and ground effect so that they can be superimposed on the map. In addition, orthophoto correction is carried out to eliminate the perspective effect on the ground. The best input format of data is the TIF format.

4.3. Experiments and Analysis

The subjective visual comparison results of three different satellite data fusion are shown in Figure 5, Figure 6 and Figure 7, and a local map is also given for further observation. The red block diagram on the center is the selected local terrain, and the lower right corner is the local enlarged view corresponding to the center side.

Because the resolution of the MS image used in this paper was inconsistent with that of the PAN image, the resolution of the MS image needed to be upsampled to be consistent with that of the PAN image, and then fused by PS method. The resolution of the fused image was consistent with that of the PAN image. In the process of visual analysis, to better reflect the advantages of this method, the upsampled image of MS image was compared with other fused images, which could better reflect the differences of various methods in quasitube vision. In quantitative analysis, there was a lack of high-resolution reference images. According to Wald’s protocol [63], the fused image was downsampled to the same size as the original MS image, and the downsampled fused image is quantitatively analyzed with MS image.

These three sets of satellite data were chosen for the following reason: the terrain shown in QuickBird satellite data includes urban roads, rural areas, and a large number of forestry areas. The terrain shown in the Gaofen-1 satellite data is mainly waterways with a large number of urban areas beside it, which is complex. The terrain displayed in Worldview-4 satellite data is mainly forestry, which can highlight the details.

Remote-sensing images obtained by QuickBird satellite include rural and forestry areas. Fusion results are shown in Figure 5. PS results obtained by the HPF and RR methods lost a lot of spectral information, resulting in overall color changing. GSA and IAIHS showed that the visual effect of these two methods is too bright in some areas. For example, the color of the boundary in the enlarged image (Figure 5c,d) was significantly different from the actual color. Fusion results obtained by the A-PNN method had similar visual effects. It can be seen from the enlarged image that the method proposed in this paper can obtain clearer PS results. The objective evaluation results of rural areas are shown in

Table 1. Experiment with six PS methods on QuickBird satellite data.

Method	RMSE	RASE	ERGAS	Q4	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{s}}$	QNR
IAIHS	26.92	8.20	1.95	0.81	0.17	0.10	0.75
GSA	34.50	10.50	2.72	0.68	0.31	0.28	0.49
HPF	60.47	18.40	4.88	0.42	0.17	0.19	0.67
RR	33.19	10.10	2.28	0.78	0.16	0.11	0.75
A-PNN	28.85	8.78	2.28	0.74	0.21	0.04	0.76
GIAIHS	25.54	7.77	1.84	0.88	0.11	0.05	0.85

. The best performance is shown in bold.

In the city image taken by Gaofen-1 satellite, the comparison diagram between this method and five other methods is given in Figure 6, which shows that the description of the outline of urban areas by the IAIHS and RR methods was not clear enough, and the roofs of some buildings turned gray. At the same time, details of the house showed that the GSA method had serious spatial distortion. The outline produced by the HPF method was too enhanced, and the image color was too bright, which produced artifacts. However, the middle road of the A-PNN method was close to white, and there is no strong color display. This method preserves a lot of spectral information in the experiment and fully uses the spatial information of PAN images. The objective evaluation of the city image is shown in

Table 2. Experiment with six PS methods on Gaofen-1 satellite data.

Method	RMSE	RASE	ERGAS	Q4	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{s}}$	QNR
IAIHS	27.47	7.36	1.86	0.84	0.14	0.29	0.61
GSA	47.27	12.67	3.24	0.68	0.33	0.50	0.33
HPF	46.05	12.34	3.10	0.71	0.09	0.32	0.62
RR	33.36	8.94	2.28	0.81	0.13	0.23	0.67
A-PNN	31.09	8.33	2.15	0.83	0.07	0.27	0.68
GIAIHS	23.66	6.34	1.61	0.89	0.07	0.15	0.79

. The best performance is shown in bold.

Figure 7 is a forest vegetation image taken by the WV-4 satellite. The fused image obtained by the HPF method was severely distorted, and the color of the trees tended to be blue. This method loses a lot of spectral information when balancing spatial and spectral information. In addition, spectral information of the GSA method was distorted due to the mismatch of spectral ranges. The spectra of the IAIHS and RR methods were relatively well-preserved, but some spatial details were lost. Spectral information of fusion results obtained by the A-PNN method was good, but some fuzzy effects could be found from the vegetation region. The method in this paper can fully use spectral information in MS images and spatial information in PAN images. The objective evaluation of forest vegetation is shown in

Table 3. Experiment with six PS methods on WorldView-4 satellite data.

Method	RMSE	RASE	ERGAS	Q4	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{s}}$	QNR
IAIHS	28.38	7.61	1.40	0.75	0.16	0.10	0.76
GSA	36.15	9.69	1.91	0.78	0.14	0.16	0.72
HPF	38.08	10.21	3.26	0.48	0.19	0.18	0.66
RR	35.38	9.49	1.80	0.68	0.16	0.14	0.72
A-PNN	26.35	7.07	1.65	0.77	0.09	0.10	0.82
GIAIHS	26.90	7.21	1.27	0.83	0.11	0.04	0.86

. The best performance is shown in bold.

Table 1 shows the comparison result of QuickBird satellite data. Table 1 shows that the GIAIHS method showed great improvement compared with other methods. For the RASE index, the GIAIHS method improved by approximately 26% compared with the GSA method. For the ERGAS index, the GIAIHS method improved by approximately 19% compared with the RR method. For Q4, GIAIHS method improved by approximately 20% over the A-PNN method. For $D_{\lambda }$, GIAIHS method improved by approximately 35% over IAIHS method.

Table 2 shows the comparison result of Gaofen-1 satellite data. Table 2 shows that, for the RASE index, the GIAIHS method improved by approximately 50% compared with the GSA method. For the ERGAS index, the GIAIHS method improved by approximately 29% compared with the RR method. For Q4, GIAIHS method improved by approximately 7% over A-PNN method. For $D_s$, GIAIHS method improved by approximately 48% over IAIHS method.

Table 3 shows the comparison result of Worldview-4 satellite data. Table 3 shows that, for the RASE index, the GIAIHS method improved by approximately 26% compared with the GSA method. For the ERGAS index, the GIAIHS method improved by approximately 29% compared with the RR method. For Q4, GIAIHS method improved by approximately 8% over A-PNN method. For $D_s$, GIAIHS method improved by approximately 60% over IAIHS method.

Table 1 shows that, for $D_s$, the GIAIHS method differed from the A-PNN method by only approximately 0.01 in the QuickBird data, but GIAIHS was greatly improved compared with other methods. At the same time, for the QNR index, GIAIHS method improved by approximately 12% over A-PNN method. Table 2 shows that, for $D_{\lambda }$, the GIAIHS and A-PNN methods achieved the same performance level in Gaofen-1 data; for the QNR index, GIAIHS method improved by approximately 16% over A-PNN method. Table 3 shows that, for $D_{\lambda }$, the GIAIHS method differed from the A-PNN method by only approximately 0.02 in Worldview-4 data; for the QNR index, GIAIHS method improved by approximately 4% over A-PNN method.

Figure 8 shows the RMSE and ERGAS evaluation results of the six methods for three different satellites’ data. The method had good stability, which was maintained in the evaluation of three different satellite data in this paper. RMSE evaluation results and the fluctuation of ERGAS displayed in Figure 8 show that the method had good robustness. This method is highly competitive with the rest of the methods. In addition, from a statistical point of view, the method proposed in this paper is better than other methods.

To better illustrate the spectral effect of the method in this paper, QB satellite data results were selected for comparative analysis. Because the resolution of the fused image was different from that of the original MS image, the downsampling of the fused image was consistent with that of the original MS image for comparative analysis. Figure 9 shows the comparative analysis of the original MS image and the results of six PS methods, in which the x axis represents the 61 selected pixels (because image resolution was 256 × 256, 61 pixels were selected in the middle part of the image), and the y axis represents the pixel value corresponding to the pixels. The fusion experiment in this paper was carried out in four bands. Figure 9a shows the comparison results of pixels selected by different methods in the R band, Figure 9b shows the comparison results of pixels selected in G band, Figure 9c shows the comparison results of pixels selected in B band, and Figure 9d shows the comparison results of pixels selected in NIR band.

The black line segment in Figure 9 is marked with 61 pixel values corresponding to the original MS image, and the six other line segments are pixel values corresponding to the IAIHS, GSA, HPF, RR, A-PNN, and GIAIHS methods, respectively. Figure 9a shows that the obtained results by the GIAIHS method were very close to the results of the original MS image, followed by A-PNN results and RR results. In Figure 9b, some pixel values of GIAIHS method are consistent with those of the original MS. There was not much difference between the GSA and HPF methods, but they are still very different from the pixel value curve of MS. Although the A-PNN method was better than the GIAIHS method, there were still some differences. Figure 9c shows that the results of the RR method and A-PNN were not very good. There was large deviation in some points, which was better than IAIHS. Only a few points deviated from the Ms results, while GIAIHS almost had a small gap with MS. The above analysis proves that GIAIHS had good performance in the spectrum.

In general, the GIAIHS method not only outperformed state-of-the-art methods, but also had some competitiveness with the DL method of A-PNN method, which also shows that the objective evaluation of the GIAIHS method is well consistent with the subjective evaluation.

5. Conclusions

In this paper, we proposed a two-stage fusion method for PS by combining the CS method with the variational idea. We used a GSG that could more fully reflect the image texture and edge information for the construction of the weight function, and further refine the fusion results to improve the PS results effectively. The GIAIHS method significantly improves the quality of the fused images, and also improves the generalization ability of the method to data from different satellites.

In performing PS, the main objective is to maintain the integrity of spectral information of MS images and spatial information of PAN images. First, we chose GSG as the weight function to fully extract the information of the image. Second, for obtained results by direct fusion, we used IKONOS data to analyze them, and performed a two-stage thinning process. Results showed that it was necessary to perform this thinning process. Experimenting on different datasets demonstrated that our proposed two-stage PS method is effective in obtaining better-quality fused images compared to other methods. Since our proposed method is required to extract more detailed information from the images, better results could be obtained when processing higher dimensional as well as more complex images. The method is better at processing spectral and spatially informative images, while it can effectively capture rich features from dense buildings and dense vegetation, which helps in generating satisfactory high-quality fusion images.

The method proposed in this paper combines the classical CS and variational methods. Experimental results showed that the idea proposed in this paper is effective. The advantage of this model is that it produced good experimental results for different satellite data from both subjective vision and quantitative indicators, and had strong generalization ability. The disadvantage is that the proposed model had more parameters. In the next step, it was changed into an adaptive parameter model. For a future research direction, because the methods that could describe image features in detail are very limited, and remote-sensing image PS based on deep learning have developed rapidly in recent years, future classical methods can be integrated into deep learning and models driven for PS. Next, we also aim to apply the plug-and-play idea [64] to the classical method. Since the MS images used in this paper had four bands, more bands of images will be PS-fused.