MS-Pansharpening Algorithm Based on Dual Constraint Guided Filtering

Abstract

The difference and complementarity of spatial and spectral information between multispectral (MS) image and panchromatic (PAN) image have laid the foundation for the fusion of the two types of images. In recent years, MS and PAN image fusion (also known as MS-Pansharpening) has gained attention as an important research area in remote sensing (RS) image processing. This paper proposes an MS-Pansharpening algorithm based on dual constraint Guided Filtering in the nonsubsampled shearlet transform (NSST) domain. The innovation is threefold. First, the dual constraint guided image filtering (DCGIF) model, based on spatial region average gradient correlation and vector correlation formed by neighborhood elements is proposed. Further, the PAN image detail information extraction scheme, based on the model, is provided, which extracts more complete and accurate detail information, thus avoiding, to some extent, the spectral distortion caused by the injection of non-adaptive information. Second, the weighted information injection model, based on the preservation of the correlation between the band spectra, is proposed. The model determines the information injection weight of each band pixel based on the spectral proportion between bands of the original MS image, which ensures the spectral correlation between bands of the fused MS image. Finally, a new MS-Pansharpening algorithm in NSST domain is proposed. The MS and PAN high frequency sub-bands of NSST are used to extract more effective spatial details. Then the proposed DCGIF model is used to extract the effective spatial detail injection information through the weighted joint method based on the regional energy matrix. Finally, the weighted information injection model is used to inject it into each band of MS to complete information fusion. Experimental results show that the proposed approach has better fusion effect than some conventional MS-Pansharpening algorithms, which can effectively improve the spatial resolution of the fused MS image and maintain the spectral characteristics of MS.

1. Introduction

The basic task of RS is to measure and record electromagnetic wave radiation signals reflected or emitted from the earth’s surface by sensors carried on airplanes or satellites, and to form different RS images according to different times, wavelengths or frequencies [1,2]. RS images with different spectral resolutions can be formed by the sensor according to the number of bands, such as PAN, MS and Hyperspectral (HS) images. Sensors can form RS images with different spatial resolutions according to the size of the distance between ground objects in the captured image [3]. In recent years, with the increasing demand and refinement of RS for earth observation, such as classification of ground features and change detection, RS image resources with both high spatial and spectral resolution are urgently needed. Currently, due to the limitation of sensor hardware technology, RS images with both high spatial and high spectral resolution cannot be obtained from a single device [4]. Fortunately, many commercial RS satellites can simultaneously carry many types of sensors to observe the same surface features and form different types of RS images. Satellites, such as IKONOS, Landsat 8, QuickBird, and World View-2, are equipped with PAN and MS sensors on the same platform to obtain both PAN and MS images of the same surface, which lays the foundation for using fusion methods to obtain RS images with both high spectral and spatial resolution.

In fact, the PAN is mixed image with visible light band ranges, which have a high spatial resolution but cannot display the colors of surfaces because of having a single band. MS covers several to more than a dozen bands, which can be used to distinguish surfaces by their spectral characteristics, and, at the same time, the type of surface can be determined by the structure and morphology of the surface reflected. However, because MS sensor wavelength range is shorter than that of PAN, MS has lower spatial resolution, which affects the accuracy of MS in earth observation applications, such as dynamic detection of ground objects, target recognition, etc. How to achieve information fusion between MS and PAN through information complementation (called MS-Pansharpening [5]) to obtain MS images with high spatial resolution has become an important research field in RS image processing [6,7].

At present, the MS-Pansharpening method mainly contains two categories: component substitution (CS) and multiresolution analysis (MRA) [5]. The CS method uses high spatial resolution components of PAN to replace low spatial resolution components of MS to obtain a fused MS with high spatial resolution. Early methods include the intensity-hue-saturation (IHS) method [8], principal component analysis (PCA) method [9], and Gram-Schmidt (GS) method [10]. Later, in order to better preserve the details of fused images, the generalized IHS (GIHS) algorithm [11] and the adaptive IHS (AIHS) algorithm [12] were proposed. This method assumes that there is a projection transformation that projects an MS image into another vector space so that its spatial structure and spectral information are well separated. Its advantage is that the spatial structure information of the fused image is well maintained, while the disadvantage is that local differences between MS and PAN often result in significant spectral distortion of the fused image [5]. MRA use multiscale transformation to extract spatial details of the PAN, and inject it into the resampled image of the MS to obtain a fused MS image with high spatial resolution. Early methods focused on obtaining spatial information of PAN using different types of wavelet and laplacian pyramid transforms [13], such as the wavelet method [14,15], nonsubsampled wavelet method [16], and generalized laplacian pyramid (GLP) method [17], etc. With the continuous development of MRA, there are some new methods, such as contourlet method [18], shearlet method [19], nonsubsampled contourlet transform (NSCT) method [20], and NSST method [21], etc. Unlike the CS method, which usually uses the difference between the PAN image and the MS image to obtain spatial information, the MRA method uses its own filters to extract spatial structure information from the PAN image independently and inject it into the MS image as equally as possible, which generally enables the fused MS to retain spectral information better, but the spatial detail structure is usually distorted.

In addition to the two MS-Pansharpening methods mentioned above, there are also some methods based on variational optimization (VO) and deep learning (DL) [6]. The former is based on the variational theory to construct an energy function by observing models or sparse representation theory, and further iteratively solving through optimization algorithms to obtain MS images with improved spatial resolution [22,23]. The latter uses deep learning theory, as in the literature [24], to assume that there is a complex nonlinear relationship between the observed image and the ideal fused MS image, and then uses some deep neural networks with significant non-linear characterization ability to automatically learn all parameters under the supervision of large training samples [25,26]. These two methods can improve the quality of fused images from different aspects, but the former generally has a high computational complexity, while the latter usually requires a large number of sample training networks, which limits the development of these two methods to a certain extent, refer to [6] for detailed analysis. In addition, the hybrid model approaches have also become hot in recent years, with deep learning combining the benefits of CS-based and MRA-based approaches in an effective and efficient way [27]. Overall, although Pansharpening fusion performance has been improved from different perspectives, how to maximize the spatial resolution of the fused image while effectively maintaining the spectral information of the original MS is still a challenging task and there are many issues to be studied.

In addition to wavelet transform, a number of multiscale transforms have successively emerged in the multiscale geometric analysis family that can effectively characterize the high-dimensional singular characteristics [28]. These transforms greatly improve the ability to capture the high-dimensional singular features, such as midline and polygon, by increasing the sensitivity to direction, which provides effective support for sparse representation of images and signals. Among them, Nonsubsampled multiscale transform [29,30,31,32] is highly valued in the fields of image enhancement, image watermarking and image fusion because of its translation invariance and information redundancy [33,34,35].

This paper presents the MS-Pansharpening method based on NSST and dual constraint guided filtering. The main contributions are listed as follows:

(1)

The Dual Constraint Guided Image Filtering (DCGIF) model, based on spatial region average gradient correlation and vector correlation formed by neighborhood elements. is proposed. On this basis, the DCGIF detail extraction scheme is proposed. On the one hand, the spatial detail information map extracted from the PAN is more adaptive under the guidance of MS. On the other hand, it has better integrity and accuracy under the constraints of element neighborhood correlation and average gradient correlation.

(2)

The MS spatial detail information injection model, based on band spectral correlation, is presented, which determines the information injection weight matrix of each band based on the spectral ratio between the original MS bands, ensuring the spectral correlation between the fused MS image bands.

(3)

A new scheme of MS-Pansharpening in NSST domain is proposed. First, a more effective source of spatial detail information is obtained from the NSST high frequency sub-bands of MS and PAN. Then, effective spatial detail injection information is extracted by weighted fusion, based on the region energy matrix, using the proposed DCGIF model. Finally, the spatial detail information is injected into each MS band according to the information injection model to complete the information fusion. A large number of simulation experiments show that the proposed algorithm can effectively improve the spatial resolution of the fused MS image, while maintaining its spectral characteristics, and achieve better fusion results than some conventional fusion algorithms.

2. Related Works

2.1. Guided Image Filter

Guided image filter (GIF) is an adaptive weighted linear filter proposed by He et al. [36]. Unlike general filters, the GIF is performed under the guidance of another so-called “guidance image”, which can be either the image to be filtered itself or another image. Its implementation is as follows.

Set the guidance image, the filter input image and the filter output image to be $X$, $p$, and $q$, respectively. Assume that there is a local linear relationship between $X$ and $q$ in the local window ${\omega }_k$ centered on pixel $k$, that is:

$$q_i=a_k{X}_i+b_k,\forall i\in {\omega }_k$$

(1)

where $q_i\in q$, $X_i\in X$. $\left(a_k,b_k\right)$ is a linear coefficient in ${\omega }_k$ and can be obtained by minimizing the following cost function:

$$E\left(a_k,b_k\right)={{\displaystyle \sum }}_{i\in {\omega }_k}({\left(a_k{X}_i+b_k-p_i\right)}^2+\eta a_k^2)$$

(2)

here, parameter$\eta$is a regularization parameter preventing $a_k$ from being too large. The values $a_k$ and $b_k$ are obtained by the linear regression method:

$$\{\begin{array}{c}{a}_k=\frac{\frac{1}{\left|{\omega }_k\right|}{{\displaystyle \sum }}_{i\in {\omega }_k}{X}_i{p}_i-{\mu }_k{\overline{p}}_k}{{\delta }_k^2+\eta }\\ b_k={\overline{p}}_k-a_k{\mu }_{k }\end{array}$$

(3)

${\delta }_k^2$ and ${\mu }_k$ represent the variance and mean of $X$within ${\omega }_k$, $\left|{\omega }_k\right|$ the number of pixels within ${\omega }_k$, and ${\overline{p}}_k$ the average of pixels within ${\omega }_k$, that is

$${\overline{p}}_k=\frac{1}{\left|{\omega }_k\right|}{\displaystyle \sum }_{i\in {\omega }_k}{p}_i$$

(4)

Further, consider that one pixel in the image may be covered by multiple windows, so $a_k$ and $b_k$ obtained by Formula (3) may be different from the ${\omega }_k$, resulting in the non-uniqueness of $q_i$. For this reason, the GIF uses the mean of $a_k$ and $b_k$ as the linear coefficient of Formula (1), the final filtering formula is:

$$q_i={\overline{a}}_i{X}_i+{\overline{b}}_i$$

(5)

where $a_k$ and $b_k$ is

$$\{\begin{array}{c}{\overline{a}}_i=\frac{1}{\left|{\omega }_i\right|}{\displaystyle {\displaystyle \sum }_{k\in {\omega }_i}}{a}_k\\ {\overline{b}}_i=\frac{1}{\left|{\omega }_i\right|}{\displaystyle {\displaystyle \sum }_{k\in {\omega }_i}}{b}_k\end{array}$$

(6)

GIF is a fast local filtering edge detection filter, which can effectively avoid the gradient inversion effect, it can keep edge detail information while smoothing the image based on the guidance image, and it has been paid attention to in the fields of image segmentation, detail enhancement and image fusion in recent years [37,38,39].

2.2. NSST and Its Sub-Band Properties

For 2D signals, the composite wavelets (CW) theory [40,41] defines the following composite expansion affine systems:

$${\mathscr{H}}_{AB}\left(\psi \right)=\left\{{\psi }_{j,l,k}\left(x\right)={\left|detA\right|}^{\frac{j}{2}}\psi \left(B^l{A}^{j}x-k\right):j,l\in \mathbb{Z},k\in {\mathbb{Z}}^2\right\}$$

(7)

where $\Psi \in L^2\left({ℝ}^2\right)$, A and B are invertible matrices of $2\times 2$, respectively, and$\left|detB\right|=1$.

For the Formula (7), $\forall f\in L^2\left({ℝ}^2\right)$ has:

$${\displaystyle \sum }_{j,l,k}{|⟨f,{\psi }_{j,l,k}⟩|}^2=\Vert f^2\Vert$$

The element ${\psi }_{j,l,k}\left(x\right)$ in ${\mathscr{H}}_{AB}\left(\psi \right)$ is called composite wavelet. As a general framework, the matrix $A^j$ is associated with scale transformation and matrix $B^l$ is associated with geometric transformation that are maintained in areas such as rotation and shearing. CW can change not only in scale and position, but also in direction as wavelets do, and it lays a foundation for constructing multi-directional multi-scale transform.

As a special case of CW, the Shearlet transform [31,42] is constructed as follows:

In Formula (7), the following anisotropic expansion and shear matrices are selected for A and B:

$$A=\left(\begin{array}{cc}4& 0\\ 0& 2\end{array}\right),B=\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)$$

The basis function ${\psi }_{j,l,k}\left(x\right)$is given by the following formula:

$$\widehat{\psi }\left(\xi \right)=\widehat{\psi }\left({\xi }_1,{\xi }_2\right)={\widehat{\psi }}_1\left({\xi }_1\right){\widehat{\psi }}_2\left(\frac{{\xi }_2}{{\xi }_1}\right)$$

(8)

where $\mathsf{\xi }=\left({\mathsf{\xi }}_1,{\mathsf{\xi }}_2\right)\in {\widehat{ℝ}}^2$, ${\mathsf{\xi }}_1\ne 0$, and ${\widehat{\psi }}_1,{\widehat{\psi }}_2\in C^{\infty }\left(\widehat{ℝ}\right)$,$supp\left({\widehat{\psi }}_1\right)\subset ⎡-\frac{1}{2},-\frac{1}{16}⎤\bigcup ⎡\frac{1}{16},\frac{1}{2}⎤$, $supp\left({\widehat{\psi }}_2\right)\subset \left[-1,1\right]$. so that $\widehat{\psi }\in C^{\infty }\left(\widehat{ℝ}\right)$, $supp\left(\widehat{\psi }\right)\subset {⎡-\frac{1}{2},\frac{1}{2}⎤}^2$. Further, assume that for $\forall j\ge 0$, the following is true:

$$\{\begin{array}{c}{\displaystyle {\displaystyle \sum }_{j\ge 0}}{\left|{\widehat{\psi }}_1\left(2^{-2j}\omega \right)\right|}^2=1, if\left|\omega \right|\ge \frac{1}{8}\\ {\displaystyle {\displaystyle \sum }_{l=-2^j}^{2^j-1}}{\left|{\widehat{\psi }}_2\left(2^j\omega -l\right)\right|}^2=1, if\left|\omega \right|\le 1\end{array}$$

(9)

This makes the base function ${\psi }_{j,l,k}\left(x\right)$ of the Shearlet transform satisfied

$$supp\left({\widehat{\psi }}_{j,l,k}\right)\subset \left\{\left\{\left({\xi }_1,{\xi }_2\right):{\xi }_1\in \left[-2^{2j-1},-2^{2j-4}\right]\bigcup \left[2^{2j-4},2^{2j-1}\right]\right\},\left|\frac{{\xi }_2}{{\xi }_1}+l{2}^{-j}\right|\le 2^{-1}\right\}$$

(10)

The discretization of Shearlet transform is achieved by combining a laplacian pyramid (LP) filter with a directional filter, where the NSST is a numerical calculation process that replaces the traditional LP with a nonsubsampled LP (NLP) filter [30].

Shearlet transform has achieved better performance than other multi-scale transforms in many aspects of image decomposition [31], such as the frequency domain and the spatial domain, which have very good compact support and localization characteristics, satisfy parabolic scale characteristics, have high directional sensitivity, and near-optimal sparse representation capabilities. Furthermore, because the NLP filtering process does not have down-sampling, it avoids distortion in the directional filtering and guarantees smooth shift invariance and information redundancy. Figure 1 shows the MS of San Clemente area from Worldview02 satellite. After 2-layer NSST decomposition, one low-frequency sub-band and two scale high-frequency sub-bands are obtained, the first scale high-frequency sub-band has two directional sub-bands in horizontal and vertical; the second scale high frequency sub-band has four directions, which are the horizontal direction and ${45}^{\circ }$direction, vertical and ${135}^{\circ }$direction.

Image fusion methods based on traditional MRA mostly use the correlation of eight neighborhood coefficients to determine the fusion rules of high-frequency sub-bands, and the directionality of these detail sub-bands is often ignored. In order to quantitatively analyze the directional characteristics of NSST high-frequency sub-bands in MS and PAN, the three groups of data sets shown in Figure 2 are selected, the San clement and Sydney area images of WorldView2 satellites, and the San Francisco region images of Quick-Bird satellite. Statistical analysis is carried out using the sub-band correlation evaluation method based on the variance of directional region [21]. The direction template [43] shown in Figure 3 corresponds to each high-frequency directional sub-band decomposed by NSST, and the variance shown in Equation (11) is used to measure and compare the magnitude of the directional neighborhood correlation with the eight neighborhood correlations. If the directional neighborhood variance is less than the eight neighborhood variances, the directional neighborhood coefficient correlation is stronger than the eight neighborhood coefficient correlations, and vice versa.

$$\{\begin{array}{c}\mu \left(i,j\right)=\frac{1}{3\times 3}{\displaystyle {\displaystyle \sum }_{m=-1}^1}{\displaystyle {\displaystyle \sum }_{n=-1}^1}c\left(i+m,j+n\right)\\ SD\left(i,j\right)=\sqrt{\frac{1}{3\times 3}{\displaystyle {\displaystyle \sum }_{m=-1}^1}{\displaystyle {\displaystyle \sum }_{n=-1}^1}{\left(c\left(i+m,j+n\right)-\mu \left(i,j\right)\right)}^2}\end{array}$$

(11)

where $c\left(i,j\right)$ is the high frequency directional subband coefficient at $\left(i,j\right)$, $\mu \left(i,j\right)$ and $SD\left(i,j\right)$ are the mean of the coefficients in the $3\times 3$ region centered on $\left(i,j\right)$ and the regional variance corresponding to the corresponding directional template,

Table 1. Statistical results of directional neighborhood correlation and eight neighborhoods correlation of the high frequency sub-bands.

			San Clement Area Image		Sydney Area Image		San Francisco Area Image
			MS	PAN	MS	PAN	MS	PAN
First-scale	Sub-band 1	$a$	32,196	32,217	32,286	32,721	32,938	32,570
		$b$	7804	7783	7714	7279	7062	7430
		$a/\left(a+b\right)(\%)$	80.49	80.54	80.72	81.80	82.35	81.43
	Sub-band 2	$a$	32,596	32,986	33,394	33,643	33,987	33,846
		$b$	7404	7014	6606	6357	6013	6154
		$a/\left(a+b\right)(\%)$	81.49	82.47	83.49	84.11	84.97	84.62
Second-scale	Sub-band 1	$a$	34,338	34,551	34,730	34,790	35,177	34,627
		$b$	5662	5449	5270	5210	4823	5373
		$a/\left(a+b\right)(\%)$	85.85	86.38	86.83	86.98	87.94	86.57
	Sub-band 2	$a$	32,509	32,630	31,260	30,672	33,262	31,877
		$b$	7491	7370	8740	9328	6738	8123
		$a/\left(a+b\right)(\%)$	81.27	81.58	78.15	76.68	83.15	79.69
	Sub-band 3	$a$	34,263	34,232	35,190	35,462	35,988	35,091
		$b$	5737	5768	4810	4538	4012	4909
		$a/\left(a+b\right)(\%)$	85.66	85.58	87.98	88.66	89.97	87.73
	Sub-band 4	$a$	31,486	30,936	30,715	30,660	31,708	30,171
		$b$	8514	9064	9285	9340	8292	9829
		$a/\left(a+b\right)(\%)$	78.72	77.34	76.79	76.65	79.27	75.43

Note: $a$ is the number of coefficients in which the directional neighborhood correlation is stronger than the eight neighborhood correlations in the high frequency subband; $b$ is the number of coefficients whose eight neighborhood correlations is stronger than the directional neighborhood correlation; $a/\left(a+b\right)$ is the proportion of directional neighborhood correlation stronger than the eight neighborhood correlations.

shows the statistical results.

It can be seen from Table 1 that the correlation of the directional neighborhood coefficients in the high frequency sub-bands of NSST was stronger than that of the eight neighborhood coefficients in both MS and PAN, which provided an effective way to further explore the correlation of the high frequency sub-band coefficients of NSST.

3. Methodology

3.1. Construction of the Dual Constraint Guided Filtering Model

The GIF model often produces halo effects when applied as a local filter to smooth edges. In reference [44], GIF is improved, and a GIF model based on edge preservation weighting is presented, inspired by this model, the GIF model based on the constraints of neighborhood elements vector correlation and regional average gradient correlation of the original and guidance images is presented, based on the region characteristics of the image. Effective and appropriate detail features can be adaptively extracted from the guidance image, which lays a foundation for improving the spatial resolution and maintaining spectral information of the MS fused image after information injection.

3.1.1. Neighborhood Elements Vector and Average Gradient Correlation Constraint

In order to effectively measure the correlation between the corresponding location of MS and the PAN, a measurement method based on the neighborhood elements vector angle is proposed, which, together with the average gradient, constrains the extraction of PAN details.

(1) Neighborhood element vector angle

For the PAN image $X$and MS image $Y$, take the $n\times n$ neighborhood with $\left(i,j\right) \left(i\in \left\{1,2,\cdots ,m\right\},j\in \left\{1,2,\cdots ,n\right\}\right)$ as the center (for the adjacent region with edge element as the center, it can be obtained by edge extension method). The neighborhood elements vector angle refers to the adjacent region elements, selected according to the “from top to bottom, left to right”, forming the angle between vectors $x=\left(\cdots ,x_{r,s},\cdots \right)$ and $y=\left(\cdots ,y_{r,s},\cdots \right)$ (see Figure 4), where $r=i-⎣n/2⎦, i-⎣n/2⎦+1, \cdots ,i+⎣n/2⎦$;$s=j-⎣n/2⎦, j-⎣n/2⎦+1, \cdots ,j+⎣n/2⎦$, the calculation formula is shown in (12), and the range of included angle is $\left[0,\pi /2\right]$.

$$SAM\left(i,j\right)\equiv \theta \left(i,j\right)=arccos\frac{{{\displaystyle \sum }}_{r=i-⎣n/2⎦}^{i+⎣n/2⎦}{{\displaystyle \sum }}_{s=j-⎣n/2⎦}^{j+⎣n/2⎦}{x}_{r,s}{y}_{r,s}}{\sqrt{\left({{\displaystyle \sum }}_{r=i-⎣n/2⎦}^{i+⎣n/2⎦}{{\displaystyle \sum }}_{s=j-⎣n/2⎦}^{j+⎣n/2⎦}{x}_{r,s}^2\right)\left({{\displaystyle \sum }}_{r=i-⎣n/2⎦}^{i+⎣n/2⎦}{{\displaystyle \sum }}_{s=j-⎣n/2⎦}^{j+⎣n/2⎦}{y}_{r,s}^2\right)}}$$

(12)

The smaller the $\theta$ value is, the stronger the correlation between the two elements is, and vice versa. The neighborhood elements vector angle reflects the correlation of the corresponding elements, which makes the measurement of the correlation of heterogeneous RS images, such as PAN and MS, more robust.

(2) Average gradient

The gradient of image pixel reflects the change rate and direction of pixel gray, which can reveal the direction and intensity of image edge. For the 2D image function $f\left(·,·\right)$, the gradient at $\left(i,j\right)$ is defined as follows:

$$\nabla f\left(i,j\right)\equiv grad\left(f\left(i,j\right)\right)={\left[\frac{\partial f\left(i,j\right)}{\partial i},\frac{\partial f\left(i,j\right)}{\partial j}\right]}^T$$

(13)

the amplitude is

$$M\left(i,j\right)\equiv mag\left(\nabla f\left(i,j\right)\right)=\sqrt{{\left(\frac{\partial f\left(i,j\right)}{\partial i}\right)}^2+{\left(\frac{\partial f\left(i,j\right)}{\partial j}\right)}^2}$$

(14)

Figure 5 shows the edge detail image obtained by gradient operator for MS and PAN.

For the $n\times n$ neighborhood (see Figure 4) in the PAN with $\left(i,j\right)$ as the center, the average gradient can reflect the clarity of the region block well, the small detail contrast and the change of texture. In this paper, the neighborhood elements vector angle and the average gradient are used as the constraint terms of the proposed improved guided filtering model to ensure the balance of extracting detail information from the PAN, in order to avoid the spectral distortion caused by over injection of detail information in the process of information fusion. The numerical expression of the average gradient of $n\times n$ neighborhood is shown in Equation (15).

$$AG\left(i,j\right)\equiv \overline{\nabla f}\left(i,j\right)=\frac{1}{n\times n}{{\displaystyle \sum }}_{r=i-⎣n/2⎦}^{j+⎣n/2⎦}{{\displaystyle \sum }}_{s=j-⎣n/2⎦}^{j+⎣n/2⎦}\sqrt{\frac{{\left(f\left(r+1,s\right)-f\left(r,s\right)\right)}^2+{\left(f\left(r,s+1\right)-f\left(r,s\right)\right)}^2}{2}}$$

(15)

3.1.2. Region Edge Recognition Function Based on the Neighborhood Elements Vector Angle and the Average Gradient

Let the size of input image $\mathit{p}$ and guidance image $X$ be $m\times n$, for any position $\left(i,j\right)$ in $p$and $X$, define the $3\times 3$ neighborhood ${\omega }_{i,j}$ with $\left(i,j\right)$ as the center (see Section 3.1.1). Let$SA{M}_{X\&p\_\omega }\left(i,j\right)$ be the neighborhood elements vector angle between adjacent area $\omega$ of $\mathrm{p}$and $X$ (see Formula (12)) and $SA{M}_{X\&p}$ be the overall element vector angle of $p$and $X$ (that is, the whole image is regarded as the adjacent area of its center point); Let $A{G}_{X\_\omega }\left(i,j\right)$ be the average gradient value of the $\omega$ in $X$ (see Equation (15)), and $A{G}_X$ be the overall average gradient value of $X$, and normalize $A{G}_{X\_\omega }\left(i,j\right)$ and $A{G}_X$ to the same value range $\left[0,\pi /2\right]$ as $SA{M}_{X\&p\_\omega }\left(i,j\right)$ and $SA{M}_{X\&p}$. The region edge recognition function $\Psi \left(i,j\right)$ is constructed as follows:

$$\Psi \left(i,j\right)\equiv \frac{sin\left( SA{M}_{X\&p\_\omega }\left(i,j\right)\right)+sin\left(A{G}_{X\_\omega }\left(i,j\right)\right)+\lambda }{\sin \left( SA{M}_{X\&p}\right)+\sin \left( A{G}_X\right)+\lambda }$$

(16)

where $\lambda$ is a small constant (1 × 10⁻⁶ is used in this article), in order to avoid zero denominator.

In order to analyze the effectiveness of the constructed edge recognition function $\Psi \left(i,j\right)$, we selected three MS and PAN images with different characteristics and make distribution statistics of their region recognition functions. Figure 6 is the visualization result of the distribution statistics of the experimental image (only PAN). It can be seen that the defined edge recognition function could recognize the smooth region and the edge texture region of the image well, which made it possible for the adaptive constraint to obtain the detail information from the guidance image.

3.1.3. Model Construction

For the same position $\left(i,j\right) \left(i\in \left\{1,2,\cdots ,m\right\},j\in \left\{1,2,\cdots ,n\right\}\right)$ of the input image$p$ and the guidance image $X$with the size of $m\times n$, a window with radius $r$centered on $\left(i,j\right)$ is defined, based on the proposed region edge recognition function $\Psi \left(i,j\right)$, the minimum cost function of the improved guide filtering id defined as follows:

$$E\left(a_{i,j},b_{i,j}\right)={\displaystyle \sum }_{\left(s,t\right)\in {\omega }_{\left(i,j\right)\_r}}\left({\left(a_{i,j}{X}_{s,t}+b_{i,j}-p_{s,t}\right)}^2+\frac{\eta }{\Psi \left(i,j\right)}{a}_{i,j}^2\right)$$

(17)

$\left(a_{i,j},b_{i,j}\right)$ is the linear coefficient of ${\omega }_{\left(i,j\right)\_r}$, $X_{s,t}\in X, p_{s,t}\in$p, $\eta$ is the regularization parameter (0.64 in this experiment).

Based on the linear regression method, $a_{i,j}$ and $b_{i,j}$ can be obtained:

$$\{\begin{array}{c}{a}_{i,j}=\frac{\frac{1}{\left|{\omega }_{\left(i,j\right)\_r}\right|}{{\displaystyle \sum }}_{\left(s,t\right)\in {\omega }_{\left(i,j\right)\_r}}{X}_{s,t}{p}_{s,t}-{\mu }_{i,j}{\overline{p}}_{i,j}}{{\delta }_{i,j}^2+\frac{\eta }{\Psi \left(i,j\right)}}\\ b_{i,j=}{\overline{p}}_{i,j}-a_{i,j}{\mu }_{i,j}\end{array}$$

(18)

where ${\delta }_{i,j}^2$ and ${\mu }_{i,j}$ represent the variance and mean value of the guidance image $X$ in the window ${\omega }_{\left(i,j\right)\_r}$, respectively, and ${\overline{p}}_{i,j}$ is shown in Equation (19):

$${\overline{p}}_{i,j}=\frac{1}{\left|{\omega }_{\left(i,j\right)\_r}\right|}{\displaystyle \sum }_{\left(s,t\right)\in {\omega }_{\left(i,j\right)\_r}}{p}_{s,t}$$

(19)

Let the output image of $p$guided by$X$ be $q$, further from Equations (17)–(19), we can obtain the Dual Constraint Guided Image Filtering (DCGIF) model based on the neighborhood elements vector angle and the average gradient as follows:

$$q_{i,j}={\overline{a}}_{i,j}{X}_{i,j}+{\overline{b}}_{i,j}$$

(20)

where $X_{i,j}\in \mathit{X}$, $q_{i,j}\in \mathit{q}$, ${\overline{a}}_{i,j}$ and ${\overline{b}}_{i,j}$ as follows:

$$\{\begin{array}{c}{\overline{a}}_{i,j}=\frac{1}{\left|{\omega }_{\left(i,j\right)\_r}\right|}{\displaystyle {\displaystyle \sum }_{\left(s,t\right)\in {\omega }_{\left(i,j\right)\_r}}}{a}_{s,t}\\ {\overline{b}}_{i,j}=\frac{1}{\left|{\omega }_{\left(i,j\right)\_r}\right|}{\displaystyle {\displaystyle \sum }_{\left(s,t\right)\in {\omega }_{\left(i,j\right)\_r}}}{b}_{s,t}\end{array}$$

(21)

The core of the guided filter is the determination of the linear coefficients in the window. Differing from the traditional GIF, which uses the regularization parameter $\eta$ to avoid the linear parameter $a_{i,j}$ in the window ${\omega }_{\left(i,j\right)\_r}$ being too large, the DCGIF model adaptively adjusts $a_{i,j}$ and $b_{i,j}$, based on the neighborhood elements vector angle and the average gradient of the region edge recognition function, to make the extracted filtered texture details more accurate. It is applied to MS-Pansharpening to ensure that the spatial information extracted from Pan has stronger adaptability under MS guidance.

Figure 7 shows a comparison between the traditional GIF model and the DCGIF model using MS as a guidance image to extract spatial detail information from the PAN. The window radius is selected as 1. From the displayed image, it can be seen that the spatial detail information extracted by the DCGIF was more delicate, and, at the same time, it had a better matching degree with the original image, which laif the foundation for improving the fusion effect of MS-Pansharpening.

3.2. MS-Pansharpening Algorithm Based on DCGIF in NSST Domain

3.2.1. Reconstruction of MS Intensity Component Guided by High Frequency Information of NSST from PAN

In order to make the spatial details extracted from the PAN better match the MS image, this paper used the NSST high frequency sub-band of the PAN as the guidance information to reconstruct the intensity component I from the IHS transform of the MS. The process is shown in Figure 8.

First, an IHS transform was performed on the MS sampled to the same size of the PAN; then, the I component of the MS was decomposed into two scales of NSST, and one low-frequency sub-band and six high-frequency direction sub-bands were obtained. The high frequency sub-bands included the horizontal $\left(0^{°}\right)$, vertical$\left({90}^{°}\right)$ direction of the first scale and the horizontal$\left(0^{°}\right)$, anti-diagonal $\left({45}^{°}\right)$, vertical $\left({90}^{°}\right)$, principal diagonal $\left({135}^{°}\right)$ of the second scale (see Figure 3). The low frequency sub-band of the I component of MS was preserved as the low frequency sub-band of the final reconstructed I component to maintain the overall overview and spatial characteristics of the original MS. For each high frequency sub-band combined with the high frequency sub-band corresponding to the PAN, a joint high-frequency sub-band was generated through the following process of maximum energy in the direction region.

For the I component of MS and each high frequency sub-band of direction $d \left(d\in \left\{0^{°},{45}^{°},{90}^{°},{135}^{°}\right\}\right)$ corresponding to the same level of PAN decomposed by NSST(denoted as “$I_d^h$” and “$PA{N}_d^h$”), the direction energy of the direction coefficient in the $3\times 3$ neighborhood $\mathrm{d}$ centered on the current position $\left(i,j\right)$ was calculated by traversing from top to bottom and from left to right (see Equation (22)). For the edge coefficient case, the symmetric continuation could be used. Figure 9 shows the high frequency sub-band neighborhood direction diagram.

$$\{\begin{array}{l}D{E}_{T,0^{°}}\left(i,j\right)=T^2\left(i,j-1\right)+T^2\left(i,j\right)+T^2\left(i,j+1\right)\\ D{E}_{T,{45}^{°}}\left(i,j\right)=T^2\left(i-1,j+1\right)+T^2\left(i,j\right)+T^2\left(i+1,j-1\right)\\ D{E}_{T,{90}^{°}}\left(i,j\right)=T^2\left(i-1,j\right)+T^2\left(i,j\right)+T^2\left(i+1,j\right)\\ D{E}_{T,{135}^{°}}\left(i,j\right)=T^2\left(i-1,j-1\right)+T^2\left(i,j\right)+T^2\left(i+1,j+1\right)\end{array}$$

(22)

where $T\in \left\{I_d^h,PA{N}_d^h\right\}$,$D{E}_{T,d}\left(i,j\right)$ represents the directional energy of $d$at the current position$\left(i,j\right)$ of the high frequency subband $T$.

Further, the joint high frequency sub-band ${\widehat{I}}_d^h$ is determined according to the Formula (23):

$${\widehat{I}}_d^h\left(i,j\right)=\{\begin{array}{c}{I}_d^h\left(i,j\right), if D{E}_{I_d^h,d}\left(i,j\right)\ge D{E}_{PA{N}_d^h,d}\left(i,j\right)\\ PA{N}_d^h\left(i,j\right), else \end{array}$$

(23)

After determining the level and direction of the joint high frequency sub-band of the I component of the MS, the I component $I_{MS\&PAN}^R$ of the reconstructed MS could be obtained by inverse NSST.

3.2.2. Extraction of Decision Texture Details for PAN Based on DCGIF Model

(1) PAN detail information coefficient fusion based on region energy weighted matrix

In order to obtain more abundant and more adaptable texture detail information from the PAN, and, at the same time, to contain as little redundant information as possible, this paper filtered the PAN twice based on the proposed DCGIF model to obtain the detail information. Further, the final decision texture details were obtained through a coefficient fusion mechanism weighted by the region energy matrix. The overall process is shown in Figure 10.

First, with the reconstructed component $I_{MS\&PAN}^R$ of MS intensity component (see Section 3.2.1) and PAN as the guidance image, the DCGIF model was used to filter the PAN twice to obtain two texture information images, $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{PAN\to PAN}$ and $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{I_{MS\&PAN}^R\to PAN}$. Further, the $3\times 3$ local region energy matrix $\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{PAN\to PAN}$ and $\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{I_{MS\&PAN}^R\to PAN}$ corresponding to the two texture information images were calculated. The two matrices were the same size as the two texture information images, and the elements of the matrix were the sum of squares of the elements in the $3\times 3$ neighborhood with the same location as the center of the corresponding texture information image.

Then, the corresponding two-weight matrix ${\omega }_{PAN\to PAN}$ and ${\omega }_{I_{MS\&PAN}^R\to PAN}$ was constructed, based on matrix $\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{PAN\to PAN}$ and $\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{I_{MS\&PAN}^R\to PAN}$:

$$\{\begin{array}{c}{\omega }_{PAN\to PAN}\left(i,j\right)=\frac{\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{PAN\to PAN}\left(i,j\right)}{\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{PAN\to PAN}\left(i,j\right)+\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{I_{MS\&PAN}^R\to PAN}\left(i,j\right)}\\ {\omega }_{I_{MS\&PAN}^R\to PAN}\left(i,j\right)=\frac{\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{I_{MS\&PAN}^R\to PAN}\left(i,j\right)}{\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{PAN\to PAN}\left(i,j\right)+\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{I_{MS\&PAN}^R\to PAN}\left(i,j\right)}\end{array}$$

(24)

Finally, the final decision texture details matrix for injection into MS was obtained by (25):

$$\begin{array}{c}\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{Final}\left(i,j\right)=\left({\omega }_{PAN\to PAN}\left(i,j\right)\times \mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{PAN\to PAN}\left(i,j\right)\right)\\ +\left({\omega }_{I_{MS\&PAN}^R\to PAN}\left(i,j\right)\times \mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{I_{MS\&PAN}^R\to PAN}\left(i,j\right)\right)\end{array}$$

(25)

(2) Discussion on window radius $r$in DCGIF Model

In order to discuss the influence of the filtering window radius $r$on the result and the fusion result after injection into the target image in DCGIF model, a statistical experiment was carried out on MS and PAN in San Clement area (see Figure 6). The filtering window radii were selected as 1, 2, 4 and 8, respectively. The corresponding PAN was filtered by DCGIF using the I component of MS as the guidance image to obtain the filtered detail image. To further observe the details injected into the target image, the details obtained were injected into the MS by simple matrix addition (see Figure 11 for specific results); root mean square error (RMSE) [45] and spectral angle mapper (SAM) [46] were used as objective evaluation indices to evaluate the fused MS (see

Table 2. Objective evaluation of DCGIF filtering results with different window radius.

Evaluation Index	$\mathit{r}\mathit{=}\mathbf{1}$	$\mathit{r}\mathbf{=}\mathbf{2}$	$\mathit{r}\mathbf{=}\mathbf{4}$	$\mathit{r}\mathbf{=}\mathbf{8}$
RMSE	23.8240	30.5323	34.8185	37.8988
SAM	0.3917	0.5453	0.6952	0.8574

).

From the above experimental results, it can be seen that as the radius of the filtering window increased, the image with filtered detail information appeared to be over-sharpened, and, at the same time, the fused image had some spectral distortion in the spectral information, such as the distortion of residential areas was more obvious, while in the spatial information, the texture detail information that was over injected created overlap in the local region. The larger the window radius was, the larger the RMSE value of the fused image, and the larger the SAM value. For this reason, the radius of the DCGIF of the PAN in the subsequent algorithm of this article was selected as 1 filtering window.

3.2.3. Detail Information Injection Model for MS Based on Spectral Correlation

Multi-band RS imagery is an image formed from reflected light from different spectral bands for the same object. The correlation between bands in a multi-band RS image is often referred to as spectral correlation. To investigate the spectral correlation between adjacent bands of MS, we selected eight sets of MS, and the correlation between adjacent bands in MS was calculated using the “spectral correlation coefficient” shown in Equation (26). The statistical results are shown in

Table 3. Correlation statistics between neighboring bands of MS.

Correlation Coefficient	MS-Test-1	MS-Test-2	MS-Test-3	MS-Test-4	MS-Test-5	MS-Test-6	MS-Test-7	MS-Test-8
$r_{12}$	0.9686	0.9891	0.9738	0.9853	0.9235	0.9611	0.9712	0.9792
$r_{23}$	0.9681	0.9930	0.9850	0.9894	0.9626	0.9868	0.9673	0.9880

.

$$r_{ij}=\frac{{{\displaystyle \sum }}_{x=1}^M{{\displaystyle \sum }}_{y=1}^N\left(\left(I_i\left(x,y\right)-{\overline{I}}_i\right)\left(I_j\left(x,y\right)-{\overline{I}}_j\right)\right)}{\sqrt{{{\displaystyle \sum }}_{x=1}^M{{\displaystyle \sum }}_{y=1}^N{\left(I_i\left(x,y\right)-{\overline{I}}_i\right)}^2{{\displaystyle \sum }}_{x=1}^M{{\displaystyle \sum }}_{y=1}^N{\left(I_j\left(x,y\right)-{\overline{I}}_j\right)}^2}}$$

(26)

where $I_i$and $I_j$are the $i$ and $j$ band images (size $M\times N$); ${\overline{I}}_i=\frac{1}{M\times N}{\displaystyle \sum }_{x=1}^M{\displaystyle \sum }_{y=1}^N{I}_i\left(x,y\right)$, ${\overline{I}}_j=\frac{1}{M\times N}{\displaystyle \sum }_{x=1}^M{\displaystyle \sum }_{y=1}^N{I}_j\left(x,y\right)$; $r_{ij}$ is the correlation coefficient of the $i$ and $j$ band.

From the statistical results in Table 3, it can be seen that the correlation coefficients between adjacent bands of MS were between$0.9~1$and there was a strong correlation between adjacent bands. Therefore, in the MS-Pansharpening algorithm based on detail information injection, if the detail injection was only applied to a certain band or a certain component of MS, the relationship between the original MS bands would be broken to some extent, so that the spectral information of the fused MS would be distorted. Based on this, a weighted detail information injection model based on spectral correlation is proposed:

Assume that the size of the MS is $M\times N$ and $I_{MS\_Band\_k }$is its$k \left(k\in \left\{1,2,\cdots ,n\right\}\right)$ band, where $n$is the total number of bands of the MS. For texture detail matrix $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{Final}$ (see Equation (25)), construct the weight matrix ${\mathit{\alpha }}_{MS\_Band\_k} \left(k\in \left\{1,2,\cdots ,n\right\}\right)$ as shown in Formula (27):

$${\mathit{\alpha }}_{MS\_Band\_k}\left(i,j\right)=\frac{I_{MS\_Band\_k }\left(i,j\right)}{{{\displaystyle \sum }}_{k=1}^n{I}_{MS\_Band\_k }\left(i,j\right)}$$

(27)

According to the following Formula (28), the texture detail information is injected to form the final fused MS ${\widehat{I}}_{MS\_Band\_k }$:

$${\widehat{I}}_{MS\_Band\_k }\left(i,j\right)=I_{MS\_Band\_k }\left(i,j\right)+{\mathit{\alpha }}_{MS\_Band\_k}\left(i,j\right)\times \mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{Final}\left(i,j\right)$$

(28)

where $k\in \left\{1,2,\cdots ,n\right\}$, $i\in \left\{1,2,\cdots ,M\right\}$, $j\in \left\{1,2,\cdots ,N\right\}$.

3.2.4. The Process of Algorithm Implementation

Based on the previous analysis and discussion, the implementation process of the MS-Pansharpening algorithm in this article is given, and the overall flow chart of the algorithm is shown in Figure 12.

The implementation process of the Algorithm 1 is as follows:

Algorithm 1 Procedures of the proposed approach.

Input: The registered PAN and MS RS images.

Initialization:  The MS is sampled to the same size as the PAN, and then the MS is transformed by IHS, the intensity component I is extracted, and the component H and component S are retained.

1: The I component and PAN are transformed by three layers of NSST to obtain the corresponding low-frequency and high-frequency sub-bands. For each direction sub-band, according to its direction, and according to Formulas (22) and (23), the fusion rule of maximum energy in the direction region is used to obtain the joint high-frequency sub-band, and then inverse NSST is used to obtain the reconstructed I component $I_{MS\&PAN}^R$;

2: Use the DCGIF model to filter the PAN twice. First, the $I_{MS\&PAN}^R$ obtained in Step 1 is used as the guidance image to obtain the texture detail image $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{I_{MS\&PAN}^R\to PAN}$. Second, the PAN is used as the guidance image to obtain the texture detail image $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{PAN\to PAN}$;

3: The $3\times 3$ local region energy matrices $\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{I_{MS\&PAN}^R\to PAN}$ and $\mathit{D}\mathit{e}\_\mathit{E}{\mathit{n}}_{PAN\to PAN}$ corresponding to$\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{I_{MS\&PAN}^R\to PAN }$ and $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{PAN\to PAN}$ are calculated respectively, and the two-weight matrices ${\omega }_{I_{MS\&PAN}^R\to PAN}$ and ${\omega }_{PAN\to PAN}$ are constructed (see Equation (24));

4: According to Formula (25), $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{I_{MS\&PAN}^R\to PAN}$ and $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{PAN\to PAN}$ are weighted average based on the weight matrix ${\omega }_{I_{MS\&PAN}^R\to PAN}$ and ${\omega }_{PAN\to PAN}$ to obtain the final decision texture detail information matrix $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{Final}$ for injection into MS;

5: For the $I_{MS\&PAN}^R$ constructed by Step 1 and the components H and S reserved by Step 1, the weight matrices ${\mathit{\alpha }}_{I_{MS\&PAN}^R}$, ${\mathit{\alpha }}_H$, and ${\mathit{\alpha }}_S$ are obtained according to Equation (27);

6: According to the weighted detail information injection model based on (28), the detail information matrix $\mathit{D}\mathit{e}\mathit{t}\mathit{a}\mathit{i}{\mathit{l}}_{Final}$ is injected into $I_{MS\&PAN}^R$, H, and S to form the components ${\widehat{I}}_{MS\&PAN}^R$, $\widehat{H}$, and $\widehat{S}$ after the detail information is injected.

7: Inverse IHS is used to transform the ${\widehat{I}}_{MS\&PAN}^R$, $\widehat{H}$, and $\widehat{S}$ to obtain the fused MS image.

Output: the fused MS image.

4. Experiment and Analysis

Experiments were conducted with an Intel Core i7 6700 CPU 3.4 GHz with 32 GB RAM in a Windows7 system, and the experimental platform was Matlab R2014a. For all the methods that required NSST, the ‘maxflat’ nonsubsampled multiscale filter and improved shear filter banks were selected, and three levels of NSST decomposition were performed on the MS and PAN. The number of sub-bands in each scale direction was 2, 4 and 8, respectively. At the same time, seven classical fusion algorithms were selected for comparison experiments, which included IHS algorithm, traditional GIF algorithm [36], Á Trous Wavelet Transform (ATWT) algorithm [47], Additive Wavelet Luminance Proportional (AWLP) algorithm [47], Adaptive Sparse Representation (ASR) algorithm [48], Deep Leaning algorithm [49] and NSST + PCNN + SR algorithm [21].

4.1. Data Set

Five sets of RS image data from two different satellites, QucikBird02 and WorldView02 (see Figure 13) were obtained. Among them, Test-RS-1, Test-RS-3 and Test-RS-5 were from the QucikBird02 satellite in the San Francisco area, while PAN had a 0.7 m spatial resolution and MS had a 2.8 m spatial resolution. The sizes of MS and PAN images were $400\times 400$ and $800\times 800$, respectively. Test-RS-2 and Test-RS-4 were from the WorldView02 satellite and were taken from the San Clemente and Sydney regions. The spatial resolution of the RS images was 0.5 m. The selected MS and PAN experimental images took into account a variety of landform information, including plains, waters, residential areas, docks, mountains and road areas. The sizes of MS and PAN images were $400\times 400$ and $800\times 800$, respectively. All the PAN and MS images in the dataset to be fused were registered images. The PAN was a single-band image and the MS contained four bands of R, G, B, Near Infrared (NIR), but, due to the three-channel image characteristics, the MS shown in this paper only contained R, G, B.

This paper divided the experiment into two parts: one was the experiment of degraded data on Test-RS-1~Test-RS-4, and the other was the real data experiment of Test-RS-5. For the real data experiment, first the MS was down-sampled to obtain a reduced spatial resolution MS image, so that the original MS could be used as a reference image. Further, the reduced spatial resolution MS was sampled up to the same size as the PAN, and then the fusion experiments were performed.

4.2. Degraded Data Experiment

(1) Subjective evaluation

Figure 14 shows the MS and the fusion results of eight different algorithms for four sets of degraded data, with a local area enlarged three times, where “NSST + DCGIF” was the algorithm in this paper.

From the perspective of spectral preservation, the spectral distortions in the result obtained by traditional IHS algorithm, ASR algorithm and deep learning algorithm were more obvious, among which the overall color of Test-RS-1, especially the green forest area, was brighter. The color information of the IHS algorithm in Test-RS-2 was lighter on roads, on the top of red houses, and on the green vegetation area than the source MS, and the whole image was darker. The results obtained by ASR and deep learning algorithms were also distorted in the spectrum, and the color of vegetation and roof was bright. The results obtained by the other five algorithms were similar in spectral information to the source MS image. From the spatial details, although the traditional GIF algorithm maintained good spectral information, overlap occurred at the edge texture, and the details were blurred, such as Test-RS-2 vegetation information appearing in blocks, and Test-RS-3 producing black patches in tree forest areas, and there was loss of some texture details. The sea surface texture details were also lost, so it appeared too smooth. The local magnification showed that the ATWT algorithm maintained the spectrum well, but some of the spatial detail information was lost and the corresponding texture detail edges were not significantly improved. The AWLP algorithm was subjectively better but the spectrum was distorted compared to the MS image and the colors were more vivid in the local area. The NSST + SR + PCNN algorithm improved the spatial detail information greatly, but the spectrum was distorted to some extent, especially in vegetation areas. The resulting image from the NSST + DCGIF algorithm proposed in this paper, kept good spectral information compared with the other comparison algorithms, and the spectral information was close to the original MS in vegetation, sea waves, land and other areas, The spatial information was also greatly enhanced. Detailed information, such as sea waves, land, forests and so on, were enhanced, the edge contour information was clearer of buildings, road and land were clearly reflected, and the texture structure of different objects could be distinguished. It effectively enhanced the spatial resolution, while maintaining the spectral information of MS.

(2) Objective evaluation

For the objective evaluation of the degraded data experimental, the commonly used RMSE, erreur relative global adimensionnelle de Synthese (ERGAS) [50], structural similarity index (SSIM) [51] and correlation coefficient (CC) [52] as the spatial quality evaluation index were applied. Relative average spectral error (RASE) [45] and SAM were used as spectral quality evaluation indices.

Table 4. Objective evaluation of degraded data.

Data Set	Evaluation Index	IHS	GIF	ATWT	AWLP	ASR	Deep Learning	NSST + SR + PCNN	NSST + DCGIF
Test-RS-1	RMSE	46.1822	36.9231	21.8892	22.0637	31.0385	24.1516	19.3740	16.6651
	ERGAS	20.8551	16.6699	9.8857	9.9650	14.0130	10.9012	8.7466	7.5239
	CC	0.8357	0.9408	0.9617	0.9612	0.9408	0.9617	0.9694	0.9776
	SSIM	0.5867	0.5276	0.8008	0.8018	0.6706	0.7981	0.7950	0.8751
	SAM	4.3361	1.0508	2.7560	4.3361	1.0067	1.4456	1.3041	0.3463
	RASE	41.7080	33.3459	19.7685	19.9261	28.0314	21.8117	17.4970	15.0506
Test-RS-2	RMSE	42.7084	44.1567	15.7518	15.7411	28.7240	24.4196	19.5112	15.6839
	ERGAS	32.2701	33.1070	11.9120	11.8041	21.4947	18.3585	14.7141	11.8500
	CC	0.8919	0.7885	0.9712	0.9725	0.9272	0.9522	0.9504	0.9714
	SSIM	0.7414	0.7176	0.9358	0.9381	0.8171	0.8834	0.8905	0.9394
	SAM	5.0471	3.5388	5.9898	3.5308	3.3412	3.6489	3.2974	0.3748
	RASE	64.1366	66.3117	23.6551	23.6389	43.1359	36.6171	29.3007	23.5530
Test-RS-3	RMSE	41.2275	44.7460	18.4704	18.6100	29.2235	21.4150	20.3299	17.5247
	ERGAS	18.5383	20.1201	8.2854	8.3358	13.1382	9.6341	9.1409	7.8784
	CC	0.6902	0.8138	0.9431	0.9423	0.8389	0.9121	0.9267	0.9467
	SSIM	0.7186	0.5209	0.8623	0.8620	0.7659	0.8807	0.8285	0.8792
	SAM	3.2074	1.4370	2.5378	2.2618	2.0981	2.4695	2.0469	0.3035
	RASE	37.0416	40.2028	16.5950	16.7205	26.2563	19.2407	18.2657	15.7453
Test-RS-4	RMSE	37.2295	38.0868	18.9726	18.9917	27.3476	21.6457	19.0765	16.8401
	ERGAS	17.6893	18.7514	9.8712	9.8679	12.7405	10.2940	9.7367	8.7608
	CC	0.9296	0.8798	0.9695	0.9695	0.9499	0.9682	0.9657	0.9745
	SSIM	0.8131	0.7847	0.9329	0.9332	0.8654	0.9277	0.9202	0.9515
	SAM	4.6999	3.5359	5.9898	4.9606	3.3908	5.3534	5.0577	0.3096
	RASE	35.2712	37.5283	19.7080	19.7073	25.5687	20.4844	19.4514	17.5329

gives the statistical results of the objective evaluation indices for the fused images of different algorithms corresponding to Figure 14.

It can be seen from Table 4 that the proposed fusion algorithm achieved a good evaluation index in most cases, and achieved better statistical results than the other methods. In particular, the objective evaluation of SAM demonstrated that the proposed NSST-DCGIF avoided, to a certain extent, the spectral distortion phenomenon caused by the injection of non-adaptive information, which showed that the algorithm could maintain the spectral characteristics of MS very well and made full use of the high spatial resolution of the PAN image to effectively improve the spatial resolution of the fused image. Compared with the other seven algorithms, it had obvious advantages.

4.3. Real Data Experiment

(1) Subjective evaluation

Figure 15 shows the target image fused with 8 different algorithms for real data. It can be seen that the GIF algorithm performed poorly in spatial detail information, the edge information of the fused result was partially lost, and the spectral information was darkened, while the color distortion was more obvious in vegetation areas. The traditional IHS algorithm, ASR algorithm and deep learning algorithm had different degrees of spectral distortion. ATWT, AWLP, NSST + SR + PCNN and the proposed algorithm were similar in spectral aspect to the source MS, but the proposed algorithm performed better in spatial detail information enhancement, with rich textural information and obvious edge information.

(2) Objective evaluation

In the real data experiment, we chose quality with no reference ($QNR$) [53] index to objectively evaluate the fused image. $QNR$ was used to calculate the cross-covariance between bands as the overall quality evaluation index. It was composed of the spectral distortion evaluation index $D_{\lambda }$ and the spatial detail distortion evaluation index $D_s$.

Table 5. Objective evaluation of real data.

Evaluation Index	IHS	GIF	ATWT	AWLP	ASR	Deep Learning	NSST + SR + PCNN	NSST + DCGIF
$QNR$	0.7561	0.2979	0.8765	0.8778	0.7765	0.8384	0.8541	0.8780
$D_{\lambda }$	0.0119	0.0073	0.0105	0.0092	0.0043	0.0037	0.0104	0.0109
$D_S$	0.2349	0.6999	0.1142	0.1140	0.2202	0.1584	0.1370	0.1123

shows the statistical results of the objective evaluation indices of the fusion images of the different algorithms corresponding to Figure 15. It can be seen from Table 5 that the proposed algorithm was better than the other algorithms in $\mathrm{QNR}$ and $D_s$, and the Deep learning algorithm performed better in $D_{\lambda }$. From the comprehensive subjective evaluation and objective evaluation, the algorithm in this paper in the real data experiment was better than the other comparative algorithms.

4.4. Computation Complexity Analysis

Table 6. Time complexity statistics of different fusion algorithms.

Data Set	Time(s)
	IHS	GIF	ATWT	AWLP	ASR	Deep Learning	NSST + SR + PCNN	NSST + DCGIF
Test-RS-1	1.50	2.45	0.61	1.00	202.38	4.06	110.07	44.74
Test-RS-2	1.42	3.04	0.95	0.96	207.59	3.69	122.39	46.13
Test-RS-3	1.55	2.34	0.59	0.60	218.66	4.13	109.54	45.85
Test-RS-4	1.60	2.27	0.58	0.64	222.32	3.94	124.96	51.44
Test-RS-5	1.55	2.29	0.60	0.62	225.18	3.92	137.39	43.79

gives a comparative analysis of the time spent by the eight algorithms on five groups of data sets. From the statistical results in the table, it can be seen that ATWT, AWLP, IHS and traditional GIF algorithms consumed less time, but the fusion results obtained by subjective and objective evaluations were not ideal, resulting in spectral information distortion, loss of spatial details and insufficient spatial information promotion. The time of the Deep learning algorithm counted in this paper did not include the time of the training process, and the training time of deep learning algorithm is generally longer. The ASR algorithm took the longest time. The NSST + SR + PCNN algorithm took about 2.5 times as long as the proposed NSST + DCGIF algorithm, but the quality of the fusion was not as good as that of this algorithm. It can be seen that NSST + DCGIF maintained the validity of the algorithm while taking into account computational efficiency.

5. Conclusions

In this paper, the MS-Pansharpening algorithm in the NSST domain, based on dual constraint guided filtering, is presented. First, the intensity component of MS and the PAN are decomposed by NSST, and then the intensity component of MS is reconstructed by using the principle of directional region energy maximization. Then the spatial detail information of PAN is extracted by using the guided filtering model proposed in this paper, which is based on the average gradient correlation and vector correlation formed by the current location neighborhood. The model ensures the applicability, integrity and accuracy of the extracted information. Finally, the extracted spatial details are injected into each component of MS to get the final fused image, based on the weighted detail information injection model, which maintains the spectral correlation proposed. The proposed NSST + DCGIF model avoids the spectral distortion phenomenon caused by the injection of non-adaptive information to a certain extent. It effectively improves the spatial resolution of the fused image maintaining the spatial detail information while maintaining the spectral information well. Through comparative experimental analysis, subjective and objective indications, especially objective evaluation, the proposed algorithm demonstrated better fusion results than some of the typical MS-Pansharpening algorithms currently available.