A Novel Dual-Branch Pansharpening Network with High-Frequency Component Enhancement and Multi-Scale Skip Connection

Abstract

In recent years, the pansharpening methods based on deep learning show great advantages. However, these methods are still inadequate in considering the differences and correlations between multispectral (MS) and panchromatic (PAN) images. In response to the issue, we propose a novel dual-branch pansharpening network with high-frequency component enhancement and a multi-scale skip connection. First, to enhance the correlations, the high-frequency branch consists of the high-frequency component enhancement module (HFCEM), which effectively enhances the high-frequency components through the multi-scale block (MSB), thereby obtaining the corresponding high-frequency weights to accurately capture the high-frequency information in MS and PAN images. Second, to address the differences, the low-frequency branch consists of the multi-scale skip connection module (MSSCM), which comprehensively captures the multi-scale features from coarse to fine through multi-scale convolution, and it effectively fuses these multilevel features through the designed skip connection mechanism to fully extract the low-frequency information from MS and PAN images. Finally, the qualitative and quantitative experiments are performed on the GaoFen-2, QuickBird, and WorldView-3 datasets. The results show that the proposed method outperforms the state-of-the-art pansharpening methods.

1. Introduction

Abundant spectral and spatial information is contained within high-resolution multispectral (HRMS) images. They can be employed in diverse practical applications, such as environmental detection, urban planning, and other fields [1,2,3]. However, with the limitations of satellite sensors, it is difficult to obtain images with both high spectral resolution and high spatial resolution from a single sensor. An effective approach to solving the problem is through pansharpening, which uses some algorithms to fuse multispectral (MS) and panchromatic (PAN) images to obtain HRMS images. The images exhibit superior spatial resolution and spectral fidelity [4], and the technique is a preprocessing step for images [5].

Over the years, numerous approaches have been introduced in pansharpening, which are usually classified into four main classes: component substitution (CS), multiresolution analysis (MRA), variational optimization (VO), and deep learning (DL).

The CS-based method is to decompose the MS images into spectral and spatial constituents in the domain of transformation, then replace the spatial components of MS images with the PAN image. Then, one obtains the HRMS images by inverse transformation. Representative methods include intensity hue saturation (IHS) [6], Gram–Schmidt (GS) [7], principal component analysis (PCA) [8], and partial replacement adaptive CS (PRACS) [9]. Although the principle of the CS-based approach is simple and has high spatial fidelity, the process of information replacement destroys the structure of the original MS information, which may result in spectral distortion.

The core concept behind the MRA methods revolves around extracting spatial information from a PAN image, followed by integrating this information with the MS images to generate the HRMS images. Representative pansharpening methods for MRA are Laplacian pyramid [10], Wavelet [11], SFIM [12], HPF [13], and Nonsubsampled contourlet [14]. Vivone et al. [15] introduced a regression-based method for pansharpening with full-scale injection coefficients. Alparone et al. [16] introduced an improved additive wavelet luminance scaling method, which achieved fast and high performance. Both MRA-based and CS-based approaches have easy realization, and good spectral information can be obtained to some extent based on the MRA method. But in the case of spatial information injection, it causes a serious loss of spatial information.

VO-based methods predominantly hinge on solving optimization problems, which are modeled using a priori knowledge, and the pansharpening results are obtained by an efficient solving algorithm. It mainly includes sparse [17,18], and model-based representation [19,20]. Zhang et al. [21] introduced a convolution sparse representation technique by representing the PAN and MS images as linear combinations of sparse coefficients and dictionaries, respectively, and then fusing the two by a convolution operation. The VO-based method obtains better fusion results than the previous two methods, but it requires more time and an a priori model that lacks preserved spatial information, so the generated images have very obvious spatial distortions.

Over thee past several years, due to benefits from the strong feature extraction and nonlinear mapping abilities of DL, many DL-based pansharpening methods have emerged. Huang et al. [22] first applied DL in pansharpening and proposed a self-coding pansharpening method that relies on sparse noise, achieving a more desirable fusion effect. Drawing inspiration from SRCNN [23], Masi et al. [24] presented the PNN network, consisting of three layers of CNNs for pansharpening; the proposed method exhibits notable performance gains over traditional approaches. Then, Yang et al. [25] presented PanNet, which introduces the idea of skip connections and trains the network within the high-pass domains. The performance is significantly improved by integrating the learned spatial information into the MS images. Yuan et al. [26] proposed MSDCNN, which improves the model performance by using multi-scale convolution kernels to process both deep and shallow information. Deng et al. [27] presented FusionNet by combining the traditional method with CNN, which fully extracts the spatial information of the images using a difference strategy. A triple-double network was introduced by Zhang et al. [28]. This structure gradually incorporates spatial details from the PAN image into the MS images, resulting in a high spatial resolution output. Tu et al. [29] proposed MMDN, which extracts spatial information on different scales through multi-distillation residual information blocks and multi-scale dilation blocks, and further improves the ability to capture detailed information by extracting residual features. Lei et al. [30] proposed a multi-scale hierarchical pansharpening network featuring adaptive optimization, which adequately captures multi-scale features by grouping dilated blocks and fuses multilevel hierarchical information for feature enhancement.

Cheng et al. [31] proposed DMFANet, which performs multilevel feature fusion within the network architecture that maximizes the utilization of features at different levels. Jian et al. [32] proposed the MMFN network for pansharpening, which fully extracts spectral and spatial information at different levels by combining multi-scale and multi-stream structures. Lu et al. [33] proposed the AWFLN network, which fully extracts detail information by embedding a multi-scale convolution module, and the accuracy of detail extraction is improved by adding feature learning blocks to adaptively learn the feature weights. Lu et al. [34] proposed SEWformer, which utilizes a cross-scale interactive encoder and spatially spectrally enhanced window transformer to fully extract detail information. Zhao et al. [35] presented an adaptive frequency-adjusted PRNet to address the spatial and frequency differences between MS and PAN images, capturing contextual information by fusing multi-scale frequency domain features.

In order to generate high-quality images, we propose a novel dual-branch pansharpening network with high-frequency component enhancement and a multi-scale skip connection. We design the high-frequency branch consisting of the high-frequency component enhancement module (HFCEM) to acquire high-frequency features from the MS and PAN images. We also design the low-frequency branch consisting of the multi-scale skip connection module (MSSCM) to extract the low-frequency features. Through two independent and synergistic branches, it can achieve accurate extraction and fusion of image information. Better fusion results are obtained at both reduced- and full-scale experiments.

The main contributions of the paper are as follows:

• We propose a novel dual-branch fusion network. While fully extracting high- and low-frequency information, it solves the problem that traditional networks ignore the differences and correlations between PAN and MS images.
• In the high-frequency branch, we designed HFCEM. Multi-scale block (MSB) processing is performed on the extracted high-frequency component to generate weights and fully extract high-frequency information. The proposed HFCEM is able to effectively utilize the correlations between images and exhibits excellent performance in recovering texture details.
• In the low-frequency branch, we further designed MSSCM. This module is able to effectively capture features at different scales by combining multi-scale convolution and skip connections, and reuse shallow information features to better extract the overall contour information.

2. Related Work

Although the above structure is able to achieve good results, it is still deficient in considering the differences and correlations of the images. The PAN and MS images possess complementary and redundant information, so constructing efficient DL models will be critical for pansharpening. Figure 1 illustrates the DL framework for pansharpening. For the Figure 1a method, using existing methods [25,33,34], MS images are directly regarded as low-frequency information, while the structural information within the image is disregarded, leading to spectral distortion. For the Figure 1b method, using existing methods [24,26], which learn features directly from the spliced PAN and MS, the differences between the two images are ignored, resulting in unspecified learned features. For the Figure 1c method, using existing method [31], which uses two independent feature extraction branches, it ignores the correlations between different images, resulting in insufficient extracted information. In summary, the core challenge we face is how to fully and accurately extract image features while also fully considering the unique differences and correlations between different images.

In response to the above problems, we present a novel framework, as shown in Figure 1d. The high-frequency and low-frequency information from MS and PAN images are obtained using module 1 and module 2 in two branches, respectively. The two branches can fully extract the feature information of the images and take into account the differences and correlations between the images.

3. Proposed Method

The overall framework of the model is shown in Figure 2. Firstly, when extracting high-frequency information, the method is no longer limited to obtaining it from PAN images only but combines MS images, effectively avoiding the problem of insufficient feature extraction that may result from a single source. At the same time, we no longer directly regard MS as low-frequency information directly, avoiding the blurring of information caused by ignoring inter-image differences.

Subsequently, different information is extracted through these two branches. The high-frequency branch comprises two layers of HFCEM, which are specifically designed to capture detailed information. The low-frequency branch consists of two layers of MSSCM, aimed at extracting information at low frequencies. Finally, after two levels of feature extraction, the reconstruction of the HRMS images is achieved by fusing the different information obtained from the two branches.

It is particularly important to perform data preprocessing prior to feature extraction, which provides rich and detailed information for subsequent feature extraction. For the convenience of modeling, we define the PAN image as P and the MS images as M. The image preprocessing module consists of three separate branches, $D_1$, $D_2$, and $D_3$. The specific formula is expressed as follows.

$$D_2=Concat(P,D_1)$$

(1)

$$D_3=P^D-D_1$$

(2)

where $D_1$ is upsampled four times using a 23-tap polynomial, $P^D$ denotes the PAN image replicated along the spectral dimension, and $Concat(·)$ refers to the concatenation function.

3.1. HFCEM

The PAN and MS images contain high-frequency information, but extracting this information adequately is a challenge. Inspired by [27,36], there is still room for optimization in extracting high-frequency information. As shown in Figure 3, we propose an improved structure for extracting high-frequency information.

The input to this structure includes $D_3$ and $D_1$, where $D_3$ was used as the high-frequency components. Subsequently, the MSBs consisting of 3 × 3, 5 × 5, and 7 × 7 are utilized to further obtain the weights of the high-frequency information. Next, these weights are multiplied with $D_1$ pixel by pixel to obtain a richly detailed output image $H_1$. In the high-frequency branch, a two-layer HFCEM is used to enhance the extraction of image features. With the above structure, the high-frequency information of the image is fully extracted.

$$\begin{array}{cc}\hfill S_1& =Con{v}_4\left(Concat(Con{v}_1\left(D_3\right),Con{v}_2\left(D_3\right),Con{v}_3\left(D_3\right))\right)\hfill \\ \hfill H_1& =S_1⨂D_2\hfill \end{array}$$

(3)

where $Con{v}_1$, $Con{v}_2$,$Con{v}_3$, and $Con{v}_4$ represent the sizes of the convolution kernels, which are 3, 5, 7, and 1. ⨂ is a pixel-by-pixel multiplication operation.

$$H_2=HFCEM(H_1,D_3)$$

(4)

where $H_2$ represents the output of the second HFCEM layer.

3.2. MSSCM

Due to the differences in information between MS and PAN images, directly treating MS images as low-frequency information to be summed with the high-frequency information causes their unique information to be ignored and the fusion result suffers from spectral distortion. To solve this issue and to take full advantage of the complementary strengths between the two, we use multi-scale skip connection extraction to capture low-frequency information. In Figure 4a, this structure can extract deeper features but does not fully utilize the information between the different levels. In Figure 4b, a multi-scale structure for feature extraction, the structure can capture more detailed information from the image, but feature extraction for low-frequency information is still lacking.

Combining the above two structures, we propose MSSCM, which aims to comprehensively capture the low-frequency information of images. As in Figure 4c, the module employs three convolution kernels of decreasing sizes in series to effectively extract the low-frequency feature information at different scale levels. The deep fusion of multilayer features is achieved by superimposing the convolution results at all levels through the Concat operation. In addition, it reduces low-frequency information loss in the convolution process by the skip connection mechanism.

First, $D_1$ is subjected to feature extraction using convolution kernels of 7 × 7, 5 × 5, and 3 × 3 sizes. Subsequently, the results after each convolution are concatenated to reuse shallow features and fuse multilayer features. Finally, the 1 × 1 convolution is used to adjust the number of channels, while reducing the parameters in the model, and $M_1$ is obtained by skip connections, which effectively mitigates the loss of information due to the increased depth of the network.

$$C_i=\left\{\begin{array}{c}{D}_1,i=0\hfill \\ Relu\left(Con{v}_j\left(C_{i-1}\right)\right),i=1,2,3\hfill \end{array}\right.$$

(5)

$$M_1=Relu\left(Conv\left(Concat(C_1,C_2,C_3)\right)\right)+C_0$$

(6)

where $Con{v}_j$ represents the size of the convolution kernel, which takes the values of 7, 5, and 3. $C_i$ indicates the output of the corresponding convolution operation.

In order to fully consider and utilize the potential correlations between the branches and to extract the low-frequency image information more efficiently, we sum the outputs of the first level of the HFCEM and MSSCM branches. The added results are used as the input of the second level of MSSCM, which enhances the richness of information. After two levels of feature extraction, the fused-images HRMS is finally obtained by fusing the features of the two branches.

$$M_2=MSSCM(M_1+H_1)$$

(7)

$$HRMS=D_1+Conv\left(Relu\left(Conv(H_2+M_2)\right)\right)$$

(8)

3.3. Loss Function

Following the network architecture design, we use the mean squared error (mse) as the loss function, and the following equation for the loss function:

$${\displaystyle Loss=\frac{1}{N}\sum _{j=1}^N{\Vert G{T}_j-H_{Net}(M_j,P_j)\Vert }^2}$$

(9)

where j represents the j-th training sample, and N represents the total number of training. $H_{Net}$ represents the network structure of this paper. $GT$ represents the ground truth (GT) images.

4. Experiments and Results

This section outlines the datasets for experimentation, the comparison methods, the evaluation index, and the parameter configurations for the conducted tests. The presented method is compared with other approaches by using three datasets. Finally, the efficacy of each introduced module is confirmed through ablation experiments.

4.1. Experimental Design

This section presents an overview of the dataset characteristics and details; we used the published dataset PanCollection [37] for training and testing. We used GaoFen-2, QuickBird, and WorldView-3 datasets to verify the robustness of the network. The MS images of the GaoFen-2 and QuickBird datasets include four bands, and the MS images of WorldView-3 contain eight bands.

Table 1. Specific settings for the datasets.

Satellite	Band	Resolution (m)	Number of Training Images	Number of Validation Images
GaoFen-2	LRMS	4	19,809	2201
	PAN	1
QuickBird	LRMS	2.44	17,139	1905
	PAN	0.61
WorldView-3	LRMS	1.2	9714	1080
	PAN	0.3

lists information about the datasets, including the spatial resolution of the remote sensing images, and the number of images in the training and validation sets. Figure 5 presents the MS and PAN images for use in the experiment. Experiments are performed on the three datasets using reduced- and full-scale experiments.

4.2. Methods of Comparison

We chose ten methods for the experiments in the three datasets introduced above, and the comparison methods included GS [7], PRACS [9], Wavelet [11], HPF [13], PNN [24], MSDCNN [26], FusionNet [27], TDNet [28], AWFLN [33], SEWformer [34], and PRNet [35].

The traditional methods are implemented on MATLAB 2018b. All DL-based methods methods are implemented using Python 3.6 with PyTorch 1.9.0 on the PC side using NVIDIA GeForce RTX 3060 GPUs. The training parameters of the DL-based pansharpening method are as follows: the batch size is 16, the epochs are set to 400, the initial value is 0.0003, and the decay is once every 100 epochs, with a decay rate of 0.8.

4.3. Evaluation Indicators

To objectively evaluate the experimental results, we adopted two strategies to quantitatively assess the fusion results. In this study, the reduced-scale experiment reference metrics include the spectral angle mapper (SAM) [38], the relative global synthesis error (ERGAS) [39], the correlation coefficient (CC) [40], the universal image quality index (Q) [41], and the extended version of Q (Q2n) [42].

SAM represents the angle between the corresponding pixels of the HRMS and GT images, which is used to assess their similarity. A lower SAM value signifies reduced spectral distortion, thus the best value is 0. ERGAS is used to assess the spectral retention of the image, reflecting the situation of spectral distortion. A smaller value of ERGAS indicates a smaller spectral loss, ideally 0. CC measures the similarity between the two images and indicates their geometric distortion, and a higher CC value signifies a better fusion effect. Q is used to evaluate quality in a variety of imaging applications, and Q2n is a widely used quality evaluation metric in generalization processing. The optimal values for Q and Q2n are 1.

The quantitative criterion for the full-scale experiment is the reference-free quality index (QNR) [43], $D_{\lambda }$, and $D_s$. $D_{\lambda }$ is the spectral quality index that evaluates the image’s spectral quality. $D_s$ is used for measuring spatial distortion. The closer these two metrics are to 0, the higher the quality of the fused image. QNR assesses the overall generalization capability of the generated images; the optimal value is 1. The indicator values of the quantitative results are the average values of the experiments.

4.4. Reduced-Scale Experiments

We designed reduced-scale experiments to validate the superiority of our proposed method on the GaoFen-2, QuickBird, and WorldView-3 datasets, with MS and PAN images of 16 × 16 and 64 × 64 sizes. In the quantitative results, we bold the best value of each metric to better observe the method corresponding to the best value. In the qualitative results, to clearly exhibit the details within the fusion images, a part of the fusion image is boxed in red and enlarged three times in this paper. The error maps show how close the fused image is to the GT images, where the more blue areas, the better the fusion result.

Table 2. Quantitative evaluation based on the GaoFen-2 dataset.

Method	SAM	ERGAS	CC	Q	Q2n
Reference	0	0	1	1	1
GS	2.1482	2.4529	0.9693	0.9825	0.8469
PRACS	1.7781	1.7027	0.9859	0.9874	0.9236
Wavelet	1.9624	2.0984	0.9761	0.9796	0.8759
HPF	1.7716	1.7993	0.9843	0.9876	0.9135
PNN	1.1663	1.2583	0.9930	0.9941	0.9639
MSDCNN	1.0074	1.0629	0.9941	0.9955	0.9711
FusionNet	1.0439	1.1522	0.9933	0.9956	0.9658
TDNet	0.9170	0.8555	0.9964	0.9965	0.9790
AWFLN	0.8978	0.8191	0.9968	0.9966	0.9822
SEWformer	0.8775	0.8059	0.9966	0.9967	0.9823
PRNet	0.8881	0.8079	0.9969	0.9966	0.9824
Proposed	0.8653	0.7972	0.9969	0.9968	0.9823

shows the quantitative results on the GaoFen-2 dataset. According to the table, the GS method has the highest SAM and ERGAS values, while PRACS and HPF methods showed better results in traditional methods. The DL-based methods exhibit notable advantages in both SAM and ERGAS and CC, Q, and Q2n. In contrast, PNN and MSDCNN fall short in pansharpening, probably due to the relative simplicity of these models producing results where the extracted features are not clear. Although AWFLN, SEWformer, and PRNet perform better than the above three methods, our proposed network better preserves spatial and spectral information.

Figure 6 shows the pseudo-color map of the GaoFen-2 dataset. We chose buildings as local details, and the figure shows that the GS method exhibits noticeable spectral distortions, such as the color of the field. The Wavelet method shows spatial aberration, especially for roads and buildings. The result of the method based on DL-based methods is much clearer, and the local zoom does not show obvious blurring. As can be seen in the error maps in Figure 7, PNN, MSDCNN, FusionNet, and TDNet have large errors with GT, and the spatial details are not well recovered. The SEWformer method and our method have smaller errors. Our proposed method fully considers the differences and correlations between images, and attains the best results in that the detail information is well handled and closest to the GT images.

Table 3. Quantitative assessment indicators for the QuickBird dataset.

Method	SAM	ERGAS	CC	Q	Q2n
Reference	0	0	1	1	1
GS	4.6296	7.0340	0.9393	0.9380	0.7617
PRACS	4.0177	6.5890	0.9456	0.9470	0.8194
Wavelet	4.8055	7.3663	0.9301	0.9294	0.7610
HPF	4.2084	6.6342	0.9454	0.9453	0.8074
PNN	3.9218	5.9846	0.9569	0.9569	0.8901
MSDCNN	3.7318	5.7833	0.9603	0.9601	0.8895
FusionNet	3.6742	5.7089	0.9585	0.9593	0.8864
TD	3.6608	5.7846	0.9569	0.9513	0.8501
AWFLN	3.6342	5.7399	0.9603	0.9601	0.8937
SEWformer	3.5948	5.5306	0.9619	0.9624	0.9005
PRNet	3.5750	5.7314	0.9621	0.9618	0.9011
Proposed	3.5110	5.3458	0.9626	0.9627	0.9012

shows the index values of various methods on the QuickBird dataset. Among the traditional methods, SAM and ERGAS based on GS and Wavelet methods are significantly weaker than other methods, with weaker spatial and spectral fidelity. The metric values of the DL-based methods do not differ much, but the metric values of our proposed methods are superior to other compared methods.

Figure 8 and Figure 9 show the visualization results and error maps for the QuickBird dataset. The Wavelet method has the least satisfactory fusion results, with obvious spectral aberrations. In contrast, the DL-based methods do not show significant spectral distortion. In Figure 9, PRNet and the proposed method have fewer residuals, but the PRNet method still needs to improve in spatial detail enhancement. Overall, the proposed method is excellent at preserving both spatial and spectral information.

The results of the quantitative evaluation for the WorldView-3 dataset are shown in

Table 4. Quantitative assessment indicators for the WorldView-3 dataset.

Method	SAM	ERGAS	CC	Q	Q2n
Reference	0	0	1	1	1
GS	4.0384	3.8543	0.9553	0.9017	0.8584
PRACS	4.0031	3.4407	0.9579	0.9083	0.8812
Wavelet	5.281	4.4258	0.9321	0.8766	0.8361
HPF	3.9317	3.7153	0.9492	0.9111	0.8725
PNN	3.4077	3.7553	0.9427	0.9280	0.8881
MSDCNN	3.2809	3.8107	0.9430	0.9320	0.8858
FusionNet	3.2728	3.8245	0.9427	0.9321	0.8836
TD	3.3098	3.7089	0.9467	0.9337	0.8911
AWFLN	3.1754	3.3648	0.9572	0.9402	0.9111
SEWformer	3.2081	3.4954	0.9542	0.9370	0.9063
PRNet	3.2711	3.2866	0.9583	0.9385	0.9124
Proposed	3.1598	3.3370	0.9584	0.9408	0.9125

. It can be seen that the Wavelet method has the worst metric value and the DL-method achieves better performance than the traditional method. Although the ERGAS value of the proposed method is second only to the PRNet method, in terms of the overall index, our proposed method performs the best. In Figure 10, the fused images of the PRACS and HPF methods are blurred; the Wavelet method has spectral distortion. The results of FusionNet and TDNet show insufficient extraction of detailed information. In contrast, PRNet and the proposed method are closer with GT images. As shown in Figure 11, we further compare the variability of the methods using error maps. Overall, our proposed method performs best in both qualitative and quantitative results.

4.5. Full-Scale Experiments

In this section, full-scale experiments are performed on the GaoFen-2, QuickBird, and WorldView-3 datasets, and the sizes of MS and PAN images are 128 × 128 and 512 × 512.

Table 5. Quantitative assessment indicators for full-scale experimental.

	GaoFen-2			QuickBird			WorldView-3
Method	QNR	${\mathit{D}}_{\lambda }$	${\mathbf{D}}_{\mathbf{s}}$	QNR	${\mathit{D}}_{\lambda }$	${\mathbf{D}}_{\mathbf{s}}$	QNR	${\mathit{D}}_{\lambda }$	${\mathbf{D}}_{\mathbf{s}}$
Reference	1	0	0	1	0	0	1	0	0
GS	0.8756	0.0683	0.0602	0.6439	0.0888	0.2934	0.8928	0.0159	0.0928
PRACS	0.8907	0.0355	0.0766	0.6837	0.0757	0.2603	0.8904	0.0259	0.0858
Wavelet	0.8699	0.0918	0.0422	0.6697	0.1491	0.2129	0.8586	0.0954	0.0509
HPF	0.9039	0.0468	0.0517	0.7424	0.0947	0.1799	0.9033	0.0303	0.0685
PNN	0.9825	0.0128	0.0048	0.8358	0.0508	0.1195	0.9436	0.0235	0.0337
MSDCNN	0.9814	0.0138	0.0049	0.8482	0.0703	0.0877	0.9545	0.0165	0.0295
FusionNet	0.9758	0.0159	0.0084	0.8818	0.0726	0.0492	0.9552	0.0114	0.0338
TDNet	0.9842	0.0102	0.0057	0.8877	0.0594	0.0562	0.9469	0.0232	0.0306
AWFLN	0.9823	0.0076	0.0102	0.8922	0.0578	0.0531	0.9559	0.0151	0.0295
SEWformer	0.9826	0.0126	0.0049	0.8967	0.0520	0.0541	0.9573	0.0119	0.0312
PRNet	0.9874	0.0074	0.0052	0.8983	0.0511	0.0533	0.9581	0.0131	0.0292
Proposed	0.9885	0.0071	0.0044	0.9070	0.0423	0.0529	0.9637	0.0087	0.0278

shows the values of the GaoFen-2, QuickBird, and WorldView-3 datasets at full-scale non-reference metrics. The GS and Wavelet methods have worse QNR metrics, and the DL-based approach performs better in terms of overall performance metrics. For the QuickBird dataset, the proposed performs in QNR and $D_{\lambda }$, with the second-best results in $D_s$. The proposed method produces better fused results on all three datasets compared to other methods.

Figure 12, Figure 13 and Figure 14 show the fusion images of the three datasets for the full-scale experiments. The fused images of the traditional methods still have deficiencies in spectral and spatial information retention. As shown in Figure 12, the HPF method suffers from spatial blurring and the Wavelet method shows white artifacts. In Figure 13, the AWFLN method is poor in details, such as vehicles on the road. As shown in the enlarged box, the proposed methods perform better in spectral and spatial details. In Figure 14, the Wavelet and HPF methods show spectral distortion, and SEWformer, PRNet, and our proposed method have higher fused image resolution. Due to the lack of GT images, it is difficult to distinguish, as can be seen from Table 5, that our method outperforms the other two methods.

4.6. Ablation Experiments

We performed ablation experiments using the GaoFen-2 dataset to assess the contribution of each network module.

Table 6. Comparison of quantitative assessment of ablation experiments on the GaoFen-2 dataset.

Index	MSSCM	HFCEM	Double layer	SAM	ERGAS	CC	Q	Q2n
1	✓		✓	1.2701	1.7005	0.9864	0.9935	0.9466
2		✓	✓	0.9886	0.8942	0.9943	0.9957	0.9754
3	✓	✓		1.0408	1.2261	0.9920	0.9946	0.9681
4	✓	✓	✓	0.8653	0.7972	0.9969	0.9968	0.9823

presents the quantitative metrics, while Figure 15 shows the results of the ablation model along with corresponding error maps. To emphasize the fusion details, we have framed several fused images and tripled their content. Ablation experiment 1 involved removing the HFCEM for feature extraction. Ablation experiment 2 replaces the MSSCM with the (b) structural multi-scale module in Figure 4. Ablation experiment 3 employed a single layer of MSSCM combined with HFCEM.

Upon comparing Table 6 and Figure 15, it is evident that removing HFCEM has the worst results for all metric values. The white buildings depicted in both fusion and error maps appear particularly blurred, underscoring HFCEM’s critical role within the network. In Experiment 2, Table 6 shows that the addition of MSSCM reduces the SAM by 0.12, indicating the validity of the proposed MSSCM.

To assess the effectiveness of the double layer structure, a single layer of HFCEM and MSSCM was used in Experiment 3. Overall comparisons showed a decrease in the values of the various category metrics. Analyses of the error maps show that improvements in detail are still lacking. In addition, we conducted experiments to study the effect of different network layers. Figure 16 and Figure 17 illustrate the number of parameters and SAM values for models with layers 1 to 6. The figures show that the model’s parameter count consistently increases as the number of layers grows. But when the number of layers is greater than 2, the change in the corresponding parameter value SAM tends to slow down. Combining the changes in the index values and the number of parameters, we chose to adopt a double-layer network as the final model.

5. Discussion

First, in Section 4.4, we performed reduced-scale experiments on the GaoFen-2, QuickBird, and WorldView-3 datasets. We outline comparisons with ten other methods through qualitative and quantitative analyses, and the best values obtained from the fused results are bolded in the quantitative assessment. We frame and triple magnify the local details in order to observe the fusion image more clearly. In addition, we use error maps to show the difference between the fused image and the GT images. From Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11, the DL-based method has high-quality fusion results in all three datasets, and the metrics values in Table 2, Table 3 and Table 4 verify its merits. In both qualitative and quantitative results, the proposed method has the best overall metrics values and produces the fused image closest to the GT images, which further proves the good performance of our proposed method.

In Section 4.5, we perform full-scale experiments. As illustrated in Figure 12, Figure 13 and Figure 14, the fused images of the traditional methods are blurry, and the results of the Wavelet method are the most obvious. The AWFLN, SEWformer, and PRNet methods retain spatial details better compared to PNN and MSDCNN, but our method has better spectral retention and spatial detail enhancement than other DL-based methods. To verify the performance of each module, we performed ablation experiments in Section 4.6. From Figure 15 and Table 6, it can be seen that the use of MSSCM and HFCEM alone is effective for the fusion results, but the simultaneous use of these two modules leads to the best fusion results.

6. Conclusions

We propose a dual-branch network with high-frequency enhancement and a multi-scale skip connection for pansharpening, to address the issue of ignoring the differences and correlations between the traditional branch images. We used two different branch structures to extract various types of information in the images. Firstly, we design HFCEM to fully extract the high-frequency information from the images. Second, we further design MSSCM to extract low-frequency information. Through the above structure, the image correlations are enhanced and the distortion caused by the difference in image information is effectively alleviated. Finally, by comparing with ten different methods on three datasets, GaoFen-2, QuickBird, and WorldView-3, the proposed method demonstrates superior performance in reduced- and full-scale experiments.