Multi-Scale and Multi-Stream Fusion Network for Pansharpening

Abstract

Pansharpening refers to the use of a panchromatic image to improve the spatial resolution of a multi-spectral image while preserving spectral signatures. However, existing pansharpening methods are still unsatisfactory at balancing the trade-off between spatial enhancement and spectral fidelity. In this paper, a multi-scale and multi-stream fusion network (named MMFN) that leverages the multi-scale information of the source images is proposed. The proposed architecture is simple, yet effective, and can fully extract various spatial/spectral features at different levels. A multi-stage reconstruction loss was adopted to recover the pansharpened images in each multi-stream fusion block, which facilitates and stabilizes the training process. The qualitative and quantitative assessment on three real remote sensing datasets (i.e., QuickBird, Pléiades, and WorldView-2) demonstrates that the proposed approach outperforms state-of-the-art methods.

1. Introduction

Remote sensing imaging is an important means by which to obtain Earth object information as high-resolution remote sensing images that are crucial in the interpretation of complex Earth objects. Due to physical limitations, a single type of satellite sensor cannot acquire a multi-spectral (MS) image with both high spatial and spectral resolution. An MS image is generally characterized by high spectral resolution but low spatial resolution, while a panchromatic (PAN) image has the opposite characteristic. However, in many practical applications, such as complex Earth feature interpretation [1,2], change detection [3,4], and land cover classification [5], high spatial and spectral resolutions are both crucial for a good analysis. Compared with hardware improvements, pansharpening provides a better solution to alleviate this problem and has gained much more attention in the remote sensing community. Pansharpening is referred to the generation of a desired high-resolution multi-spectral (HR MS) image from an MS image and a simultaneously acquired PAN image. Over the past decades, various pansharpening methods have been developed [6,7,8]. Classic and deep learning (DL)-based methods are two significant categories among the existing pansharpening methods.

Classic methods mostly use hand-crafted priors derived from basic assumptions or variational optimization procedures to fuse the low-resolution MS (LR MS) image and the PAN image. For instance, the component substitution (CS)-based methods assume that high-resolution PAN images can entirely or partially replace the principal component after the linear transformation of LR MS images. However, due to the differences in the spectral responses between the MS and PAN sensors, CS-based methods often suffer from severe spectral distortions. Another historical category is based on the use of multi-resolution analysis (MRA) frameworks to extract PAN details. The hypothesis behind the approaches in this class is that the missing high spatial details of the LR MS image can be obtained by extracting them from the PAN image via multi-scale decomposition. However, MRA-based pansharpening results are affected by both detailed extraction and injection procedures. For instance, unreasonable extraction approaches and ineffective injection coefficients may produce insufficient details, which can cause blur effects or artifacts in the final results. To achieve a better tradeoff between spectral information and spatial details, variational model (VM)-based methods use data and regularization terms to guide the pansharpening process. However, several approaches rely upon complex variational models requiring time-consuming optimization procedures to be solved.

Recently, the advanced capabilities of getting non-linear representations using convolutional neural networks (CNNs) and deep learning have achieved breakthroughs in solving the pansharpening task. However, the existing deep learning (DL)-based pansharpening methods often encounter three challenges, which are stated as follows:

• Some DL-based methods combine all the input data together, but, in this way, they lose the possibility to independently analyze the inputs of the fusion process that are represented by the MS and PAN images;
• Most fusion models process the source images separately and neglect their spectral/spatial correlations;
• The use of the fixed-scale information of the input images limits the pansharpening performance.

Therefore, there is still room for improvement in pansharpening to enhance the spatial resolution while preserving the spectral information.

To address the above issues, in this paper, we propose a multi-scale and multi-stream fusion network (MMFN) for pansharpening. More specifically, we developed a multi-stream fusion block that gets the best from the input images’ spectral/spatial correlation and from the data, taking them as a separate source of information. The MS, PAN, and the concatenation of the two above-mentioned images were separately inputted to shallow convolution layers and deep convolution networks. Moreover, we downsampled the source images twice to extract multi-scale features, which can avoid the loss of spectral and spatial information. Additionally, we designed a multi-stage image reconstruction to recover the desired HR MS image. That is, for each multi-stream fusion block, a loss function was built, which can boost the training process and promote high-resolution MS images close to ground-truth images.

The main contributions of this paper can be summarized as follows:

• Multi-scale and multi-stream strategies. We combined the multi-scale and multi-stream strategies to build a hybrid network structure for pansharpening extracting both shallow and deep features at different scales;
• Multi-stream fusion network. On the basis of multi-scale and multi-stream strategies, we introduced a multi-stream fusion network, which separately leverages spectral and spatial information in the MS and PAN images. Simultaneously, we considered the pansharpening problem as a super-resolution task that concatenates the PAN and MS images to further extract spatial details;
• Multi-scale information injection. We make full use of the multi-scale information of the input (MS and PAN) images by exploiting downsampling and upsampling operations. At each scale, the information of the original MS image is injected through a multi-scale loss.

The remainder of this paper is organized as follows. Section 2 briefly reviews the existing pansharpening methods. Section 3 presents the proposed pansharpening approach. Section 4 shows the experimental results. Finally, in Section 5, the conclusions are drawn.

2. Related Works

2.1. Traditional Pansharpening

The first model-based methods devoted to solving the pansharpening problem belong to the component substitution (CS) and the multi-resolution analysis (MRA) classes [6]. Early stage CS-based techniques include the intensity–hue-saturation (IHS) [9], the principle component analysis (PCA) [10], and the Gram–Schmidt (GS) transform [11]. These methods assume that the HR PAN image can substitute the principal component of the LR MS image projected into a new domain using one of the above-mentioned transformations. Aiazzi et al. [12] proposed an adaptive CS-based pansharpening method using multivariate regression analysis of the two inputs. Garzelli et al. [13] designed a band-dependent-spatial-detail (BDSD) model that extracts the optimal detail image from the PAN image for each MS band. A robust estimator based on this model has recently been proposed in [14]. Choi et al. [15] exploited the idea of the partial replacement proposing the so-called PRACS approach. Instead, Kang et al. presented in [16] an image matting model-based (MMP) component substitution pansharpening method. To alleviate the well-known spectral distortion problem for CS outcomes, researchers developed MRA-based solutions. Representative MRA-based methods include the generalized Laplacian pyramid (GLP) [17], the contourlet transformation [18], the curvelet transformation [19], and the use of non-linear morphological filters [20]. Recently, a new view of seeing the MRA framework has been developed and the detail extraction is efficiently addressed by the simple difference between the PAN image and its low-pass version [21,22] leading to new solutions achieving a high performance with a limited computational burden [23,24,25,26,27]. In this case, to overcome the issue about the limited knowledge of the shape of the spatial filters to generate the low-pass versions of the PAN image, deconvolution approaches have recently been designed [28,29].

Some promising model-based methods are the VM ones [30], which use regularization terms based on images’ prior information to solve the pansharpening (ill-posed) problem. Palsson et al. [31] presented a total variation (TV)-based pansharpening method, which encourages noise removal and preserves the edge detail information of an image. To further reduce spectral distortion, Duran et al. [32] utilized the image self-similarity, which has been applied to the PAN image to establish the nonlocal relationships among the patches of the final result. Chen et al. [33] combined local spectral consistency and dynamic gradient sparsity to simultaneously implement image registration and fusion (SIRF). Liu et al. [34] exploited structural sparsity between the LR MS image and the desired HR MS image and spectral–spatial low-rank priors. In [35], Khademi et al. incorporated an adaptive Markov random field (MRF) prior into the Bayesian framework to restore pansharpened results. Finally, hyper-Laplacian prior-based [36], local gradient constraints-based [37], texture space-based [38], and gradient sparse representation-based [39] approaches have also achieved interesting performances.

2.2. Deep Learning-Based Pansharpening

Deep learning-based methods have recently shown great potential for pansharpening thanks to their powerful nonlinear mapping capability. A comprehensive review about the topic with a critical comparison of widely-used approaches together with a freely distributed toolbox can be found in [40]. The deep learning-based pansharpening methods can be divided roughly into two categories: supervised and unsupervised.

Supervised pansharpening methods require the generation of low-resolution MS images by exploiting the original MS images as ground-truth. Masi et al. [41] attempted first to apply a network for single-image super-resolution [42] to fuse PAN and MS images. This pansharpening method, the so-called PNN, uses a three-layer convolutional neural network (CNN) to construct the map between the inputs and the desired HRMS image. Similarly, Yang et al. [43] proposed a deep network architecture (PanNet) to improve the pansharpening accuracy in terms of spatial and spectral preservation. Scarpa et al. [44] proposed a fast and high-quality target-adaptive CNN (TACNN)-based pansharpening method. To fully make use of the respective information of MS and PAN images, Liu et al. [45] adopted a two-stream fusion network (ResTFNet-$l_1$) for pansharpening. Xu et al. [46] also designed a shallow and deep feature-based spatial-spectral fusion network to enhance the pansharpened results. Liu et al. [47] attempted first to explore the combination of pansharpening and generative adversarial networks (GANs), i.e., the so-called PSGAN, to produce high-quality pansharpened results. Shao et al. [48] combined the idea of an encoder with GANs for pansharpening. A first attempt to integrate classical CS and MRA frameworks with deep convolutional neural networks has been provided in [49]. A more general framework that can fuse images with an arbitrary number of bands exploiting recurrent neural networks has recently been proposed for pansharpening [50]. Wang et al. [51] used an explicit spectral-to-spatial convolution that operates on the MS data to produce an HR MS image. However, the above methods only process single-scale images and do not perform multi-scale feature extraction. Exploiting multi-scale features is of crucial importance for pansharpening, even representing a hot topic for deep neural networks. Yuan et al. [52] introduced a multi-scale feature extraction and multidepth network for pansharpening. Wei et al. [53] proposed a two-stream coupled multi-scale network to fuse MS and PAN images. Huang et al. [54] utilized a non-subsampled contourlet transform to decompose MS and PAN images into low- and high-frequency components, then trained a network with high-frequency images and obtained the fused image by combining the output of the network and the low-frequency components of the MS image. In [55], multi-scale perception dense coding has been integrated into a neural network for pansharpening. Hu et al. [56] combined multi-scale features extraction and dynamic convolutional networks to fuse MS and PAN images. Wang et al. [57] employed a multi-scale deep residual network for pansharpening. In [58], a grouped multi-scale dilated convolution was designed to sharpen MS images. Peng et al. [59] adopted multi-scale dense blocks in a global dense connection network for pansharpening. Multi-scale feature extraction and multi-scale densely connection were employed in [60] to fuse MS and PAN images. Lai et al. [61] extracted multi-scale features in an encoder-decoder network for pansharpening. Zhou et al. [62] proposed a multi-scale invertible network and heterogeneous task distilling to fully utilize the information at full resolution. A multi-scale grouping dilated block was designed in [63] to obtain fine-grain representations of multi-scale features for pansharpening. Tu et al. [64] introduced a clique-structure-based multi-scale block and a multidistillation residual information block for MS and PAN image fusion. A parallel multi-scale attention network is presented in [65] to learn residuals to be injected into the LR MS image. Huang et al. [66] combined MRA and deep learning methods, achieving the injection coefficients with multi-scale residual blocks. Jin et al. [67] decomposed the input images using Laplacian pyramids, then exploited a multi-scale network to fuse each image scale. In [68], a pyramid attention was applied to capture multi-scale features and then the latter were fused by a feature aggregation module to obtain the fused product. A multi-scale transformer with an interaction attention module was introduced in [69]. Zhang et al. [70] proposed a 3D multi-scale attention network for MS and PAN image fusion, in which 2D and 3D convolutions were compared for this task. Although these methods all use multi-scale feature extraction, they either directly perform multi-scale processing on the concatenated MS and PAN images or separately extract features from the MS and PAN images and then perform multi-scale processing. None of the state-of-the-art approaches in the literature can effectively fuse the features extracted from the combined MS and PAN images and the features separately extracted from the MS and PAN images.

The other sub-class of deep learning-based pansharpening approaches relies upon unsupervised methods. Indeed, due to the lack of ground-truth MS images, Qu et al. [71] and Uezato et al. [72] presented unsupervised ways to train models for pan-sharpening.

Finally, there are the promoting methods, belonging to the class of hybrid pansharpening solutions, that combine variational optimization-based and deep learning models, thus benefiting from both the philosophies and increasing the generalization ability of deep learning approaches, see, e.g., Refs. [73,74,75].

3. Proposed Method

The proposed method utilizes multi-scale information and a multi-stream fusion strategy to fully use different levels of features in PAN and MS images. We will introduce the problem formulation of the pansharpening task and depict the overview framework. Afterwards, the network architecture and loss function are presented in detail.

3.1. Problem Formulation

The LR MS image is denoted as M with a size of $w\times h\times c$ ($M\in {\mathbb{R}}^{w\times h\times c}$), while the high-resolution PAN image is denoted as P with a size of $rw\times rh\times 1$ ($P\in {\mathbb{R}}^{rw\times rh\times 1}$), where w, h, and c are the width, height, and number of channels of the MS image, r is the spatial scale factor between the PAN and MS images. The goal of pansharpening is to obtain an HR MS image, X, as close as to the ground-truth image, G, with a size of $rw\times rh\times c$. We designed the overall framework, which learns the pansharpened process, namely, $X=f_{\mathsf{\Theta }}^{net}\left(M,P\right)$, where $f^{net}$ and $\mathsf{\Theta }$ represent the MMFN architecture and its parameters, respectively.

3.2. Network Architecture

Figure 1 shows the overall framework of the proposed method, including the multi-scale feature extraction, the multi-stream feature fusion, and the multi-stage image reconstruction. To fully extract the features from the input PAN and MS images, we used a multi-scale strategy to downsample the input images twice, which allows the extracted features to represent original images at different levels. We used the same multi-stream block (MSB) for each scale to fuse the downsampled PAN and MS images. Finally, the pansharpened sub-images were upsampled to recover the HR MS image using a multi-stage strategy. A detailed description is given in the following sections.

3.2.1. Multi-Scale Feature Extraction

We used a fixed scale factor to downsample the PAN and MS images. This operation can extract different levels of spectral and spatial features in both PAN and MS images and it can maximize the information usage at different scales. We denote $D\downarrow \left(·\right)$ as the downsampling operation. Thus, we have:

$$I_t=D\downarrow \left(I_{t-1},s\right),t=2,3,$$

(1)

where $I_1$ is $\tilde{M}$ or P when the downsampling of the upsampled version of the MS image or the PAN image is considered, respectively, $I_t$ (with $t>1$) represents the downsampled version of the MS or PAN images, t and s are the scale index and the downsampling factor (equal to 2 in our case), respectively.

3.2.2. Multi-Stream Feature Fusion

Many researchers regard the pansharpening task as a super-resolution problem that concatenates the MS and PAN images to form a single stream network for spatial information improvement, as shown in Figure 2a. However, in this way, the related spectral characteristics of the input products are ignored. Some other researchers consider that the MS and PAN images convey various independent information. The typical operation is to use the high-frequency information of the PAN image to build the missing spatial details of the MS image. However, it is hard to state that the spatial information is a typical feature related to the sole PAN image and the spectral features are only related to the MS image. This challenge motivates us to focus on developing a dual-stream fusion network (see Figure 2b), which combines all the features of the MS and PAN images.

To avoid losing information as in the case of the single-stream or the dual-stream, we leveraged a multi-stream fusion strategy to comprehensively exploit information from the spectral and spatial domains. As shown in Figure 2c, we utilized three streams, namely, PAN, MS, and fusion streams, to extract the features from the PAN data, the MS image, and the concatenation of the two inputs (i.e., MS and PAN), respectively. First, the PAN and MS streams built by three sequential convolutional layers were used to extract spatial and spectral features as follows:

$$F=Conv\left(Conv\right(Conv\left(P\right)\left)\right),$$

(2)

where $Conv(·)$ denotes the convolutional layers, P is the (input) MS/PAN image, and F is the corresponding output. The fusion stream is instead fed by the concatenation of the MS, $M_t$, and the PAN, $P_t$, images to extract spatial-spectral features by a ResNet between two convolutional layers. Hence, we have that the spatial–spectral features, $F_{PMS}$, are obtained by:

$$F_{PMS}=Conv\left(ResNet\left(Conv\left([P_t,M_t]\right)\right)\right),$$

(3)

where $[·]$ denotes the concatenation operation and $ResNet(·)$ is the function of ResNet as shown in Figure 2. Finally, we further fused the outputs from the three streams by the same structure as in the fusion stream, thus obtaining the output of the MSB by concatenating $F_{PMS}$ with the further fused features, i.e.,

$$H_t=[F_{PMS},Conv\left(ResNet\left(Conv\left([P_t,M_t,F_{PMS}]\right)\right)\right)],$$

(4)

where $H_t$ is the output of the MSB.

3.2.3. Multi-Stage Image Reconstruction

In our work, we used the original source images and their two downsampled versions to perform a three-stage pansharpening network. All the image pairs were fed into the same multi-stream fusion block using Equation (4) for the extraction of the multi-scale fused features. As shown in Figure 1, these multi-scale features were adopted to reconstruct the multi-scale image by using reconstruction blocks, which have the same structure as the one in the fusion stream (i.e., Net 1 in Figure 2c).

$$F_t=Conv\left(ResNet\left(Conv\left([P_t,M_t,F_{PMS}]\right)\right)\right),$$

(5)

where $F_t$ is the fused image at the t-th scale.

Afterwards, the fusion result of each RB was upsampled and concatenated with the corresponding downsampled MS image as follows:

$$X_t=[U\uparrow \left(F_{t+1},s\right),\widetilde{M_t}],t=1,2,$$

(6)

where $U\uparrow \left(·\right)$ represents the upsampling operation. In our work, we have that $X_3=\widetilde{M_3}$.

After the multi-stage fusion, the final HRMS image can be reconstructed in a residual way, thus preserving the spectral information. Hence, we have:

$$X=F_1+\tilde{M},$$

(7)

where X is the fused (pansharpened) product.

3.3. Loss Function

During the training phase, we adopted the $l_1$-norm as loss function to measure the distance between the pansharpened product and the related ground-truth (reference) image. Comparing it with the $l_2$-norm, the $l_1$-norm can overcome local minima reaching stable training [76]. Our work had three MSB blocks that generated pansharpening images at three different resolutions. Thus, a loss function was adopted for each block to constrain its training. Hence, the overall loss function can be written as:

$$L\left(\mathsf{\Theta }\right)=\frac{1}{3}\sum _{t=1}^3{\left|F_t-HRM{S}_t\right|}_1,$$

(8)

where $\mathsf{\Theta }$ is the set of parameters for the proposed framework, $HRM{S}_t$ is the downsampled version of the ground-truth image at the t-th scale, and ${\left|·\right|}_1$ is the $l_1$-norm.

4. Experiment and Evaluations

In this section, we provide extensive experiments to validate the effectiveness of our proposed method. Moreover, three groups of ablation studies were investigated to further demonstrate the superiority of each claim.

4.1. Datasets and Metrics

Three real remote sensing datasets, captured by QuickBird, Pléiades, and WorldView-2, were used. To boost the pansharpening capability of the proposed network, we adopted data argumentation approaches, i.e., rotation, cropping, and flipping. The total number of training and testing datasets for each satellite were 4000 and 40, respectively.

Due to the lack of ground-truth datasets, we used Wald’s protocol [77] to obtain the downsampling version of both MS and PAN images (with a downsampling factor equal to 4). The degraded MS images were upsampled to the original size so that the MS image could serve as the HR MS (ground-truth) image for quality assessment.

In our study, to quantitatively evaluate the pansharpened results, six metrics with reference and a metric without reference were employed to evaluate the products at reduced resolution and at full resolution, respectively. For the metrics with reference, the correlation coefficient (CC) [12], the relative dimensionless error in synthesis (ERGAS) [78], the $Q_{2}n$ [79], the spectral angle mapper (SAM) [80], the relative average spectral error (RASE) [81], and the root mean squared error (RMSE) were used. Additionally, the metric without reference consisted of the combination of a spectral distortion ($D_{\lambda }$) index and a spatial distortion ($D_S$) index. It is commonly-used and known as a quality with no reference (QNR) index [82,83]. In general, the largest values of CC, $Q_4$, and QNR indicate the best performance. On the other hand, the smaller the ERGAS, SAM, RASE, and RMSE are, the closer the results are to the ground-truth (reference) image.

4.2. Implementation Details

We trained the proposed framework using PyTorch [84]. The configuration of the hardware device was composed of an NVIDIA GTX 1070Ti (GPU), 48-GB RAM (Memory), and an Intel Core i5-8500 (CPU). For the training phase, an Adam optimization algorithm is used. The initial learning rate, the moment decay, and the batch size were set to 0.0001, 0.9, and 4, respectively. The input size of the LR MS image and the PAN image were set to $64\times 64$ and $256\times 256$, respectively. The parameters of each convolutional layer in the MSB block and the RB block are described in

Table 1. Parameters of each convolutional layer in the MSB block and RB block.

Layer Name	Kernel Size	Stride	Padding	Input Channels	Output Channels
Conv0	3 × 3	1	1	1 (PAN) or # MS bands (MS)	32
Conv1	3 × 3	1	1	32	64
Conv2	3 × 3	1	1	64	32
Conv1_1	3 × 3	1	1	1 + # MS bands	32
ResNet1_1	3 × 3	1	1	32	32
Conv1_2	3 × 3	1	1	32	# MS bands
Conv2_1	3 × 3	1	1	64	32
ResNet2_1	3 × 3	1	1	32	32
Conv2_2	3 × 3	1	1	32	# MS bands
Conv(RB)	3 × 3	1	1	2 × # MS bands	32
ResNet(RB)	3 × 3	1	1	32	32
Conv(RB)	3 × 3	1	1	32	# MS bands

. The training epochs were 400.

4.3. Results and Discussion

To fairly compare the effectiveness of the proposed MMFN, we selected thirteen state-of-the-art pansharpening methods, including seven widely used traditional ones, i.e., Inter23 [6], which is the simple upsampling to the PAN scale using a 23-tap polynomial interpolator, AWLP [6,85], BDSD [13], GSA [12], MMP [16], MTF-GLP-HPM (MGH) [21], and SIRF [33], and six deep learning-based ones, i.e., PNN [41], PanNet [43], PSGAN [47], ResTFNet-$l_1$ [45], TACNN [44], and MUCNN [51]. All the experiments were conducted using the set of parameters as originally indicated by the authors in the related papers. In the following, we will present the results for the two classical assessments, i.e., at reduced and full resolutions.

4.3.1. Assessment at Reduced Resolution

In this section, we compare the performance at reduced resolution of the proposed method with the proposed benchmark. Figure 3, Figure 4 and Figure 5 present the pansharpened results for the three datasets acquired by QuickBird, Pléiades, and WorldView-2, respectively. For the visual comparison, all the images are shown in true (RGB) color. Additionally, mean absolute errors (MAEs, by using heatmaps), namely, residual maps between pansharpened results and the related reference images, are also given. The results show that the deep learning-based methods outperform the traditional ones in terms of MAEs.

In Figure 3, compared with the ground-truth image (Figure 3o), the traditional approaches, i.e., Inter23, AWLP, BDSD, GSA, MMP, MGH, suffer from various degrees of spatial detail loss, especially some serious blurring effects generated by Inter23 and GSA methods. This is because the Inter23-based method directly utilizes a polynomial kernel (with 23 coefficients) to upsample the LRMS image without any information from the PAN image. The GSA-based method may fail to estimate the high-frequency details during the component replacement operation. Deep learning-based techniques also show poor performance in preserving spatial details, as seen from the results produced by PNN, PanNet, TACNN, and MUCNN. Although PSGAN and ResTFNet-$l_1$ can obtain promising pansharpened results that are close to the ground-truth image from a visual point of view, some differences still exist when observing the residual maps. By contrast, our MMFN method can achieve the best trade-off in preserving spatial details and spectral signatures, which are the closest with respect to the ground-truth ones.

Table 2. Average quantitative performance on the QuickBird dataset. The best and second best performances are shown in Red and Blue, respectively.

Methods	CC (1)	ERGAS (0)	${\mathit{Q}}_{\mathbf{2}}\mathit{n}$ (1)	RASE (0)	RMSE (0)	SAM (0)	${\mathit{D}}_{\mathit{\lambda }}$ (0)	${\mathit{D}}_{\mathit{S}}$ (0)	QNR (1)
Inter23	0.8746 ± 0.0607	2.2640 ± 0.7606	0.8599 ± 0.0728	9.2008 ± 3.3102	26.4504 ± 11.4306	0.0388 ± 0.0158	0.0021 ± 0.0013	0.0699 ± 0.0303	0.9282 ± 0.0301
AWLP	0.9375 ± 0.0308	1.5913 ± 0.5495	0.9346 ± 0.0322	6.5544 ± 2.4926	19.1237 ± 10.1874	0.0304 ± 0.0106	0.0547 ± 0.0344	0.0971 ± 0.0699	0.8558 ± 0.0931
BDSD	0.9279 ± 0.0373	1.8970 ± 0.7729	0.9211 ± 0.0407	7.5889 ± 3.2623	22.0196 ± 12.5966	0.0374 ± 0.0146	0.0178 ± 0.0105	0.0436 ± 0.0326	0.9394 ± 0.0330
GSA	0.9321 ± 0.0382	1.8597 ± 0.7802	0.9234 ± 0.0434	7.3530 ± 3.0796	21.1029 ± 11.3929	0.0359 ± 0.0136	0.0417 ± 0.0272	0.0974 ± 0.0726	0.8666 ± 0.0884
MMP	0.9385 ± 0.0326	1.4731 ± 0.5839	0.9344 ± 0.0367	5.8352 ± 2.4399	16.9974 ± 9.7580	0.0302 ± 0.0112	0.0209 ± 0.0142	0.0451 ± 0.0404	0.9354 ± 0.0481
MGH	0.9380 ± 0.0312	1.6173 ± 0.5230	0.9358 ± 0.0316	6.2571 ± 2.1460	18.2318 ± 9.0696	0.0297 ± 0.0095	0.0637 ± 0.0359	0.1064 ± 0.0709	0.8391 ± 0.0944
SIRF	0.8252 ± 0.1064	2.4460 ± 0.9231	0.8007 ± 0.1247	9.6182 ± 3.8658	28.3468 ± 16.3364	0.0511 ± 0.0314	0.0668 ± 0.0451	0.0706 ± 0.0674	0.8687 ± 0.0893
PNN	0.9654 ± 0.0175	1.0973 ± 0.3758	0.9646 ± 0.01782	4.3596 ± 1.6335	12.6981 ± 6.3785	0.0243 ± 0.0086	0.0250 ± 0.0269	0.0527 ± 0.0614	0.9252 ± 0.0808
PanNet	0.9715 ± 0.0125	0.9736 ± 0.2891	0.9712 ± 0.0127	3.8504 ± 1.2341	11.2352 ± 5.0074	0.0222 ± 0.0071	0.0181 ± 0.0188	0.0208 ±0.0220	0.9619 ± 0.0384
PSGAN	0.9691 ± 0.0156	1.0041 ± 0.3031	0.9686 ± 0.0158	4.0294 ± 1.3248	11.7302 ± 5.0467	0.0230 ± 0.0073	0.0424 ± 0.0488	0.0545 ± 0.0609	0.9082 ± 0.0953
ResTFNet-$l_1$	0.9762 ± 0.0108	0.8729 ± 0.2616	0.9758 ± 0.0111	3.4346 ± 1.1012	10.0058 ± 4.4144	0.0203 ± 0.0064	0.0226 ± 0.0224	0.0335 ± 0.0469	0.9456 ± 0.0646
TACNN	0.9602 ± 0.0203	1.1075 ± 0.2862	0.9589 ± 0.0217	4.2747 ± 1.1669	12.3099 ± 4.3565	0.0254 ± 0.0076	0.0228 ± 0.0273	0.0406 ± 0.0471	0.9387 ± 0.0667
MUCNN	0.9700 ± 0.0145	1.0255 ± 0.3683	0.9693 ± 0.0150	4.0755 ± 1.5800	11.9330 ± 6.5289	0.0226 ± 0.0078	0.0206 ± 0.0256	0.0291 ± 0.0411	0.9520 ± 0.0619
MMFN (ours)	0.9783 ± 0.0099	0.8368 ± 0.2575	0.9778 ± 0.0102	3.3083 ± 1.0834	9.6204 ± 4.3155	0.0192 ± 0.0061	0.0203 ± 0.0172	0.0273 ± 0.0314	0.9535 ± 0.0455

reports the average quantitative performance on the QuickBird dataset including 40 groups of images by using different pansharpening methods. As shown in the table, the proposed MMFN achieves the best results for all the metrics, indicating that our approach has outstanding sharpening capabilities for reduced resolution images. The ResTFNet is also a promising method. From both a quantitative and qualitative point of view, we can state that our proposed method can achieve better pansharpening outcomes.

Similarly, in Figure 4 and Figure 5, the pansharpening results for Pléiades and WorldView-2 are depicted. Again, it is easy to observe that the performance of traditional methods is inferior with respect to deep learning-based ones. In Figure 4, the conventional methods, except for Inter23, show various degrees of spectral distortion, mainly in the wooded areas. Moreover, the color of the road is also inconsistent with the reference image. It can be found that the BDSD and GSA methods generate block effects in the vegetation areas. Deep learning-based methods instead produce better results (closer to the ground-truth image). However, spatial distortion still exists, at seen from the MAE maps. Although the performance of PSGAN and the proposed approach are similar at first glance, the residual map of the proposed approach is closer to zero, see the red boxes in the figure. In Figure 5, the SIRF method can efficiently perform spatial details enhancement. However, it suffers from severe spectral distortion, see Figure 5g. Compared with the ground-truth image, the results of AWLP, BDSD, PNN, and TACNN generate visible color distortion. Additionally, the benchmarking methods still have a loss of spatial information on the WorldView-2 dataset in terms of MAEs, see, e.g., the result of the MUCNN method. Thus, from this analysis, it is clear that the best performance is obtained by the proposed MMFN. This statement is further corroborated by the quantitative results in

Table 3. Average quantitative performance on the Pléiades dataset. The best and second best performances are shown in Red and Blue, respectively.

Methods	CC (1)	ERGAS (0)	${\mathit{Q}}_{\mathbf{2}}\mathit{n}$ (1)	RASE (0)	RMSE (0)	SAM (0)	${\mathit{D}}_{\mathit{\lambda }}$ (0)	${\mathit{D}}_{\mathit{S}}$ (0)	QNR (1)
Inter23	0.9247 ± 0.0344	4.0650 ± 1.0726	0.9178 ± 0.0403	16.4545 ± 3.3143	93.9909 ± 23.9316	0.0554 ± 0.0154	0.0007 ± 0.0006	0.0871 ± 0.0366	0.9122 ± 0.0368
AWLP	0.9402 ± 0.0240	4.0773 ± 0.8803	0.9376 ± 0.0244	18.6537 ± 4.3317	106.7633 ± 29.1444	0.0666 ± 0.0289	0.0354 ± 0.0182	0.0863 ± 0.0405	0.8821 ± 0.0538
BDSD	0.9154 ± 0.0340	5.0235 ± 1.6217	0.9069 ± 0.0397	20.0637 ± 4.1547	113.9981 ± 26.7171	0.0777 ± 0.0241	0.0133 ± 0.0072	0.0628 ± 0.0153	0.9247 ± 0.0158
GSA	0.9282 ± 0.0344	4.8284 ± 1.7408	0.9173 ± 0.0444	19.6190 ± 4.6651	111.9132 ± 31.0864	0.0705 ± 0.0239	0.0610 ± 0.0269	0.1637 ± 0.0505	0.7864 ± 0.0671
MMP	0.9363 ± 0.0310	3.7855 ± 1.1154	0.9353 ± 0.0322	15.2021 ± 3.1425	86.6501 ± 21.4939	0.0601 ± 0.0208	0.0077 ± 0.0031	0.0243 ± 0.0298	0.9682 ± 0.0296
MGH	0.9382 ± 0.0238	4.2363 ± 1.3904	0.9338 ± 0.0250	17.2398 ± 3.7794	97.8060 ± 22.3985	0.0531 ± 0.0146	0.0409 ± 0.0205	0.0958 ± 0.0396	0.8680 ± 0.0545
SIRF	0.9499 ± 0.0271	3.1350 ± 0.4913	0.9474 ± 0.0305	12.5454 ± 1.7723	71.7516 ± 14.5913	0.0595 ± 0.0156	0.0195 ± 0.0146	0.0874 ± 0.0442	0.8964 ± 0.0536
PNN	0.9824 ± 0.0082	1.9860 ± 0.5974	0.9821 ±0.0084	8.2626 ± 2.0612	47.1546 ± 13.7918	0.0406 ± 0.0120	0.0286 ± 0.0170	0.0734 ± 0.0330	0.9006 ± 0.0465
PanNet	0.9876 ± 0.0076	1.6766 ± 0.7228	0.9873 ± 0.0081	7.0452 ± 2.5112	39.6260 ± 12.8293	0.0357 ± 0.0112	0.0167 ± 0.0143	0.0289 ± 0.0154	0.9551 ± 0.0283
PSGAN	0.9922 ± 0.0039	1.2959 ± 0.3253	0.9921 ± 0.0040	5.4914 ± 1.1744	31.1803 ± 7.3758	0.0305 ± 0.0086	0.0258 ± 0.0208	0.0736 ± 0.0258	0.9029 ± 0.0424
ResTFNet-$l_1$	0.9923 ± 0.0040	1.2974 ± 0.3251	0.9922 ± 0.0040	5.5191 ± 1.1801	31.4281 ± 7.8812	0.0302 ± 0.0085	0.0194 ± 0.0159	0.0514 ± 0.0153	0.9304 ± 0.0284
TACNN	0.9871 ± 0.0055	1.7147 ± 0.3856	0.9870 ± 0.0056	7.4884 ± 1.4363	42.7377 ± 10.0400	0.0417 ± 0.0110	0.0195 ± 0.0158	0.0587 ± 0.0244	0.9233 ± 0.0377
MUCNN	0.9896 ± 0.0053	1.5371 ± 0.5308	0.9893 ± 0.0056	6.5236 ± 1.9152	37.2740 ± 12.6312	0.0338 ± 0.0100	0.0165 ±0.0157	0.0229 ±0.0169	0.9611 ± 0.0311
MMFN (ours)	0.9930 ± 0.0037	1.2197 ± 0.3277	0.9930 ± 0.0037	5.1791 ± 1.1225	29.3613 ± 6.9001	0.0281 ± 0.0080	0.0173 ± 0.0123	0.0223 ± 0.0096	0.9609 ± 0.0189

and

Table 4. Average quantitative performance on the WorldView-2 dataset. The best and second best performances are shown in Red and Blue, respectively.

Methods	CC (1)	ERGAS (0)	${\mathit{Q}}_{\mathbf{2}}\mathit{n}$ (1)	RASE (0)	RMSE (0)	SAM (0)	${\mathit{D}}_{\mathit{\lambda }}$ (0)	${\mathit{D}}_{\mathit{S}}$ (0)	QNR (1)
Inter23	0.8729 ± 0.0602	4.0245 ± 1.9083	0.8588 ± 0.0711	15.4794 ± 7.7455	53.3841 ± 25.8446	0.0515 ± 0.0231	0.0035 ± 0.0050	0.0545 ± 0.0333	0.9423 ± 0.0370
AWLP	0.9307 ± 0.0428	2.9811 ± 1.8452	0.9283 ± 0.0442	12.8921 ± 7.0297	43.5665 ± 17.8291	0.0484 ± 0.0201	0.0640 ± 0.0653	0.0833 ± 0.0845	0.8629 ± 0.1236
BDSD	0.9248 ± 0.0503	3.2997 ± 2.2302	0.9213 ± 0.0517	12.9515 ± 8.2815	43.3519 ± 20.5032	0.0566 ± 0.0269	0.0347 ± 0.0342	0.0456 ± 0.0426	0.9224 ± 0.0657
GSA	0.9301 ± 0.0542	3.2834 ± 2.6487	0.9239 ± 0.0585	12.8705 ± 9.4822	42.4991 ± 22.8362	0.0559 ± 0.0303	0.0621 ± 0.0602	0.1165 ± 0.1015	0.8335 ± 0.1330
MMP	0.9393 ± 0.0437	2.6916 ± 1.8941	0.9331 ± 0.0486	10.8338 ± 6.9976	36.3995 ± 17.6618	0.0467 ± 0.0235	0.0269 ± 0.0139	0.0386 ± 0.0275	0.9357 ± 0.0353
MGH	0.9359 ± 0.0417	3.0157 ± 2.0017	0.9340 ± 0.0425	11.9062 ± 7.2632	39.9244 ± 17.5511	0.0451 ± 0.0197	0.0689 ± 0.0614	0.0900 ± 0.0818	0.8519 ± 0.1187
SIRF	0.8431 ± 0.1578	4.7626 ± 1.8849	0.8091 ± 0.1624	18.5698 ± 9.0005	62.0451 ± 22.7993	0.1146 ± 0.0784	0.1013 ± 0.0823	0.0873 ± 0.0801	0.8263 ± 0.1377
PNN	0.9659 ± 0.0267	1.8580 ± 0.9584	0.9649 ± 0.0278	7.5961 ± 3.7324	26.0741 ± 11.5782	0.0360 ± 0.0181	0.0340 ± 0.0597	0.0534 ± 0.0649	0.9178 ± 0.1024
PanNet	0.9704 ± 0.0257	1.6760 ± 0.7625	0.9696 ± 0.0267	6.8436 ± 2.9796	23.6430 ± 10.0350	0.0330 ± 0.0151	0.0395 ± 0.0539	0.0547 ± 0.0720	0.9115 ± 0.1105
PSGAN	0.9703 ± 0.0275	1.6735 ± 0.7520	0.9694 ± 0.0286	6.7857 ± 2.9530	23.4621 ± 10.2050	0.0322 ± 0.0146	0.0454 ± 0.0716	0.0533 ± 0.0743	0.9084 ± 0.1205
ResTFNet-$l_1$	0.9722 ± 0.0262	1.5961 ± 0.7276	0.9716 ± 0.0272	6.5399 ± 2.8571	22.6806 ± 10.0196	0.0309 ± 0.0141	0.0376 ± 0.0693	0.0411 ± 0.0667	0.9271 ± 0.1133
TACNN	0.9562 ± 0.0333	2.2537 ± 0.9703	0.9537 ± 0.0379	9.1814 ± 3.4089	30.9212 ± 9.9047	0.0440 ± 0.0172	0.0344 ± 0.0442	0.0379 ± 0.0464	0.9309 ± 0.0800
MUCNN	0.9694 ±0.0256	1.7244 ± 0.8290	0.9685 ± 0.0268	7.0707 ± 3.2251	24.4422 ±10.7436	0.0332 ± 0.0156	0.0305 ± 0.0548	0.0354 ± 0.0576	0.9381 ± 0.0963
MMFN (ours)	0.9753 ± 0.0226	1.5102 ± 0.6750	0.9746 ± 0.0237	6.1888 ± 2.6461	21.4883 ± 9.4222	0.0291 ± 0.0132	0.0332 ± 0.0716	0.0348 ± 0.0675	0.9379 ± 0.1137

. Indeed, the proposed MMFN method achieves the best results on both the Pléiades and WorldView-2 datasets and for all the metrics. In conclusion, the generalization ability of the proposed method for reduced-resolution images is better than that of the state-of-the-art methods.

4.3.2. Full Resolution Assessment

We further assess the proposed method at full resolution using datasets at the original scale without any spatial degradation to generate the ground-truth (reference) image. Figure 6, Figure 7 and Figure 8 show the qualitative comparison. The $D_{\lambda }$, $D_s$, and QNR metrics were used to evaluate the pansharpened performance. Inter23 obtains the best $D_{\lambda }$ value, which indicates the absence of spectral distortion. Therefore, we used the results of the Inter23-based method as the basis images from which to observe the detailed information of PAN images injected into the output images. For visual comparison, we still adopted a residual representation calculating the difference between the pansharpened result and Inter23 to display the injected details. It is observed that the DL-based methods inject more structures and textures than the compared traditional methods, see, e.g., the residual images for the QuickBird dataset in Figure 6. BDSD transfers the wrong details into the pansharpened product, leading to noise effects on green areas, see Figure 6c. From a visual point of view, lawn areas should be smooth without containing any detail. Significant spectral distortion still exists in the result of the SIRF-based method, see Figure 6g. Among the traditional methods, the AWLP, GSA, MMP, and MGH-based approaches can hardly transfer precise edges into the pansharpened results, as shown in the related MAEs in Figure 6. It is clear from Figure 6j that PSGAN produces spectral distortion in the buildings with an orange-like color (the right side of the image). A similar phenomenon is that partial edges around green areas can be retained in the MAE results of PNN, PanNet, and TACNN-based methods. Although the ResTFNet, MUCNN, and the proposed method have highly similar results, the blurring effects and the noise cannot be removed in green vegetation areas. Objective evaluations are reported in the last three columns of Table 2. We find that the proposed method obtains a high performance in terms of $D_S$, and almost the best QNR metrics. Overall, the full-resolution experiments on the QuickBird dataset are satisfying.

A similar visual analysis can be performed in Figure 7 and Figure 8 for the other two datasets (i.e., Pléiades and WorldView-2). It is easy to see that the proposed MMFN achieves the best trade-off between spatial and spectral consistency. Furthermore, objective results mark the high performance of the proposed approach; see the last three columns of Table 3 and Table 4, again. Indeed, the proposed MMFN has a promising fusion performance, balancing the trade-off between spatial enhancement and spectral fidelity, thanks to the use of the multi-scale information and the multi-stream fusion strategy.

4.3.3. Parameter Numbers vs. Model Performance

We added a comparative experiment evaluating the running times of deep learning methods. We conducted experiments on a hardware device with an Intel(R) Xeon(R) Gold 5117 CPU @ 2.00 GHz and 128 GB memory. We used Python 3.9 and PyTorch 1.11 as the programming language and deep learning framework, respectively. We tested running times for both full and reduced resolution MS images with a batch size of 4. We measured the average execution times for processing a single input on the CPU. We used the Python built-in timer to measure the execution times, repeating each measurement 10 times to ensure statistical significance. The results are shown in

Table 5. Parameter numbers and running times for the compared deep learning methods on datasets at different scales.

Methods	Scale (Time)		# Parameters
	RR (s)	FR (s)
PNN	0.14	1.67	80,420
PanNet	0.26	3.84	21,384
PsGan	1.11	13.77	2,367,716
ResTFNet-$l_1$	0.85	12.00	2,362,869
TACNN	0.15	1.72	80,420
MUCNN	0.51	7.21	1,362,832
MMFN (ours)	0.62	8.97	328,142

. Although our method is not the most time-efficient one (because it processes more features generated at different scales), it exploits fewer parameters than PSGAN and ResTFNet-$l_1$, thus having a shorter running time. Furthermore, while our method is not as time-efficient as PNN, PanNet, MUCNN, and TACNN, it achieves a better pansharpening performance.

4.4. Ablation Studies

In this section, we provide three groups of ablation studies to demonstrate each claim of the proposed framework.

4.4.1. Ablation Study about Different Scales

To investigate the influence of the number of scales on the pansharpening performance, we compared the results of six different scales by using a fixed multi-stream fusion network. From Figure 9, it can be observed that increasing the number of scales cannot continuously improve the pansharpening performance in terms of objective evaluation. Most evaluation metric values indicate that the performance of the multi-scale (scale number = 3) results is better than that of the other scales. Additionally, as the number of scales increases, the processing time increases. Therefore, the scale number equal to three represents a good choice for our approach.

As shown in Figure 10, by observing the residual maps between the results of the different scales and the reference image, the spectral distortion and spatial distortion can be significantly reduced, increasing the number of scales. It is also straightforward to see that three scales represent the best result among the three presented in the figure.

4.4.2. Ablation Study about Different Streams

To verify the effectiveness of the choice of a number of streams equal to three, the single-stream (stream number = 1) fusion and the dual-stream (stream number = 2) fusion were investigated, fixing the multi-scale network.

The quantitative comparison is presented in Figure 11. The results of the multi-stream block using three streams are better than those of single-stream and dual-stream fusion blocks. This is because the sole use of an architecture with single-stream or dual-stream fusion cannot guarantee the full merge of the spatial details of the PAN image together with the spectral information of the MS image. In addition, Figure 12 shows the qualitative results on the three real remote sensing datasets, varying the number of streams. As the number of streams increases, the spectral and spatial distortions decrease. Thus, the above-mentioned ablation studies confirm the validity of the choices made (the number of scales and streams both equal to three).

4.4.3. Ablation Study about Different numbers of Residual Blocks

To assess the effect of the number of residual blocks in the MSB module on the results, we experimentally tested the impact of using one, two, and three residual blocks. The experimental results shown in Figure 13 indicate that increasing the number of residual blocks leads to a slight improvement in performance on WorldView-2 and QuickBird datasets. However, on the Pléiades dataset, there was a decrease in performance as the number of residual blocks increases. This is because the simplest model structure is enough for the Pléiades dataset. Considering that increasing the number of residual blocks also increases the complexity and computational cost of the network, leading to overfitting and longer training times, the simplest structure was chosen.

5. Conclusions

This paper proposed a multi-scale and multi-stream fusion network (MMFN), which is simple but effective for pansharpening. Different levels of spectral/spatial information contained in the MS and PAN images was obtained first by using a multi-scale strategy. Afterwards, a multi-stream fusion block was adopted to fully fuse the MS and PAN images, preserving their spatial and spectral characteristics. Additionally, to constrain the training process, we developed a multi-stage reconstruction approach, setting the same loss function for each fusion block. The proposed method was assessed on three real remote sensing datasets acquired by three different sensors. The reduced and full resolution assessments demonstrated the validity of the proposed approach.