Pan-Sharpening Network of Multi-Spectral Remote Sensing Images Using Two-Stream Attention Feature Extractor and Multi-Detail Injection (TAMINet)

Abstract

Achieving a balance between spectral resolution and spatial resolution in multi-spectral remote sensing images is challenging due to physical constraints. Consequently, pan-sharpening technology was developed to address this challenge. While significant progress was recently achieved in deep-learning-based pan-sharpening techniques, most existing deep learning approaches face two primary limitations: (1) convolutional neural networks (CNNs) struggle with long-range dependency issues, and (2) significant detail loss during deep network training. Moreover, despite these methods’ pan-sharpening capabilities, their generalization to full-sized raw images remains problematic due to scaling disparities, rendering them less practical. To tackle these issues, we introduce in this study a multi-spectral remote sensing image fusion network, termed TAMINet, which leverages a two-stream coordinate attention mechanism and multi-detail injection. Initially, a two-stream feature extractor augmented with the coordinate attention (CA) block is employed to derive modal-specific features from low-resolution multi-spectral (LRMS) images and panchromatic (PAN) images. This is followed by feature-domain fusion and pan-sharpening image reconstruction. Crucially, a multi-detail injection approach is incorporated during fusion and reconstruction, ensuring the reintroduction of details lost earlier in the process, which minimizes high-frequency detail loss. Finally, a novel hybrid loss function is proposed that incorporates spatial loss, spectral loss, and an additional loss component to enhance performance. The proposed methodology’s effectiveness was validated through experiments on WorldView-2 satellite images, IKONOS, and QuickBird, benchmarked against current state-of-the-art techniques. Experimental findings reveal that TAMINet significantly elevates the pan-sharpening performance for large-scale images, underscoring its potential to enhance multi-spectral remote sensing image quality.

1. Introduction

Currently, remote sensing images are widely used to monitor agriculture, environmental protection, industry, military protection, and other fields [1,2,3]. The accuracy of remote sensing technology applications is closely related to the quality of remote sensing images, so its improvement is the major focus of research in the field of remote sensing image processing. Due to physical constraints, it is difficult to achieve a balance between spectral resolution and spatial resolution of multispectral remote sensing images. To improve the spatial features of remote sensing image details and spectral features, many pan-sharpening methods are proposed. Their goal is to achieve panchromatic imaging with high spatial resolution but low spatial resolution multi-spectral (LRMS) images and single-band panchromatic (PAN) images are merged to obtain high-resolution multi-spectral images (HRMS) [4].

Many research efforts have been devoted to developing pan-sharpening algorithms during the last few decades. The most widely used method is the component substitution (CS) method. The CS method is the most classic and basic of the pan-sharpening methods. It mainly includes the following types: the traditional intensity–hue–saturation (IHS) algorithm [5], principal component analysis (PCA) [6], the Gram–Schmidt algorithm (GS) [7], the adaptive Gram–Schmidt algorithm (GSA) [8], and the method based on band-dependent spatial details (BDSD) [9]. While the CS method is straightforward to implement and capable of preserving the spatial details of the PAN image without sacrificing its spectrum, it does introduce distortions to both the extracted spatial information and the merged PAN image. Moreover, discrepancies arise among the components derived from the CS method [10], ultimately culminating in image distortions over time.

The MRA-based method uses spatial filters to inject spatial features into the LRMS. Representative algorithms include the generalized Laplacian pyramid (GLP) [11,12], smoothing filter-based intensity modulation (SFIM) [13], additive wavelet luminance proportional (AWLP) [14], “A-trous” wavelet transform (ATWT) [15], robust regression to GLP [16] and GLP with full-scale regression (GLP-Reg) [17], and other strategies decompose the LRMS and PAN images into multi-scale spaces and then inject the decomposed PAN images into the corresponding LRMS images for information fusion. The MRA-based method can better maintain spectral fidelity in pan-sharpening results, but high-frequency information is easily lost and cannot guarantee spatial accuracy.

When compared to CS and MRA techniques, VO-based methods exhibit a competitive edge in addressing pan-sharpening challenges. This method uses previous regularization constraints based on sparse representation theory [18] or Bayesian theory [19] to build a variational fusion energy function and uses iterative optimization algorithms such as gradient descent to minimize the energy function. The VO method is mathematically elegant but requires a considerable amount of computation, causing high costs. Therefore, most benchmarking and practical applications still use MRA and CS methods.

Considering the shortcomings of the three traditional methods above, and inspired by the super-resolution convolutional neural network (SRCNN) [20], Masi et al. [21] proposed pan sharpening using the convolutional neural network (PNN), according to the specific prior information in remote sensing images. It was the first study to apply the neural network to the pan-sharpening task. In subsequent studies, inspired by the pioneering work of PNN networks, deep-learning-based super-resolution methods have made significant progress, and researchers have proposed many advanced methods, among which residual learning, dense connections, and generative adversarial networks are commonly used. Inspired by residual learning, Yang et al. [22] proposed a deeper network than the PNN, the PanNet network. It adopts the skip connection idea from ResNet [23] to design a deeper network structure and further proposes using high-frequency information to compensate for clear spatial details. Similarly, Wei et al. [24] introduced the concept of residual learning into deep residual generalized sharpening neural networks (DRPNN) to form a very deep convolutional neural network, which can further improve the performance of generalized sharpening.

In addition to using the form of deepening the network to fully extract information, the multi-scale architecture can also be used to achieve the goal of sharpness. Yuan et al. [25] proposed a multi-scale and multi-depth convolutional neural network (MSDCNN) to explore convolutional neural network filters of different sizes. Jin et al. [26] proposed a Laplacian pyramid panchromatic network architecture, which uses the Laplacian pyramid method to split the image into multiple scales and develops a fusion convolutional neural network (FCNN) for each scale to combine them to form the final multi-scale network architecture. Cai and Huang [27] proposed a super-resolution-guided progressive pan-sharpening neural network (SRPPNN) to combine multi-scale features and obtain better pan-sharpening performance.

Furthermore, the pan-sharpening task can be considered as an imaging task, and the method based on the generative adversarial network (GAN) [28] provides striking advantages. This method uses a discriminator to distinguish the generated image from the GroundTruth image to improve image fusion quality. Due to powerful deep learning technology and a large amount of remote sensing data, pan-sharpening technology has developed rapidly. For example, PSGAN [29] uses generators and condition discriminators to reconstruct multi-band images with high spatial resolution. Ma et al. [30] proposed a new unsupervised framework for pan-sharpening based on generative adversarial networks, called Pan-GAN, through generative adversarial networks. It does not rely on so-called basic facts during network training.

Although the method based on deep learning has received wide attention and achieved reliable application results, deep-learning-based approaches have been increasingly transformed in the direction of deep networks. Therefore, this raises two questions. Deep networks focus on local features but ignore the connections between the overall and local features. Deep networks have difficulty finding which feature is the target area to focus on. Based on the problem listed above, we might consider the following ways to improve:

• Multi-spectral images are 3D data cubes, and it is difficult for ordinary CNN to extract high-fidelity detailed information. The attention mechanism can capture information from orientation and position perception, which can help the model locate and identify the target of interest more accurately.
• The traditional pan-sharpening method has the advantage of high fidelity to spatial-spectral feature information. The DL-based approach relies on large-scale dataset training to extract spectral information from LRMS images and spatial details from the PAN images. After the training phase, pan-sharpening images can be easily predicted or calculated by learning nonlinear mapping. Thus, it is an innovative idea to combine the traditional method with the DL method.

Building on the above analysis, this article proposes a new DL-based method called the multi-spectral remote sensing image pan-sharpening network (TAMINet) with two-stream attention and multi-detail injection. The main characteristics and contributions of this study are as follows:

• This study integrates the coordinate attention block in the feature extraction module, which, in turn, can effectively extract mode-specific features by encoding channel relationships and remote dependencies through accurate position information using a two-stream feature extractor to obtain mode-specific features from PAN and LRMS images.
• Our approach pays special attention to CS and MRA frameworks, and inspired by this traditional method, using a high-pass filter for detail extraction, the spectral direction features contained in LRMS images are merged (injected) with the high-resolution spatial detail pass information from PAN images several times to solve the problem of losing details in the fusing process.
• We present a combination of three simple optimization terms to constrain the spectral fidelity and spatial accuracy of pan-sharpening results. The first two optimization terms constrain the difference between predicted HRMS and LRMS, as well as PAN images to generate a similar structural distribution. Another optimization constraint provides spatial and spectral consistency between HRMS and GroundTruth images.

2. Related Work

2.1. Pan-Sharpening

Over the past few decades, various pan-sharpening algorithms have been proposed and studied. Although many scholars have developed new ways to improve the traditional method, the effect is still not ideal. Considering the different features contained in PAN and LRMS images, Liu et al. [31] proposed to use two-stream networking (TFNet) for feature extraction and perform information fusion in the feature domain. Inspired by this idea, this paper uses the two-stream network (TFNet) and adds a coordinate attention mechanism to the feature extractor to improve the ability to extract mode-specific features from PAN and LRMS images.

At the same time as the development of the multispectral remote sensing image fusion algorithm based on deep learning, the method of integrating the ideas of traditional methods has become a focus of research in recent years. Wu et al. [32] propose a general fusion framework that can perform weighted pan-sharpness tasks by combining VO with deep learning (DL), where these key weights, which directly determine the relative contribution of DL to each pixel, are estimated adaptively. Liu et al. [33] proposed a method of injecting high-pass detail from PAN images into upsampled MS images, that is, using classical injection to improve the details of merged images. This method resembles the scheme of traditional CS and MRA methods, but high-pass detail extraction is inconsistent with the classical process of CS and MRA methods. In addition, as high-pass filtering is used in the reconstruction process, the proposed architecture also shows the generalization capability of the corresponding network. He et al. [34] proposed a convolutional neural network (DiCNN) based on detail injection. The study develops an architecture based on detail injection, DiCNN1, which relies on MS and PAN images for detail injection. Benzenati et al. [35] proposed a gain injection prediction method based on CNN (GIP-CNN), which performs the injection gain by estimating the GIP feature information on a conventional grid and regularizes the grid by injecting MS image details. Compared to CS/MRA-based techniques, the GIP-CNN model shows better generalization results and provides more competitive fusion performance compared to PNN and DRPNN.

Therefore, inspired by previous innovative studies [31,32,33,34,35], we use two-stream neural networks to adjust the extraction of spatial and spectral details through the estimation of nonlinear and local injection models.

2.2. Coordinate Attention

Currently, the attention mechanism has been widely used in deep neural networks to improve model performance. However, in lightweight networks where model capacity is strictly limited, applying attention is very slow, mainly because the computational overhead of most attention mechanisms is inaccessible to lightweight networks. Considering the limited computational power of lightweight networks, Hu et al. [36] proposed squeeze-and-excitation (SE) attention, which is still the most popular attention mechanism at present. Unfortunately, SE attention only considers encoding information between channels and ignores the importance of location information, which is crucial for many visual tasks that require capturing object structure. Therefore, the convolutional block attention module (CBAM) [37] later added the spatial attention module to obtain position information through convolution. However, convolution can only capture local position relationships and cannot model long-range dependency. As the two networks outlined above still have problems, Qibin Hou et al. [38] proposed coordinate attention to solve the above problems as it provides the following advantages. First, it can capture not only channel information but also direction and position awareness information, which can help the model locate and identify the target of interest more accurately. Secondly, coordinate attention is flexible and lightweight, plug and play. Finally, coordinate attention can be a significant gain for downstream tasks based on lightweight networks.

Due to the limitation of convolution operators, it is often impossible to obtain the long-range space features accurately, thus limiting the overall performance. Therefore, coordinate attention is added to the feature extraction network to capture the remote spatial information between the channels, which enables the model to obtain the spectral information from the LRMS image and the spatial information of the PAN image.

3. Methods

3.1. Overall Network Architecture

The main task of pan-sharpening technology is to merge the LRMS image and PAN image, to obtain the same spatial resolution and PAN image and spectral fidelity and LRMS image from the same sharpness HRMS image. In this work, $M\in R^{w\times h\times c}$ is used to represent LRMS images, where $w$ and $h$ represent the width and height of low-resolution images, respectively. $P\in R^{W\times H}$ is used to represent the corresponding PAN images and $\widehat{M}\in R^{W\times H\times c}$ represent the HRMS images after pan-sharpening, where $W$ and $H$ respectively represent the width and height of the high-resolution image and $c$ represent the number of spectral bands of the multi-spectral image, usually $c=4$. The hyperspatial resolution scale factor can be defined as $r=W/w=H/h$, and the scale factor $r=4$ is usually set.

In addition, inspired by DR-NET [39,40], we designed a feature extraction network, a feature fusion network and an image reconstruction network. First, the LRMS images were upsampled using interpolation methods such as bicubic interpolation to obtain preliminary LRMS images $M\in R^{w\times h\times c}\stackrel{\uparrow upsampling}{\to }{M}^{\prime }\in R^{W\times H\times c}$ with the same resolution as the PAN image. Second, this study adds the coordinate attention (CA) block in the feature extraction network (FEN), which can capture cross-channel data and information from direction and position perception. The efficiency of acquiring spectral features from the LRMS image and spatial feature details from the PAN image is improved. The process is mainly expressed as input from $M^{\prime }$ and $P$ into the FEN to obtain spectral and spatial features, respectively. Third, the feature fusion network (FN) is used to perform a feature-level fusion of the PAN image and the LRMS image. Finally, the reconstruction network (REC) is inserted to reconstruct the $\widehat{M}$ HRMS image after pan-sharpening. Furthermore, in the pan-sharpening reconstruction network, the spatial details of the PAN image after high-pass processing are injected many times, and the spectral feature information lost in the fusing process is injected with gain. The general form of the overall fusion process is shown in Formula (1):

$$\widehat{M}=f(M^{\prime },P,\theta )$$

(1)

where $f(\cdot )$ describes the two-stream encoder–decoder fusion model, that is, $M^{\prime }$ and $P$ are taken as inputs to generate the necessary HRMS image $\widehat{M}$, with $\theta$ as the set of model parameters.

Among them, high-pass represents the high-pass filter, FEN represents the feature extraction network, FN represents the feature fusion network and REC represents the image reconstruction network. In this study, high-pass filtering is used to extract high-frequency details and inject high-frequency details into the FN module and REC module several times to achieve the goal of preserving many spatial feature details.

Specifically, FEN is used to extract features from the upsampled LRMS and PAN images, contributing to subsequent fusion steps. Therefore, with $M^{\prime }$ or $P$ as input, corresponding characteristics $F_m$ or $F_p$ can be obtained, according to Formulas (2) and (3):

$$F_m=f_{FEN}(M^{\prime })$$

(2)

$$F_p=f_{FEN}(P)$$

(3)

where $f_{FEN}$ represents the operation of the FEN. It should be mentioned that $f_{FEN}(M^{\prime })$ and $f_{FEN}(P)$ have the same structure but different parameters and extract different features from the LRMS and PAN images, respectively. After that, those obtained $F_m$ and $F_p$ were, respectively, fed into the feature fusion network. Moreover, the first PAN image detail injection gain was performed in the FN module, according to Formula (4):

$$F_{FN}=f_{FN}(F_m,F_p,F_{HP})$$

(4)

where $f_{FN}$ represents the feature fusion network operation, $F_{FN}$ is the result obtained from the network structure, and $F_{HP}$ is the high-frequency details extracted from the PAN image after the high-pass. Finally, the merged data are incorporated into the REC. Spectral and spatial feature details from the LRMS and PAN images are injected into the image reconstruction for enhancement. The network is therefore formulated as shown in Formula (5):

$$\widehat{M}=f_{REC}(F_{FN},F_{HP},M^{\prime })$$

(5)

$f_{REC}$ represents the reconstruction network and $\widehat{M}$ is the high-resolution multi-spectral image generated after reconstruction. Detailed architectures of the TAMINet are shown in Figure 1.

In Figure 1, “High-Pass” represents a high-pass filter, FEN represents a feature extraction network, FN represents a feature fusion network, REC represents an image reconstruction network. The Basic Unit is a convolution unit composed of two $3\times 3$ convolution kernels and two PRelu activation layers. FEN consists of FEN-1 and FEN-2, where FEN-2 consists of CA block, $2\times 2$ convolution kernel and PRelu. FN comprises two modules, FN-1 and FN-2. FN-1 consists of a Basic Unit, and FN-2 consists of a $2\times 2$ convolution layer and a PRelu activation layer. REC comprises four components: REC-1, REC-2, REC-3, and REC-4. All are composed of the Basic Unit, but there are transposed convolution layers.

3.2. Loss Function

Spatial loss ($L_{spatial}$) is designed to search for gradient mapping between HRMS and PAN images. The loss function is defined as follows:

$$L_{spatial}=\left|D\nabla \widehat{{\rm M}}-\nabla P\right|$$

(6)

where $\widehat{M}$ represents the HRMS image, $P$ is the PAN image, $\left|\cdot \right|$ is the norm of ${\ell }_1$, and $D$ represents the diagonal matrix used to weight each channel so that the size of $\widehat{M}$ is scaled to the size of $P$. Note that $D$ can be learned by other parameters in the frame. $\nabla$ represents the image gradient operator. $L_{spatial}$ represents spatial similarity, that is, whether finer details of spatial features can be obtained.

Spectral loss ($L_{spectral}$) is designed to search in this case for gradient mapping between HRMS and LRMS images. The specific loss function of this part is defined as follows:

$$L_{spectral}={‖\widehat{M}S-M‖}_F^2$$

(7)

where $M$ is the LRMS image, ${‖\cdot ‖}_F^2$ is the Frobenius norm and $S$ is the fuzzy spatial downsampling operator. Specifically, the main objective of $L_{spectral}$ is to promote spectral similarity and spatial fidelity.

The norm loss ($L_1$ norm) is the average of the absolute errors of HRMS and GroundTruth. The loss function formula for this part is defined as (8):

$$L_1=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left|\widehat{M}-GT\right|}$$

(8)

where $GT$ represents the GroundTruth image and $N$ represents the number of image pairs in the training set. $\left|\cdot \right|$ represents the norm of ${\ell }_1$. The main purpose of $L_1$ is to maintain the similarity between the HRMS image and the GroundTruth image in potential features.

The total loss function ($L$) uses spectral loss ($L_{spectral}$) and spatial loss ($L_{spatial}$) to simultaneously retrieve spatial details and retain input spectral information. In addition, a $L_1$ loss is used to further refine the spectral quality. Finally, the proposed pan-sharpening model is applied to minimize the following loss function (9):

$$L=\alpha L_{spetral}+\beta L_{spatial}+\mu L_1$$

(9)

where $\alpha$, $\beta$ and $\mu$ are the weights defined according to experience in the experiment. The setting of weights is based on references [30,41]. The purpose of the loss function used in this paper is to control the spatial accuracy through $L_{spatial}$ and spectral accuracy through $L_{spectral}$, $L_1$ norm controls of the similarity of potential features between the HRMS image and the GroundTruth image to improve the model optimization effect and speed.

4. Results

To verify the effectiveness of the proposed method, simulation experiments and analysis of actual data are conducted and described in this section.

4.1. Experiment Settings

4.1.1. Datasets

We use training datasets from three different satellites to conduct simulation experiments.

The first is the IKONOS dataset [42], which utilizes multi-spectral (LRMS) and panchromatic (PAN) data from the IKONOS satellite in the United States. The IKONOS satellite, also known as IKONOS, is the first generation of high-resolution commercial earth observation satellites developed by Eartheye to provide high-resolution satellite remote sensing imagery to military and civilian users. Panchromatic band imaging band range: 0.45 μm~0.9 μm. Multi-spectral images of each wavelength range: 0.45 μm~0.53 μm (blue), 0.52 μm~0.61 μm (green), 0.64 μm~0.72 μm (red), 0.76 μm~0.86 μm (near infrared). Spatial resolution: 1 m (panchromatic), and 4 m (multi-spectral). Land cover types in this dataset include cities, vegetation, rivers, and lakes.

The second is the QuickBird dataset [27], which uses LRMS and PAN images from the QuickBird satellite. Quickbird uses the Global Aerial Imaging System 2000 (BGIS2000), which has the fourth highest Earth image resolution in the world at 0.61–0.72 m in full color and 2.44–2.88 m in multispectral. Product type: panchromatic, multi-spectral, panchromatic enhancement, panchromatic + multi-spectral bundle, and so forth. Panchromatic band imaging range: 0.405 μm~1.053 μm. Multi-spectral imaging of each wavelength range: 0.45 μm~0.520 μm (blue), 0.52 μm~0.60 μm (green), 0.63 μm~0.69 μm (red), 0.76 μm~0.90 μm (near infrared). The land cover types in this dataset include forests, farmland, buildings, and rivers.

The third is the WorldView-2 dataset [29], which uses LRMS and PAN images from the WorldView-2 satellite, providing unique 8-band high-resolution commercial satellite imagery. The spatial resolutions of LRMS and PAN are 1.84 m and 0.46 m, respectively. In this study, the panchromatic band and four commonly used bands were selected: 0.45 μm~0.74 μm (panchromatic). The wavelength range: 0.45 μm~0.51 μm (blue), 0.51 μm~0.58 μm (green), 0.63 μm~0.69 μm (red), and 0.77 μm~0.895 μm (near infrared).

Following the Wald protocol, PAN and LRMS in the training set and test set are downsampled with operator 4 to obtain new PAN and LRMS. In the simulation experiment, the original LRMS can be considered as GroundTruth, that is, the target LRMS image approximated by model training is proposed. The four pre-processing steps of the proposed network framework dataset are as follows: (1) The LRMS image is cut into an image block. (2) The PAN image is cropped into a big and small $1024\times 1024$ image block. (3) the training set and test set for network training are obtained by random division according to the proportion of 80% and 20%, respectively. (4) The training set is divided into training and verification sets according to the ratio of 70% and 30%, respectively. The resulting IKONOS dataset contains 200 data pairs, the QuickBird dataset contains 721 pairs, and the WorldView-2 dataset contains 1174 pairs. Details of the specific dataset are summarized in

Table 1. Details of datasets.

Satellite	Type of Image	Spatial Accuracy	Number of Spectral Bands	Size	Total Numbers	Training Numbers	Verification Numbers	Testing Numbers
IKONOS	PAN	1 m	1 band	1024 × 1024	200	112	28	60
	LRMS	4 m	4 bands	256 × 256 × 4
QuickBird	PAN	0.7 m	1 band	1024 × 1024	721	403	403	217
	LRMS	2.8 m	4 bands	256 × 256 × 4
WorldView-2	PAN	0.46 m	1 band	1024 × 1024	1173	657	657	352
	LRMS	1.84 m	4 bands	256 × 256 × 4

.

4.1.2. Comparison Method and Evaluation Index

To verify the advantages of the proposed method, we compare it with nine more advanced pan-sharpening methods proposed in recent years. The first four methods (GS, IHS, Brovey [43,44], and PRACS [45]) are the traditional pan-sharpening methods. The fifth to tenth methods (PNN, PanNet, TFNet, MSDCNN, SRPPNN, $\lambda$-PNN [46]) are deep learning methods.

Eight indicators are used to quantitatively evaluate the performance of the remote sensing image pan-sharpening network proposed in this paper and the corresponding comparison methods. The eight indicators are spectral angle mapper ($SAM$) [47], relative dimensionless global error in synthesis, $ERGAS$ [48], quality without reference ($QNR$) and its related indices are compared with $D_s$ and $D_{\lambda }$ [49], universal image quality index ($UIQI$) [50], the four-band expansion of the $Q$ index ($Q_4$ index, $Q_4$) [49], and the spatial correlation coefficient ($sCC$) [51].

$SAM$ is used to evaluate the spectral difference between the reference image and the pan-sharpened image. It is defined as the angle between the spectral vectors of the enhanced image and the reference image at the same pixel. The smaller the $SAM$ value, the more similar the spectral distribution of the enhanced image to that of the reference image. The calculation formula is (10):

$$SAM(x,y)=arccos(\frac{x\cdot y}{‖x‖\cdot ‖y‖})$$

(10)

where $x$ and $y$ are the pan-sharpened enhanced image and GroundTruth, respectively.

The relative dimensionless global error ($ERGAS$) is a commonly used global quality index. $ERGAS$, like $SAM$, mainly reflects spectral distortion in enhanced images. The calculation formulas are (11) and (12):

$$RMSE(x,y)=\sqrt{\frac{1}{m}{\displaystyle \sum _{i=1}^m{(x_i-y_i)}^2}}$$

(11)

$$ERGAS(x,y)=100\frac{h}{l}\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{(\frac{RMSE(x_i,y_i)}{MEAN(y_i)})}^2}}$$

(12)

where $x$ and $y$ are the pan-sharpened and enhanced images and GroundTruth, respectively; $x_i,y_i$, respectively, represent the values of the pan-sharpened enhanced image and the GroundTruth image in the first band; $m$ is the number of pixels in an image; $RMSE(x,y)$ is the root-mean-square error between the $i$-th band of the merged image and the reference image; $h$ and $l$ are the spatial resolution of the PAN image and the MS image, respectively; $MEAN(y_i)$ is the average of the $i$-th band of an LRMS image with a total of $N$ bands.

The Universal Image Quality Index ($UIQI$) is an index for estimating the global spectral quality of a pan-sharpened image. It is defined as (13):

$$UIQI(x,y)=\frac{4{\sigma }_{xy}\cdot {\mu }_x\cdot {\mu }_y}{({\sigma }_x{}^2+{\sigma }_y{}^2)({\mu }_x{}^2+{\mu }_y{}^2)}$$

(13)

For $UIQI$, $x$ and $y$ represent the pan-sharpened and enhanced image and GroundTruth, respectively; ${\mu }_x$ and ${\mu }_y$ are the average values of $x$ and $y$, respectively; ${\sigma }_x$ and ${\sigma }_y$ are the variances of $x$ and $y$, respectively; ${\sigma }_{xy}$ represents the covariance between $x$ and $y$.

Regarding the Universal Image Quality Index ($Q_4$), the $Q$ index explains the correlation, average deviation, and contrast change of the resulting image relative to GroundTruth. $Q_4$ is an enhanced iteration of $Q$ designed for multi-spectral imaging with four spectral bands. The calculation formula is shown in (14):

$$Q_4=\frac{4\left|{\sigma }_{z_1{z}_2}\right|\cdot \left|{\mu }_{z_1}\right|\cdot \left|{\mu }_{z_2}\right|}{({\sigma }_{z_1}{}^2+{\sigma }_{z_2}{}^2)({\mu }_{z_1}{}^2+{\mu }_{z_2}{}^2)}$$

(14)

where $Z_1$ and $Z_2$ are two quaternions formed by the spectral vector of the MS image. The quaternions are made up of one real number and three imaginary numbers, $i$, $j$, $k$, and are generally expressed as $Z=a+ib+jc+kd$, where $a$, $b$, $c$ and $d$ are real numbers. ${\mu }_{z_1}$ and ${\mu }_{z_2}$ are the average values of $Z_1$ and $Z_2$; ${\sigma }_{z_1{z}_2}$ represents the covariance between $Z_1$ and $Z_2$, ${\sigma }_{z_1}{}^2$ and ${\sigma }_{z_2}{}^2$ are the variances of $Z_1$ and $Z_2$.

There is no reference index ($QNR$), which mainly reflects the fusion performance without GroundTruth, including the spectral distortion evaluation index ($D_{\lambda }$) and spatial distortion evaluation index ($D_s$). The closer the $D_{\lambda }$ index is to 0, the better the degree of spectral fusion. The closer the $D_s$ index is to 0, the better the structure. The closer the $QNR$ index is to 1, the better the pan-sharpening image performs. The calculation formula of $D_{\lambda }$ index is shown in (15) and the formula of $D_s$ index in (16). The calculation formula of $QNR$ index in (17).

$$D_{\lambda }(x,M)=\sqrt[p]{\frac{1}{C(C-1)}{\displaystyle \sum _{c=1}^C{\displaystyle \sum _{r=1}^C{\left|UIQI(x_c,y_r)-UIQI(M_c,M_r)\right|}^p}}}$$

(15)

$$D_s(x,P)=\sqrt[q]{\frac{1}{C}{\displaystyle \sum _{c=1}^C{\left|UIQI(x_c,P)-UIQI(M_c,P)\right|}^q}}$$

(16)

$$QNR(x,M,P)={(1-D_{\lambda }(x,M))}^i\cdot {(1-D_s(x,P))}^j$$

(17)

where $p$ and $q$ represent positive integer exponents; $P$ and $M$ are PAN and MS images, respectively; $i$ and $j$ are weighted parameters to quantify spectral and spatial distortions, respectively. $C$ is the number of strips in the MS image. In this test, $p$, $q$, $i$, $j$ are set to 1.

$$F=\left. [\begin{array}{ccc}-1& -1& -1\\ -1& 8& -1\\ -1& -1& -1\end{array}\right]$$

(18)

$$sCC=\frac{{\displaystyle \sum _{i=1}^w{\displaystyle \sum _{j=1}^h(F{(x)}_{i,j}-\mu F(x))(F{(y)}_{i,j}-\mu F(y))}}}{\sqrt{{\displaystyle \sum _{i=1}^w{\displaystyle \sum _{j=1}^h{(F{(x)}_{i,j}-\mu F(x))}^2{(F{(y)}_{i,j}-\mu F(y))}^2}}}}$$

(19)

where $c$ computes the pan-sharpened image, $y$ represents the corresponding reference image; $F$ is a high-frequency core used to process images; $w$ and $h$ are the width and height of the image, respectively. $\mu F(x)$ and $\mu F(y)$ are the mean values of $F(x)$ and $F(y)$, respectively.

4.1.3. Optimize the Environment and Details

This paper uses three sets of images collected from IKONOS, QuickBird and WorldView-2 to train and test the proposed method. To make a fair comparison, all traditional methods are tested in MATLAB R2017a. All DL-based comparison methods were simulated using Python 3.7 and PyTorch 1.12.1 in the Windows 10 environment. We run the program using an NVIDIA GeForce GTX 1650 graphics card. The Adam optimizer is used to adjust the entire network. The learning rate is set to 0.0001 with exponential decay and the batch size is set to 16. The MATLAB Toolbox in the MATLAB R2017 is used as a framework to obtain evaluation indicators in the experiment.

During the training process, various data enhancement techniques such as random horizontal flip, random vertical flip, 90-degree random rotation, and random cropping are used. In the process of random cropping, each training image is subsampled using bicubic interpolation with operator 4, and it is cropped into an LRMS image, PAN image and GroundTruth image.

4.2. Comparative Experiment

4.2.1. IKONOS Experiment Results

This section mainly describes the results of evaluating the eight indicators carried out using ten algorithms on the IKONOS dataset and visually displays the improved results of each algorithm. Each image is cropped and projected into a 24-bit true color image for display. Figure 2 shows the visualized results of the IKONOS dataset.

(I) A visualization of the LRMS image and the PAN image are illustrated in Figure 2. Section (II) of Figure 2, documents some spectral distortion in the fusion results of the traditional methods such as GS, IHS, Brovey, and PRACS. In terms of spatial details, the sharpened images of GS, IHS, Brovey, PRACS, PNN, PanNet and TFNet methods differ significantly from the PAN image. Observing the spot (as highlighted in the box), it is evident that deep learning approaches excel the traditional methods in terms of spectral consistency. The reconstruction of ground objects by these methods is notably superior. However, an overview of the reconstructed pan-sharpening image reveals that some degree of spectral distortion remains. Regarding the final output, SRPPNN, $\lambda$-PNN and TAMINet can more effectively restore buildings and generally capture more accurate spectral information. For instance, the building restoration details of the TAMINet method (highlighted in the red box in the bottom-left corner) exhibit striking advantages over other algorithms. While both the SRPPNN and TAMINet methods excel at retrieving spatial feature details, our proposed TAMINet method is slightly superior to SRPPNN. With TAMINet, the contours of roads and buildings in remote sensing images are distinctly clear, and details of buildings are vividly visible, offering an impressive visualization effect.

We analyzed the difference graph and histogram of the fusion result graph of pan-sharpening and the GroundTruth image in the blue band in the IKONOS dataset and illustrated them in Figure 2 (III) and (IV). Among them, the GroundTruth image and the pan-sharpening fusion result produce the difference map closest to 0 (that is, the difference image closer to green). When the fusion image exhibits a smaller difference compared to the GroundTruth image; conversely, a smaller value implies a greater disparity between the images. Therefore, we can see whether the difference map is close to the full green and observe whether the image has bump detail from other colors. The closer to green the image is, the cleaner it will be and the better the fusion effect will be. From the difference graphs (III) and (IV) in the IKONOS dataset in Figure 2 the color of the difference graph of the TAMINet method is almost completely green, while the difference graph of other methods still has a lot of visible color details, thus, the difference from GroundTruth is relatively large, and the detail loss comparably severe. It is evident that the four traditional methods in line (III) of Figure 2 and the DL-based PNN method show relatively large differences in the comparison of details. Many details of building are blue or yellow. Compared to other DL algorithms that are almost entirely green, traditional methods show more loss of detail and increased spectral distortion. The spatial structure details extracted by the TAMINet method are closer to the GroundTruth image, and the histogram difference is smaller than in other methods. Therefore, the TAMINet method is superior to the other methods in extracting details of spatial features in the IKONOS dataset.

The quantitative evaluation of the IKONOS dataset is shown in

Table 2. Quantitative evaluation of the IKONOS dataset.

Method	$\mathit{S}\mathit{A}\mathit{M}$ ↓	$\mathit{E}\mathit{R}\mathit{G}\mathit{A}\mathit{S}$ ↓	${\mathit{Q}}_4$ ↑	$\mathit{U}\mathit{I}\mathit{Q}\mathit{I}$ ↑	$\mathit{s}\mathit{C}\mathit{C}$ ↑	${\mathit{D}}_{\mathit{\lambda }}$ ↓	${\mathit{D}}_{\mathit{s}}$ ↓	$\mathit{Q}\mathit{N}\mathit{R}$ ↑
GS	2.6098	2.0241	0.7586	0.7753	0.9078	0.1028	0.1911	0.7319
IHS	2.8214	2.1569	0.7202	0.7435	0.8838	0.1721	0.2441	0.6352
Brovey	2.7520	2.1136	0.7231	0.7469	0.8905	0.1516	0.2276	0.6629
PRACS	2.7562	2.1330	0.8029	0.8009	0.8901	0.1257	0.1619	0.7332
PNN	2.1375	1.5205	0.8349	0.8456	0.9300	0.0856	0.1057	0.8251
PanNet	2.4550	1.8111	0.7973	0.8075	0.9061	0.1343	0.1249	0.7605
TFNet	2.3028	1.6740	0.8279	0.8397	0.9278	0.0926	0.0593	0.8571
MSDCNN	2.0119	1.4374	0.8502	0.8571	0.9387	0.0950	0.1071	0.8177
SRPPNN	1.7580	1.2817	0.8695	0.8757	0.9489	0.0816	0.0983	0.8358
$\lambda$-PNN	2.0174	1.4455	0.8551	0.8613	0.9388	0.0819	0.0889	0.8382
TAMINet	1.6407	1.3159	0.8445	0.8889	0.9568	0.0795	0.1007	0.8364

. Note that the sub-optimal value is underlined and the optimal value of each indicator is marked in bold.

Compared with traditional methods, the deep learning method can achieve fusion results of hyperspectral and spatial precision, which is significantly better than traditional methods (Table 2). Another problem is that PNN, PanNet, TFNet, MSDCNN and the four other methods are obviously inferior to SRPPNN, $\lambda$-PNN and TAMINet. In the case of less training data in the IKONOS dataset, although the TAMINet method still has advantages, minor gaps still remain in several evaluation indicators compared to the SRPPNN method.

To summarize, in case of minor training data in the IKONOS dataset, the TAMINet method excels the SRPPNN in stability and fit, although there are small gaps in various evaluation indices compared to the SRPPNN. Thus, the TAMINet method has obvious advantages compared to the other nine methods.

4.2.2. QuickBird Experiment Results

As for the IKONOS dataset, a representative pan-sharpening result is selected for the QuickBird dataset test and illustrated in Figure 3 for visual comparison. We analyze the difference graph and histogram of the fusion result graph of the GroundTruth image and pan-sharpening in the near-infrared band with the QuickBird dataset and show them in Figure 3 (III) and (IV). Among them, the difference graph information of the GroundTruth image and pan-sharpening fusion result graph is the same as in Figure 2.

From the difference plots and histograms (III) (IV) in Figure 3, the fusion effect of the traditional methods (GS, IHS, Brovey and PRACS) is weak in the QuickBird dataset. The detail loss and spectral distortion of the two methods (GS and IHS) are comparably severe, and the colors and values of the difference charts are relatively large. However, compared to the traditional methods, the deep-learning-based method has relatively better spectral fidelity, spatial and edge details of the merged remote sensing image, and the difference map is close to full green. Despite the advantages of deep learning methods, PNN, PanNet and TFNet still exhibit serious spectral distortion and detail loss. Only the spatial structure details extracted by SRPPNN and TAMINet methods are close to the GroundTruth diagram, with comparably lower loss, and the effect of extracting spatial ground object details is better compared to the other methods.

The quantitative evaluation of the QuickBird dataset is shown in

Table 3. Objective assessment of the QuickBird dataset.

Method	$\mathit{S}\mathit{A}\mathit{M}$ ↓	$\mathit{E}\mathit{R}\mathit{G}\mathit{A}\mathit{S}$ ↓	${\mathit{Q}}_4$ ↑	$\mathit{U}\mathit{I}\mathit{Q}\mathit{I}$ ↑	$\mathit{s}\mathit{C}\mathit{C}$ ↑	${\mathit{D}}_{\mathit{\lambda }}$ ↓	${\mathit{D}}_{\mathit{s}}$ ↓	$\mathit{Q}\mathit{N}\mathit{R}$ ↑
GS	2.9600	2.2395	0.7152	0.7311	0.8836	0.0337	0.0701	0.8995
IHS	3.3501	2.4878	0.6402	0.6747	0.8122	0.1019	0.1222	0.7894
Brovey	3.2152	2.3940	0.6554	0.6846	0.8330	0.0843	0.1137	0.8125
PRACS	3.7735	2.9583	0.7728	0.7668	0.8571	0.0478	0.0678	0.8885
PNN	1.7489	1.3446	0.8674	0.8723	0.9393	0.0484	0.0440	0.9108
PanNet	1.7779	1.3766	0.8684	0.8706	0.9401	0.0360	0.0453	0.9210
TFNet	1.5284	1.1881	0.8882	0.8917	0.9555	0.0701	0.0463	0.8884
MSDCNN	1.7528	1.3024	0.8786	0.8821	0.9451	0.0588	0.0504	0.8948
SRPPNN	1.3692	1.0353	0.9009	0.9025	0.9703	0.0378	0.0390	0.9252
$\lambda$-PNN	1.6612	1.3022	0.8766	0.8834	0.9436	0.0217	0.0466	0.8840
TAMINet	1.2935	0.9819	0.9094	0.9122	0.9737	0.0468	0.0306	0.9248

. The sub-optimal value is underlined, and the optimal value of each indicator is marked in bold.

Compared to the traditional methods, deep learning methods significantly outperform traditional methods (Table 3). With the increase in training data in the QuickBird dataset, the TAMINet method has only one evaluation index, which is slightly inadequate compared to the SRPPNN method. Meanwhile, it has a relatively better effect on spectral fidelity and detail recovery of spatial features. The TAMINet method introduced in this paper significantly outperforms the conventional fusion technique.

4.2.3. WorldView-2 Experimental Results

Similar to the IKONOS dataset, a representative pan-sharpening result was selected from the test set of the WorldView-2 data and illustrated in Figure 4 for visual evaluation.

In the results (II) of Figure 4, in addition to the striking color distortion, the spectral information from the IHS and Brovey algorithm is also missing. The overall reconstruction results of the GS and PRACS algorithm are comparably poor, the visual effect is not ideal and low-quality spatial details appear. Due to the poor spectral fidelity effect, comparably bright colored spots are generated at the edges of the building. For example, both the IHS method in Figure 4d and the Brovey method in Figure 4e show the yellow road color in the enlarged image in the lower left corner, but the road in the original MS image is white. PNN, PanNet, TFNet, MSDCNN and other deep neural network algorithms exhibit better reconstruction effects. Although loss of details and incomplete information content occur in the image, the basic features of ground objects are identifiable. The SRPPNN, $\lambda$-PNN and TAMINet methods show the best spectral and structural information. However, the SRPPNN method exhibits minor boundary blurring problems and detail loss problems. As shown in (IV), the proposed TAMINet method has the lowest error range and lowest detail loss. For example: the building restoration details of the TAMINet method in the red box in the bottom left corner have obvious advantages compared to the other algorithms.

The difference graph and histogram of the fusion result graph of the GroundTruth image and pan-sharpening in the near-infrared band were analyzed for the WorldView-2 dataset and displayed in Figure 4 (III) and (IV). The difference graph information of the fusion result graph of the GroundTruth image and pan-sharpening is the same as in Figure 3. As shown in (III) and (IV) of Figure 4, the fusion results of traditional methods in the WorldView-2 dataset are quite different from the reference images. For example, the details of the white road in the lower right corner are slightly different from the details of the reference images. Although the spectral fidelity of traditional methods is better than in other datasets, we observe from the details that IHS and Brovey exhibit color distortion and blurring. Due to the noise suppression function of PNN, PanNet, TFNet and MSDCNN in the super-resolution process, the edge of the remote sensing image is very smooth. At the same time, in the reconstruction results, some missing details, displacements and other situations occurred in the GS, Brovey and PRACS methods, causing unrealistic visual effects in the images. SRPPNN has significantly improved these deficiencies, but the reconstruction results are still unsatisfactory. The spatial structure details extracted by the proposed TAMINet method are closer to those of GroundTruth. Thus, the effect of extracting the spatial ground object details is superior to the other methods. As indicated by the difference map, the overall effect is close to full green.

Table 4. Objective assessment of the WorldView-2 dataset.

Method	$\mathit{S}\mathit{A}\mathit{M}$ ↓	$\mathit{E}\mathit{R}\mathit{G}\mathit{A}\mathit{S}$ ↓	${\mathit{Q}}_4$ ↑	$\mathit{U}\mathit{I}\mathit{Q}\mathit{I}$ ↑	$\mathit{s}\mathit{C}\mathit{C}$ ↑	${\mathit{D}}_{\mathit{\lambda }}$ ↓	${\mathit{D}}_{\mathit{s}}$ ↓	$\mathit{Q}\mathit{N}\mathit{R}$ ↑
GS	2.6649	2.2789	0.8010	0.8152	0.8747	0.0214	0.0767	0.8746
IHS	2.7153	2.3693	0.7442	0.7856	0.8744	0.1019	0.0974	0.8153
Brovey	2.5360	2.3218	0.7696	0.8007	0.8782	0.0832	0.0938	0.8333
PRACS	2.7545	2.2170	0.8392	0.8265	0.8530	0.0569	0.1035	0.8466
PNN	1.7284	1.3545	0.9014	0.9037	0.9367	0.0159	0.0590	0.9267
PanNet	1.7027	1.3581	0.9045	0.9075	0.9412	0.0233	0.0652	0.9137
TFNet	1.3475	1.0733	0.9253	0.9260	0.9590	0.0337	0.0615	0.9088
MSDCNN	1.6110	1.2770	0.9099	0.9105	0.9436	0.0205	0.0702	0.9117
SRPPNN	1.3991	1.1338	0.9185	0.9226	0.9560	0.0172	0.0635	0.9215
$\lambda$-PNN	1.6808	1.3525	0.8976	0.9089	0.9388	0.0129	0.0617	0.9271
TAMINet	1.3110	1.0459	0.9280	0.9286	0.9608	0.0281	0.0568	0.9191

lists the results of eight evaluation indicators for ten pan-sharpening methods in the WorldView-2 dataset.

Table 4 enumerates the results of eight evaluation indices for ten pan-sharpening techniques tested on the WorldView-2 dataset. The best value for each exponent is marked in bold and the lowest value is underlined. The results show that the TAMINet method outperforms the traditional fusion methods. With the increase in training data in the WorldView-2 dataset, the advantages of the TAMINet method are further expanded compared to the SRPPNN method, and the effect is better on spectral fidelity and retrieval of details of spatial features.

Table 5. The FLOPS, Time and #Params of the IKONOS datasets.

Method	FLOPS	Time (s)	#Params
PNN	5263.85	0.0078	0.08 M
PanNet	5135.93	0.0117	0.08 M
TFNet	30,749.49	0.0268	2.36 M
MSDCNN	12,423.53	0.0238	0.19 M
SRPPNN	8823.77	0.0189	1.83 M
$\lambda$-PNN	15,022.17	0.0320	0.23 M
TAMINet	30749.49	0.0268	2.36 M

discusses the computational cost and number of parameters for different models on the IKONOS test set. Note that the size of the pan-sharpened image is about 256 × 256 × 4, and in order to avoid the accident of the calculation, we compare the average training time at a batch size of 16.

It is evident from the experiments that although the IKONOS dataset is only slightly better than other algorithms, in the QuickBird dataset, all evaluation indices of the proposed TAMINet method significantly exceed those of the other algorithms. However, after testing on the WorldView-2 dataset, the spectral fidelity and detail sharpening of the proposed TAMINet method are significantly superior to the other algorithms. From this series of improvements, we noticed that the IKONOS dataset has only 200 data pairs, the QuickBird dataset has 721 data pairs, and the WorldView-2 training dataset has 1174 pairs of data. Therefore, although the TAMINet method is better than other algorithms by a minor margin in the case of a small amount of data, the larger the amount of data, the greater the accuracy of the TAMINet method. Thus, the algorithm introduced in this paper demonstrates more pronounced benefits when applied to a sizable dataset.

4.3. Ablation Experiments

4.3.1. Selection of the Attention Mechanism

In response to the question of innovation point 1 regarding why the coordinate attention (CA) mechanism is chosen in this study, this experiment uses the original network as the benchmark and replaces the CA mechanism in the network with SE and CBAM mechanisms for the experiment. With the exception of the CA, SE and CBAM mechanisms, all variables are trained and tested on the same network.

Table 6. Quantitative assessment of the attention mechanism.

Dataset	Method	$\mathit{S}\mathit{A}\mathit{M}$ ↓	$\mathit{E}\mathit{R}\mathit{G}\mathit{A}\mathit{S}$ ↓	${\mathit{Q}}_4$ ↑	$\mathit{U}\mathit{I}\mathit{Q}\mathit{I}$ ↑	$\mathit{s}\mathit{C}\mathit{C}$ ↑	${\mathit{D}}_{\mathit{\lambda }}$ ↓	${\mathit{D}}_{\mathit{s}}$ ↓	$\mathit{Q}\mathit{N}\mathit{R}$ ↑
IKONOS	SE	1.9694	1.9694	0.8597	0.8630	0.9404	0.1038	0.1305	0.7899
	CBAM	2.0676	1.4612	0.8497	0.8566	0.9379	0.1150	0.1477	0.7636
	CA	1.6407	1.3159	0.8445	0.8889	0.9568	0.0795	0.1007	0.8364
QuickBird	SE	1.2999	0.9950	0.9058	0.9098	0.9735	0.0457	0.0339	0.9226
	CBAM	1.3434	1.0448	0.9018	0.9081	0.9703	0.0521	0.0328	0.9225
	CA	1.2690	0.9682	0.9098	0.9130	0.9745	0.0438	0.0302	0.9279
WorldView-2	SE	1.3145	1.0454	0.9281	0.9285	0.9604	0.0273	0.0626	0.9146
	CBAM	1.3145	1.0453	0.9272	0.9287	0.9605	0.0272	0.0619	0.9144
	CA	1.3110	1.0459	0.9284	0.9288	0.9609	0.0281	0.0568	0.9191

summarizes the findings from the ablation analysis.

The TAMINet method uses three different attention mechanisms on the backbone network and the results of the CA mechanism are superior to the SE and CBAM attention mechanisms in most indicators including benchmark indicators, such as $SAM$, $ERGAS$ and $Q_4$ (Table 6). The results show that the CA module contributes to a better performance in increasing the model accuracy and it significantly improves the quality of visual sensation and refines the quantitative results. Thus, the application of CA mechanism is the ideal method to achieve the best model performance.

4.3.2. Coordinate Attention and Detail Injection Modules

The ablation experiment was conducted based on the backbone network after removing the two main modules, the coordinate attention (CA) and detail injection (DI). CA indicates the network to which the CA module is added, DI indicates the network to which the DI module is added, and ALL indicates the network to which both modules are added. Except for the CA and DI modules, all variables are trained and tested on the same schema.

Table 7. Quantitative evaluation of the CA and DI.

Dataset	Method	$\mathit{S}\mathit{A}\mathit{M}$ ↓	$\mathit{E}\mathit{R}\mathit{G}\mathit{A}\mathit{S}$ ↓	${\mathit{Q}}_4$ ↑	$\mathit{U}\mathit{I}\mathit{Q}\mathit{I}$ ↑	$\mathit{s}\mathit{C}\mathit{C}$ ↑	${\mathit{D}}_{\mathit{\lambda }}$ ↓	${\mathit{D}}_{\mathit{s}}$ ↓	$\mathit{Q}\mathit{N}\mathit{R}$ ↑
IKONOS	baseline	2.3028	1.6740	0.8279	0.8397	0.9278	0.0926	0.0593	0.8571
	CA	2.2129	1.6537	0.8323	0.8380	0.9273	0.1009	0.0650	0.8449
	DI	1.9497	1.3847	0.8601	0.8650	0.9417	0.0952	0.1238	0.8020
	ALL	1.6407	1.3159	0.8445	0.8889	0.9568	0.0795	0.1007	0.8364
QuickBird	baseline	1.5284	1.1881	0.8882	0.8917	0.9555	0.0701	0.0463	0.8884
	CA	1.5313	1.1972	0.8887	0.8919	0.9523	0.0649	0.0457	0.8934
	DI	1.2935	0.9819	0.9094	0.9122	0.9737	0.0468	0.0306	0.9248
	ALL	1.2690	0.9682	0.9098	0.9130	0.9745	0.0438	0.0302	0.9279
WorldView-2	baseline	1.3475	1.0733	0.9253	0.9260	0.9590	0.0337	0.0615	0.9088
	CA	1.3358	1.0650	0.9262	0.9271	0.9591	0.0279	0.0562	0.9185
	DI	1.3123	1.0443	0.9280	0.9286	0.9608	0.0262	0.0634	0.9142
	ALL	1.3110	1.0459	0.9284	0.9288	0.9609	0.0281	0.0568	0.9191

displays the results of the ablation analysis.

When the proposed framework adds CA modules to the backbone network, the benchmark metrics including $SAM$, $ERGAS$, and $Q_4$ increase slightly. The results show that the CA module helps the model perform well in increasing accuracy. The main contribution of the CA module is to improve the quality of visual perception and quantitative results. When the DI module is added to the backbone network, most indicators obviously increase, and the increase is greater than that of the CA module. Therefore, the DI module contributes much more to the improvement of network fidelity and the quality of visual perception than the CA module. The results show that using the CA and the DI modules simultaneously is the best method to achieve the best model performance.

4.3.3. Detail Injection Module

From the above ablation experiment, the DI module achieved the highest improvement accuracy among the two main modules (CA module and DI module) added to our study. Therefore, to verify the influence and performance of each component in the DI module, three variants of the framework have been designed. The first is to use high-pass filtering to extract spatial feature details from PAN images and inject them into the backbone network, which is called DI-high-pass. The other part uses the spectral features of the LRMS image to inject into the image reconstruction stage, which is called DI-up-LRMS. Finally, two components are simultaneously added, termed DI-all. All variables are trained and tested on the same schema, except the detail injection module.

Table 8. Quantitative evaluation of DI module.

Dataset	Method	$\mathit{S}\mathit{A}\mathit{M}$ ↓	$\mathit{E}\mathit{R}\mathit{G}\mathit{A}\mathit{S}$ ↓	${\mathit{Q}}_4$ ↑	$\mathit{U}\mathit{I}\mathit{Q}\mathit{I}$ ↑	$\mathit{s}\mathit{C}\mathit{C}$ ↑	${\mathit{D}}_{\mathit{\lambda }}$ ↓	${\mathit{D}}_{\mathit{s}}$ ↓	$\mathit{Q}\mathit{N}\mathit{R}$ ↑
IKONOS	DI-high-pass	2.0993	1.4712	0.8446	0.8534	0.9359	0.1260	0.1471	0.7583
	DI-up-LRMS	2.0320	1.4589	0.8498	0.8567	0.9376	0.0998	0.1098	0.8092
	DI3-all	1.9641	1.3946	0.8541	0.8653	0.9423	0.0910	0.1211	0.8085
QuickBird	DI-high-pass	1.3797	1.0442	0.9013	0.9042	0.9705	0.0604	0.0335	0.9091
	DI-up-LRMS	1.2733	0.9820	0.9071	0.9110	0.9739	0.0475	0.0283	0.9260
	DI3-all	1.2935	0.9819	0.9094	0.9122	0.9737	0.0468	0.0306	0.9248
WorldView-2	DI-high-pass	1.3430	1.0765	0.9264	0.9270	0.9590	0.0301	0.0602	0.9127
	DI-up-LRMS	1.3053	1.3053	0.9281	0.9292	0.9602	0.0223	0.0568	0.9250
	DI3-all	1.3110	1.0459	0.9280	0.9286	0.9608	0.0281	0.0545	0.9191

shows the results of the ablation study.

The image quality of the DI-high-pass model is refined after fusion and the accuracy is also improved. The image quality of the DI-up-LRMS model is also improved and the accuracy is markedly increased. However, the effect is less significant compared to the use of the DI-high-pass. Ultimately, combining DI-high-pass and DI-up-LRMS simultaneously yields the optimal model performance, resulting in a notably enhanced effect.

4.3.4. Weight of the Loss Function

The sensitivities of these parameters are discussed in detail in the following subsections and detailed experimental evidence is provided. Weights are defined empirically, in Formula (9), based on studies in the field of pan-sharpening and discussions in references [30,39]. We add a discussion of these parameter choices and a detailed analysis of their impact on the model performance in this section. We use the peak signal-to-noise ratio (PSNR) to evaluate the model. Since $\alpha +\beta +\mu =1$, we fix the $\alpha$ value and keep changing $\beta$ and $\mu$ to identify the optimal ratio. Figure 5 shows the values of weights $\alpha$, $\beta$ and $\mu$.

The experimental result show that $L_1$ is the main factor that affects the model effect. $L_{spectral}$ and $L_{spatial}$ are less significant factors. Therefore, we selected the optimal ratio of weights $\alpha$, $\beta$ and $\mu$ of $1:1:2$.

5. Conclusions

In this study, we introduce a pan-sharpening network for multi-spectrum remote sensing images, utilizing two-stream coordinate attention and multi-detail injection. Our exploration of various architectures reveals that the most effective enhancement in pan-sharpening is derived from the multi-detail injection. The proposed TAMINet effectively leverages the details from the original LRMS data. By incorporating the CA block in the feature extraction network, it encodes channel relationships and long-range dependency using precise position data, thereby extracting original data details more efficiently. During fusion, this extracted information undergoes multiple injections to enhance details, significantly augmenting the spatial resolution and spectral fidelity of the combined image. Furthermore, the architecture and loss function of the TAMINet improves the fusion result reconstruction quality. Through ablation studies and comparisons with other leading methods, the superiority of the TAMINet framework for practical uses becomes evident. While TAMINet considerably improves accuracy, its depth makes it less lightweight. To address this, future work will delve deeper into lightweight strategies, aiming to further refine the architecture and boost the network’s pan-sharpening performance.