Sparse Mix-Attention Transformer for Multispectral Image and Hyperspectral Image Fusion

Abstract

Multispectral image (MSI) and hyperspectral image (HSI) fusion (MHIF) aims to address the challenge of acquiring high-resolution (HR) HSI images. This field combines a low-resolution (LR) HSI with an HR-MSI to reconstruct HR-HSIs. Existing methods directly utilize transformers to perform feature extraction and fusion. Despite the demonstrated success, there exist two limitations: (1) Employing the entire transformer model for feature extraction and fusion fails to fully harness the potential of the transformer in integrating the spectral information of the HSI and spatial information of the MSI. (2) HSIs have a strong spectral correlation and exhibit sparsity in the spatial domain. Existing transformer-based models do not optimize this physical property, which makes their methods prone to spectral distortion. To accomplish these issues, this paper introduces a novel framework for MHIF called a Sparse Mix-Attention Transformer (SMAformer). Specifically, to fully harness the advantages of the transformer architecture, we propose a Spectral Mix-Attention Block (SMAB), which concatenates the keys and values extracted from LR-HSIs and HR-MSIs to create a new multihead attention module. This design facilitates the extraction of detailed long-range information across spatial and spectral dimensions. Additionally, to address the spatial sparsity inherent in HSIs, we incorporated a sparse mechanism within the core of the SMAB called the Sparse Spectral Mix-Attention Block (SSMAB). In the SSMAB, we compute attention maps from queries and keys and select the K highly correlated values as the sparse-attention map. This approach enables us to achieve a sparse representation of spatial information while eliminating spatially disruptive noise. Extensive experiments conducted on three synthetic benchmark datasets, namely CAVE, Harvard, and Pavia Center, demonstrate that the SMAformer method outperforms state-of-the-art methods.

1. Introduction

Multispectral image (MSI) and hyperspectral image (HSI) fusion (MHIF) aims to obtain a high-resolution (HR) HSI by fusing a low-resolution (LR) HSI and an HR-MSI in the same scene. Compared to traditional images containing only a few bands (such as RGB), HSIs offer the advantage of capturing more spectral information about the same scene. Research has demonstrated the significant benefits of HSIs in various vision tasks, including tracking [1], segmentation [2], and classification [3].

Early methods of MHIF drew inspiration from pansharpening techniques used in remote-sensing images. These methods, such as component substitution (CS) [4,5] and multiresolution analysis (MRA) [6,7,8], were computationally efficient. However, they face challenges in terms of the fusion quality and easily lead to spectral distortion. This can be attributed to the fact that a single panchromatic image inherently contains less spectral information compared to HSIs. The subsequent model-based techniques, e.g., Bayesian-based [9,10], tensor-based [11,12], and matrix factorization-based methods [13,14], rely on prior knowledge of HSIs to address this issue. For example, Wei et al. [15] suppose the target image resides in a lower-dimensional subspace. A carefully designed sparse regularization term is incorporated, and a fusion problem is addressed through optimization algorithms. Kawakami et al. [13] assume that each spectrum can be linearly represented by a set of spectral atomic components. Applying the decomposition algorithm to the hyperspectral input enables the estimation of the basis representing the reflection spectra, which is subsequently integrated with the MSI image to yield the final result. However, it is important to note that these prior knowledge sources are artificially designed and may not encompass all the characteristics of real data. As a result, the ability of the model to generalize is significantly constrained.

In recent years, deep learning methods have shown remarkable advancements in various domains of computer vision, including super-resolution (SR) tasks [16,17]. The field of MHIF has also witnessed significant exploration in applying deep learning methods. One of the earliest deep learning approaches in this field is based on a 3D convolutional neural network (CNN) [18], which has achieved impressive results. Xie et al. [19] have successfully combined deep learning algorithms with optimization algorithms to enhance interpretability. These methods can be broadly categorized into two types, as shown in Figure 1. The first type (see Figure 1a) directly concatenates the upsampled HSI and MSI as the input and passes them through a neural network to obtain the reconstructed image [20,21]. The second type (see Figure 1b) employs two independent neural networks to extract features from HSIs and MSIs separately, and a fusion module is then utilized to generate the final output [22,23]. Although these methods have shown advancements compared to traditional approaches, they still exhibit certain limitations. The restricted receptive field of CNNs hinders the effective capture of global information, resulting in constraints on the final fusion outcomes.

In order to address the aforementioned issues, researchers have applied the transformer model to the MHIF. The transformer model proves to be capable of capturing global information from the images, effectively solving long-range dependencies and accurately capturing fine details between HSIs and MSIs. Hu et al. [24] were among the first to adopt the transformer model for MHIF. Their framework resembles the first type of CNN architecture, where the HSI and MSI images are merged and the self-attention mechanism is employed to establish relationships between the two images. Following this, Jia et al. [25] proposed a two-stream network, which is similar to the second type of CNN network. This approach involves extracting spectral features from HSIs and spatial features from MSIs separately. Both of these methods show improvement over CNN models; nonetheless, they still have two main problems. Firstly, they remain constrained by conventional frameworks and do not fully harness the transformer model, which has the potential to establish a comprehensive mapping relationship between HSIs and MSIs. Secondly, the earlier transformer models include all pixels in the image in the computation process, resulting in a dense calculation process. However, due to factors such as uneven feature distribution and the impact of the sensor resolution, HSIs display spatial sparsity, resulting in lower correlations between certain pixels. Furthermore, HSIs possess a higher number of dimensions, thereby enhancing the spectral correlations among its components [26,27]. Solely relying on raw transformers not only escalates the computational complexity but also introduces the potential for system noise interference.

To overcome the limitations of previous methods, we propose a novel MHIF framework named the Sparse Mix-Attention Transformer (SMAformer). There are two main technologies included in the framework. Firstly, in order to fully utilize the flexibility of the attention mechanism in the transformer model, we combine feature extraction and feature fusion to generate a unified framework called the Spectral Mix-Attention Block (SMAB). Specifically, as illustrated in Figure 1c, we expand both the upsampled LR-HSI and MSI in the channel dimension, treating each channel as a token. Subsequently, we concatenate the K and V of the two images and perform attention operations separately, combining self-attention and cross-attention. By treating the channels as tokens, we are able to emphasize the close relationship between the spectra of the HSI. The use of two attention mechanisms allows us to pay attention to both the intrinsic information of the HSI and the reference information from the MSI. During this process, the extracted feature information can be more efficiently integrated. Secondly, in the past, transformers utilized all available information in their calculations. Nonetheless, the spatial sparsity inherent in the HSI often results in significant computational inefficiency when following these approaches. To mitigate the unnecessary computational overhead, we introduced a sparse-attention mechanism to replace the original self-attention. More specifically, after calculating the self-attention for the MSI at an HR stage, we select the Top-k correlation elements for subsequent computations while setting the rest to zero. In lower resolutions, we employ a conventional transformer. This approach allows information from the MSI to selectively complement the HSI rather than being entirely combined with it. This not only aligns with the physical characteristics of the HSI but also minimizes the impact of irrelevant information from the MSI on the restored image. The main contributions of this article can be summarized as the following three points:

• We introduce a novel transformer-based network called SMAformer; it takes advantage of the flexibility of the attention mechanism in the transformer to effectively extract intricate long-range information in both spatial and spectral dimensions.
• We use the sparse-attention mechanism to replace the self-attention mechanism. This enables us to achieve a sparse representation of spatial information while eliminating interfering noise in space.
• Extensive experiments on three synthetic benchmark datasets, CAVE, Harvard, and Pavia Center, show that SMAformer outperforms state-of-the-art methods.

2. Related Work

We divide the existing MHIF methods into two categories, traditional and deep-learning-based methods. In the following, we review them in detail.

2.1. Traditional Work

Early methods were based on the principle of linear transformations: Principal Component Analysis (PCA) and Wavelet Transformation. For example, Nunez et al. [28] proposed two wavelet decomposition-based MSI pansharpening methods: an additive method and a replacement method. Matrix-factorization-based methods assume that each spectrum can be linearly represented by a few spectral atoms. Naoto et al. [29] decomposed the HSI and the MSI alternately into an endmember matrix and an abundance matrix and then built the sensor observation model related to these two data into the initialization matrix of each non-negative matrix factorization (NMF) unmixing process. Finally, they obtained the HR-HSI. The spatial sparsity of the hyperspectral input by Kawakami et al. [13], who unmixed the hyperspectral input and combined it with the RGB input to produce the desired result, treats the unmixing problem as decomposing the input into a basis and a search for a set of maximally sparse coefficients. Another important method of HSI-SR is based on tensor factorization. Tensor factorization technology can convert traditional 2D matrix images into 4D or even higher-order tensors without losing information. Zhang et al. [30] regularized the low-rank tensor decomposition based on the spatial spectrogram, derived the spatial domain map by using MSIs, derived the spectral domain map by using HSIs, and finally fused the two parts. Xu et al. [31] proposed a coupled tensor ring (TR) representation model to fuse HSIs and MSIs while introducing graph Laplacian regularization into the spectral kernel tensor to preserve spectral information.

Although the above traditional methods are successful to a certain extent, they are all based on certain priors or assumptions, which may not match the complex display environment, thus causing many negative effects, such as spectral distortion.

2.2. Deep-Learning-Based Work

Due to the powerful representation ability of CNNs, the method of deep learning has also been applied to the super-resolution task of HSIs. Dian et al. [32] use a deep residual network to directly learn image priors and finally use an optimization algorithm to reconstruct the HSI. Considering the rich spectral information of HSIs, Palsson [18] uses a 3D convolutional network to fuse MSIs and HSIs. In order to reduce the amount of calculation, it uses PCA to reduce the dimension of the HSI. This method realizes end-to-end training and has achieved very good results. Yang et al. [23] designed a dual-branch network: one branch extracts MSI spatial information and the other extracts HSI spectral information and finally fuses to obtain ideal results. Considering the spatial and spectral degradation of the HSI image, Xie et al. [19] established a deep model and used an optimization algorithm to iteratively solve it. Due to the inherent defects of convolution, the above CNN-based deep learning method is limited by the receptive field and cannot fully mine the correlation between MSIs and HSIs, resulting in the loss of some structural details.

A transformer can solve the shortcomings of the CNN due to its ability to perceive full image information and establish long-range details. Hu et al. [24] first applied it to the field of MHIF called Fusformer, which can use the transformer to globally explore the internal relationship within the feature. Jia et al. [25] used the transformer to establish a dual-branch fusion network: one network extracts spectral information and the other branch extracts spatial information. Although the above methods have made significant improvements, their fusion framework still complies with the previous CNN network and does not use the flexibility of the transformer network to establish the connection between HSIs and MSIs. We directly establish the relationship between the two within the transformer to fully obtain the relevant information of the two.

3. Method

3.1. Network Architecture

Figure 2a illustrates our overall framework. Given a pair of HR-MSI and LR-HSI, represented by $\mathbf{X}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{c}}$ and $\mathbf{Y}\in {\mathbb{R}}^{\mathrm{h}\times \mathrm{w}\times \mathrm{C}}$, respectively, where $\mathrm{H}$ and $\mathrm{h}$ represent the height of the MSI and HSI, respectively; $\mathrm{W}$ and $\mathrm{w}$ represent the width; and $\mathrm{C}$ and $\mathrm{c}$ represent the number of channels. The network takes the upsampled HSI $\mathbf{Y}\uparrow$ and $\mathbf{X}$ as inputs and generates the corresponding reconstructed HR-HSI ($\mathbf{Z}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$). To enhance the recovery of intricate details in $\mathbf{Y}\uparrow$ and concurrently diminish the computational overhead of the subsequent network, we chose to implement Bicubic interpolation [33] as the upsample function. Firstly, we duplicate the image on each side of $\mathbf{Y}$ and concatenate these duplications to expand its dimensions to $(\mathrm{h}+4)\times (\mathrm{w}+4)\times \mathrm{C}$. We denote this resulting image as ${\mathbf{Y}}_{pad}$. We employ Bicubic upsampling for each channel $\gamma =\left\{1,2,···,C\right\}$ of ${\mathbf{Y}}_{pad}$. To execute an operation using Bicubic interpolation, it necessitates the utilization of 16 surrounding points. The corresponding points of ${\mathbf{Y}\uparrow }^{\gamma }(i,j)$ are $\mathrm{Points}(i,j)$, $i=\{1,2,···,H\}$, $j=\{1,2,···,W\}$. They can be expressed as the following matrix:

$$\begin{array}{c}\hfill \mathrm{Points}(i,j)=\left[\begin{array}{cccc}{\mathbf{Y}}_{pad}^{\gamma }(i_d+1,j_d+1)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+1,j_d+2)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+1,j_d+3)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+1,j_d)+4\\ {\mathbf{Y}}_{pad}^{\gamma }(i_d+2,j_d+1)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+2,j_d+2)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+2,j_d+3)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+2,j_d)+4\\ {\mathbf{Y}}_{pad}^{\gamma }(i_d+3,j_d+1)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+3,j_d+2)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+3,j_d+3)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+3,j_d)+4\\ {\mathbf{Y}}_{pad}^{\gamma }(i_d+4,j_d+1)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+4,j_d+2)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+4,j_d+3)& {\mathbf{Y}}_{pad}^{\gamma }(i_d+4,j_d)+4\end{array}\right],\end{array}$$

(1)

where $i_d=\lfloor i/8\rfloor$ and $j_d=\lfloor j/8\rfloor$, $\lfloor ·\rfloor$ indicates floor. The constant 8 represents the ratio of upsampling. Their coefficient matrices are

$$\begin{array}{c}\hfill {\Gamma }_1(i,j)=\left[\begin{array}{cccc}\mathrm{FB}(1+u)& \mathrm{FB}\left(u\right)& \mathrm{FB}(1-u)& \mathrm{FB}(2-u)\end{array}\right],\end{array}$$

(2)

$$\begin{array}{c}\hfill {\Gamma }_2(i,j)=\left[\begin{array}{c}\mathrm{FB}(1+v)\\ \mathrm{FB}\left(v\right)\\ \mathrm{FB}(1-v)\\ \mathrm{FB}(2-v)\end{array}\right],\end{array}$$

(3)

where $u=\left(i\%8\right)/8$, $v=\left(j\%8\right)/8$, % represents the remainder operation. $\mathrm{FB}\left(\right)$ is the kernel function, which can be expressed by the following formula:

$$\begin{array}{c}\hfill \mathrm{FB}\left(x\right)=\left\{\begin{array}{cc}(e+2){\left|x\right|}^3-e+3{\left|x\right|}^2+1\hfill & \left|x\right|\le 1\hfill \\ e{\left|x\right|}^3-5e{\left|x\right|}^2+8e\left|x\right|-4e\hfill & 1<\left|x\right|<2,\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(4)

where the constant $e=-0.5$. Finally, the ${\mathbf{Y}\uparrow }^{\gamma }\left(i,j\right)$ can be expressed as

$$\begin{array}{c}\hfill {\mathbf{Y}\uparrow }^{\gamma }\left(i,j\right)={\Gamma }_1\left(i,j\right)·\mathrm{Points}\left(i,j\right)·{\Gamma }_2\left(i,j\right).\end{array}$$

(5)

Perform the above-mentioned operations for each channel of ${\mathbf{Y}}_{pad}$, generating $\mathbf{Y}\uparrow \in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$. Then, we feed $\mathbf{Y}\uparrow$ and $\mathbf{X}$ into separate $3\times 3$ convolutional layers to achieve the same shape. Following concatenation, the concatenated output is directed into N Single-stage Mixed Spectral-wise Transformer (SMST) modules. The SMST adopts a U-shaped structure consisting of an encoder, decoder, and bottleneck. The input is first passed through the mapping to obtain the initial features $\mathbf{F}(\mathbf{Y}\uparrow )$ and $\mathbf{F}\left(\mathbf{X}\right)$. Considering the spatially sparse nature of HSIs, the first stage of the encoder utilizes Sparse Spectral Mix-Attention Blocks (SSMAB) at a high resolution, while the SMAB is used in the subsequent low-resolution stage and bottleneck. The same strategy is employed in the decoder. Both the embedding and mapping layers employ a $3\times 3$ convolutional layer, while downsampling uses a $4\times 4$ convolutional layer with a stride of 2. To minimize information loss during downsampling, skip connections are utilized between the encoder and decoder. Specifically, after upsampling through a $\mathrm{deconv}2\times 2$ convolutional layer, we acquire features at an identical resolution as the encoding step and subsequently concatenate them along the channels, and a $1\times 1$ convolution layer is employed for the dimensionality reduction as the input for the subsequent step. The output of the N SMST is further processed through a $3\times 3$ convolution layer to yield a feature matrix with the same dimensions as $\mathbf{Y}\uparrow$. Ultimately, this feature matrix is summed with $\mathbf{Y}\uparrow$ to produce the final $\mathbf{Z}$. The overall network structure can be expressed as the following formula:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\mathbf{Y}\uparrow =\mathrm{Upsample}{}_{\times 8}\left(\mathbf{Y}\right),\hfill \\ \mathrm{F}(\mathbf{Y}\uparrow ),\mathrm{F}\left(\mathbf{X}\right)=\mathrm{conv}3\times 3(\mathrm{conv}3\times 3((\mathbf{Y}\uparrow ),\mathrm{conv}3\times 3\left(\mathbf{X}\right)\left)\right),\hfill \\ \mathbf{Z}=\mathrm{conv}3\times 3\left({\mathrm{SMST}}_{\times \mathrm{N}}(\mathrm{F}(\mathbf{Y}\uparrow ),\mathrm{F}\left(\mathbf{X}\right)))\right.+\mathbf{Y}\uparrow ,\hfill \end{array}\right.\end{array}$$

(6)

where $\mathrm{conv}3\times 3$ represents a $3\times 3$ convolution layer and ${\mathrm{SMST}}_{\times \mathrm{N}}$ indicates N SMST modules. We will provide a more detailed explanation of the SMAB and the sparse attention in the SSMAB in the next two sections.

3.2. Spectral Mix-Attention Block

Different from the previous use of one-stream and two-stream networks, our network employs a completely new architecture. The SMAB, as illustrated in Figure 2b, plays a crucial role in the system. It takes input features from $\mathbf{Y}\uparrow$ and $\mathbf{X}$, extracting their long-range features simultaneously, and integrating the interactive information between them. This module differs from the traditional transformer in two key ways. First, taking into account the spatial sparseness and interspectral correlation in HSIs [34], we perform the attention operation on the image channels. This approach not only reduces the computational complexity but also proves to be more suitable for the current task. Second, we employ two parallel attention modules: self-attention and cross-attention. These modules not only focus on their own long-range information but also capture the cross-information between the two inputs. Specifically, we expand $\mathbf{F}(\mathbf{Y}\uparrow )$ and $\mathbf{F}\left(\mathbf{X}\right)$ along the channel dimension and apply linear transformations to obtain ${\mathbf{Q}}_y$, ${\mathbf{K}}_y$, ${\mathbf{V}}_y$ for $\mathbf{F}(\mathbf{Y}\uparrow )$ and ${\mathbf{Q}}_x$, ${\mathbf{K}}_x$, ${\mathbf{V}}_x$ for $\mathbf{F}\left(\mathbf{X}\right)$. Then, we concatenate the ${\mathbf{K}}_y$ and ${\mathbf{K}}_x$, as well as the ${\mathbf{V}}_y$ and ${\mathbf{V}}_x$, and perform the attention operation by using their respective $\mathbf{Q}$. This approach enables us to effectively capture not only the intrinsic relationships within $\mathbf{Y}\uparrow$ and $\mathbf{X}$ themselves but also the dynamic exchange of information between them. The following formula illustrates this process:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\left[{\mathbf{Q}}_y,{\mathbf{K}}_y,{\mathbf{V}}_y\right]=\mathbf{Y}\uparrow \left[{\mathbf{W}}_y^{\mathbf{Q}},{\mathbf{W}}_y^{\mathbf{K}},{\mathbf{W}}_y^{\mathbf{V}}\right],\hfill \\ \left[{\mathbf{Q}}_x,{\mathbf{K}}_x,{\mathbf{V}}_x\right]=\mathbf{X}\left[{\mathbf{W}}_x^{\mathbf{Q}},{\mathbf{W}}_x^{\mathbf{K}},{\mathbf{W}}_x^{\mathbf{V}}\right],\hfill \\ {\mathbf{K}}_c=\mathrm{Concat}({\mathbf{K}}_x,{\mathbf{K}}_y),\hfill \\ {\mathbf{V}}_c=\mathrm{Concat}({\mathbf{V}}_x,{\mathbf{V}}_y),\hfill \\ {\mathrm{Attention}}_{\mathrm{y}}=\mathrm{Softmax}\left(\frac{{\mathbf{Q}}_y{\mathbf{K}}_c^T}{\sqrt{d}}\right){\mathbf{V}}_c,\hfill \\ {\mathrm{Attention}}_{\mathrm{x}}=\mathrm{Softmax}\left(\frac{{\mathbf{Q}}_x{\mathbf{K}}_c^T}{\sqrt{d}}\right){\mathbf{V}}_c,\hfill \end{array}\right.\end{array}$$

(7)

where ${\mathbf{W}}_y^{\mathbf{Q}}$, ${\mathbf{W}}_y^{\mathbf{K}}$, ${\mathbf{W}}_y^{\mathbf{V}}$ and ${\mathbf{W}}_x^{\mathbf{Q}}$, ${\mathbf{W}}_x^{\mathbf{K}}$, ${\mathbf{W}}_x^{\mathbf{V}}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{n}}$ denote the learnable parameters of $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ of $\mathbf{Y}\uparrow$ and $\mathbf{X}$, respectively; ${\mathrm{Attention}}_{\mathrm{y}}$ and ${\mathrm{Attention}}_{\mathrm{y}}$ denote the attention of $\mathbf{Y}\uparrow$ and $\mathbf{X}$, respectively, which include self-attention and cross-attention. Subsequently, after concatenating the features obtained from the two pathways, we apply a linear projection, followed by reshaping it to match the same dimensions as the input. Linear projection can be expressed as the following formula:

$$\begin{array}{c}\hfill {\mathbf{F}}_{out}={\mathbf{F}}_{in}{\mathbf{W}}^T+b\end{array}$$

(8)

where ${\mathbf{F}}_{out}$ and ${\mathbf{F}}_{in}$ represent the input and output features, respectively; ${\mathbf{W}}^T$ represents the learnable parameters; and b represents bias. Finally, after the layer normal operation, it passes through the Feed Forward Neural Network (FFN), which is shown in Figure 2d.

3.3. Sparse Spectral Mix-Attention Block

The global self-attention of the transformer is not well-suited for MHIF. This is primarily due to two reasons. Firstly, HSIs exhibit spatial sparsity, making dense self-attention unsuitable for capturing their physical characteristics. Secondly, the spectral correlation in HSIs is stronger than the spatial correlation. Therefore, it is necessary to reduce the proportion of spatial-reference-information branches to mitigate the interference of irrelevant features and noise.

To overcome these limitations, We introduce a sparse-attention [35] module that leverages the sparsity inherent in neural networks, as shown in Figure 2c. This module is applied to the branch of MSI and integrated into the overall framework. The initial feature-extraction stage and the acquisition of $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ remain consistent with the SMAB. However, after calculating the $\mathbf{Q}\times {\mathbf{K}}_c^{\top }$, we select only the Top-k elements with the largest attention coefficients in the attention map. Here, the value of Top-k is an adjustable dynamic parameter, and we extract the largest portion of data from the attention map; for instance, $\frac{1}{2}$ or $\frac{2}{3}$. Other small values are replaced with zeros. Then, these processed attention maps are individually multiplied by a learnable parameter. Subsequently, we take the average of these results to derive the final attention map. The objective of this step is to retain the most important components and discard redundant or irrelevant ones. The final formulation for sparse attention will take the following form:

$$\mathrm{SparseAtt}=\mathrm{softmax}\left({\mathcal{T}}_k\left(\frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d}}\right)\right)\mathbf{V},$$

(9)

where ${\mathcal{T}}_k$ is an operator that selects the Top-k values; it can be represented by the following formula:

$${\left[{\mathcal{T}}_k\left(\mathbf{M}\right)\right]}_{ij}=\left\{\begin{array}{cc}{M}_{ij}\hfill & M_{ij}\ge t_i,\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(10)

where $t_i$ is the k-th largest value in the j-th row of ${\mathrm{Amap}}_{\mathrm{x}}$.

After applying this method in the HR step, the rest of the process is always the same as the previous method. We selected real and fake apples from the CAVE dataset [36] and plotted the correlation coefficients of the 20th spectral band in the HSI. Figure 3 illustrates the impact of employing various multihead self-attention (MSA) techniques, i.e., the original global MSA (G-MSA) [37], Mix MSA [38], Spectral MSA (S-MSA) [34], and our MSA. It is evident that the heat map generated through our utilization strategy exhibits the highest resemblance to the Ground Truth (GT).

3.4. Loss Function

After obtaining the reconstructed HR-HSI, we proceed to train the network by using the following loss-function formula:

$$\mathcal{L}={\mathcal{L}}_1+\lambda {\mathcal{L}}_{ssim}.$$

(11)

The loss function comprises two components: the first component is the ${\mathcal{L}}_1$ loss function, aimed at enhancing the clarity of the recovered edges and details. The formula is as follows:

$${\mathcal{L}}_1={∥\widehat{\mathbf{Z}}-\mathbf{Z}∥}_1,$$

(12)

where $\widehat{\mathbf{Z}}$ and $\mathbf{Z}$ denote the GT and the fusion result, respectively.

The second component is the ${\mathcal{L}}_{ssim}$ loss function [39], which contributes to the overall image performance improvements. It is defined as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{ssim}& =1-SSIM(\widehat{\mathbf{Z}}-\mathbf{Z}),\hfill \\ \hfill & =1-\frac{1}{S}\sum _{p=1}^S\frac{\left(2{\mu }_{{\widehat{\mathbf{Z}}}_p}{\mu }_{{\mathbf{Z}}_p}+d_1\right)\left(2{\sigma }_{{\widehat{\mathbf{Z}}}_p{\mathbf{Z}}_p}+d_2\right)}{\left({\mu }_{{\widehat{\mathbf{Z}}}_p}^2+{\mu }_{{\mathbf{Z}}_p}^2+d_1\right)\left({\sigma }_{{\widehat{\mathbf{Z}}}_p}^2+{\sigma }_{{\mathbf{Z}}_p}^2+d_1\right)},\hfill \end{array}$$

(13)

where p represents different bands from 1 to S, and S represents the number of channels of the image. ${\mu }_{{\widehat{\mathbf{Z}}}_k}$ and ${\mu }_{{\mathbf{Z}}_k}$ represent the average values of the GT and the fusion result, respectively, while ${\sigma }_{{\widehat{\mathbf{Z}}}_k}$ and ${\sigma }_{{\mathbf{Z}}_k}$ signify the standard deviation of the image pixel values. ${\sigma }_{{\widehat{\mathbf{Z}}}_k{\mathbf{Z}}_k}$ represents covariance, and $d_1$ and $d_2$ are two constants introduced to prevent a denominator of 0; $d_1$ and $d_2$ take the values $0.{01}^2$ and $0.{03}^2$, respectively. The parameter $\lambda$ in the middle serves as a coefficient used to balance the two components, and in this paper, a value of $0.1$ is employed.

4. Experiments

In this section, to demonstrate the effectiveness of the method presented in this paper, we carried out a series of experiments. Firstly, we introduce the experimental environment and configuration, as well as present three datasets and various evaluation metrics. Following this, a comprehensive comparative analysis is performed to assess the performance of the method proposed in this paper in comparison to several advanced methods on these datasets. Finally, in order to validate the efficacy of our proposed module, we conducted ablation experiments

4.1. Experimental Setup and Model Details

This method utilizes PyTorch 1.10.1 and Python 3.6.15 for training on a single NVIDIA GPU GeForce GTX 2080Ti. The Adam optimizer is employed to minimize the loss function, and for the learning rate, a dynamic approach is adopted. Initially, the learning rate is set to 1 × 10⁻⁴ and then decreased by a factor of 0.8 every 100 epochs. Based on the initial configuration outlined above, we conducted experiments by using the CAVE dataset and generated curves illustrating the relationship between the training loss, testing loss, and epochs in Figure 4. As the decrease in the loss function gradually plateaus in the later stages, we introduced an enlarged image specifically focusing on the epoch range from 1500 to 2000 in the upper right corner.

It is evident that both training and testing losses experienced a substantial decline within the initial 250 epochs. However, during the subsequent phases, the rate of decline decelerated. The enlarged images provide further insight, indicating occasional fluctuations in the middle epochs; nevertheless, the overarching trend remains downward, reassuringly indicating an absence of overfitting within this specified range. Despite these positive observations, we opted for a training epoch limit of 2000, considering the pragmatic constraint of time.

4.2. Datasets

To fully validate the effectiveness of the method proposed in this paper, we conducted a comparison by using three synthetic benchmark datasets, including two real-world datasets: CAVE [36] and Harvard [40], and a remote-sensing dataset: Pavia Center. The characteristics of these datasets are as follows:

CAVE: Captured by an Apogee Alta U260 CCD camera, this dataset comprises spectral bands ranging from 400 nm to 700 nm with 10 nm intervals, representing real-world objects. It consists of a total of 32 scenes, categorized into five groups: Stuff, Skin and Hair, Paints, Food and Drinks, and Real and Fake. Each scene contains 512 × 512 pixels and 31 spectral bands. Additionally, RGB images corresponding to these scenes are also provided. For our experiments, we selected the first 20 scenes as the training set and the remaining 12 scenes as the test set.

Harvard: Captured by using Nuance FX, CRI Inc., this dataset comprises both indoor and outdoor real scenes captured under sunlight. It contains the same 31 spectral bands as the CAVE dataset, but the spectral band ranges from 420 nm to 720 nm. The resolution of each scene is 1392 × 1040 pixels. We designated the last 12 scenes as the test set and used the remaining scenes for training.

Pavia Center: The Pavia Center dataset was obtained through satellite remote sensing by using ROSIS sensors, capturing imagery from the heart of Pavia, Italy. Pavia is a small city encompassing urban regions, agricultural land, and diverse geographic features, and the scenes are even more intricate. This dataset comprises 102 spectral bands, and the images have a resolution of 1096 × 1096 pixels.

Data Simulation: For the CAVE and Harvard datasets, to train the network, paired LR-HSI and HR-MSI data are required. However, the dataset only provides one HSI image, necessitating data simulation. In this process, the original image serves as the GT. The HR-MSI image is calculated by using the response function of the Nikon D700 camera and the original HSI image. For LR-HSI, a Gaussian convolution kernel with size r × r is applied to blur the original HSI image. The resulting image is then downsampled by a factor of eight, yielding the final LR-HSI. Due to the Pavia Center dataset consisting of only a single image, it was divided into two segments. The 512 × 216 pixel block located in the lower left corner serves as the test set while the remaining sections are allocated for training. The HR-MSI was generated by utilizing the spectral response functions of the IKONOS satellite and raw images. The method used to generate the LR-HSI remains consistent with the previous two methods.

4.3. Evaluation Metrics

To comprehensively assess the efficacy of our method, we chose five evaluation metrics to appraise the images restored by our approach. These metrics include PSNR, the Structural Similarity Index Metric (SSIM) [39], the Spectral Angle Mapper (SAM) [41], Erreur Relative Global Adimensionnelle Synthese (ERGAS) [42], and the Universal Image-Quality Index (UIQI) [43]. Subsequently, we will introduce each of these evaluation indicators individually.

4.3.1. PSNR

PSNR calculates the peak error between the corresponding pixels of the generated image and the reference image. The formula for this calculation is as follows:

$$\mathrm{PSNR}\left(\widehat{\mathbf{Z}},\mathbf{Z}\right)=\frac{10}{S}\sum _{p=1}^{S}lo{g}_{10}\left(\frac{MA{X}_{{\mathbf{Z}}_p}^2}{\mathrm{MSE}\left({\widehat{\mathbf{Z}}}_p,{\mathbf{Z}}_p\right)}\right),$$

(14)

where p represents different bands from 1 to S, and S represents the number of channels of the image. $MA{X}_{{\mathbf{Z}}_p}$ represents the maximum pixel value of the two input images, with the maximum value for the images in this paper being one. $\mathrm{MSE}\left({\widehat{\mathbf{Z}}}_p,{\mathbf{Z}}_p\right)$ stands for the Mean Square Error between the GT and the fusion result, and its formula is as follows:

$$\mathrm{MSE}\left(\widehat{{\mathbf{Z}}_p},{\mathbf{Z}}_p\right)=\frac{1}{HW}\sqrt{\sum _{i=0}^{H-1}\sum _{j=0}^{W-1}{\left[{\widehat{\mathbf{Z}}}_p\left(i,j\right)-{\mathbf{Z}}_p\left(i,j\right)\right]}^2},$$

(15)

where H and W represent the height and width of the image, respectively. A higher PSNR value indicates a smaller error between the generated image and the real image, indicating better image quality.

4.3.2. SSIM

The SSIM measures the similarity between two images, focusing on improved image quality in terms of brightness, contrast, and structure. The calculation formula is as follows:

$$\mathrm{SSIM}\left(\widehat{\mathbf{Z}},\mathbf{Z}\right)=\frac{1}{S}\sum _{p=1}^S\frac{\left(2{\mu }_{{\widehat{\mathbf{Z}}}_p}{\mu }_{{\mathbf{Z}}_p}+d_1\right)\left(2{\sigma }_{{\widehat{\mathbf{Z}}}_p{\mathbf{Z}}_p}+d_2\right)}{\left({\mu }_{{\widehat{\mathbf{Z}}}_p}^2+{\mu }_{{\mathbf{Z}}_p}^2+d_1\right)\left({\sigma }_{{\widehat{\mathbf{Z}}}_p}^2+{\sigma }_{{\mathbf{Z}}_p}^2+d_1\right)},$$

(16)

where ${\mu }_{{\widehat{\mathbf{Z}}}_p}$ and ${\mu }_{{\mathbf{Z}}_p}$ represent the average values of the image pixels, serving to quantify image brightness, while ${\sigma }_{{\widehat{\mathbf{Z}}}_p}$ and ${\sigma }_{{\mathbf{Z}}_p}$ signify the standard deviation of the image pixel values, used to gauge the image contrast. ${\sigma }_{{\widehat{\mathbf{Z}}}_p{\mathbf{Z}}_p}$ represents covariance, indicating the similarity in image structure, and $d_1$ and $d_2$ are two constants introduced to prevent a denominator of 0; $d_1$ and $d_2$ take the values $0.{01}^2$ and $0.{03}^2$, respectively. This metric aligns more closely with human-visual-perception characteristics than PSNR. A larger SSIM value corresponds to a better human-visual-perception effect and a smaller disparity between the enhanced image and the original image.

4.3.3. SAM

The SAM treats the spectrum of each pixel within the image as a high-dimensional vector and quantifies the similarity between these spectra by determining the angle between the two vectors. A smaller angle indicates a higher degree of similarity between the two spectra. The formula for the SAM is expressed as follows:

$$\mathrm{SAM}\left(\widehat{\mathbf{Z}},\mathbf{Z}\right)=\arccos \left(\frac{{\sum }_{p=1}^S{\widehat{\mathbf{Z}}}_p{\mathbf{Z}}_p}{\sqrt{{\sum }_{p=1}^S{\left({\widehat{\mathbf{Z}}}_p\right)}^2}\sqrt{{\sum }_{p=1}^S{\left({\mathbf{Z}}_p\right)}^2}}\right).$$

(17)

4.3.4. ERGAS

ERGAS is a metric used to assess the quality of remote-sensing images. It is typically employed to compare the quality difference between two remote-sensing images. A lower ERGAS value indicates a smaller difference between the reconstructed image and the original image, indicating higher image quality. It is defined as follows:

$$\mathrm{ERGAS}(\widehat{\mathbf{Z}},\mathbf{Z})=\frac{100}{r}\sqrt{\frac{1}{S}\sum _{p=1}^S\frac{\mathrm{MSE}\left({\widehat{\mathbf{Z}}}_p,{\mathbf{Z}}_p\right)}{{\mu }_{{\widehat{\mathbf{Z}}}_p}^2}},$$

(18)

where r represents the magnification of the image; the value of r is set to eight in this paper. ${\mu }_{{\widehat{\mathbf{Z}}}_p}^2$ represents the square of the mean value of the pixel value in the p-th spectral dimension of the image.

4.3.5. UIQI

The UIQI is a metric used to assess the quality of an image by comparing it to a reference or original image. It measures the similarity between the two images in terms of three key factors: the correlation loss, brightness distortion, and contrast distortion. The UIQI is a mathematical metric and does not explicitly consider the human visual system. However, it has been found to be consistent with subjective quality assessments for a wide range of image distortions. It is defined as follows:

$$\mathrm{UIQI}(\widehat{\mathbf{Z}},\mathbf{Z})=\frac{1}{S}\sum _{p=1}^S\frac{4{\mu }_{{\widehat{\mathbf{Z}}}_p}{\mu }_{{\mathbf{Z}}_p}{\sigma }_{{\mathbf{Z}}_p}}{\left({\mu }_{{\widehat{\mathbf{Z}}}_p}^2+{\mu }_{{\mathbf{Z}}_p}^2\right)\left({\sigma }_{{\widehat{\mathbf{Z}}}_p}^2+{\sigma }_{{\mathbf{Z}}_p}^2\right)}.$$

(19)

The greater the UIQI value, the higher the quality of the corresponding image.

4.4. Quantitative Analysis

To demonstrate the effectiveness of our approach in this study, we conducted comprehensive experiments on two datasets. Furthermore, we compared our method with several state-of-the-art approaches, encompassing a variety of research methods, including matrix factorization-based LTTR [44], CNN-based MHF-Net [19], DBIN [45], ADMM-HFNet [20], Spf-Net [21], GuidedNet [46], and transformer-based Fusformer [24].

Table 1. Average quantitative results by all the compared methods on the CAVE testing set.

Title	PSNR↑	SSIM ↑	SAM↓	ERGAS↓	UIQI↑
LTTR [44]	41.20	0.980	4.25	1.90	0.984
MHFnet [19]	45.23	0.988	4.88	0.71	0.981
DBIN [45]	45.02	0.981	3.38	0.71	0.992
ADMM-HFNet [20]	45.48	0.992	3.39	0.71	0.992
SpfNet [21]	46.29	0.990	4.24	1.46	0.980
Fusformer [24]	42.18	0.993	3.07	1.25	0.992
GuidedNet [46]	45.41	0.991	4.03	0.97	-
Ours	46.56	0.994	2.92	0.64	0.995

The best values are highlighted, and the second-best values are underlined. ↑ signifies better performance as the corresponding metric values increase, while ↓ indicates enhanced performance with decreasing metric values.

provides an overview of the average quality metrics for our method and other methods on the CAVE dataset. It is evident that our method consistently claims the top position across all performance indicators. Specifically, in terms of the PSNR metric, we outperform the second-ranked method by 0.3 dB, signifying a more robust noise-suppression capability in our approach. Furthermore, in the SSIM metric, we also maintain a lead, indicating superior perceptual quality in the images generated by our method. In the realm of spectral recovery, our SAM score significantly surpasses that of the second-ranked method, resulting in minimal spectral distortion. Lastly, when considering the two comprehensive evaluation metrics, ERGAS and the UIQI, we maintain a substantial lead, underlining that the images restored by our method exhibit superior overall quality, closely resembling real-world images.

Similarly,

Table 2. Average quantitative results by all the compared methods on the Harvard testing set.

Title	PSNR↑	SSIM ↑	SAM↓	ERGAS↓	UIQI↑
LTTR [44]	40.06	0.999	4.69	1.29	0.993
MHFnet [19]	44.50	0.981	3.68	1.21	0.991
DBIN [45]	45.33	0.983	3.04	1.09	0.995
ADMM-HFNet [20]	45.53	0.983	3.04	1.08	0.995
SpfNet [21]	45.09	0.984	2.31	0.65	0.997
Fusformer [24]	41.96	0.995	3.33	2.86	0.995
GuidedNet [46]	41.64	0.981	2.85	1.20	-
Ours	47.86	0.995	2.25	0.75	0.997

The best values are highlighted, and the second-best values are underlined. ↑ signifies better performance as the corresponding metric values increase, while ↓ indicates enhanced performance with decreasing metric values.

presents the performance metrics for our method and other methods on the Harvard dataset, where our method also excels across various indicators. Although we rank slightly lower than the first-place method in the SSIM and ERGAS metrics, we still maintain a significant lead over the third-place method. In all other metrics, we significantly outperform other methods.

In summary, our method showcases an exceptional performance in image-quality restoration, as demonstrated through evaluations on the CAVE and Harvard datasets. Whether it is the noise reduction, perceptual quality, spectral recovery, or overall image-quality assessment, our method consistently emerges as the leader in these domains, underscoring its remarkable potential in the field of MHIF.

4.5. Qualitative Analysis

To better exhibit the effectiveness of our method, we conducted comparisons with four other four methods. In Figure 5, we chose three representative images from CAVE, each displaying its pseudo-RGB representation, utilizing the 3rd, 13th, and 2nd dimensions. The error maps below demonstrate their corresponding images; in the error map, a stronger blue color indicates that the image has fewer errors, while a stronger red color indicates that the image has more errors. From the figures, it is evident that MHF-net and Fusformer exhibit noticeable color deviations in their restored images, indicating significant spectral errors in the recovered HSI. This discrepancy is also apparent in their respective error maps when compared to the GT. LTTR performs better compared to the first two methods, yet still shows pronounced distortions in challenging areas, such as the colored spheres in the image. The SPF-net method yields relatively favorable results with overall minimal distortions and relatively realistic details.

In contrast, our method demonstrates the smallest difference when compared to the GT. In terms of color, our images exhibit no discernible differences, suggesting that our method preserves spectral information with minimal distortion. Furthermore, in terms of details, our method accurately restores the object structures, even in areas where other methods struggle to provide clear representations.

To provide a more comprehensive display of our restoration results, we conducted experiments by using the Harvard dataset. Figure 6 showcases the images restored by our method and the results of the other methods. It can be observed that MHF-net performs similarly to the CAVE dataset, with a better overall performance, but still exhibits notable spectral distortions in some images. The other three methods produce images with fewer distortions, although some deficiencies remain in certain details. Our method clearly outperforms the others both globally and locally, with hardly any evident errors in the error maps.

To showcase the superior performance of our method, we selected an image from both the CAVE and Harvard datasets. We then computed the PSNR values for each spectral dimension and visualized them in Figure 7. As Figure 7 demonstrates, our SMAformer method (depicted by the red curve) consistently outperforms other methods, clearly leading the pack. This indicates that our method not only excels in the overall image-recovery quality but also surpasses others in every spectral dimension.

Furthermore, to validate the practicality of our approach, we conducted comparative experiments on real remote-sensing imagery from Pavia Center, and the results are shown in Figure 8. The resulting image depicts a 216 × 216 pixel section situated on top of the test set. We assigned the 65th band to red, the 30th band to green, and the 15th band to blue, creating a pseudo-RGB representation. To highlight details, we applied a $3\times$ magnification factor to a specific area by using a red box within the image. Additionally, error maps were generated and displayed beneath the respective pseudo-RGB image.

After a thorough image analysis, it is evident that the other four methods indeed produced satisfactory results, effectively revitalizing the original visual fidelity. Nevertheless, upon meticulous scrutiny of the magnified image, it becomes apparent that certain fine details have not been fully restored. Common issues such as chromatic aberration and blurred edges persist in their results. In stark contrast, our method excels in the meticulous restoration of these intricate details. From the error map, it is evident that there is virtually no distinction between our method and the actual image. Our approach adeptly and precisely rejuvenates the true appearance, achieving the highest scores across various performance indicators.

In Figure 9, we present the index curves representing images generated by different methods alongside the GT in each spectral band. Figure 9a illustrates the PSNR values. It is evident that our method excels in lower spectral bands and still delivers a strong performance in higher dimensions, even though there is a slight decline. This demonstrates that images produced by our method exhibit reduced noise levels. Figure 9b showcases the performance based on the SSIM values. With the exception of the LTTR method, which exhibits significant fluctuations, the other methods maintain a relatively consistent performance. Figure 9c illustrates the performance of ERGAS curves across various dimensions. Our approach demonstrates excellent results in the middle and low bands, suggesting that the image quality restored by our method excels particularly in these bands. In Figure 9d, we observe the UIQI indicator. Our method lags slightly behind in the middle bands but consistently leads in other bands. This underlines the effectiveness of our approach in preserving the image quality across different spectral bands.

In summary, based on the performance on these three synthetic datasets, our method not only excels in terms of metrics but also produces more realistic images with richer detailed information.

4.6. Ablation Study

To validate the effectiveness of our proposed method, we conducted ablation studies by using the CAVE dataset.

Our primary module, the SMST, underwent a total of three repetitions. To determine the most appropriate number of repetitions, we conducted experiments, including one, two, three, and four repetitions. The results of experiments with different repetition numbers N on the CAVE dataset are presented in

Table 3. Average quantitative results by different N on the CAVE testing set.

N	PSNR↑	SSIM ↑	SAM↓	ERGAS↓	UIQI↑	Time↓
1	43.06	0.986	3.69	1.19	0.993	0.45
2	45.38	0.990	3.20	0.97	0.991	0.51
3	46.56	0.994	2.92	0.64	0.995	0.55
4	46.01	0.995	2.91	0.78	0.993	0.68

The best values are highlighted. ↑ signifies better performance as the corresponding metric values increase, while ↓ indicates enhanced performance with decreasing metric values.

. From the table, it can be observed that increasing the repetition number N to a certain extent enhances the performance of the model. The SSIM and SAM metrics reached their peak results at $N=4$. However, the PSNR, ERGAS, and UIQI metrics exhibited superior performance at $N=3$. Simultaneously, with an increase in N, the parameters of the model also increased, leading to a significant extension in the time required for image generation. Therefore, to strike a balance between the performance metrics and generation time, we opted for three repetitions, i.e., $N=3$. In this scenario, our model comprises 6.2 million learnable parameters and operates at a floating-point operations per second (FLOPS) rate of 330 billion. The average training duration for our model spans 22 h.

In this paper, we utilized two distinct loss functions, ${\mathcal{L}}_1$ loss and ${\mathcal{L}}_{ssim}$ loss, to optimize our model. To strike a balance between the effects of these two loss functions, we introduced a hyperparameter, denoted as $\lambda$. The purpose of this experiment is to systematically explore the impact of varying $\lambda$ values 1, 0.5, 0.1, and 0.01 on the performance of the model and determine the most effective balancing configuration. Our model was trained from scratch by using the CAVE dataset, and the final experimental results are presented in

Table 4. Average quantitative results by different $\lambda$ on the CAVE testing set.

$\mathit{\lambda }$	PSNR↑	SSIM ↑	SAM↓	ERGAS↓	UIQI↑
1	42.13	0.920	4.12	1.67	0.986
$0.5$	44.25	0.986	3.49	1.03	0.990
$0.1$	46.56	0.994	2.92	0.64	0.995
$0.01$	45.21	0.992	3.03	0.66	0.994

The best values are highlighted. ↑ signifies better performance as the corresponding metric values increase, while ↓ indicates enhanced performance with decreasing metric values.

. Based on the data presented in the table, it is evident that larger values of $\lambda$, such as 1 and $0.5$, lead to convergence issues and adversely impact the final performance of the model. Conversely, when $\lambda$ is set to a smaller value, such as 0.01, although acceptable results are achieved, the distinctive impact of the ${\mathcal{L}}_{ssim}$ loss is not effectively demonstrated. Consequently, we carefully selected the optimal value of 0.1. This choice strikes a balance, avoiding convergence disruptions while still allowing the ${\mathcal{L}}_{ssim}$ loss to exert its intended influence on the performance of the model.

We proposed two modules: the SMAB and SSMAB. The SMAB leverages the flexibility of the transformer to establish self-generated and mutual correlations between HR-MSIs and LR-HSIs within the module. The SSMAB builds upon the SMAB by incorporating a sparse-attention module to better accommodate the spatial sparse characteristics of HSIs. To assess the effectiveness of our proposed modules, we conducted experiments. We replaced the SMAB with residual convolutional blocks [47] in the original module to obtain the w/o SMAB method, and we replaced the SSMAB with the SMAB to create the w/o SSMAB method. While keeping the other parameters constant, the final results are presented in

Table 5. Average quantitative results by different modules on the CAVE testing set.

Method	PSNR↑	SSIM ↑	SAM↓	ERGAS↓	UIQI↑
w/o SMAB	44.38	0.991	3.24	0.96	0.990
w/o SSMAB	45.02	0.990	2.96	0.77	0.992
Ours	46.56	0.994	2.92	0.64	0.995

The best values are highlighted. ↑ signifies better performance as the corresponding metric values increase, while ↓ indicates enhanced performance with decreasing metric values.

.

Table 5 reveals that substituting the SMAB with residual convolutional blocks led to a notable drop in PSNR by 2.18 dB. This suggests that our module excels in extracting crucial information from both images compared to its predecessors. When the SSMAB was omitted, the PSNR decreased by 1.54 dB, underscoring the idea that sparsity is better suited for handling HSIs.

5. Conclusions

In this paper, we proposed a novel model for the MHIF task called SMAformer. Different from previous fusion methods, we focused on fusion super-resolution tasks and gave full play to the flexibility of the transformer model. We designed a multistage fusion module that can effectively capture the information between space and the spectrum to improve the fusion effect. In addition, considering the characteristics of HSIs, we performed sparse processing inside the model to make the generated images more realistic. Extensive experimental results on three benchmark datasets demonstrated that our SMAformer surpassed the state-of-the-art methods.

Although we made significant progress in our current research, there are still challenges that need to be addressed. Firstly, supervised training requires a large number of paired high- and low-resolution images, which are not easily obtainable in real-world scenarios, especially in specific domains or applications. This limitation hinders the universality of our model. To tackle this dilemma, we plan to shift our research focus towards unsupervised MHIF tasks. This approach does not rely on precise paired data but instead depends on the self-learning of the model to enhance resolution. Research in this direction holds a distinct advantage in handling large-scale datasets lacking GT in real-world situations.

In our future work, we will also explore and incorporate advanced Generative Adversarial Networks (GANs) [48] and self-supervised learning techniques to improve the performance of the model in unsupervised environments. These methods can assist us in better capturing latent structures and features within images, thereby enhancing the effectiveness of the MHIF task.

Furthermore, we intend to optimize the generalization ability of the model to adapt to the HSI SR requirements in different environments and scenarios. By delving into the exploration of distinctive features in various domains, we aim to make the model more adaptable and versatile, better addressing challenges in practical applications.