Hyperspectral Image Super-Resolution via Adaptive Factor Group Sparsity Regularization-Based Subspace Representation

Abstract

Hyperspectral image (HSI) super-resolution is a vital technique that generates high spatial-resolution HSI (HR-HSI) by integrating information from low spatial-resolution HSI with high spatial-resolution multispectral image (MSI). However, existing subspace representation-based methods face challenges, including adaptive subspace dimension determination, inadequate spectral correlation capture, and expensive computation. In this paper, we propose a novel factor group sparsity regularized subspace representation (FGSSR)-based method for HSI super-resolution that can simultaneously address these issues encountered in previous methods. Specifically, by incorporating the factor group sparsity regularization into the subspace representation model, we first propose an FGSSR model to capture the spectral correlation property of the HR-HSI. The key advantage of FGSSR lies in its equivalence to the Schatten-p norm and its adaptive determination of the accurate subspace dimension, enabling it to capture spectral correlation more effectively. To preserve the spatial self-similarity prior in the HR-HSI, the tensor nuclear norm regularization on the low-dimensional coefficients is also incorporated into the proposed FGSSR-based model. Finally, an effective proximal alternating minimization-based algorithm is developed to solve the FGSSR-based model. Experimental results on the simulated and real datasets demonstrate that the proposed FGSSR-based method outperforms several state-of-the-art fusion methods with significant improvements.

1. Introduction

Hyperspectral (HS) imaging is an advanced technology that enables the acquisition of a high-dimensional image dataset encompassing numerous diverse bands. Due to their rich spectral information, HS images (HSIs) have become extensively utilized for numerous earth observation tasks, such as image classification [1,2], ecosystem monitoring [3], and change detection [4]. Nevertheless, HS imaging sensors struggle to balance their spatial and spectral resolutions due to finite sun irradiation [5,6,7]. As a result, HSIs often prioritize spectral resolution over spatial resolution, restricting their applications in the study of land surface dynamics occurring in small areas [8,9,10]. Fortunately, multispectral (MS) imaging sensors generally outperform HS imaging sensors in terms of spatial resolution. This provides an opportunity to perform HSI super-resolution by integrating complementary information of the high spatial-resolution MS images (HR-MSIs) and the low spatial-resolution HSIs (LR-HSIs) captured from the same scene, resulting in the generation of the desired HR-HSIs. Current HSI super-resolution approaches are primarily categorized into three groups [11]: deep learning (DL)-based methods [12,13,14], matrix factorization-based methods [15,16], tensor-based methods [17,18,19].

The remarkable learning performance and effective computing capabilities of DL contributed to the development of DL-based HSI super-resolution techniques [13,20]. For example, the work in [21] designed a deep residual learning network for HSI super-resolution, which effectively learns the spectral prior by using the deep convolution neural network (CNN). The work in [14] presented a simple and efficient CNN architecture that can maintain the spatial and spectral specifics of the observations. The main benefit of these techniques lies in their ability to achieve satisfactory results by leveraging the powerful feature capturing capabilities of CNNs. However, it is worth noting that these methods are supervised and rely on a substantial amount of labeled samples to pretrain the CNN. In order to tackle this matter, numerous scholars have put forth unsupervised deep learning-based approaches [22,23]. For example, in work [22], an unsupervised and non-pretraining encoder–decoder architecture for HSI super-resolution was designed. However, these unsupervised methods may produce slightly worse results than the supervised methods.

Matrix factorization-based methods approximate the latent HR-HSI using a dictionary and its corresponding coefficients, assuming that every spectral component is able to be expressed as a composition of a small set of distinct spectral patterns. Within this theoretical framework, the primary focus centers around the precise estimation of both the dictionary and its accompanying coefficients. Several methods [24,25,26] estimated the dictionary and its corresponding coefficients using certain predetermined priors. For instance, Xue et al. [26] exploited the subspace low-rank priors in both spatial and spectral dimensions to accomplish HSI super-resolution. In addition, previous studies [27,28] successfully addressed the HSI super-resolution problem through the utilization of spatial structures inherent in HR-HSI. For example, Simoes et al. [27] attempted to capture the spatial self-similarity features through the total variation regularization. However, fusion approaches based on matrix factorization encounter difficulties in fully leveraging the spatial and spectral correlations of the HR-HSI due to their approach of unfolding the high-dimensional HSI into matrices without considering its underlying data structure. As a result, those methods often yield unsatisfactory fusion outcomes [29,30].

Given that HSI is a three-order tensor, methods based on tensor theory have gained significant attention in addressing the HSI super-resolution problem [31]. These methods address the HSI super-resolution problem using various tensor decompositions, including tensor ring decomposition [17,18,19,31], tensor train decomposition [9], Canonical polyadic decomposition [32,33,34], and Tucker decomposition [8,17,29,30,35,36,37], while incorporating specific priors. For example, Dian et al. [35] developed an HSI super-resolution approach via Tucker decomposition, which reconstructs the latent HR-HSI by utilizing dictionary learning and sparse coding. He et al. [18] developed a coupled tensor ring factorization-based approach that can simultaneously learn tensor ring core tensors from the observations for reconstructing the latent HR-HSI. Although these tensor decomposition-based methods have yielded promising performances, they still encounter certain challenges and limitations. One of the challenges is the high computational cost associated with representing the strong spectral and spatial correlations in the original HSI domain through tensor decompositions, particularly when dealing with a large number of bands in HSI [6].

The spectral bands in the HR-HSI exhibit a strong correlation and typically reside inside a subspace of reduced dimensionality [38]. This implies that HSI super-resolution is able to be modeled by projecting the high-dimensional HR-HSI into a lower-dimensional subspace. This strategy can largely decrease the computing complexity and enhance the spectral correlation property [6]. Recently, many HSI super-resolution methods [6,38,39,40] have been proposed through subspace representation. Xu et al. [6] exploited a tensor subspace representation to effectively leverage the nonlocal and global spectral–spatial low-rank priors to accomplish HSI super-resolution. Xing et al. [39] presented a fusion approach based on variational tensor subspace decomposition. This method effectively captures the distinctions and interrelationships among the three modes of the HR-HSI. To exploit both the global and nonlocal priors, Xu et al. [40] developed a tensor ring decomposition-based approach that blends spectral subspace learning, nonlocal similarity learning, and the tensor nuclear norm. Compared with matrix/tensor decomposition-based methods, subspace representation-based methods project the high-dimensional HR-HSI into the low-dimensional spectral subspace, consuming less computational complexity and generating better results. However, the current subspace representation-based methods often encounter challenges in adaptively selecting the appropriate subspace dimension, making them highly sensitive to changes in subspace dimension and affecting their overall performance.

To tackle the aforementioned concerns, this study develops a factor group sparsity regularized subspace representation-based method for HSI super-resolution. Specifically, we first propose a factor group sparsity regularized subspace representation (FGSSR) model, which automatically selects the appropriate subspace dimension, resulting in a precise depiction of the high spectral correlations in the latent HR-HSI. The proposed FGSSR model has several advantages that can be summarized as follows: (1) FGSSR is established based on subspace representation framework, resulting in an optimization process with low computational cost. (2) FGSSR is rank-revealing by seeking to minimize the number of nonzero frontal slices of the coefficient tensor. Thus, it provides a solution to the challenge of adaptively selecting the appropriate subspace dimension faced by previous subspace representation methods. (3) FGSSR corresponds to the Schatten-p norm, allowing it the selection of a subspace dimension that is closer to the true rank of the latent HR-HSI compared to the nuclear norm. Under the FGSSR framework, the depiction of the spatial structure prior in the high-dimensional HR-HSI domain can be translated into an exploration of the low-dimensional coefficient domain. Driven by the efficacy of the tensor nuclear norm (TNN) [41], we impose it on the low-dimensional coefficient tensor to capture and retain the spatial information inherent in the latent HR-HSI. Furthermore, we design an effective optimization algorithm under the proximal alternating minimization (PAM) framework to address our FGSSR-based HSI super-resolution model. This study makes several notable contributions in the following areas:

• We propose an FGSSR-based HSI super-resolution method that overcomes the sensibility of the subspace dimension determination of earlier subspace representation-based methods. Specifically, by incorporating factor group sparsity regularization into subspace representation, FGSSR becomes rank-revealing, enabling it to adaptively and accurately approximate the subspace dimension to the rank of the latent HR-HSI.
• The FGSSR-based method utilizes the FGSSR model to explore spectral correlation and employs TNN regularization to capture spatial self-similarity in the latent HR-HSI. FGSSR can adaptively and effectively capture the high spectral correlation by fully leveraging the advantages of subspace representation and the Schatten-p norm while mitigating their limitations. TNN regularization on the low-dimensional coefficients can effectively capture the spatial self-similarity and facilitate efficient computations through dimensionality reduction.
• An effective PAM-based algorithm is developed with the aim of addressing the FGSSR-based model. Extensive studies carried out with simulated and real-world datasets reveal that our FGSSR-based method outperforms the state-of-the-art (SOTA) HSI super-resolution methods in both quantitative and visual judgements.

The ensuing section of this paper is structured as follows: Section 2 introduces the related work. Section 3 presents the FGSSR-based method, outlining its key features and methodology. Section 4 and Section 5 present the experimental results and model discussions, respectively. Finally, Section 6 concludes this paper and discusses the potential future research directions.

2. Related Work

2.1. Notations

We let $\mathcal{M}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ be a three-dimensional tensor. $\mathcal{M}(:,:,i)$ is the ith frontal slice of $\mathcal{M}$, denoted ${\mathbf{M}}^{\left(i\right)}$. Mode-n unfolding operator of $\mathcal{M}$ is given by ${\mathbf{M}}_{\left(n\right)}={\mathtt{Unfold}}_n\left(\mathcal{M}\right)$, and $\mathcal{M}$ can be retrieved by reversing this process as $\mathcal{M}={\mathtt{fold}}_n\left({\mathbf{M}}_{\left(n\right)}\right)$. The ${\mathsf{\ell }}_1$ norm of $\mathcal{M}$ is represented as ${\parallel \mathcal{M}\parallel }_1={\sum }_{ijk}\left|{\mathcal{M}}_{ijk}\right|$, the Frobenius norm is represented as ${\parallel \mathcal{M}\parallel }_F=\sqrt{{\sum }_{ijk}{\left|{\mathcal{M}}_{ijk}\right|}^2}$, and  the ${\mathsf{\ell }}_{2,1}$ norm is denoted by ${\parallel \mathcal{M}\parallel }_{2,1}={\sum }_i{\parallel {\mathbf{M}}^{\left(i\right)}\parallel }_F$. $\overline{\mathcal{M}}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ is the outcome of performing the fast Fourier transformation (FFT) on $\mathcal{M}$ along the third dimension, denoted as $\overline{\mathcal{M}}=\mathcal{F}\left(\mathcal{M}\right)$. Likewise, $\mathcal{M}$ can be retrieved through inverse FFT, denoted as $\mathcal{M}={\mathcal{F}}^{-1}\left(\overline{\mathcal{M}}\right)$. Additionally, we define a pair of unfolding–folding operators, that is,

$$\mathtt{Unfold}\left(\mathbf{M}\right)=\left[\begin{array}{c}{\mathbf{M}}^{\left(1\right)}\\ {\mathbf{M}}^{\left(2\right)}\\ ⋮\\ {\mathbf{M}}^{\left(I_3\right)}\end{array}\right],\mathtt{fold}\left(\mathtt{Unfold}\left(\mathcal{M}\right)\right)=\mathcal{M}.$$

(1)

Definition 1

(Tensor Product [42]). The tensor product of tensors $\mathcal{M}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ and $\mathcal{N}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ is given by

$$\mathcal{M}\ast \mathcal{N}=\mathtt{fold}\left(\mathtt{bcirc}\right(\mathcal{M})·\mathtt{Unfold}(\mathcal{N}\left)\right).$$

(2)

Definition 2

(Orthogonal Tensor [42]). An orthogonal tensor $\mathcal{S}\in {\mathbb{R}}^{n\times n\times I_3}$ is a special type of tensor that satisfies the following conditions: ${\mathcal{S}}^T\ast \mathcal{S}=\mathcal{S}\ast {\mathcal{S}}^T=\mathcal{I}$, where $\mathcal{I}$ and ${\mathcal{S}}^T$ are the identity tensor and the conjugate transpose of $\mathcal{S}$, respectively.

Theorem 1

(Tensor Singular Value Decomposition (t-SVD) [42]). t-SVD decomposes $\mathcal{M}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ as

$$\mathcal{M}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^T,$$

(3)

where $\mathcal{U}\in {\mathbb{R}}^{I_1\times I_1\times I_3}$ and $\mathcal{V}\in {\mathbb{R}}^{I_2\times I_2\times I_3}$ are orthogonal, and $\mathcal{S}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ is f-diagonal.

Definition 3

(TNN [41]). Let tensor $\mathcal{M}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$, with its t-SVD given by $\mathcal{M}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^T$. The TNN of tensor $\mathcal{M}$, denoted as ${\parallel \mathcal{M}\parallel }_{\ast }$, is the aggregate of the singular values of all frontal slice of tensor $\mathcal{S}$, that is,

$${\parallel \mathcal{M}\parallel }_{\ast }=\mathrm{tr}\left({\overline{\mathit{S}}}^{\left(1\right)}\right)={\parallel {\overline{\mathit{M}}}^{\left(1\right)}\parallel }_{\ast }$$

(4)

where ${\overline{\mathit{S}}}^{\left(1\right)}$ and ${\overline{\mathit{M}}}^{\left(1\right)}$ are the block diagonal matrices of $\mathcal{M}$ and $\mathcal{S}$, respectively. Following Definition 3, the TNN can be accomplished efficiently by employing a single matrix SVD, as opposed to the use of the complex t-SVD method for obtaining $\mathcal{S}$.

Definition 4

(Singular Value Thresholding (SVT) [41]). The SVT operator performed on tensor $\mathcal{M}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$ is given by

$$\left\{\begin{array}{c}{\mathcal{D}}_{\tau }\left(\mathcal{M}\right)=\mathcal{U}\ast {\mathcal{D}}_{\tau }\left(\mathcal{S}\right)\ast {\mathcal{V}}^T,{\mathcal{D}}_{\tau }\left(\mathcal{S}\right)={\mathcal{F}}^{(-1)}\left({\mathcal{D}}_{\tau }\left(\overline{\mathcal{S}}\right)\right),\hfill \\ {\mathcal{D}}_{\tau }\left({\overline{\mathit{S}}}^{\left(i\right)}\right)=\mathrm{diag}(\max {\{{\sigma }_j^{\left(i\right)}-\tau \}}_{1\le j\le r}),i=1,2,\cdots ,I_3,\hfill \\ \mathcal{M}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^T,\hfill \end{array}\right.$$

(5)

where ${\mathcal{D}}_{\tau }$ denotes the SVT operator, and τ is a positive scalar.

Theorem 2

([43]). For any $\mathcal{M}\in {\mathbb{R}}^{I_1\times I_2\times I_3}$, the following problem:

$$\underset{\mathcal{M}}{\arg \min }{\tau \parallel \mathcal{M}\parallel }_{\ast }+\frac{1}{2}{\parallel \mathcal{M}-\mathcal{B}\parallel }_F^2$$

(6)

has a closed-form solution $\mathcal{M}={\mathcal{D}}_{\tau }\left(\mathcal{B}\right)$.

Definition 5

(Schatten-p Norm [44]). For any matrix $\mathit{X}\in {\mathbb{R}}^{m\times n}$, its Schatten-p norm ($0<p<1$) can be defined as

$${\parallel \mathit{X}\parallel }_{S_p}={\left(\sum _{i=1}^{min(m,n)}{\sigma }_i^p\left(\mathit{X}\right)\right)}^{1/p},$$

(7)

where ${\sigma }_i$ is the ith singular value of $\mathit{X}$. In special cases at the boundaries, the Schatten-p norm simplifies into the nuclear norm when $p=1$, and it reduces to the rank function when p is 0. When $0<p<1$, the Schatten-p norm can be seen as a rank approximation lying between rank function and nuclear norm.

Theorem 3

([45]). For any matrix $\mathit{X}\in {\mathbb{R}}^{m\times n}$ having the rank $r\le d\le min(m,n)$, it can be decomposed into two low-dimensional matrices $\mathit{U}\in {\mathbb{R}}^{m\times d}$ and $\mathit{V}\in {\mathbb{R}}^{d\times n}$,

$$\begin{array}{cc}\hfill {\parallel \mathit{X}\parallel }_{{\mathit{S}}_{1/2}}^{1/2}& =\sum _{i=1}^r{\sigma }_i^{1/2}\left(\mathit{X}\right)=\frac{1}{2}\underset{\mathit{X}=\mathit{U}\mathit{V}}{min}{(\parallel \mathit{U}\parallel }_{2,1}+\parallel {\mathit{V}}^T{\parallel }_{2,1}),\hfill \end{array}$$

(8)

where ${\parallel ·\parallel }_{2,1}$ is the ${\mathsf{\ell }}_{2,1}$ norm, ${\parallel \mathit{X}\parallel }_{{\mathit{S}}_{1/2}}^{1/2}$ denotes the Schatten-p norm of $\mathit{X}$ with $p=\frac{1}{2}$, and ${\sigma }_i\left(\mathit{X}\right)$ denotes the ith singular of $\mathit{X}$. Let the SVD of $\mathit{X}$ denote $\mathit{X}={\mathit{U}}_x{\Sigma }_x{\mathit{V}}_x^T$. When $\mathit{U}={\mathit{U}}_x{\Sigma }_x^{1/2}$ and $\mathit{V}={\Sigma }_x^{1/2}{\mathit{V}}_x^T$, equality holds in Equation (8). Here, the low-rank approximation of (8) is called the factor group sparsity (FGS) regularizer.

2.2. Problem Formulation

(1) Observation Model: $\overline{\mathcal{Y}}\in {\mathbb{R}}^{\mathit{w}\times \mathit{h}\times \mathit{S}}$ is the observed LR-HSI, which contains a wealth of spectral information. Current HSI super-resolution techniques mainly rely on the assumption that the LR-HSI $\overline{\mathcal{Y}}$ is the spatially degraded version of the latent HR-HSI $\mathcal{Z}\in {\mathbb{R}}^{\mathit{W}\times \mathit{H}\times \mathit{S}}$  [39,46,47]. However, this assumption may not be totally justifiable due to the fact that the blur kernel is often unknown in real-world scenarios. To eliminate the need for an empirical blur kernel, this paper employs interpolation technology to upsample the observed LR-HSI, resulting in an upsampled LR-HSI $\mathcal{Y}\in {\mathbb{R}}^{W\times H\times S}$, which can be represented as the combination of the latent HR-HSI $\mathcal{Z}$ and the difference image $\mathcal{S}\in {\mathbb{R}}^{W\times H\times S}$, that is,

$$\mathcal{Y}=\mathcal{Z}+\mathcal{S}+{\mathcal{N}}_1,$$

(9)

where ${\mathcal{N}}_1$ denotes the noise within the upsampled LR-HSI.

The observed HR-MSI $\mathcal{X}\in {\mathbb{R}}^{W\times H\times s}$ contains a rich amount of spatial structure information. It can be considered the spectrally degraded form of the latent HR-HSI $\mathcal{Z}$, i.e.,

$$\mathcal{X}=\mathcal{Z}{\times }_3\mathbf{P}+{\mathcal{N}}_2,$$

(10)

where $\mathbf{P}\in {\mathbb{R}}^{s\times S}$ and ${\mathcal{N}}_2$ represent the spectral response and noise in the observed HR-MSI, respectively.

(2) HSI Super-resolution Model: Indeed, directly reconstructing the latent HR-HSI $\mathcal{Z}$ from the observations is ill-posed. To achieve a robust reconstruction of the desired HR-HSI, we can employ the maximum a posteriori approach to incorporate regularization models and convert the ill-posed issue into a well-posed one. In addition, our previous work [48] has demonstrated that the gradients along the horizontal, vertical, and spectral modes of the difference image $\mathcal{S}$ follow hyper-Laplacian distributions which can be depicted by ${\mathsf{\ell }}_{1/2}$ norm. Therefore, we impose an ${\mathsf{\ell }}_{1/2}$ norm on the gradients of $\mathcal{S}$ to construct the spectral fidelity term. Therefore, HSI super-resolution model is given by

$$\begin{array}{c}\hfill \underset{\mathcal{Z},\mathcal{S}}{min}\frac{\alpha }{2}{\parallel \mathcal{Y}-\mathcal{Z}-\mathcal{S}\parallel }_{\mathit{F}}^2+\eta \sum _{n=1}^3\parallel {\nabla }_n{\mathcal{S}\parallel }_{1/2}+\frac{\beta }{2}{\parallel \mathcal{X}-\mathcal{Z}{\times }_3\mathbf{P}\parallel }_{\mathit{F}}^2+w\varphi \left(\mathcal{Z}\right),\hfill \end{array}$$

(11)

where the combination of the first two terms is the spectral fidelity term, and the third term is the spatial fidelity term. $\varphi \left(\mathcal{Z}\right)$ is a regularizer that depicts the potential property of the HR-HSI. ${\parallel ·\parallel }_{1/2}$ is the ${\mathsf{\ell }}_{1/2}$ norm, and ${\nabla }_n$ represents the gradient operators along the mode-n. $\alpha$, $\beta$, $\eta$ and w are positive parameters used to achieve a balance between the functions involved in the fusion process. The ${\mathsf{\ell }}_{1/2}$ norm-based spectral fidelity term can skip the empirical blur kernel and reduce the computing complexity.

3. Proposed FGSSR-Based Method

In this section, we present an FGSSR-based method for fusing the LR-HSI and HR-MSI to generate the target HR-HSI. In this method, we first propose a FGSSR model that combines the subspace learning of a low-dimensional spectral subspace with the imposition of group sparsity regularization on a low-dimensional coefficient tensor to depict the high spectral correlations of HS-HSI. The proposed FGSSR model offers several advantages, including enhanced rank minimization, automatic subspace dimension selection, and low computational cost. These advantages allow us adaptive and effective capture of the high spectral correlations. Furthermore, since each frontal slice of the coefficient tensor can be regarded as the eigen-image [38], we also introduce TNN regularization on the low-dimensional coefficient tensor to depict the spatial self-similarity prior, which offers high efficiency while successfully maintaining the spatial features. The flowchart of the proposed FGSSR-based method is demonstrated in Figure 1.

3.1. Subspace Learning

For the latent HR-HSI, there is typically a significant correlation among spectral vectors. This indicates that spectral vectors in the latent HR-HSI often reside within a low-dimensional subspace [38,40,49]. Hence, the latent HR-HSI can be approximated by adopting a subspace representation strategy, that is,

$$\mathcal{Z}=\mathcal{B}{\times }_3\mathbf{A},$$

(12)

where $\mathbf{A}\in {\mathbb{R}}^{\mathit{S}\times \mathit{d}}$ and $\mathcal{B}\in {\mathbb{R}}^{\mathit{W}\times \mathit{H}\times \mathit{d}}$ represent the spectral subspace and its coefficients, respectively. Notably, the subspace dimension $\mathit{d}$ is significantly smaller than $\mathit{S}$, which indicates that the spectral bands reside inside a low-dimensional subspace. This reduction in dimensionality drastically improves computational efficiency. However, the effectiveness of Subspace representation model (12) heavily relies on accurately determining the appropriate subspace dimension $\mathit{d}$, namely the number of the frontal slices of $\mathcal{B}$. Therefore, the subspace dimension $\mathit{d}$ should be precisely chosen to approximate the rank of ${\mathbf{Z}}_{\left(3\right)}$ (i.e., the unfolding matrix of $\mathcal{Z}$ along the 3rd dimension) as closely as possible. Motivated by the FGS regularizer (8), we define a factor group sparsity regularized subspace representation (FGSSR) model (see Figure 2) for HSI super-resolution, that is,

$$\underset{\mathcal{Z}=\mathcal{B}{\times }_3\mathbf{A}}{min}\frac{1}{2}{(\parallel \mathbf{A}\parallel }_{2,1}+{\parallel \mathcal{B}\parallel }_{2,1}).$$

(13)

Based on Theorem 3, FGSSR is equivalent to the Schatten-p norm $\parallel {\mathbf{Z}}_{\left(3\right)}{\parallel }_{{\mathit{S}}_{1/2}}^{1/2}$ when $\mathbf{A}={\mathbf{U}}_z{\Sigma }_z^{1/2}$ and ${\mathbf{B}}_{\left(3\right)}={\Sigma }_z^{1/2}{\mathbf{V}}_z^T$, where ${\mathbf{B}}_{\left(3\right)}$ is the unfolding matrix of $\mathcal{B}$ along the third dimension, ${\mathbf{U}}_z$, ${\Sigma }_z$, and ${\mathbf{V}}_z^T$ come from the SVD of ${\mathbf{Z}}_{\left(3\right)}$, i.e., ${\mathbf{Z}}_{\left(3\right)}={\mathbf{U}}_z{\Sigma }_z{\mathbf{V}}_z^T$. From Observation model (9), the upsampled LR-HSI consists of three components: the latent HR-HSI, the difference image, and noise. This suggests that a significant portion of spectral information of the latent HR-HSI is preserved in the upsampled LR-HSI. Consequently, it can be inferred that both LR-HSI and HR-HSI are situated inside the identical spectral subspace. As a result, the low-dimensional spectral subspace $\mathbf{A}$ can be learned by SVD and Theorem 3 from the upsampled LR-HSI. We represent the upsampled LR-HSI using its SVD as follows:

$$\begin{array}{c}\hfill {\mathbf{Y}}_{\left(3\right)}={\mathbf{U}}_y{\Sigma }_y{\mathbf{V}}_y^T,\end{array}$$

(14)

where ${\mathbf{U}}_y\in {\mathbb{R}}^{\mathit{S}\times \mathit{S}}$ and ${\mathbf{V}}_y\in {\mathbb{R}}^{\mathit{WH}\times \mathit{WH}}$ are semiunitary, and ${\Sigma }_y\in {\mathbb{R}}^{\mathit{S}\times \mathit{WH}}$ is a diagonal matrix comprising the singular values sorted in a non-increasing manner. By saving the $\mathit{d}$ largest singular values and deleting the remaining ($\mathit{S}-\mathit{d}$) smallest ones from ${\Sigma }_y$, a low-rank approximation of ${\mathbf{Y}}_{\left(3\right)}$ can be given by

$$\begin{array}{c}\hfill {\widehat{\mathbf{Y}}}_{\left(3\right)}={\mathbf{U}}_y(:,1:\mathit{d}){\Sigma }_y(1:\mathit{d},1:d){\mathbf{V}}_y{(:,1:d)}^T.\end{array}$$

(15)

Based on Theorem 3, the spectral subspace $\mathbf{A}$ can be achieved by

$$\mathbf{A}={\mathbf{U}}_y(:,1:d){\Sigma }_y{(1:d,1:d)}^{1/2}.$$

(16)

Compared to Direct subspace representation model (12), the FGSSR model applies the group sparsity regularization to the coefficients $\mathcal{B}$, resulting in certain frontal slices in $\mathcal{B}$ being quickly forced to zero during the iterative process. This allows the FGSSR model automatic reduction in the number of non-zero frontal slices and dynamic approximation of the rank $\mathit{r}$ of ${\mathbf{Z}}_{\left(3\right)}$. Regarding to the spectral subspace $\mathbf{A}$, we make two remarks: (1) The spectral subspace $\mathbf{A}$ is dynamic since its subspace dimension $\mathit{d}$ reduces with the reduction in the number of nonzero frontal slices of $\mathcal{B}$ in the iterative optimization process. (2) As the iteration progresses, the FGSSR model dynamically reduces the number of columns in $\mathbf{A}$, allowing it the consideration as the spectral basis matrix of $\mathcal{Z}$. This leads to $\mathcal{B}$ inheriting the potential property of the latent HR-HSI $\mathcal{Z}$. Therefore, the spatial property in the HR-HSI can be captured and modeled in its low-dimensional coefficient domain, which can reduce the computational complexity.

To sum up, the FGSSR model has the following advantages:

• Enhanced Rank Minimization: The FGSSR model corresponds to the Schatten-p norm, which enables it to provide a more precise approximation to rank minimization compared to the nuclear norm.
• Reduced Computational Complexity: Unlike other rank minimization problems that require SVD calculation at each iteration, the FGSSR model can be effectively solved through the soft-threshold shrinkage operator or tiny linear equations, resulting in enhanced computational efficiency. In addition, the FGSSR model employs the subspace learning strategy to learn the spectral subspace $\mathbf{A}$, which can also reduce the computational complexity.
• Automatic Subspace Dimension Selection: Compared to the direct subspace representation model (12), the FGSSR model (13) imposes the group sparsity regularization on the coefficients $\mathcal{B}$. This leads to the rapid elimination of specific frontal slices in $\mathcal{B}$ during the iterative optimization process, automatically reducing the number of non-zero frontal slices (i.e., subspace dimension d), and dynamically approximating the rank r of ${\mathbf{Z}}_{\left(3\right)}$.

3.2. Adaptive FGSSR-Based HSI Super-Resolution Model

By incorporating FGSSR model (13) into Model (11), we have

$$\begin{array}{c}\underset{\mathcal{B},\mathcal{S}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\eta \sum _{n=1}^3{\parallel {\nabla }_n\mathcal{S}\parallel }_{1/2}\hfill \\ \hfill \hspace{1em}\hspace{1em} +\frac{\beta }{2}\parallel \mathcal{X}-\mathcal{B}{\times }_3{\mathbf{P}\mathbf{A}\parallel }_F^2+\frac{1}{2}{(\parallel \mathbf{A}\parallel }_{2,1}+{\parallel \mathcal{B}\parallel }_{2,1})+w\varphi \left(\mathcal{B}\right),\hfill \end{array}$$

(17)

where $\frac{1}{2}{(\parallel \mathbf{A}\parallel }_{2,1}+{\parallel \mathcal{B}\parallel }_{2,1})$ is the FGSSR model, which serves to depict the high spectral correlations of the target HR-HSI $\mathcal{Z}$. $\varphi \left(\mathcal{B}\right)$ is a regularization term, which serves to capture the self-similarity property of the target HR-HSI $\mathcal{Z}$. The idea of applying the TNN-based regularization to the low-dimensional coefficient tensor $\mathcal{B}$ to capture the nonlocal self-similarity has been successfully used for HSI super-resolution  [16,38,39], but often demands significant computational resources [50]. In this paper, the TNN is employed to capture the global spatial self-similarity property of the latent HR-HSI by directly performing on the low-dimensional coefficient tensor $\mathcal{B}$. This strategy offers a reduction in computational complexity while effectively preserving the image details. Moreover, through the subspace learning strategy, the spectral subspace $\mathbf{A}$ is essentially known. Therefore, the proposed FGSSR-based HSI super-resolution model can ultimately be expressed as follows:

$$\begin{array}{c}\hfill \underset{\mathcal{B},\mathcal{S}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\eta \sum _{n=1}^3{\parallel {\nabla }_n\mathcal{S}\parallel }_{1/2}\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\frac{\beta }{2}\parallel \mathcal{X}-\mathcal{B}{\times }_3{\mathbf{P}\mathbf{A}\parallel }_F^2+\frac{1}{2}{\parallel \mathcal{B}\parallel }_{2,1}+w{\parallel \mathcal{B}\parallel }_{\ast },\hfill \end{array}$$

(18)

where ${\parallel \mathcal{B}\parallel }_{\ast }$, defined in (4), is the TNN regularization term.

In our proposed method, the spectral correlation property is adaptively and effectively depicted by the FGSSR model, and the spatial self-similarity property is effectively captured by imposing the TNN regularization on the low-dimensional coefficients $\mathcal{B}$. Specifically, we first propose the FGSSR model by combining the subspace learning of subspace $\mathbf{A}$ with the imposition of group sparsity regularization on coefficients $\mathcal{B}$. The proposed FGSSR model offers several advantages, including enhanced rank minimization, automatic subspace dimension selection, and low computational cost. These advantages allow us adaptive and effective capture of the high spectral correlations. Furthermore, we introduce TNN regularization on the low-dimensional coefficients $\mathcal{B}$ to depict the spatial self-similarity prior, which offers high efficiency while successfully maintaining the spatial features. Next, we design an efficient PAM-based algorithm for the proposed FGSSR-based HSI super-resolution model.

3.3. Optimization Algorithm

In this section, we employ the PAM framework [51] to address our Fusion model (18). We choose the PAM framework for two reasons. Firstly, the PAM framework has a proven ability that can guarantee convergence to a critical point given specific criteria. Secondly, the PAM framework provides us with an opportunity to update subspace dimension d at each PAM iteration. It should be noted that $\mathbf{A}$ is initialized using the subspace learning strategy and subsequently updated only according to the subspace dimension d, which is determined by applying the group sparsity regularization to the coefficients $\mathcal{B}$ during the iterative optimization process. In other words, $\mathbf{A}$ is known with the subspace dimension d determined in each PAM iteration. As a result, solving Model (18) can be changed into an alternative optimization of the following subproblems:

$$\left\{\begin{array}{c}\mathcal{B}=\underset{\mathcal{B}}{argmin}f(\mathcal{B},\mathcal{S})+\frac{\rho }{2}{\parallel \mathcal{B}-{\mathcal{B}}_{pre}\parallel }_F^2\hfill \\ \mathcal{S}=\underset{\mathcal{S}}{argmin}f(\mathcal{B},\mathcal{S})+\frac{\rho }{2}{\parallel \mathcal{S}-{\mathcal{S}}_{pre}\parallel }_F^2\hfill \end{array}\right.$$

(19)

where $f(\mathcal{B},\mathcal{S})$ is the objective function implicitly defined in (18), ${(·)}_{pre}$ is the variable predicted in the preceding iteration, and $\rho >0$ is a proximal parameter. Next, we address the two subproblems by using a two-step iterative scheme:

(1) Step 1: Optimization with respect to $\mathcal{B}$

From (19), we have

$$\begin{array}{c}\hfill \underset{\mathcal{B}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\frac{\beta }{2}{\parallel \mathcal{X}-\mathcal{B}{\times }_3\mathbf{P}\mathbf{A}\parallel }_F^2\hfill \\ \hfill \hspace{1em} +\frac{1}{2}{\parallel \mathcal{B}\parallel }_{2,1}+{w\parallel \mathcal{B}\parallel }_{\ast }+\frac{\rho }{2}{\parallel \mathcal{B}-{\mathcal{B}}_{pre}\parallel }_F^2,\hfill \end{array}$$

(20)

where ${\mathcal{B}}_{pre}$ is the coefficients determined in the preceding PAM iteration. To effectively deal with Problem (20), the alternating direction method of the multiplier (ADMM) framework [52] is adopted. By introducing the auxiliary valuables $\mathcal{R}=\mathcal{B}$ and $\mathcal{U}=\mathcal{B}$, we obtain the augmented Lagrangian function as follows:

$$\begin{array}{cc}\hfill L(\mathcal{B},\mathcal{R},\mathcal{U},\mathcal{W},\mathcal{V})& =\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\frac{\beta }{2}\parallel \mathcal{X}-\mathcal{B}{\times }_3{\mathbf{P}\mathbf{A}\parallel }_F^2+\frac{1}{2}{\parallel \mathcal{R}\parallel }_{2,1}\hfill \\ & +\frac{\mu }{2}{\parallel \mathcal{R}-\mathcal{B}+\mathcal{W}\parallel }_F^2+{w\parallel \mathcal{U}\parallel }_{\ast }+\frac{\mu }{2}{\parallel \mathcal{U}-\mathcal{B}+\mathcal{V}\parallel }_F^2+\frac{\rho }{2}{\parallel \mathcal{B}-{\mathcal{B}}_{pre}\parallel }_F^2,\hfill \end{array}$$

(21)

where $\mathcal{W}$ and $\mathcal{V}$ are Lagrange multipliers, and $\mu >0$ is a penalty parameter. The ADMM iteration process is given by

$$\begin{array}{cc}\hfill \mathbf{for}& k=1,2,\cdots \hfill \end{array}$$

$$\begin{array}{cc}\hfill & \mathcal{B}\leftarrow \underset{\mathcal{B}}{argmin}L(\mathcal{B},\mathcal{R},\mathcal{U},\mathcal{W},\mathcal{V}),\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & \mathcal{R}\leftarrow \underset{\mathcal{R}}{argmin}L(\mathcal{B},\mathcal{R},\mathcal{U},\mathcal{W},\mathcal{V}),\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & \mathcal{U}\leftarrow \underset{\mathcal{U}}{argmin}L(\mathcal{B},\mathcal{R},\mathcal{U},\mathcal{W},\mathcal{V}),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & \mathcal{W}\leftarrow \mathcal{W}+(\mathcal{R}-\mathcal{B}),\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & \mathcal{V}\leftarrow \mathcal{V}+(\mathcal{U}-\mathcal{B}),\hfill \\ \hfill \mathbf{end}\end{array}$$

(26)

For the $\mathcal{B}$ subproblem, we have

$$\begin{array}{c}\underset{\mathcal{B}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\frac{\beta }{2}{\parallel \mathcal{X}-\mathcal{B}{\times }_3\mathbf{P}\mathbf{A}\parallel }_F^2\hfill \\ \hspace{1em}\hspace{1em} +\frac{\mu }{2}{\parallel \mathcal{R}-\mathcal{B}+\mathcal{W}\parallel }_F^2+\frac{\mu }{2}{\parallel \mathcal{U}-\mathcal{B}+\mathcal{V}\parallel }_F^2+\frac{\rho }{2}{\parallel \mathcal{B}-{\mathcal{B}}_{pre}\parallel }_F^2.\hfill \end{array}$$

(27)

Problem (27) is quadratic, and it can be addressed via the Sylvester matrix equation

$$\begin{array}{cc}\hfill \alpha (\mathcal{Y}-\mathcal{S}){\times }_3{\mathbf{A}}^T+\beta \mathcal{X}{\times }_3{\mathbf{D}}^T& +\mu (\mathcal{R}+\mathcal{W}+\mathcal{U}+\mathcal{V})+\rho {\mathcal{B}}_{pre}\hfill \\ & =(\rho +2\mu )\mathcal{B}+\mathcal{B}{\times }_3(\alpha {\mathbf{A}}^T\mathbf{A}+\beta {\mathbf{D}}^T\mathbf{D}),\hfill \end{array}$$

(28)

where $\mathbf{D}=\mathbf{P}\mathbf{A}$.

For the $\mathcal{R}$ subproblem, we have

$$\begin{array}{c}\hfill \underset{\mathcal{R}}{min}\frac{1}{2}{\parallel \mathcal{R}\parallel }_{2,1}+\frac{\mu }{2}{\parallel \mathcal{R}-\mathcal{B}+\mathcal{W}\parallel }_F^2.\end{array}$$

(29)

The efficient solution to Problem (29) can be achieved through the utilization of the group sparse soft-threshold shrinkage operator. Subsequently, the closed-form solution for each frontal slice of $\mathcal{R}$ can be derived as follows:

$$\begin{array}{c}\hfill \mathcal{R}(:,:,i)=\left\{\begin{array}{cc}\frac{{\parallel \mathcal{Q}(:,:,i)\parallel }_F-\frac{1}{2\mu }}{{\parallel \mathcal{Q}(:,:,i)\parallel }_F}{\parallel \mathcal{Q}(:,:,i)\parallel }_F,\hfill & {\mathrm{if}\parallel \mathcal{Q}(:,:,i)\parallel }_F>\frac{1}{2\mu }\hfill \\ \mathbf{0},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.,\end{array}$$

(30)

where $\mathcal{Q}=\mathcal{B}-\mathcal{W}$, and $i=1,2,\cdots ,\mathit{d}$.

The $\mathcal{U}$ subproblem can be updated by

$$\begin{array}{c}\hfill \underset{\mathcal{U}}{min}{w\parallel \mathcal{U}\parallel }_{\ast }+\frac{\mu }{2}{\parallel \mathcal{U}-\mathcal{B}+\mathcal{V}\parallel }_F^2.\end{array}$$

(31)

Based on the Definition 4 and Theorem 2, Problem (31) yields closed-form solution

$$\begin{array}{c}\hfill \mathcal{U}={\mathcal{D}}_{\tau }(\mathcal{B}-\mathcal{V}),\tau =\frac{w}{\mu },\end{array}$$

(32)

where ${\mathcal{D}}_{\tau }$ is the SVT operator.

To enhance comprehension, the process of optimizing $\mathcal{B}$ subproblem (20) is succinctly outlined in Algorithm 1.

Algorithm 1: ADMM Algorithm for $\mathcal{B}$ Subproblem

Input: $\mathcal{X}$, $\mathcal{Y}$, $\mathcal{S}$, $\mathbf{A}$, ${\mathcal{B}}_{pre}$, $\mathbf{P}$, $\alpha$, $\beta$, $\rho$, $\mu$, and $k_{\max }.$

Initialization: Let $k=0$, $\epsilon ={10}^{-5}$, $\mathcal{R}=\mathcal{U}=\mathcal{W}=\mathcal{V}=\mathbf{0}$.

1: while not converged and $k<k_{\max }$ do

2: $k=k+1$

3: Update $\mathcal{B}$ via (28).

4: Update $\mathcal{R}$ via (30).

5: Update $\mathcal{U}$ via (32).

6: Update $\mathcal{W}$ and $\mathcal{V}$ via (25) and (26), respectively.

7: Check the convergence condition: $\frac{\parallel {\mathcal{B}}^{k+1}-{\mathcal{B}}^k{\parallel }_F^2}{\parallel {\mathcal{B}}^k{\parallel }_F^2}\le \epsilon$.

8: end while

Output: Low-dimensional coefficient tensor $\mathcal{B}$

(2) Step 2: Optimization with respect to $\mathcal{S}$

Updating the difference image $\mathcal{S}$ is solved by

$$\begin{array}{c}\hfill \underset{\mathcal{S}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\eta \sum _{n=1}^3\parallel {\nabla }_n{\mathcal{S}\parallel }_{1/2}+\frac{\rho }{2}{\parallel \mathcal{S}-{\mathcal{S}}_{pre}\parallel }_F^2,\end{array}$$

(33)

where ${\mathcal{S}}_{pre}$ represents the difference image determined in the preceding PAM iteration. Similar to the $\mathcal{B}$ subproblem, we also employ ADMM to solve $\mathcal{S}$ subproblem (33). By introducing the auxiliary valuable ${\mathcal{C}}_n={\nabla }_n\mathcal{S}$, we have

$$\begin{array}{cc}\hfill L(\mathcal{S},{\mathcal{C}}_n,{\mathcal{G}}_n)& =\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\eta \sum _{n=1}^3{\parallel {\mathcal{C}}_n\parallel }_{1/2}\hfill \\ & +\sum _{n=1}^3\frac{\mu }{2}\parallel {\mathcal{C}}_n-{\nabla }_n\mathcal{S}+{\mathcal{G}}_n{\parallel }_F^2+\frac{\rho }{2}{\parallel \mathcal{S}-{\mathcal{S}}_{pre}\parallel }_F^2,\hfill \end{array}$$

(34)

where ${\mathcal{G}}_n$ is the Lagrangian multipliers. ADMM iterations follow this procedure:

$$\begin{array}{cc}\hfill \mathbf{for}& i=1,2,\cdots \hfill \end{array}$$

$$\begin{array}{cc}\hfill & \mathcal{S}\leftarrow \underset{\mathcal{S}}{argmin}L(\mathcal{S},{\mathcal{C}}_n,{\mathcal{G}}_n),\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & {\mathcal{C}}_n\leftarrow \underset{{\mathcal{C}}_n}{argmin}L(\mathcal{S},{\mathcal{C}}_n,{\mathcal{G}}_n),n=1,2,3,\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & {\mathcal{G}}_n\leftarrow {\mathcal{G}}_n+({\mathcal{C}}_n-{\nabla }_n\mathcal{S}),n=1,2,3,\hfill \\ \hfill \mathbf{end}\end{array}$$

(37)

The $\mathcal{S}$ subproblem can be updated by

$$\begin{array}{c}\hfill \underset{\mathcal{S}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\sum _{n=1}^3\frac{\mu }{2}\parallel {\mathcal{C}}_n-{\nabla }_n\mathcal{S}+{\mathcal{G}}_n{\parallel }_F^2+\frac{\rho }{2}{\parallel \mathcal{S}-{\mathcal{S}}_{pre}\parallel }_F^2.\end{array}$$

(38)

Problem (38) can be effectively solved by FFT strategy:

$$\begin{array}{c}\hfill \mathcal{S}={\mathcal{F}}^{-1}\left(\frac{\mathcal{F}(\alpha (\mathcal{Y}-\mathcal{B}{\times }_3\mathbf{A})+{\sum }_{n=1}^3\mu {\nabla }_n^T({\mathcal{C}}_n+{\mathcal{G}}_n)+\rho {\mathcal{S}}_{pre})}{\mathcal{F}({\sum }_{n=1}^3\mu {\nabla }_n^T{\nabla }_n+(\alpha +\rho )\mathcal{O})}\right),\end{array}$$

(39)

where $\mathcal{O}\in {\mathbb{R}}^{W\times H\times S}$ is a tensor with all elements equal to 1.

For the ${\mathcal{C}}_n$ subproblem, we have

$$\begin{array}{c}\hfill \underset{{\mathcal{C}}_n}{min}\eta \sum _{n=1}^3\parallel {\mathcal{C}}_n{\parallel }_{1/2}+\sum _{n=1}^3\frac{\mu }{2}{\parallel {\mathcal{C}}_n-{\nabla }_n\mathcal{S}+{\mathcal{G}}_n\parallel }_F^2.\end{array}$$

(40)

Problem (40) that involves the ${\mathsf{\ell }}_{1/2}$ norm is nonconvex. Here, we employ the generalized shrinkage/thresholding (GST) operation [53] to address Problem (40) due to its efficiency and low computational cost. Consequently, Nonconvex problem (40) yields the following solution:

$$\begin{array}{c}\hfill {\mathcal{C}}_n={\mathbf{T}}_{1/2}^{GST}({\nabla }_n\mathcal{S}-{\mathcal{G}}_n,\frac{\eta }{\mu }),n=1,2,3,\end{array}$$

(41)

where ${\mathbf{T}}_{1/2}^{GST}(a,b)$ is the GST operator.

The process of optimizing the $\mathcal{S}$ subproblem is succinctly outlined in Algorithm 2. Combining the solutions for the aforementioned two block subproblems, we derive the PAM algorithm (summarized in Algorithm 3) to solve the FGSSR-based model (18). Algorithm 3 has a strong capability to converge towards a critical point of the objective function, as proven in [51].

Algorithm 2: ADMM Algorithm for $\mathcal{S}$ Subproblem

Input: $\mathcal{Y}$, ${\mathcal{S}}_{pre}$, $\mathbf{A}$, $\mathcal{B}$, $\alpha$, $\eta$, $\rho$, $\mu$, and $i_{\max }.$

Initialization: Let $i=0$, $\epsilon ={10}^{-5}$,${\mathcal{C}}_n={\mathcal{G}}_n=\mathbf{0}$, $n=1,2,3$.

1: while not converged and $i<i_{\max }$ do

2: $i=i+1$.

3: Update $\mathcal{S}$ via (39).

4: Update ${\mathcal{C}}_n$, $n=1,2,3$, via solving (41) with the GST operator.

5: Update ${\mathcal{G}}_n$, $n=1,2,3$, via (37).

6: Check the convergence condition: $\frac{\parallel {\mathcal{S}}^{i+1}-{\mathcal{S}}^i{\parallel }_F^2}{\parallel {\mathcal{S}}^i{\parallel }_F^2}\le \epsilon$.

7: end while

Output: Difference image $\mathcal{S}$

Algorithm 3: PAM Algorithm for FGSSR-based Model

Input: $\mathcal{X}$, $\mathcal{Y}$, $\mathbf{P}$, $\mathrm{rank}r\le d\le min\{WH,S\}$, $\alpha$, $\beta$, w, $\eta$, $\rho$, $\mu$, and $t_{\max }.$

Initialize: $t=0$, $\epsilon ={10}^{-5}$, $\mathbf{A}={\mathbf{U}}_y(:,1:d){\Sigma }_y{(1:d,1:d)}^{1/2}$,    $\mathcal{B}={\mathtt{Fold}}_3\left({\Sigma }_y{(1:d,1:d)}^{1/2}{\mathbf{V}}_y{(:,1:d)}^T\right)$, where ${\mathbf{U}}_y$, ${\Sigma }_y$, and ${\mathbf{V}}_y$ come from the    SVD of ${\mathbf{Y}}_{\left(3\right)}$.

1: while not converged and $t<t_{\max }$ do

2: $t=t+1$.

3: Update $\mathcal{B}$ via solving (20) with Algorithm 1.

4: Update $\mathcal{S}$ via solving (33) with Algorithm 2.

5: Update $d=\mathrm{nnzc}\left(\mathcal{B}\right)$, where $\mathrm{nnzc}\left(\mathcal{B}\right)$ is the number of the nonzero frontal     slices of $\mathcal{B}$

6: Remove the zero frontal slices of $\mathcal{B}$.

7: Update $\mathbf{A}$ via (16).

8: Check the convergence condition: $\frac{\parallel {\mathcal{B}}^{t+1}{\times }_3{\mathbf{A}}^{t+1}-{\mathcal{B}}^t{\times }_3{\mathbf{A}}^t{\parallel }_F^2}{\parallel {\mathcal{B}}^t{\times }_3{\mathbf{A}}^t{\parallel }_F^2}\le \epsilon$.

9: end while

Output: Target HR-HSI: $\mathcal{Z}=\mathcal{B}{\times }_3\mathbf{A}$

3.4. Computational Complexity

Let us denote the acquired LR-HSI as $\overline{\mathcal{Y}}\in {\mathbb{R}}^{w\times h\times S}$ and the acquired HR-MSI as $\mathcal{X}\in {\mathbb{R}}^{W\times H\times s}$. We let $d\in \{\mathrm{rank}r,S\}$ represent the subspace dimension. In Algorithm 3, the most computationally intensive parts involve solving the $\mathcal{B}$ and $\mathcal{S}$ subproblems. In Algorithm 1, updating $\mathcal{R}$ involves a group soft threshold shrinkage operation, which costs $\mathcal{O}\left(WHd\right)$. Updating $\mathcal{B}$ costs $\mathcal{O}\left(WH{d}^2\right)$, and updating $\mathcal{U}$ costs $\mathcal{O}(dmin{W}^{2}H,W{H}^2)$. In Algorithm 2, updating $\mathcal{S}$ involves an FFT operation, which costs $\mathcal{O}(WHSlog(WHS\left)\right)$. The computational complexity of updating ${\mathcal{C}}_n$ through a soft threshold shrinkage operation is $\mathcal{O}\left(WHd\right)$. To summarize, the overall computational complexity of Algorithm 3 at every iteration can be expressed as $\mathcal{O}(2WHd+WH{d}^2+dmin{W}^{2}H,W{H}^2+WHSlog\left(WHS\right))$. It is noteworthy that the subspace dimension d reduces as the iteration progresses during the execution of Algorithm 3.

4. Experiments

We conducted several experiments on both simulated and real datasets to evaluate the effectiveness of our FGSSR-based method. Furthermore, we compared our FGSSR-based method against six SOTA methods, including matrix factorization-based methods (HySure [27], the coupled nonnegative matrix factorization method (CNMF) [54], the nonnegative structured sparse representation method (NSSR) [28]), tensor factorization-based methods (joint-structured sparse block-term tensor decomposition-based fusion method (JSSLL1) [55], the group sparsity constrained fusion method (GSFus) [16], the low tensor multirank-based method (LTMR) [38]) and subspace representation-based methods (HySure [27], GSFus [16], and LTMR [38]). The codes of the compared methods are obtained from the authors’ homepages, and their settings are fine-tuned to produce optimal outputs, following the reference recommendations. For our FGSSR-based method, we set $\alpha =1\times {10}^{-2}$, $\beta =0.5$, $\eta =1\times {10}^{-4}$, $w=1\times {10}^{-2}$, $\rho =7$, $\mu =1\times {10}^{-2}$, and $\epsilon =1\times {10}^{-5}$ for all the following experiments. A comprehensive analysis of the parameters of our FGSSR-based method is presented in Section 5.2.

4.1. Datasets

4.1.1. Datasets for Simulated Experiments

Pavia University (PU) dataset: PU data are a hyperspectral image with dimensions of $610\times 340$ spatial pixels and 115 spectral bands, which was captured by the reflective optics spectrographic imaging system (ROSIS-3) optical airborne sensor over Pavia University, Italy. After eliminating the spectral bands that are influenced by water vapour absorption, a total of 103 spectral bands remain. In our study, we choose the up-middle $256\times 256\times 93$ cube-size PU image as the reference image. PU data are publicly available and can be downloaded from the website: https://rslab.ut.ac.ir/data (accessed on 28 September 2023).

Washington DC Mall (WDCM) dataset: WDCM data are acquired using the hyperspectral digital imagery collection experiment sensor over the Mall in Washington, DC, USA. It comprises $1280\times 307$ spatial pixels and initially contains 210 spectral bands. The elimination of low signal-to-noise ratio (SNR) spectral bands results in a reduction in the total number of spectral bands to 191. In our study, we specifically choose a WDCM HSI with dimensions of $256\times 256\times 93$ as the reference image to perform our analysis and experiments. WDCM data are a public dataset and can be downloaded from the website: https://rslab.ut.ac.ir/data (accessed on 28 September 2023).

Chikusei dataset: Chikusei data are taken over Chikusei, Ibaraki, Japan, by using the Headwall’s Hyperspec-VNIR-C imaging sensor. It comprises $2517\times 2335$ spatial pixels and consists of 128 spectral bands, covering both urban and agricultural regions. For our study, a Chikusei HSI with dimensions of $256\times 256\times 93$ is extracted as the reference image to perform our analysis and experiments. Chikusei data are publicly available and can be downloaded from the website: https://naotoyokoya.com/Download.html (accessed on 28 September 2023).

4.1.2. Dataset for Real Experiment

The real dataset comprises a real HSI and a real MSI over the area of Arkansas City, Cowley County, Kansas, USA. The real HSI obtained from the Earth-Observing-1 (EO-1) satellite consists of 89 spectral bands, with the exclusion of bands exhibiting low SNR. For our study, we select a space of size $100\times 100$ as the observed LR-HSI, i.e., $\overline{\mathcal{Y}}\in {\mathbb{R}}^{100\times 100\times 89}$. The real MSI is acquired by the Sentinel-2A satellite, and its red, green, blue, and near-infrared bands with a spatial size of $300\times 300$ are selected as the HR-MSI, i.e., $\mathcal{X}\in {\mathbb{R}}^{300\times 300\times 4}$.

4.2. Evaluation Index

Five evaluation indices are employed to quantitatively assess the quality of the fusion results.

(1) The peak signal-noise-ratio (PSNR) [6]:

$$\mathrm{PSNR}(\mathcal{Z},\widehat{\mathcal{Z}})=\frac{1}{S}\sum _{i=1}^S\mathrm{PSNR}({\mathbf{Z}}_i,{\widehat{\mathbf{Z}}}_i),$$

(42)

where $\mathcal{Z}\in {\mathbb{R}}^{W\times H\times S}$ and $\widehat{\mathcal{Z}}\in {\mathbb{R}}^{W\times H\times S}$ are the reference and fused images, respectively. ${\mathbf{Z}}_i\in {\mathbb{R}}^{W\times H}$ and ${\widehat{\mathbf{Z}}}_i\in {\mathbb{R}}^{W\times H}$ are the ith spectral bands of $\mathcal{Z}$ and $\widehat{\mathcal{Z}}$, respectively.

(2) The Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS) [56]:

$$\mathrm{ERGAS}(\mathcal{Z},\widehat{\mathcal{Z}})=\frac{100}{q}\sqrt{\frac{1}{S}{\sum }_{i=1}^S\frac{\mathrm{MSE}({\mathbf{Z}}_i,{\widehat{\mathbf{Z}}}_i)}{{\mu }_{{\widehat{\mathbf{Z}}}_i}^2}},$$

(43)

where q denotes the spatial downsampling factor, ${\mu }_{{\widehat{\mathbf{Z}}}_i}$ denotes the mean value of ${\widehat{\mathbf{Z}}}_i$, and $\mathrm{MSE}({\mathbf{Z}}_i,{\widehat{\mathbf{Z}}}_i)$ represents the mean square error between ${\mathbf{Z}}_i$ and ${\widehat{\mathbf{Z}}}_i$. The best value is zero.

(3) The structure similarity (SSIM) [57]:

$$\mathrm{SSIM}(\mathcal{Z},\widehat{\mathcal{Z}})=\frac{1}{S}\sum _{i=1}^S\mathrm{SSIM}({\mathbf{Z}}_i,{\widehat{\mathbf{Z}}}_i),$$

(44)

which measures the similarity of the overall structures between the reference and fused images, with a value of one indicating the best similarity.

(4) The spectral angle mapper (SAM) [58]:

$$\mathrm{SAM}(\mathcal{Z},\widehat{\mathcal{Z}})=\frac{1}{WH}\sum _{i=1}^{WH}arccos\frac{{\widehat{\mathbf{z}}}_i^T{\mathbf{z}}_i}{\parallel {\widehat{\mathbf{z}}}_i{\parallel }_2{\parallel {\mathbf{z}}_i\parallel }_2},$$

(45)

where ${\widehat{\mathbf{z}}}_i$ and ${\mathbf{z}}_i$ are pixels in $\widehat{\mathcal{Z}}$ and $\mathcal{Z}$, respectively. SAM is given in degrees, where smaller SAM indicates less spectral distortions between the reference and fused images.

(5) The universal image quality index (UIQI) [59]:

$$\mathrm{UIQI}({\mathbf{Z}}_i,{\widehat{\mathbf{Z}}}_i)=\frac{1}{M}\sum _{j=1}^M\frac{{\sigma }_{{\mathbf{Z}}_j^i{\widehat{\mathbf{Z}}}_j^i}^2}{{\sigma }_{{\mathbf{Z}}_j^i}{\sigma }_{{\widehat{\mathbf{Z}}}_j^i}}\frac{2{\mu }_{{\mathbf{Z}}_j^i}{\mu }_{{\widehat{\mathbf{Z}}}_j^i}}{{\mu }_{{\mathbf{Z}}_j^i}^2+{\mu }_{{\widehat{\mathbf{Z}}}_j^i}^2}\frac{2{\sigma }_{{\mathbf{Z}}_j^i}{\sigma }_{{\widehat{\mathbf{Z}}}_j^i}}{{\sigma }_{{\mathbf{Z}}_j^i}^2+{\sigma }_{{\widehat{\mathbf{Z}}}_j^i}^2},$$

(46)

where M is the total number of sliding windows with a size of $32\times 32$, ${\mathbf{Z}}_j^i$ and ${\widehat{\mathbf{Z}}}_j^i$ are the jth window in the ith band of the reference and fused images, respectively. ${\sigma }_{{\mathbf{Z}}_j^i}$ denotes the standard deviation of ${\mathbf{Z}}_j^i$, and ${\sigma }_{{\mathbf{Z}}_j^i{\widehat{\mathbf{Z}}}_j^i}$ is the covariance of ${\mathbf{Z}}_j^i$ and ${\widehat{\mathbf{Z}}}_j^i$. UIQI ranges from zero to one, where a higher UIQI value signifies a higher quality of the fusion result.

4.3. Experimental Results on Simulated Datasets

4.3.1. Results on Uniform Blur Simulated Datasets

In this part, we utilize the selected PU and WDCM HSIs as the reference HR-HSIs. To simulate the LR-HSIs, we downsample the reference HR-HSIs by uniformly averaging over disjoint $q\times q$ blocks, where q is the spatial downsampling factor (e.g., $q=4,8,16$). The HR-MSIs are produced through the process of degrading the reference HR-HSIs along the third mode, utilizing the IKONOS-like spectral reflectance response filter (SRRF) [60].

Table 1. Quantitative results on simulated PU dataset with uniform blur.

Evaluation Index	PSNR	SAM	ERGAS	UIQI	SSIM
Best Value	$+\infty$	0	0	1	1
$q=4$
CNMF [54]	42.107	1.9039	1.1855	0.9945	0.9896
HySure [27]	41.635	2.0066	1.2248	0.9940	0.9888
NSSR [28]	43.131	2.2042	1.1417	0.9921	0.9825
JSSLL1 [55]	44.022	1.8815	1.0145	0.9938	0.9896
GSFus [16]	43.697	2.0110	1.0409	0.9937	0.9891
LTMR [38]	44.953	1.6761	0.8867	0.9955	0.9912
Proposed	45.910	1.5669	0.7898	0.9963	0.9923
$q=8$
CNMF [54]	39.692	2.3796	0.7644	0.9907	0.9848
HySure [27]	39.087	2.5558	0.8246	0.9909	0.9841
NSSR [28]	42.757	2.3663	0.5939	0.9915	0.9820
JSSLL1 [55]	43.381	1.9963	0.5114	0.9935	0.9863
GSFus [16]	43.357	2.0910	0.5396	0.9933	0.9883
LTMR [38]	44.342	2.0232	0.5004	0.9946	0.9895
Proposed	45.101	1.7139	0.4357	0.9956	0.9913
$q=16$
CNMF [54]	39.161	2.4714	0.4103	0.9904	0.9844
HySure [27]	38.909	2.8017	0.4162	0.9893	0.9814
NSSR [28]	42.507	2.4716	0.3109	0.9911	0.9813
JSSLL1 [55]	42.737	2.2773	0.2920	0.9927	0.9862
GSFus [16]	43.253	2.1154	0.2732	0.9932	0.9880
LTMR [38]	43.979	2.1211	0.2522	0.9937	0.9889
Proposed	44.555	1.8265	0.2334	0.9950	0.9907

and

Table 2. Quantitative results on simulated WDCM dataset with uniform blur.

Evaluation Index	PSNR	SAM	ERGAS	UIQI	SSIM
Best Value	$+\infty$	0	0	1	1
$q=4$
CNMF [54]	40.486	2.0643	1.2656	0.9927	0.9890
HySure [27]	40.731	2.2544	1.2953	0.9925	0.9865
NSSR [28]	41.151	2.3756	1.3732	0.9929	0.9821
JSSLL1 [55]	41.886	2.2584	1.2521	0.9933	0.9858
GSFus [16]	41.815	2.3285	1.3101	0.9923	0.9867
LTMR [38]	43.110	2.0396	1.1182	0.9949	0.9884
Proposed	43.752	1.6558	1.0408	0.9959	0.9914
$q=8$
CNMF [54]	39.452	2.4361	0.7205	0.9903	0.9863
HySure [27]	38.929	2.8177	0.8370	0.9887	0.9828
NSSR [28]	40.777	2.6519	0.7233	0.9903	0.9817
JSSLL1 [55]	41.068	2.594	0.7187	0.9918	0.9842
GSFus [16]	40.644	2.8770	0.7554	0.9899	0.9860
LTMR [38]	41.919	2.5506	0.7215	0.9924	0.9837
Proposed	42.452	2.0956	0.6218	0.9938	0.9886
$q=16$
CNMF [54]	37.418	3.1868	0.4567	0.9850	0.9823
HySure [27]	36.335	3.4996	0.5335	0.9816	0.9760
NSSR [28]	39.522	3.2146	0.4298	0.9869	0.9779
JSSLL1 [55]	39.809	2.8308	0.4135	0.9870	0.9789
GSFus [16]	38.449	3.3989	0.5049	0.9818	0.9799
LTMR [38]	40.335	2.8135	0.4611	0.9881	0.9793
Proposed	41.499	2.4277	0.3539	0.9917	0.9870

provide the quality metrics of the test methods on the simulated PU and WDCM datasets, respectively. The optimal values are emphasized in bold to enhance understanding. From Table 1 and Table 2, matrix factorization-based methods (e.g., CNMF, HySure, and NSSR) yield inferior results compared with the tensor factorization-based methods (e.g., JSSLL1, GSFus, LTMR, and the proposed FGSSR-based method). GSFus, JSSLL1, and LTMR consistently yield satisfactory results across all spatial downsampling factor levels. However, the proposed FGSSR-based method demonstrates even more competitive performance across all spatial downsampling factor cases. For instance, when the spatial downsampling factor q is set to four, the PSNR of the fused PU HSI using the proposed FGSSR-based method is observed to be more than 0.9 dB higher compared to the suboptimal method, and the PSNR of the fused WDCM HSI using the proposed FGSSR-based method is observed to be more than 0.6 dB higher compared to the suboptimal method.

Figure 3 and Figure 4 demonstrate the fused results and their corresponding pixel-wise root mean square error (RMSE) images [61] of the simulated PU and WDCM datasets with uniform blur, respectively. The color images for the PU are generated using the 30th, 73rd, and 13th spectral bands. Similarly, the color images for the WDCM are created using the 75th, 45th, and 8th spectral bands. Meanwhile, we enlarge a specific region of interest (highlighted with a red block) to facilitate a more detailed visual comparison. Upon closer examination of the magnified regions, it becomes evident that all of the fused images display a notable improvement in spatial details. However, there are also noticeable spectral inconsistencies between the reference and fused images. For instance, we can see that the fused images of CNMF and HySure show noticeable spectral distortions in the building areas. Similarly, NSSR, GSFus, and LTMR produce a slight degree of spectral distortion in the tree areas. Additionally, the fused image of JSSLL1 clearly exhibits the presence of box effects on the PU dataset. Compared to other methods, our FGSSR-based method performs better in preserving spatial features and maintaining consistent spectral information.

To assess the spectral preservation capabilities of the test methods, the spectral curves of the fused results are compared in Figure 5. A clear observation from the comparison of spectral curves is that the fused results generated by the FGSSR-based method exhibit higher spectral consistency with the reference images across the majority of spectral bands. This finding suggests that the FGSSR-based method outperforms other compared methods in maintaining spectral accuracy. This can be attributed to the superior capability of the FGSSR model in capturing and preserving spectral correlation properties.

4.3.2. Results on Gaussian Blur Simulated Datasets

Meanwhile, we also verify the effectiveness of our FGSSR-based method in handling the simulated datasets generated by Gaussian blur. To obtain the LR-HSIs, we downsample every eight-pixel cluster along the spatial modes after applying an $8\times 8$ Gaussian blur (standard deviation 2.25) to the reference HR-HSIs obtained from the PU and WDCM datasets. The HR-MSIs are simulated by utilizing the IKONOS-like SRRF [60]. The results on the Gaussian blur simulated datasets are reported in

Table 3. Quantitative results on simulated PU and WDCM datasets with Gaussian blur and q = 8.

Evaluation Index	PSNR	SAM	ERGAS	UIQI	SSIM
Best Value	$+\infty$	0	0	1	1
PU dataset
CNMF [54]	39.144	2.3847	0.8216	0.9906	0.9847
HySure [27]	38.679	2.6830	0.8640	0.9901	0.9827
NSSR [28]	42.079	2.2543	0.6734	0.9914	0.9852
JSSLL1 [55]	42.851	2.1845	0.5923	0.9921	0.9863
GSFus [16]	43.195	2.1028	0.5467	0.9934	0.9884
LTMR [38]	44.125	1.9953	0.4924	0.9942	0.9893
Proposed	44.727	1.8090	0.4567	0.9953	0.9910
WDCM dataset
CNMF [54]	38.932	2.5809	0.7642	0.9892	0.9848
HySure [27]	38.845	2.8448	0.8451	0.9885	0.9825
NSSR [28]	40.426	2.6366	0.7965	0.9899	0.9818
JSSLL1 [55]	40.543	2.5385	0.7145	0.9912	0.9834
GSFus [16]	40.804	2.5341	0.7234	0.9905	0.9864
LTMR [38]	41.225	2.6241	0.7939	0.9912	0.9809
Proposed	42.052	2.2358	0.6584	0.9931	0.9879

. The optimal values are emphasized in bold to enhance understanding. From the table, matrix factorization-based methods have poor performance in Gaussian blur cases compared with tensor factorization-based methods. For example, the best-performing matrix factorization-based method NSSR achieved a SAM of 2.2543 on the Gaussian blur-simulated PU dataset, while the worst performing tensor factorization-based method JSSLL1 achieved a SAM of 2.1845. Table 3 also demonstrates that for both of the Gaussian blur simulated datasets, the FGSSR-based method outperforms the competition on the basis of all quality measures. This suggests that the FGSSR-based method exhibits superior performance in comparison to other SOTA methods in handling various blur functions and scaling factors.

4.3.3. Results on Noisy Simulated Datasets

In real-world scenarios, both HS and MS imaging processes typically introduce additive noise, with the noise level in the HR-MSI usually being lower than that in the LR-HSI. To evaluate the robustness of the FGSSR-based method under the influence of noise, we conducted tests using the noisy simulated datasets generated from the Chikusei dataset. Specifically, the noisy LR-HSI was simulated by adding Gaussian white noise after downsampling the reference Chikusei HSI using uniform averaging over $8\times 8$ blocks. For the noisy HR-MSI, it can be simulated by adding Gaussian white noise after filtering the reference Chikusei HR-HSI with an IKONOS-like SRRF. Specifically, the FGSSR-based method was verified in two different noisy cases:

Noisy case 1: SNRh = 35 dB, SNRm = 40 dB;

Noisy case 2: SNRh = 30 dB, SNRm = 35 dB,

where SNRh and SNRm represent the SNR of the noisy LR-HSI and HR-MSI, respectively.

Figure 6 shows the fused results and their corresponding RMSE images of the noisy cases on the Chikusei. The color images for the Chikusei were generated using the 60th, 39th, and 10th spectral bands. As seen in Figure 6, noise artefacts are presented in the fused results of CNMF, HySure, JSSLL1, GSFus, and LTMR. NSSR is sensitive to noise variations, and some noticeable block artifacts can be found in its fused images. The proposed FGSSR-based method achieved superior results with fewer errors compared to the other test methods.

Table 4. Quantitative results on the noisy simulated datasets of the Chikusei.

Evaluation Index	PSNR	SAM	ERGAS	UIQI	SSIM
Best Value	$+\infty$	0	0	1	1
Noisy case 1
CNMF [54]	41.192	1.4237	1.6872	0.9692	0.9855
HySure [27]	40.635	1.7182	1.8944	0.9631	0.9796
NSSR [28]	42.736	1.6560	1.5704	0.9644	0.9815
JSSLL1 [55]	43.365	1.4429	1.7172	0.9665	0.9828
GSFus [16]	43.723	1.4587	1.5946	0.9668	0.9861
LTMR [38]	44.517	1.3465	1.5730	0.9736	0.9867
Proposed	45.489	1.1652	1.4243	0.9742	0.9883
Noisy case 2
CNMF [54]	39.414	1.6403	1.7888	0.9623	0.9787
HySure [27]	39.596	2.0064	2.1211	0.9509	0.9678
NSSR [28]	39.450	2.4438	1.9339	0.9401	0.9624
JSSLL1 [55]	41.176	1.6936	1.9087	0.9584	0.9734
GSFus [16]	41.947	1.8207	1.6702	0.9581	0.9775
LTMR [38]	42.294	1.6918	1.7407	0.9610	0.9756
Proposed	43.295	1.4812	1.6072	0.9635	0.9794

reports the results on the noisy simulated datasets, further demonstrating that the FGSSR-based method performed better than other compared methods in dealing with noisy scenarios.

4.4. Experimental Results on Real Dataset

To verify the practical validity of the FGSSR-based method in real-life scenarios, we conducted an experiment using a real dataset. The details of this dataset can be found in Section 4.1. During the real experiment, we encountered the challenges of the unknown spectral response and the spatial downsampling method. To address these issues, we employed the method provided in [27] to calculate the unknown spectral response and spatial blur kernel. Figure 7 shows the fused results on the real dataset. From Figure 7, the fused images of JSSLL1 and GSFus exhibit aliasing effects in the road, implying that the preservation of spatial structures and details is not entirely satisfactory. Additionally, there are noticeable artificial effects and spectral distortions present in the LTMR results. The FGSSR-based method strikes a better balance between preserving spatial structures and minimizing spectral distortions than other methods, resulting in superior visual quality in its fused result. Figure 8 demonstrates the spectral curves of the fused images and the real LR-HSI. In Figure 8, we can see that the spectral curves of the compared methods are consistent with the spectral curve of the real LR-HSI, while the spectral curve of the fused result of the proposed FGSSR-based method exhibits the highest level of consistency with that of the real LR-HSI. This demonstrates the excellent performance of the FGSSR-based method in real-life scenarios, showcasing its practical effectiveness.

5. Discussion

In this part, the PU and WDCM datasets are selected as the experimental objects. The LR-HSIs are simulated by using an $8\times 8$ Gaussian blur function (Standard deviation 3) and a spatial Downsampling factor 8, and the HR-MSIs are generated by using an IKONOS-like SRRF. Additionally, Gaussian white noise is injected into the simulated LR-HSIs (SNRh = 35 dB) and HR-MSIs (SNRm = 40 dB).

5.1. The Effectiveness of the Proposed FGSSR Model

To demonstrate the effectiveness of the proposed Factor group sparsity regularized subspace representation (FGSSR) model (13), we carried out a comprehensive analysis by comparing our FGSSR-based fusion method with the fusion method based on the Direct subspace representation (DSR) model (12). By replacing the FGSSR model (13) with the DSR model (12), the DSR-based fusion method can be written as

$$\begin{array}{c}\hfill \underset{\mathcal{B},\mathcal{S}}{min}\frac{\alpha }{2}\parallel \mathcal{Y}-\mathcal{B}{\times }_3{\mathbf{A}-\mathcal{S}\parallel }_F^2+\eta \sum _{n=1}^3\parallel {\nabla }_n{\mathcal{S}\parallel }_{1/2}+\frac{\beta }{2}\parallel \mathcal{X}-\mathcal{B}{\times }_3{\mathbf{P}\mathbf{A}\parallel }_F^2+w{\parallel \mathcal{B}\parallel }_{\ast }.\end{array}$$

(47)

In the experiments, the subspace dimension of the DSR-based fusion method was set to be consistent with the subspace dimension adaptively determined by the FGSSR-based fusion method.

Table 5. Quantitative results of the FGSSR-based and DSR-based methods on the simulated PU and WDCM datasets.

Methods	PU Dataset						WDCM Dataset
	PSNR	SAM	ERGAS	UIQI	SSIM	PSNR	SAM	ERGAS	UIQI	SSIM
Best Value	$+\infty$	0	0	1	1	$+\infty$	0	0	1	1
DSR-based	42.380	2.2899	0.5761	0.9923	0.9833	38.677	3.2126	0.8330	0.9821	0.9692
FGSSR-based	43.764	1.9899	0.4991	0.9942	0.9884	41.001	2.5193	0.7023	0.9911	0.9848

presents the quantitative results of the FGSSR-based and DSR-based fusion methods on the simulated PU and WDCM datasets. From Table 5, it is evident that the FGSSR model (13) exhibited superior performance compared to the DSR model (12), delivering significantly better results on both of the two simulated datasets.

5.2. Analysis of Model Parameters

Parameter $\alpha$ is utilized to constrain the spectral fidelity term. Figure 9a illustrates the PSNR curves corresponding to various values of $\alpha$. In Figure 9a, the PSNR values for the simulated PU and WDCM datasets remain relatively stable within the range of $\alpha$ from $5\times {10}^{-5}$ to $1\times {10}^{-2}$, but sharply decrease for $\alpha$ greater than $1\times {10}^{-2}$. This reveals that the proposed FGSSR-based method exhibits relatively robust performance for both of the two simulated datasets when $\alpha \in [5\times {10}^{-5},1\times {10}^{-2}]$. Therefore, $\alpha =1\times {10}^{-2}$ is set for the proposed method in all experiments.

Parameter $\beta$ serves to restrict the spatial fidelity term. Figure 9b displays the PSNR curves corresponding to various values of $\beta$. In the experiment, $\beta$ is varied within the range of $[1\times {10}^{-4},1]$. We can see from Figure 9b that there are obvious similarities between the trends of changes in the PSNR curves of two simulated datasets. This reveals that our FGSSR-based method is robust and stable across different datasets. The PSNR values for both the two simulated datasets experience a significant increase when $\beta$ varies from $1\times {10}^{-4}$ to $5\times {10}^{-2}$, and then stabilize at the maximum level when $\beta$ varies from $5\times {10}^{-2}$ to $1\times {10}^0$. As a result, we assign a value of $\beta =0.5$ to the proposed FGSSR-based method in all conducted experiments.

To evaluate the impact of the parameter $\eta$ on the FGSSR-based method, experiments are conducted on the simulated PU and WDCM datasets using different values of $\eta$. In the experiment, $\eta$ is selected from the set $\{1\times {10}^{-6},1\times {10}^{-5},1\times {10}^{-4},1\times {10}^{-3},1\times {10}^{-2},1\times {10}^{-1}$, $1\times {10}^0\}$. Figure 9c illustrates the PSNR curves for different $\eta$. According to Figure 9c, the PSNR values for the simulated PU and WDCM datasets remain relatively stable within the range of $\eta$ from $1\times {10}^{-6}$ to $1\times {10}^{-3}$, but decrease when $\eta$ exceeds $1\times {10}^{-3}$. Based on these observations, $\eta =1\times {10}^{-4}$ is selected as the optimal value for the proposed method in all experiments.

Parameter w is utilized to constrain the TNN regularization term. To examine the effectiveness of the TNN regularization term, the proposed FGSSR-based method is applied to the simulated PU and WDCM datasets using varying values of w. Figure 9e illustrates the PSNR curves corresponding to various values of parameter w. According to Figure 9e, the PSNR values for both the simulated PU and WDCM datasets show increasing trends as w increases. This demonstrates that the TNN regularization applied to the low-dimensional coefficients $\mathcal{B}$ enhances the ability of the proposed method in capturing spatial self-similarity features, ultimately leading to improved HR-HSI reconstruction. Additionally, the optimal outcomes for both of the two simulated datasets are attained when the value of w lies within the interval of $[1\times {10}^{-3},1\times {10}^{-1}]$. Taking this into consideration, $w=1\times {10}^{-2}$ is selected as the optimal value for the proposed method in all experiments.

$\rho$ is a proximal parameter that ensures the theoretical convergence of the PAM solver, while parameter $\mu$ serves as a penalty parameter. Figure 9f,g display the PSNR of the fused results for the simulated PU and WDCM datasets as functions of $\rho$ and $\mu$, respectively. According to Figure 9f,g, the optimal results for both of the two simulated datasets are achieved when the value of $\rho$ is within the interval of $[5\times {10}^{-1},1\times {10}^1]$, while the value of $\mu$ is within the range of $[1\times {10}^{-6},1\times {10}^{-1}]$. Based on these observations, $\rho =7$ and $\mu =1\times {10}^{-2}$ are set for the proposed method in all experiments.

5.3. Automatic Adjustment of Subspace Dimension $\mathit{d}$

The proposed FGSSR-based method adaptively determines the subspace dimension $\mathit{d}$ by minimizing the number of nonzero frontal slices of coefficients $\mathcal{B}$ during the iteration process. This adaptive strategy allows the FGSSR-based method dynamic adjustment of the subspace dimension $\mathit{d}$ to approximate the rank of the latent HR-HSI. Therefore, the FGSSR-based method can effectively capture the spectral correlation property and enhance the overall quality of the fused results. Figure 10 shows the variation of subspace dimension $\mathit{d}$ as the iteration increases. As shown in Figure 10, the subspace dimension $\mathit{d}$ exhibits a notable decline as the iteration progresses, ultimately reaching a state of convergence where it stabilises. Moreover, the convergence behavior of the subspace dimension differs between the simulated PU dataset and the simulated WDCM dataset. This emphasizes the significance of our proposed FGSSR-based method, which has the ability to dynamically identify the optimal subspace dimension, ensuring accurate and robust fused results across a wide range of datasets.

5.4. Computation Efficiency

Table 6. Calculation time of test methods on simulated datasets (in seconds).

Dataset	CNMF	HySure	NSSR	JSSLL1	GSFus	LTMR	Proposed
PU	28.87	88.87	142.29	174.68	236.47	155.87	24.35
WDCM	37.14	94.40	145.68	211.45	243.33	160.92	25.84

provides the calculation time in seconds for the test methods. All experiments were executed on a computer featuring a 1.7 GHz Intel Core i5-1240P CPU and 16 GB of RAM using MathWorks MATLAB R2016a. The primary components contributing to the computational complexity of Algorithm 3 are the $\mathcal{B}$ and $\mathcal{S}$ subproblems. For the $\mathcal{B}$ subproblem, tensor product and TNN computations take up the majority of CPU time. The $\mathcal{S}$ subproblem is a nonconvex sparse problem that can be efficiently addressed using the GST algorithm. From Table 6, it is evident that our FGSSR-based method consumes the least calculation time compared to other methods, despite the presence of inter and outer iterations in Algorithm 3.

5.5. Numerical Convergence

Figure 11 illustrates the variation in relative change (Rech), defined as $(\parallel {\mathcal{B}}^{k+1}\times {\mathbf{A}}^{k+1}-{\mathcal{B}}^k\times {\mathbf{A}}^k{\parallel }_F)/(\parallel {\mathcal{B}}^k\times {\mathbf{A}}^k{\parallel }_F)$, of the FGSSR-based method on the simulated PU and WDCM datasets as the iteration progresses. As shown in Figure 11, the Rech quickly converges to zero within a few iterations, demonstrating the fast and favorable numerical convergence of our FGSSR-based method.

6. Conclusions

In this study, we present a novel FGSSR-based method for HSI super-resolution. In our FGSSR-based method, the high spectral correlation property in HR-HSI is effectively modeled by the proposed FGSSR model, which offers several advantages, including enhanced rank minimization, adaptive subspace dimension determination, and low computational cost. Moreover, we employ TNN regularization on the low-dimensional coefficients to depict the spatial self-similarity property in HR-HSI, which offers high efficiency while effectively preserving the spatial features of HR-HSI. An efficient PAM-based algorithm is developed to solve the FGSSR-based model. The FGSSR-based method is evaluated against the SOTA HSI super-resolution methods using the simulated and real-world datasets. The comparison is conducted through visual and quantitative analysis, which reveals the superior performance of our FGSSR-based method.

Despite these promising results, there are several areas that require further investigation. For example, our proposed method currently concentrates solely on the global spatial self-similarity of HR-HSI, which aids in reducing computational complexity but may limit its ability to reconstruct spatial structure and preserve spatial details. Therefore, our future work will focus on employing the nonlocal self-similarity property of HR-HSI under the FGSSR framework to enhance the performance of HSI super-resolution. In addition, we plan to further extend our proposed method to a broader range of hyperspectral datasets for comparative analysis, aiming to provide a more comprehensive stability assessment of our proposed method. Furthermore, work [62] provides a novel sight to improve the quality of remote sensing images. Thus, our other future work is focused on enhancing the spatial resolution of HSI from the perspective of dynamic differential evolution theory.