Coupled Tensor Block Term Decomposition with Superpixel-Based Graph Laplacian Regularization for Hyperspectral Super-Resolution

Abstract

Hyperspectral image (HSI) super-resolution aims at improving the spatial resolution of HSI by fusing a high spatial resolution multispectral image (MSI). To preserve local submanifold structures in HSI super-resolution, a novel superpixel graph-based super-resolution method is proposed. Firstly, the MSI is segmented into superpixel blocks to form two-directional feature tensors, then two graphs are created using spectral–spatial distance between the unfolded feature tensors. Secondly, two graph Laplacian terms involving underlying BTD factors of high-resolution HSI are developed, which ensures the inheritance of the spatial geometric structures. Finally, by incorporating graph Laplacian priors with the coupled BTD degradation model, a HSI super-resolution model is established. Experimental results demonstrate that the proposed method achieves better fused results compared with other advanced super-resolution methods, especially on the improvement of the spatial structure.

1. Introduction

Hyperspectral remote sensing has been widely used in the fields of geological exploration, agricultural production, urban planning, environmental monitoring and so on. Due to the limitation of the optical imaging mechanism of airborne spectrometers, hyperspectral image (HSI) with high spectral resolution is often accompanied by lower spatial resolution, which brings inconvenience to the applications, such as classification, anomaly detection, and object recognition [1,2]. Therefore, improving the spatial resolution of HSI has become an urgent issue. Fortunately, multispectral image (MSI) has high spatial resolution but with low spectral resolution. Therefore, the fusion of HSI and MSI (HSI–MSI), also called HSI super-resolution, provides an effective and efficient way to improve the spatial resolution of HSI and results in a high-resolution hyperspectral image (HR-HSI). Furthermore, by fusing HSI–MSI, the shortcoming of a single imaging device in spatial-spectral resolution has been overcome, and a more comprehensive and accurate understanding of the observed environment can be obtained [3,4].

1.1. Relates Works

In general, the HSI–MSI fusion methods can be categorized as four classes [5,6,7], pansharpening-based methods [8,9], matrix-based methods [10,11,12], tensor-based methods [13,14,15,16,17], and deep CNN-based methods [18,19]. Among them, matrix and tensor methods depend on the degradation model of high-resolution HSI combined with priors of the decomposition matrix or tensor factors. Regularization priors, such as low-rank [20,21,22,23,24], sparsity [25,26,27], or graph Laplacian [28,29], are widely used in HSI–MSI fusion models. However, in matrix-based methods, the HSI is unfolded into matrices, which ignores the inherent three-dimensional features. Tensor, as a multi-dimension array, provides a flexible representation of HSI and has made tensor-based HSI super-resolution methods popular recently.

In tensor-based HSI super-resolution, the HSI is modeled as a third-order tensor with two spatial dimensions and one spectral dimension, which can fully exploit the dependence across different dimensions or modes. Moreover, tensor has more flexible decomposition forms, such as canonical polyadic decomposition (CPD) [30], Tucker decomposition (TD) [23], singular value decomposition (t-SVD) [31,32], tensor ring decomposition (TRD) [33,34], tensor block term decomposition (BTD) [35,36], etc. Each decomposition leads to a different perspective in understanding the correlation between the different modes of the tensor. Canonical polyadic decomposition (CPD) represents a tensor with a sum of R tensors, in which each tensor is rank-1. Kanatsoulis et al. [37] first formulated a coupled tensor CPD-based HSI–MSI fusion model by establishing the relations between tensor mode-product and the HSI degradation model. Furthermore, the blind and semi-blind HSI–MSI fusion models were solved by the alternative optimization algorithm, which is a well-known STEREO algorithm. Xu et al. [38] proposed a HSI super-resolution model based on non-local coupled tensor patches, in which the constructed fourth-order low-rank tensor is guided by MSI, and the HR-HSI and MSI share the same nonlocal tensor CPD factor matrices. Although CPD can represent the three-dimensional data structure, it has a high computational complexity and the CP rank R is hard to compute. Tucker decomposition (TD) provides a flexible decomposition with a core tensor and three matrix factors. Each factor of TD is easy to compute. Assuming the HR-HSI has a low Tucker multilinear rank, a coupled TD-based HSI–MSI fusion model with blind and semi-blind SCOTT algorithms is proposed in [16]. Furthermore, the decomposed core tensor was estimated by solving the generalized Sylvester equation, and three factor matrices were computed by truncated singular value decomposition. In addition, TD factors can be viewed from spatial and spectral dimensions, so a dictionary learning strategy was introduced to train spatial and spectral dictionaries from MSI and low-resolution HSI, respectively [13]. Furthermore, by considering the spectral smoothness and spatial consistency as priors, a graph regularized low-rank tensor fusion method was developed in [17]. Borsoi et al. [39] assumed that there is spectral variability in HSI–MSI fusion and introduced the variability into the TD fusion model with an additive term. In general, tensor CPD and TD can be unified as tensor BTD. Specifically, tensor BTD provides a clear physical explanation for the factor matrices from the perspective of unmixing, and makes the prior of the abundance and endmember easy to model [40,41,42]. Therefore, a coupled BTD-based HSI–MSI fusion technique has become popular for the linear unmixing model involved [43,44]. The endmember and low-rank abundance map are represented by potential factor matrices under BTD. However, due to the lack of constraints on the factor matrix, the quality of the fused HSI is reduced. Therefore, other regularization terms have the potential to be combined with BTD in fusion models to improve the recovery performance.

1.2. Motivations and Contributions

In terms of regularization, factors such as total variation [45], low-rank, sparse, graph Laplacian [46], etc. have been widely exploited combined with tensor decomposition as mentioned. Among them, manifold Laplacian is an effective strategy to improve the spatial structure of images. In [46], a graph Laplacian is integrated with BTD for HSI–MSI fusion, resulting in an impressive performance. However, the local weight matrix of the graph is calculated in a pixel-wise manner, which can easily be affected by noise and cannot describe the local geometry of the image well. Therefore, to exploit the spatial neighborhood structure by generating homogeneous segmented regions and reduce the sensitivity to noise and outliers, a superpixel-guided graph Laplacian regularization is constructed in this paper. Furthermore, considering the merits of the BTD, the graph Laplacian is introduced to the BTD-based HSI–MSI fusion framework, resulting in the regularization of the proposed superpixel-based graph Laplacian with the BTD fusion method, named as SGLCBTD for short. The main contributions are as follows.

(1)

The MSI is segmented by regional clustering according to spectral–spatial distance measurements;

(2)

Two-directional tensor graphs are designed via the features of the segmented MSI superpixel blocks, whose local geometric structure is consistent with HSI;

(3)

The similarity weights of the superpixel blocks are calculated and graph Laplacian matrices are constructed, which is used to convey the spatial manifold structures from MSI to the factor matrices of HSI;

(4)

The proposed superpixel graph Laplacian BTD model is solved by the block coordinate descent algorithm, and the experimental results are displayed.

2. Background

For convenience, some necessary definitions and preliminaries of tensor are introduced first. A scalar, a vector, a matrix, and a tensor are denoted as $x,\mathit{x},\mathit{X}$ and $\mathcal{X}$, respectively. $x_i,{\mathit{X}}_{ij}$ and ${\mathcal{X}}_{ijk}$ denote the $i$-th, $\left(i,j\right)$-th and $\left(i,j,k\right)$-th element of $\mathit{x}\in {ℝ}^I,X\in {ℝ}^{I\times J}$ and $\mathcal{X}\in {ℝ}^{I\times J\times K}$, respectively. The n-mode unfolding of tensor $\mathcal{X}$ is represented by ${\mathit{X}}^{\left(n\right)}$. The one-mode product of tensor $\mathcal{X}\in {ℝ}^{I\times J\times K}$ with a matrix $\mathit{A}\in {ℝ}^{M\times I}$ is denoted by $\mathcal{Y}=\mathcal{X}{\times }_1\mathit{A}=\mathit{A}{\mathit{X}}^{\left(1\right)}$, thus, for the n-mode products, n = 1, 2, 3. $\mathit{A}\circ \mathit{x}$ stands for the outer product of a matrix $\mathit{A}\in {ℝ}^{M\times \mathrm{N}}$ and a vector $\mathit{x}\in {ℝ}^I$, resulting in an $M\times N\times I$ tensor. For two matrices $\mathit{A}\in {ℝ}^{I\times K}$ and $\mathit{B}\in {ℝ}^{I\times K}$, the Khatri–Rao product is $\mathit{A}\odot \mathit{B}\in {ℝ}^{IJ\times K}$, and ${\odot }_p$ stands for the column-wise Khatri–Rao product along the column.

2.1. Block Term Decomposition

The block term decomposition (BTD) in rank-$\left(L_r,L_r,1\right)$. terms is defined that a third-order tensor $\mathcal{X}\in {ℝ}^{I\times J\times K}$. can be decomposed as the sum of rank-$\left(L_r,L_r,1\right)$ terms [35]

$$\mathcal{X}\approx {{\displaystyle \sum }}_{r=1}^R\left({\mathit{A}}_r{\mathit{B}}_r^T\right)\circ {\mathit{c}}_r$$

(1)

where ${\mathit{A}}_r\in {ℝ}^{I\times L_r}$ ${\mathit{B}}_r\in {ℝ}^{J\times L_r}$ are full-column matrices with rank-Lr, and ${\mathit{c}}_r\in {ℝ}^K,$ R is the rank of the tensor $\mathcal{X}$.

Let $={\mathit{B}}_r^T\in {ℝ}^{I\times \mathrm{J}}$ and replace it into formula (1), so

$$\mathcal{X}\approx {{\displaystyle \sum }}_{r=1}^R{\mathit{S}}_r\circ {\mathit{c}}_r$$

(2)

Furthermore, unfolding the tensor along the third mode results in the following expression from the matrix point of view:

$${\mathit{X}}^{\left(3\right)}=\mathit{C}\mathit{S}+\mathit{\epsilon }$$

(3)

where ${\mathit{X}}^{\left(3\right)}\in {ℝ}^{K\times IJ}$ is the three-mode unfolding of tensor $\mathcal{X}$, matrices $\mathit{C}=\left[{\mathit{c}}_1,\cdots ,{\mathit{c}}_R\right]\in {ℝ}^{K\times R}$ and $\mathit{S}=\left[\mathit{S}\left(1,:\right),\cdots ,\mathit{S}\left(R,:\right)\right]\in {ℝ}^{R\times IJ}$ have rank-R, each element $\mathit{S}\left(r,:\right)\in {ℝ}^{IJ}$ of $\mathit{S}$ constitutes the matrix ${\mathit{S}}_r$ with size of $I\times \mathrm{J}$, the error term $\epsilon$ is the Gaussian noise.

The Formula (3) can be easily connected with the linear unmixing model (LMM) of HSI. Specifically, matrix $\mathit{C}$ is the endmember matrix containing the spectral signatures of $R$ endmembers ${\mathit{c}}_1,{\mathit{c}}_{2 },\dots {\mathit{c}}_R,$ matrix $\mathit{S}$ represents the abundance coefficient matrix, each ${\mathit{S}}_r$ is the abundance corresponding to endmember ${\mathit{c}}_r$. Therefore, decomposition factors ${\mathit{A}}_r$ and ${\mathit{B}}_r$ of matrix ${\mathit{S}}_r$ also represent the spatial abundance information.

2.2. Problem Formulation

Let ${\mathcal{Y}}_h\in {ℝ}^{I_h\times J_h\times K_H} ,{\mathcal{Y}}_m\in {ℝ}^{I_M\times J_M\times K_m}$ represent the low spatial resolution hyperspectral image (LR-HSI) and MSI, respectively. Let ${\mathcal{Y}}_{\mathit{s}}\in {ℝ}^{I_M\times J_M\times K_H}$ be the super-resolution hyperspectral image (SRI) to be estimated. $I_h,$ $I_M$ and $J_h, J_M$ represent the dimensions of the spatial width and height, respectively, $K_H,K_m$ represent the spectral dimension, and $I_h\ll I_M, J_h\ll J_M$,$K_m\ll K_H$. It is often assumed that LR-HSI is obtained by spatial degradation of SRI, and MSI is the spectral degradation of SRI, namely,

$${\mathcal{Y}}_h={\mathcal{Y}}_{\mathit{S}}{\times }_1{\mathit{P}}_1{\times }_2{\mathit{P}}_2$$

(4)

$${\mathcal{Y}}_m={\mathcal{Y}}_{\mathit{S}}{\times }_3{\mathit{P}}_3$$

(5)

where ${\mathit{P}}_1\in {ℝ}^{I_h\times I_M}$ and ${\mathit{P}}_2\in {ℝ}^{J_h\times J_M}$ are the spatial blurring and downsampling matrices, ${\mathit{P}}_3\in {ℝ}^{K_m\times K_H}$ is a spectral response matrix.

Then, the HSI–MSI fusion can be formulated as

$${\min }_{{\mathcal{Y}}_{\mathit{S}}}\left|\right|{\mathcal{Y}}_h-{\mathcal{Y}}_{\mathit{S}}{\times }_1{\mathit{P}}_1{\times }_2{\mathit{P}}_2|{|}_F^2+||{\mathcal{Y}}_m-{\mathcal{Y}}_{\mathit{S}}{\times }_3{\mathit{P}}_3|{|}_F^2$$

(6)

By assuming ${\mathcal{Y}}_{\mathit{S}}$ follows the rank-$\left(L_r,L_r,1\right)$ model as shown in (1), the HSI–MSI fusion model (6) can be rewritten as follows:

$$mi{n}_{{\mathit{A}}_r,{\mathit{B}}_r,{\mathit{C}}_r}\Vert {\mathcal{Y}}_h-{{\displaystyle \sum }}_{r=1}^R\left({\mathit{P}}_1{\mathit{A}}_r{\left({\mathit{P}}_2{\mathit{B}}_r\right)}^T\right)\circ {\mathit{c}}_r{\Vert }_F^2+\Vert {\mathcal{Y}}_m-{{\displaystyle \sum }}_{r=1}^R\left({\mathit{A}}_r{\mathit{B}}_r^T\right)\circ {\mathit{P}}_3{\mathit{c}}_r{\Vert }_{\mathrm{F}}^2$$

(7)

Formula (7) tells us that estimating SRI ${\mathcal{Y}}_{\mathit{S}}$ with BTD is equal to estimating the high-resolution abundance map ${\left\{{\mathit{A}}_r{\mathit{B}}_r^T\right\}}_{r=1}^R$ and endmember matrix ${\mathit{C}}_r$. The theorem in [44,45] gives the recoverability of a hyperspectral super-resolution problem under the BTD of rank-$\left(L_r,L_r,1\right)$.

3. Proposed Methods

Generally, the HSI–MSI fusion model (7) is often added with more constraints to achieve a stable and accurate solution, such as total variation, sparsity, low-rank, graph Laplacian, etc. Among them, graph Laplacian can preserve the local manifold structure of high-dimensional image data. Moreover, considering that the superpixel provides local homogeneous regions with geometric structure involved, a superpixel-based graph Laplacian combined with a coupled BTD HSI–MSI fusion model (SGLCBTD) is proposed.

The flowchart is given in Figure 1. As it shows, the core idea of the proposed method is the construction of two graph Laplacian. To achieve this goal, four steps are performed: (1) segmenting the MSI to generate superpixel blocks; (2) extracting the features of the superpixel blocks to form two feature tensors; (3) constructing two graphs based on feature tensors, including compute the weights; and (4) establishing two graph Laplacian terms. Finally, the regularized graph Laplacian terms are introduced to the HSI–MSI fusion framework with the BTD formula.

3.1. Superpixel-Based Graph Laplician Construction

3.1.1. Regional Clustering-Based Superpixel Segmentation

Compared with the pixel-wise segmentation method, the superpixel segmentation method shows faster speed and more accurate segmentation results. One of the most popular superpixel segmentation methods is the simple linear iterative clustering (SLIC) algorithm. Inspired by the SLIC method, a regional clustering superpixel segmentation method is designed for HSI unmixing by integrating spatial correlation and spectral similarity at clustering procedure [47]. Specifically, the combination of spectral information divergence (SID) and spectral angler mapper (SAM) is employed as a spectral distance measurement, and Euclidean distance is used as a spatial distance. Taking this merit of the superpixel segmentation method into account, in this paper, the regional clustering superpixel segmentation method is employed to generate superpixel blocks.

3.1.2. Two-Directional Feature Tensors Extraction

Similarly to the MSI, the resultant superpixel block shows geometric spatial structure as well as spectral information. Along horizontal and vertical spatial directions, the superpixel has different features. Therefore, it is necessary to design two-directional feature tensors to represent the geometric structure.

For each irregular superpixel block $S_k$, along the horizontal direction, feature tensor ${\mathcal{S}}_h^k$ is constructed by its horizontal feature vectors (such as average, maximum, median or difference, etc.). Then, all horizontal feature tensors are arranged according to the second dimension, resulting in a feature tensor ${\mathcal{S}}_h$ with a size of $I_M\times N\times K_m,$ where N is the number of the superpixel block. Meanwhile, the vertical feature tensor ${\mathcal{S}}_{\mathit{V}}\in {ℝ}^{I_M\times J_M\times K_m}$ is generated in the same manner.

3.1.3. Two Graph Generation

A horizontal graph ${\mathit{G}}_h=\left({\mathit{V}}_h,{\mathit{E}}_h\right)$ with vertex ${\mathit{V}}_h$ and edge ${\mathit{E}}_h$ is defined by horizontal feature tensor ${\mathcal{S}}_h$ derived from superpixel blocks. To be specific, taking the one-mode factorization of tensor ${\mathcal{S}}_h$ as elements of the vertex ${\mathit{V}}_h=\left({\mathit{S}}_h^{\left(1\right)} \left(1,:\right) ,\cdots ,{\mathit{S}}_h^{\left(1\right)} \left(I_M,:\right) \right)$, the weigh ${\mathit{w}}_h^{pq}$ of the edge ${\mathit{E}}_h=\left\{e_{p,q}^h\right\}$ can be calculated as follows

$$w_h^{pq}={\mathit{e}}^{-\frac{|\left|{\mathit{S}}_h^{\left(1\right)} \left(p,:\right)-{\mathit{S}}_h^{\left(1\right)} \left(q,:\right) \right|{|}^2}{{\sigma }^2}}$$

(8)

where ${\mathit{S}}_h^{\left(1\right)} \left(\mathit{p},:\right)$ and ${\mathit{S}}_h^{\left(1\right)} \left(\mathit{q},:\right)$ represents the $\mathit{p}$-th row and $\mathit{q}$-th row of ${\mathit{S}}_h^{\left(1\right)}$, respectively, and $\sigma$ is the bandwidth of Gaussian kernel.

Meanwhile, a vertical graph $=$ with vertex ${\mathit{V}}_{\mathit{v}}$ and edge ${\mathit{E}}_{\mathit{v}}$ can be defined according to the vertical feature tensor ${\mathcal{S}}_{\mathit{v}}$ in the same way.

3.1.4. Two Graph Laplacian Construction

The graph shows the spatial correlation between superpixels, which is the same as that of SRI, for the fused SRI and MSI have the same spatial structure. The manifold structure in MSI is incorporated into SRI as illustrated in Figure 2, where ${\mathit{A}}_r$ and ${\mathit{B}}_r$ are the decomposition factor of BTD. With the BTD factorization, the factor matrix ${\mathit{A}}_r$ represents the spatial information along the horizontal direction (first mode of the tensor), so the row relationship of HR-HSI and MSI is the same as that of ${\mathit{A}}_r$. Similarly, factor matrix ${\mathit{B}}_r$ represents the vertical (second mode of tensor) information, so the relationship between columns in HR-HSI and MSI is the same of that of the columns of ${\mathit{B}}_r.$

Therefore, the similarity between rows in ${\mathit{A}}_r$ can be formulated as:

$${{\displaystyle \sum }}_{\mathit{p}=1}^{I_M}{{\displaystyle \sum }}_{\mathit{q}=1}^{I_M}\left|\right|{\mathit{A}}_r \left(\mathit{p},:\right)-{\mathit{A}}_r \left(\mathit{q},:\right) {\Vert }^2{\mathit{w}}_h^{\mathit{p}\mathit{q}}$$

(9)

where ${\mathit{A}}_r \left(\mathit{p},:\right)$ and ${\mathit{A}}_r \left(\mathit{q},:\right)$ represents the $\mathit{p}$-th row and $\mathit{q}$-th row of $A_r,$ respectively, ${\mathit{w}}_h^{\mathit{p}\mathit{q}}$ is the weight of the horizontal graph.

Let ${\mathit{W}}_h=\left({\mathit{w}}_h^{\mathit{p}\mathit{q}}\right) \in {ℝ}^{I_M\times I_M}$, which stands for the horizontal weighted adjacency matrix. Then, the horizontal graph Laplacian matrix ${\mathit{L}}_h$ can be defined as ${\mathit{L}}_h={\mathit{D}}_h-{\mathit{W}}_{h },$ where ${\mathit{D}}_h$ is a diagonal matrix with diagonal elements $d_h^{ii}={\displaystyle \sum }_j{\mathit{w}}_h^{ij}$.

Therefore, Formula (9) can be rewritten as follows:

$${{\displaystyle \sum }}_{\mathit{p}=1}^{I_M}{{\displaystyle \sum }}_{\mathit{q}=1}^{I_M}\left|\right|{\mathit{A}}_r \left(\mathit{p},:\right)-{\mathit{A}}_r \left(\mathit{q},:\right) {\Vert }^2{\mathit{w}}_h^{\mathit{p}\mathit{q}}=\mathrm{Tr}\left({\mathit{A}}_r^T{\mathit{L}}_h{\mathit{A}}_r\right)$$

(10)

Extending this constraint to $\mathit{A}=\left[{\mathit{A}}_1,\cdots ,{\mathit{A}}_R\right]$, the horizontal graph Laplacian can be written as

$${{\displaystyle \sum }}_{r=1}^R\mathrm{Tr}\left({\mathit{A}}_r^T{\mathit{L}}_h{\mathit{A}}_r\right)=\mathrm{Tr}\left({\mathit{A}}^T{\mathit{L}}_h\mathit{A}\right)$$

(11)

In the same way, the vertical graph Laplacian related to ${\mathit{B}}_r$ is defined as

$${{\displaystyle \sum }}_{r=1}^R\mathrm{Tr}\left({\mathit{B}}_r^T{\mathit{L}}_v{\mathit{B}}_r\right)=\mathrm{Tr}\left({\mathit{B}}^T{\mathit{L}}_v\mathit{B}\right)$$

(12)

where ${\mathit{L}}_v$ is the vertical graph Laplacian matrix, $\mathit{B}=\left[{\mathit{B}}_1,\cdots ,{\mathit{B}}_R\right]$.

3.2. Proposed SGLCBTD Model and Algorithm

Incorporating the two mentioned graph Laplacian terms (11–12) with the coupled BTD super-resolution model (7), a superpixel graph Laplacian regularization with coupled BTD fusion model is proposed (SGLCBTD):

$$\begin{gathered} {\min }_{{\mathit{A}}_r\ge 0,{\mathit{B}}_r\ge 0,c_r\ge 0}\Vert {\mathcal{Y}}_h-{{\displaystyle \sum }}_{r=1}^R\left({\mathit{P}}_1{\mathit{A}}_r{\left({\mathit{P}}_2{\mathit{B}}_r\right)}^T\right)\circ {\mathit{c}}_r{\Vert }_F^2+\Vert {\mathcal{Y}}_m-{{\displaystyle \sum }}_{r=1}^R\left({\mathit{A}}_r{\mathit{B}}_r^T\right)\circ \\ {\mathit{P}}_3{c}_r{\Vert }_F^2+{\lambda }_1\mathrm{Tr}\left({\mathit{A}}^T{\mathit{L}}_h\mathit{A}\right)+{\lambda }_2\mathrm{Tr}\left({\mathit{B}}^T{\mathit{L}}_{\mathit{v}}\mathit{B}\right) \end{gathered}$$

(13)

where ${\lambda }_1$ and ${\lambda }_2$ are regularization parameters.

Let us define the objective function in Equation (13) as $J \left(\mathit{A},\mathit{B},\mathit{C}\right)$, the above super-resolution model is solved alternately by block coordinate descent (BCD) algorithm [48], i.e., the matrices $\mathit{A},\mathit{B},\mathit{C}$ are iteratively updated via solving subproblems w.r.t. while fixing other variables as follows

$${\mathit{A}}_{t+1}\leftarrow {\mathrm{argmin}}_{A\ge 0} J\left(\mathit{A},{\mathit{B}}^t,{\mathit{C}}^t\right)$$

(14a)

$${\mathit{B}}_{t+1}\leftarrow {\mathrm{argmin}}_{\mathit{B}\ge 0} J\left({\mathit{A}}^{t+1},\mathit{B},{\mathit{C}}^t\right)$$

(14b)

$${\mathit{C}}_{t+1}\leftarrow {\mathrm{argmin}}_{\mathit{C}\ge 0} J\left({\mathit{A}}^{t+1},{\mathit{B}}^{t+1},\mathit{C}\right)$$

(14c)

where $t$ is the number of current iteration steps.

Each subproblem is a quadratic optimization problem, which leads to the generalized Sylvester equation and can be transformed into a large-scale sparse linear system of equations by Kronecker product.

Consider the subproblem (14a), fix $\mathit{B}$ and $\mathit{C}$, the subproblem of $\mathit{A}$ can be rewritten as

$$\begin{gathered} {\min }_{\mathit{A}\ge 0}\left|\right|{\mathit{Y}}_h^{\left(1\right)}-\left(\mathit{C}{\odot }_p{\mathit{P}}_2\mathit{B}\right){\left({\mathit{P}}_1\mathit{A}\right)}^T|{|}_F^2+||Y_m^{\left(1\right)}-\left({\mathit{P}}_3\mathit{C}{\odot }_{p}B\right){\mathit{A}}^T|{|}_F^2 \\ +{\lambda }_1\mathrm{Tr}\left({\mathit{A}}^T{\mathit{L}}_A\mathit{A}\right) \end{gathered}$$

(15)

The optimization problem is quadratic, and its solution is equivalent to compute the following general Sylvester equation [49]:

$${\mathit{P}}_1^T{\mathit{P}}_1\mathit{A}{\mathit{Q}}^T\mathit{Q}+\mathit{A}{\mathit{K}}^T\mathit{K}+2{\lambda }_1{\mathit{L}}_A\mathit{A}={\mathit{P}}_1^T{\mathit{Y}}_h^{\left(1\right)T}\mathit{Q}+{\mathit{Y}}_m^{\left(1\right)T}\mathit{K}$$

(16)

where $\mathit{Q}=\mathit{C}{\odot }_p{\mathit{P}}_2\mathit{B},\mathit{K}={\mathit{P}}_3\mathit{C}{\odot }_p\mathit{B}$.

For the solution of subproblem of matrices B and C, the algorithms are the same as that of matrix A.

4. Experimental Results

4.1. Experiment Setup

In this section, to demonstrate the effectiveness of the proposed HSI super-resolution method, numerical experiments are carried out on two popular datasets: Indian Pines and Pavia University datasets (https://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes, accessed on 20 March 2022). Both qualitative and quantitative analysis are used in our experiments.

4.1.1. Quality Assessment Indices

For quantitative performance, seven indices are employed to evaluate the performance [7].

(1)

Normalized mean square error (NMSE) is defined as

$$NMSE=\frac{\left|\right|\mathcal{Y}-\widehat{\mathcal{Y}}|{|}_F^2}{\left|\right|\mathcal{Y}|{|}_F^2}$$

(17)

where $\mathcal{Y}\in {ℝ}^{I\times J\times K}$ is the ideal HSI and $\widehat{\mathcal{Y}}\in {ℝ}^{I\times J\times K}$ is the resulted SRI.

(2)

Reconstruction signal-to-noise ratio (R-SNR) is inversely proportional to NMSE with the formulation as follows:

$$R\text{-}SNR=10\lg \left(\frac{\left|\right|\mathcal{Y}|{|}_F^2}{\left|\right|\mathcal{Y}-\mathcal{Y}|{|}_F^2}\right)$$

(18)

(3)

Spectral angle mapper (SAM) evaluates the spectral distortion and is defined as

$$SAM=\frac{1}{IJ}{{\displaystyle \sum }}_{n=1}^{IJ}\arccos \left(\frac{{\mathit{Y}}^{\left(3\right)}\left(n,:\right),\widehat{{\mathit{Y}}^{\left(3\right)}}\left(n,:\right)}{|\left|{\mathit{Y}}^{\left(3\right)}\left(n,:\right)\right|{|}_2\cdot |\left|\widehat{{\mathit{Y}}^{\left(3\right)}}{\left(n,:\right)}^T\right|{|}_2}\right)$$

(19)

where ${\mathit{Y}}^{\left(3\right)}\left(n,:\right)$ is the spectral vector, $\langle .\rangle$ is inner product of two vectors.

(4)

Relative global dimensional synthesis error (ERGAS) reflects the global quality of the fused results and is defined as

$$ERGAS=100d\sqrt{\frac{1}{K}{{\displaystyle \sum }}_{k=1}^K\frac{|\mathcal{Y}\left(:,:,k\right)-\widehat{\mathcal{Y}}\left(:,:,k\right)|{|}_F^2}{{\mu }_k^2}}$$

(20)

where $d$ is spatial upsampling factor, ${\mu }_k$ is the mean of $\widehat{\mathcal{Y}}$.

(5)

Correlation coefficient (CC) is computed as follows

$$CC=\frac{1}{K}{{\displaystyle \sum }}_{k=1}^K\rho \left(\mathcal{Y}\left(:,:,k\right),\widehat{\mathcal{Y}}\left(:,:,k\right)\right)$$

(21)

where $\rho$ is the Pearson correlation coefficient.

(6)

Peak signal-to-noise rate (PSNR) for each band of HSI is defined as

$$PSNR=10\lg \left(\frac{\max \left(\left|\right|\mathcal{Y}\left(:,:,k\right)|{|}_F^2\right)}{\left|\right|\mathcal{Y}\left(:,:,k\right)-\widehat{\mathcal{Y}}\left(:,:,k\right)|{|}_F^2}\right)$$

(22)

(7)

Structural similarity index measurement (SSIM) for each band of HSI is defined as follows:

$$SSIM=\frac{\left(2{\mu }_{\mathcal{Y}}{\mu }_{\widehat{\mathcal{Y}}}+c_1\right)\left(2\sigma +c_2\right)}{\left({\mu }_{\mathcal{Y}}^2+{\mu }_{\widehat{\mathcal{Y}}}^2+c_1\right)\left({\sigma }_{\mathcal{Y}}^2+{\sigma }_{\widehat{\mathcal{Y}}}^2+c_2\right)}$$

(23)

where ${\mu }_1,{\mu }_2$ and ${\sigma }_1^2,{\sigma }_2^2$ are the mean and variance of the k-th band image $\mathcal{Y}\left(:,:,k\right)$ and $\widehat{\mathcal{Y}}\left(:,:,k\right)$, respectively, $\sigma$ is the covariance between $\mathcal{Y}\left(:,:,k\right)$ and $\widehat{\mathcal{Y}}\left(:,:,k\right)$, and $c_1,c_2$ are constant.

4.1.2. Methods for Comparison

The proposed SGLCBTD method is compared with the state-of-the-art HSI–MSI fusion methods, including: coupled nonnegative matrix factorization (CNMF) [3], super-resolution tensor-reconstruction (STEREO) [37], STEREO with non-negative CP decomposition (CNN-CPD) [37], coupled non-negative tensor block term decomposition (CNN-BTD) [43], and graph Laplacian regularization with coupled block term decomposition (GLCBTD) [46].

In the experiment, the optimal parameters involved in different methods are set according to the author’s suggestion. Specially, in tensor-based method, considering that the Pavia University dataset has rich spatial structures, the rank of tensor R is set to 15, but for the Indian Pines dataset, the rank is 10. Meanwhile, in the related BTD rank-$\left(L_r,L_r,1\right)$ methods, including CNN-BTD, GLCBTD and SGLCBTD, the rank of the factor matrices is $L_r=10$. In addition, regularization parameters ${\lambda }_1$ and ${\lambda }_2$ in SGLCBTD are set to ${10}^{-5}$. More details about the parameters are shown in Section 4.3.

4.2. Performance Comparison of Different Methods

4.2.1. Indian Pines Dataset

The Indian Pines dataset was captured by the NASA AVIRIS instrument. The size of the underlying SRI ${\mathcal{Y}}_{\mathit{S}}$ is 145 × 145 × 220 and the wavelength covers 400–2500 nm. The HSI and MSI are obtained according to the Wald protocol [50]. The degradation from SRI to HSI is that blurring with 9 × 9 Gaussian and downsampling with factor 5, resulting in ${\mathcal{Y}}_h$ with size of 29 × 29 × 220. The size of the MSI is 145 × 145 × 4. Finally, zero-mean iid. Gaussian noise is added to HSI and MSI, and the signal-to-noise ratio (SNR) is set to 30 dB.

Table 1. Quantitative indices comparison on the Indian Pines dataset.

Algorithm	R-SNR	NMSE	SAM	ERGAS	CC
CNMF	25.24	0.0546	0.0398	1.3320	0.7589
STEREO	26.25	0.0487	0.0413	1.1254	0.7970
CNN-CPD	26.41	0.0478	0.0365	1.0891	0.7980
CNN-BTD	27.32	0.0430	0.0346	1.0394	0.8070
GLCBTD	28.02	0.0397	0.0319	0.9507	0.8193
SGLCBTD	29.78	0.0324	0.0277	0.9610	0.8128

lists the values of R-SNR, NMSE, SAM, ERGAS and CC of the compared methods on the Indian Pines dataset, and the best values are marked in bold. It can be shown that the tensor-based methods outplay matrix-based methods. The indexes of BTD fusion methods are much better than that of CPD. Moreover, the regularized models, CNN-CPD, GLCBTD and SGLCBTD are more advantageous than STEREO and CNN-BTD methods, for the latter two have rather low indices. Most of indices of the proposed SGLCBTD achieve first place, which shows that the method performs well in spatial detail and spectral preservation. In addition, compared with the GLCBTD, the R-SNR is significantly improved by about 1.76 dB.

The PSNR and SSIM values for each spectral band of various fusion methods are compared in Figure 3. As it can be seen, most of CNMF’s curves achieve the lowest values in each band, which means the performance of the matrix-based method is significantly lower than that of tensor-based methods. The curves of SREREO and CNN-CPD are very similar except for a higher SSIM index of CNN-CPD. Three BTD-based methods, CNN-BTD, GLCBTD and SGLCBTD perform better than other methods. The overall performances of GLCBTD and SGLCBTD are excellent, especially in SSIM curves, which shows that the graph-based method can preserve the geometric structure well. It should be mentioned, in the PSNR curve, from the 100-th band, the curve of SGLCBTD has unsatisfactory results, which is likely due to the water absorption, noise and parameters’ setting in our view. Even so, the curve is the most stable in each band of all compared methods. In general, the PSNR and SSIM curves show that the proposed SGLCBTD method has the superiority of preserving geometric structures.

Figure 4 and Figure 5 show the fusion results on the Indian Pines dataset in 10-th and 100-th bands, respectively. From the visual effect, it can be seen that SGLCBTD is closer to the original image with reduced red areas and increased blue areas, moreover, the edge details are clearer than those of other methods.

4.2.2. Pavia University Dataset

The Pavia University dataset was captured by the ROSIS instrument. The size of the original HSI is 610 × 340 × 115, where the spectral band is from 430 nm to 860 nm. The spatial size of 200 × 200 is cropped due to hardware limitations, and the spectral band is 103 after removing the vapor absorption bands. Therefore, the underlying SRI ${\mathcal{Y}}_{\mathit{S}}$ is 200 × 200 × 103. A 9 × 9 Gaussian blurring and downsampling factor 4 are performed as degradation to obtain a HSI ${\mathcal{Y}}_h$ with size of 50 × 50 × 103. The size of MSI ${\mathcal{Y}}_m$ is 200 × 200 × 4. Finally, zero mean iid. Gaussian noise is added to HSI and MSI, the signal-to-noise ratio (SNR) is set to 25 dB.

Table 2. Quantitative indices comparison on the Pavia University dataset.

Algorithm	R-SNR	NMSE	SAM	ERGAS	CC
CNMF	17.36	0.1355	0.1105	1.0453	0.9526
STEREO	18.21	0.1228	0.1465	1.0154	0.9624
CNN-CPD	18.75	0.1154	0.1052	0.8867	0.9652
CNN-BTD	19.41	0.1070	0.1042	0.8447	0.9702
GLCBTD	21.08	0.0883	0.0873	0.7100	0.9792
SGLCBTD	21.38	0.0853	0.0855	0.6919	0.9807

shows the numerical results of several comparison methods on the Pavia University dataset. It can be seen that for the Pavia University dataset with tensor rank-10, the fusion performance fusion of CNN-BTD and GLCBTD have been improved. GLCBTD is still better than CNN-BTD due to the regularization term, while SGLCBTD improves the R-SNR value of GLCBTD by 0.3 dB, and other indices have also been improved, which shows that the proposed SGLCBTD method can improve the spatial–spectral assessment values effectively.

Figure 6 shows the comparison of PSNR and SSIM curves on the Pavia University dataset of different fusion methods, in which the red solid line represents the proposed SGLCBTD method. It can be seen that the PSNR and SSIM curves of GLCBTD and SGLCBTD are roughly similar, and both of them have higher values than that of other methods. Furthermore, it should be noticed that the overall curves of SGLCBTD are the highest and most stable, which means the proposed SGLCBTD performs best among the compared methods.

As shown in Figure 7 and Figure 8, the 10-th band and the 80-th band of the fusion results on the Pavia University dataset image are displayed, respectively. CNMF, CNN-CPD and STEREO have more noise unremoved as well as more blur. Relatively, BTD-based methods achieve better fusion results, among them, regularized GLCBTD and SGLCBTD methods achieve better visual effects than that of CNN-BTD. In addition, the spatial edge and texture structures of the fused image of SGLCBTD is clearer with less noise, which shows the effectiveness of SGLCBTD fusion method.

4.3. Discussions

4.3.1. Parameter Analysis

The parameters involved in the SGLCBTD including the number of superpixel $N$, the tensor rank $R,$ the rank of factor matrices $L$, and ${\lambda }_1,{\lambda }_2$. The main contribution of this paper is the tensor BTD with a superpixel-based graph. Therefore, the parameter analysis is performed on R, L and N.

(1)

Analysis of R and L

Taking the Indian Pines dataset as an example, R-SNR is used to evaluate the performance of parameters R and L as shown in Figure 9. Given the range of R and L form 7–13 with increment 1. It can be seen that both of two curves reach the highest peak values around 10. In addition, with R or L decrease, the performance becomes worse. Considering the high computational complexity with high rank increase, for the Indian Pines, R = L = 10, while for the Pavia University dataset, R = 15, L = 10 on account of the complex geometric structures in the image.

(2)

Analysis of N

The proposed SGLCBTD method is a superpixel-based method, the number of superpixel N is crucial to the efficient as well as effectiveness. For two datasets, the parameter N is selected from the set {225, 324, 400, 625, 841, 900} and {225, 324, 400, 484, 625,784}, respectively, by experience. Figure 10 shows R-SNR curves as parameter N changes. From the Figure, for the Indian Pines dataset, R-SNR reaches the highest value when N = 400. When N > 400, the curve drops down then rises at the point N = 625. For the Pavia University dataset, R-SNR is relatively stable in the data range. In addition, when $N\in \left[400,700\right]$, the highest value is reached at N = 625. Compared to the Indian Pines dataset, the Pavia University dataset has a richer geometry, and the segmentation results should be finer, resulting in a larger number of superpixel blocks. It is also noted that large superpixel numbers increase the time computation and do not lead to better performance. Therefore, 400 and 625 are set as superpixel numbers for the Indian Pines and Pavia University datasets, respectively. More adaptive and accurate estimation of the parameter is still an open issue to be researched further.

4.3.2. Time Complexity Analysis

The computational time of the compared methods on the two datasets is listed in

Table 3. Computational time of the compared methods (seconds).

Method	Indian Pines Dataset	Pavia University Dataset
CNMF	9.02	11.34
STEREO	2.97	4.76
CNN-CPD	5.05	5.60
CNN-BTD	65	89
GLCBTD	534	771
SGLCBTD	1023	1245

. The running time results are recorded in MATLAB R2018b, using a GPU server with NVIDIA RTX 2080Ti/11GB. Owing to the advantages of the block coordinate descent (BCD) algorithm or the alternating direction method of multipliers (ADMM), all the compared methods converge quickly after 10 to 25 iterations. In addition, for each dataset, the time given in Table 3 is obtained by averaging the five times. As can be seen, the STEREO is the fastest method. GLCBTD and the proposed SGLCBTD methods cost more time than those of other methods due to the massive calculation of superpixel segmentation and adjacency matrix of the graph, which is a progressive work to be considered for us in the future.

5. Conclusions

In this paper, a HSI super-resolution method is proposed based on tenor block term decomposition, known as SGLCBTD. To preserve the spatial manifold structure of the fused HSI, two-directional spectral–spatial graphs are constructed according to feature tensors induced by the MSI segmented superpixel. Then, the manifold graph Laplacian is utilized to regularize the super-resolution HSI, resulting in the proposed HSI–MSI fusion method. In addition, the model is solved alternately by block coordinate descent algorithm. Stimulation experiments are conducted on different datasets. Compared with the state-of-the-art methods, the proposed SGLCBTD obtains better fusion performance with more spatial details retained.

For future work, two aspects should be taken into consideration. On one hand, the feature of superpixel blocks is limited, more powerful features can be exploited to further improve the preservation of the manifold structures of the HSI. On the other hand, as mentioned in Section 4.3.2, more efficient optimization algorithms need to be developed to reduce time complexity.