Multispectral and Hyperspectral Image Fusion Based on Regularized Coupled Non-Negative Block-Term Tensor Decomposition

Abstract

The problem of multispectral and hyperspectral image fusion (MHF) is to reconstruct images by fusing the spatial information of multispectral images and the spectral information of hyperspectral images. Focusing on the problem that the hyperspectral canonical polyadic decomposition model and the Tucker model cannot introduce the physical interpretation of the latent factors into the framework, it is difficult to use the known properties and abundance of endmembers to generate high-quality fusion images. This paper proposes a new fusion algorithm. In this paper, a coupled non-negative block-term tensor model is used to estimate the ideal high spatial resolution hyperspectral images, its sparsity is characterized by adding 1-norm, and total variation (TV) is introduced to describe piecewise smoothness. Secondly, the different operators in two directions are defined and introduced to characterize their piecewise smoothness. Finally, the proximal alternating optimization (PAO) algorithm and the alternating multiplier method (ADMM) are used to iteratively solve the model. Experiments on two standard datasets and two local datasets show that the performance of this method is better than the state-of-the-art methods.

1. Introduction

Hyperspectral sensors are capable of imaging target regions in multiple parts of the electro-magnetic spectrum; for example, ultraviolet, visible, near-infrared, and infrared, in dozens or even several hundred consecutive, relatively narrow bands simultaneously, acquiring images with high spectral resolution but low spatial resolution [1]. In other words, the images are acquired by perceiving the sunlight reflected from the object or scene, and the spatial resolution is inevitably reduced due to the limited solar irradiance [2]. Multispectral sensors, in contrast, acquire images with higher spatial resolution, but with insufficient spectral resolution. Thus, hyperspectral imagers (HSIs) are three-dimensional data cubes with two spatial dimensions and one spectral dimension, where the spectral dimension provides the power to identify materials in the scene. It makes HSI abundant in spectral information, which can be effectively used to improve the performance of computer vision and remote sensing. Additionally, multispectral imagers (MSIs) contain abundant spatial information. However, spectral and spatial information is crucial in hyperspectral imaging applications (e.g., image recognition, agricultural monitoring, etc.), and so multispectral and hyperspectral image fusion (MHF) techniques are catching the interest of researchers. MHF can fuse HSI and MSI to obtain high spatial resolution hyperspectral images (HRHSs). Among them, HSI has abundant spectral information and poor spatial resolution, while MSI is mainly characterized by less spectral information but high spatial resolution. Due to physical constraints, HRHS cannot be acquired by a single sensor at present.

Early spatial–spectral fusion research was extended via panchromatic sharpening, which produces high-resolution multispectral images (HR-MSI) mainly by fusing panchromatic images (PANs) with low spatial resolution MSIs. Existing panchromatic sharpening methods mainly add details from panchromatic images to MSIs to generate HR-MSIs. Meng et al. in [3] divided the panchromatic sharpening methods into component replacement (CS) [4,5] and multi-resolution analysis (MRA) [6,7,8]. They are mainly based on wavelet techniques [9,10], intensity hue saturation (IHS) methods [11], principal component analysis (PCA) [12], Laplace’s pyramid [13], or attempts at the generalized sharpening algorithms of CS and MRA for MHF problems [14]. However, in the course of extensive practice, it was found that these methods do not improve the spatial resolution of each hyperspectral band very well [15]. In recent years, with the update of algorithms, more and more researchers are interested in the MHF problems, which makes the solution for the MHF problem continuously in development. To a great extent, it promotes the development of diversity in MHF techniques. Low-rank matrix decomposition-based methods [16,17,18,19,20,21,22] and tensor-based methods [23,24,25,26,27,28,29,30] are widely popular.

In 2011, Yokoya et al. proposed a coupled non-negative matrix decomposition (CNMF) [16] method for MSI and HSI fusion problems in which the alternate spectral unmixing of the non-negative matrix decomposition (NMF) was used to estimate the abundance matrix and the endmembers matrix, but the fusion results were not satisfactory because the cost function of NMF was nonconvex. Simoes and Wei et al. proposed Hysure [19] and FUSE [20] successively by considering the target image as a low-rank matrix, but the results were also unsatisfactory. In 2018, Kanatsoulis [31] et al., based on the fact that both HSI and MSI are composed of a “cube” of “space × space × spectrum”, a new idea based on the coupled tensor decomposition (CTD) framework is proposed to solve the MHF problem. Here, the spectral image is formulated as a third-order tensor, the HSI-MSI fusion problem is solved using the coupled canonical polyadic decomposition (CPD) model, and the uniqueness of CPD is exploited. The recoverability of HRHS under mild conditions is demonstrated. In the literature [23], a similar idea was adopted to solve the MHF problem using Tucker decomposition. In 2020, Xu [25] et al. proposed a new hyperspectral super-resolution method based on unidirectional total variational (TV). The method uses the Tucker decomposition to decompose the target HRHS using a sparse core tensor, which is decomposed using a three-mode dictionary matrix, and then applies the $L_1$ norm to the core tensor to express the sparsity of the objective HRHS, and applies the unidirectional TV canon to the three-mode dictionary to represent the segmental stability of the objective HRHS. Although CPD and Tucker show good recoverability for MHF problems and both frameworks exhibit excellent performance, the physical interpretation of the underlying factors is unintroduced into the framework. In the presence of noise or modeling errors, the endmembers and abundance information cannot be used to produce better fusion results. Therefore, the canonical polyadic (CP) and Tucker methods are by no means necessarily better than the low-ranked matrix-based estimation methods, which do without recoverable support.

In contrast, the linear mixed model (LMM), which utilizes matrixed HSI and MSI low-rank structures, excels in this regard with a strong physical interpretation of the underlying factors (i.e., endmembers and abundance). In addition, many structural constraints and regularizations can be applied to improve algorithm capabilities. This information becomes more important when the data is noisy. However, the key to solving MHF using LMM is whether recovered HRHS can be identified. Some work in the literature [30] shows that under relatively restricted conditions, coupled matrix decomposition models are identifiable to recover HRHS, such as: the abundance matrix of MSI has sparsity. Based on this, Zhang [32] et al. in 2019 proposed to solve the hyperspectral super-resolution (HSR) problem using the block-term tensor decomposition (BTD) [33,34,35] model. When modeling the spectral image as a BTD model, the potential factors have explicit physical meaning, while the spectral image key components are also incorporated into the model, such as the endmembers and non-negative abundance maps, and spectrally and spatially smooth a priori information. The algorithm does not consider more types of regularization and constraints, and does not test the algorithm in multiple scenarios, which has limitations. In 2021, Ding [36] et al. formulated the HSR problem as a coupled LL1 tensor decomposition model, proved BTD recoverability, ensured that theoretical guarantees were not lost when using the LL1 model, and added regularization terms to improve the algorithm performance. However, the algorithm does not take the individual abundance matrices into account and the implementation scenario is limited to satellite images. Therefore, to take full advantage of the physical interpretation of the potential factors in the MHF problem, the algorithm performance is improved. In this paper, we propose a coupled non-negative BTD model to estimate HRHS and inscribe its sparsity by adding parametrization while introducing TV to inscribe segmental smoothness. The difference operators in the horizontal and vertical directions are added to the algorithm in this paper; the proposed algorithm in this paper is called 2D-CNBTD. The contributions of the proposed method are as follows.

(1)

In the process of recovering high spatial resolution hyperspectral images (HRHS), considering the sparsity of the endmember matrix and the abundance matrix, the $L_1$ norm representing sparsity is used to improve the performance of the fusion algorithm.

(2)

Because there is always noise and a non-smooth phenomenon in the acquired image, in order to avoid this phenomenon, two different operators are defined, into which are described their segmental smoothing.

(3)

During the experiment, not only are two different standard datasets, but also two local datasets about Ningxia are used to make the algorithm adaptability more extensive and its performance more convincing.

2. Background

Since HSI can be considered as a third-order tensor (three-dimensional array); here, it can be understood as a generalization of vectors and matrices. In existing research on fusion algorithms, low-ranked tensor-based methods are commonly used, while tensor-based methods are broadly classified into CP and Tucker decomposition models.

The CPD model [37] is arguably the most popular tensor rank decomposition model, and is also called the parallel factor analysis model (PARAFAC), which decomposes a tensor into a sum of multiple single-rank tensors; the process is shown in Figure 1. Under the CP model, a third-order tensor of rank R is transformed into a sum of tensors of rank one, which could be built as (see Appendix A for details of symbolic remarks in the text):

$$\underline{X}=\sum _{r=1}^R{\mathbf{A}}_{\mathbf{r}}\circ {\mathbf{B}}_{\mathbf{r}}\circ {\mathbf{C}}_{\mathbf{r}}$$

(1)

where ${\mathbf{A}}_{\mathrm{r}}\in {\mathbb{R}}^{I\times 1}$, ${\mathbf{B}}_{\mathrm{r}}\in {\mathbb{R}}^{J\times 1}$, ${\mathbf{C}}_{\mathrm{r}}\in {\mathbb{R}}^{K\times 1}$ are the factor matrices of the third-order tensor $\underline{X}\in {\mathbb{R}}^{I\times J\times K}$, respectively.

In addition, the Tucker model [38], which has been extensively applied in the imaging field, could be considered as a high-order version of PCA [12], which decomposes the tensor into a core tensor and three factor matrices; the process is shown in Figure 2. It is capable of being expressed as:

$$\underline{X}=\underline{G}\circ {}_{\mathbf{1}}\mathbf{A}\circ {}_{\mathbf{2}}\mathbf{B}\circ {}_{\mathbf{3}}\mathbf{C}$$

(2)

where $\mathbf{A}\in {\mathbb{R}}^{I\times n_1}$, $\mathbf{B}\in {\mathbb{R}}^{J\times n_2}$, $\mathbf{C}\in {\mathbb{R}}^{K\times n_3}$ is a set of factor matrices and $\underline{G}\in {\mathbb{R}}^{I\times J\times K}$ is a third-order core tensor.

By combining the advantages of the CP decomposition and the Tucker decomposition, the BTD [39] of rank-$\left(L_r,M_r,N_r\right)$ can be formulated generally in terms of:

$$\underline{X}=\sum _{r=1}^R{\underline{G}}_r\times {}_{\mathbf{1}}{\mathbf{A}}_{\mathbf{r}}\times {}_{\mathbf{2}}{\mathbf{B}}_{\mathbf{r}}\times {}_{\mathbf{3}}{\mathbf{C}}_{\mathbf{r}}$$

(3)

where ${\mathbf{A}}_{\mathbf{r}}\in {\mathbb{R}}^{I\times L_r}$, ${\mathbf{B}}_{\mathbf{r}}\in {\mathbb{R}}^{J\times M_r}$, ${\mathbf{C}}_{\mathbf{r}}\in {\mathbb{R}}^{K\times N_r}$, ${\underline{G}}_r\in {\mathbb{R}}^{L_r\times M_{\mathrm{r}}\times N_r}$. It is observed that BTD aims to approximate $\underline{X}$ through R component tensors; each tensor is represented by the Tucker decomposition of rank-$\left(L_r,M_r,N_r\right)$. Assume that the rank of every component tensor of BTD is one; it can be considered as a CPD. Alternatively, when $R=1$, the above model can be viewed as the Tucker decomposition of rank-$\left(L_r,M_r,N_r\right)$. Therefore, the BTD can be regarded as a combo of CPD and Tucker decomposition, and the decomposition model is shown in Figure 3.

In this paper, we select a special model of the BTD, the BTD of rank-$\left(L,L,1\right)$, which can be formulated as:

$$\underline{X}=\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}$$

(4)

where ${\mathbf{A}}_{\mathbf{r}}\in {\mathbb{R}}^{I\times L_r}$, ${\mathbf{B}}_{\mathbf{r}}\in {\mathbb{R}}^{J\times L_r}$ for $r=1,\dots ,R$ and $\mathbf{C}=\left[{\mathbf{c}}_{\mathbf{1}},\dots ,{\mathbf{c}}_{\mathbf{R}}\right]\in {\mathbb{R}}^{K\times R}$. We further stated $\mathbf{A}=[{\mathbf{A}}_{\mathbf{1}},\dots ,{\mathbf{A}}_{\mathbf{R}}]\in {\mathbb{R}}^{I\times {\sum }_{r=1}^R{L}_r}$, $\mathbf{B}=[{\mathbf{B}}_{\mathbf{1}},\dots ,{\mathbf{B}}_{\mathbf{R}}]\in {\mathbb{R}}^{J\times {\sum }_{r=1}^R{L}_r}$; where the partition matrices $\mathbf{A}=[{\mathbf{A}}_{\mathbf{1}},\dots ,{\mathbf{A}}_{\mathbf{R}}]$ and $\mathbf{B}=[{\mathbf{B}}_{\mathbf{1}},\dots ,{\mathbf{B}}_{\mathbf{R}}]$ have full column rank and do not have any colinear columns, with $\mathbf{C}=\left[{\mathbf{c}}_{\mathbf{1}},\dots ,{\mathbf{c}}_{\mathbf{R}}\right]$. The usual full column rank requirement for A and B is $min\left(I,J\right)\ge {\sum }_{r=1}^R{L}_r$. Under a mild condition, the BTD of rank-$\left(L,L,1\right)$ also has the nice property where ${\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}$ and ${\mathbf{c}}_{\mathbf{r}}$ $r=1,\dots ,R$ are identifiable until the alignment and scaling ambiguities arise, with the following lemma ([34] Theorem 4.7):

Proof of Theorem 1.

Let $\left(\mathbf{A},\mathbf{B},\mathbf{C}\right)$ represent a BTD of $\underline{X}$ in rank-$\left(L,L,1\right)$ terms. Assume that $\left(\mathbf{A},\mathbf{B},\mathbf{C}\right)$ are drawn from certain joint absolutely continuous distributions. If $IJ\ge L^{2}R$ and $min\left(⌊\frac{I}{L}⌋,R\right)+min\left(⌊\frac{J}{L}⌋,R\right)+min\left(K,R\right)\ge 2R+2$, then ${\left\{{\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}},{\mathbf{c}}_{\mathbf{r}}\right\}}_{r=1}^R$ are essentially and almost definitely unique.    □

Using the above notation, Equation (4) is unfolded to be expressed as:

$${\mathbf{X}}_{\left(\mathbf{1}\right)}=(\mathbf{C}\odot \mathbf{B}){\mathbf{A}}^{\mathbf{T}}$$

(5)

$${\mathbf{X}}_{\left(\mathbf{2}\right)}=(\mathbf{C}\odot \mathbf{A}){\mathbf{B}}^{\mathbf{T}}$$

(6)

$${\mathbf{X}}_{\left(\mathbf{3}\right)}=\left[\right({\mathbf{B}}_{\mathbf{1}}{\odot }_c{\mathbf{A}}_{\mathbf{1}}){\mathbf{1}}_{{\mathbf{L}}_{\mathbf{1}}},\cdots ,({\mathbf{B}}_{\mathbf{R}}{\odot }_c{\mathbf{A}}_{\mathbf{R}}){\mathbf{1}}_{\mathbf{LR}}]{\mathbf{C}}^{\mathbf{T}}$$

(7)

3. MHF Method Based on the Coupled LL1 Model

3.1. Linear Mixed Model (LMM) and Multilinear Rank Model (LL1)

The classical LMM is first introduced to facilitate an understanding of how the matrix decomposition-based approach utilizes physical interpretation. Set HRHS ${\underline{Y}}_S\in {\mathbb{R}}^{I_M\times J_M\times K_H}$ as the objective image. According to the LMM, the three-mode matrix ${{\mathbf{Y}}_{\mathbf{S}}}_{\left(\mathbf{3}\right)}$ is denoted [40] as follows:

$${\mathbf{Y}}_{\mathbf{S}\left(\mathbf{3}\right)}=\mathbf{S}{\mathbf{C}}^{\mathbf{T}}\in {\mathbb{R}}^{{\mathbf{I}}_{\mathbf{M}}{\mathbf{J}}_{\mathbf{M}}\times {\mathbf{K}}_{\mathbf{H}}}$$

(8)

where $\mathbf{C}\in {\mathbb{R}}^{K_H\times R}$ contains the matrix of spectral properties of the endmembers, and $R\ll min\left\{I_M{J}_M,K_H\right\}$; $\mathbf{S}$ is the abundance matrix. The abundance map of an endmembers r can be expressed as ${\mathbf{S}}_{\mathbf{r}}=mat\left(\mathbf{S}\left(:,\mathbf{r}\right)\right)$, where the $mat\left(·\right)$ operation is to reshape the vector of size $I_M{J}_M\times 1$ into a matrix of size $I_M\times J_M$. Assumptions: each abundance map ${\mathbf{S}}_{\mathbf{r}}\in {\mathbb{R}}^{I_M\times J_M}$ is obtained via an approximation of a low-rank matrix of rank-$L_r$. i.e.,:

$${\mathbf{S}}_{\mathbf{r}}\approx {\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}$$

(9)

where ${\mathbf{A}}_{\mathbf{r}}\in {\mathbb{R}}^{I\times L_r}$, ${\mathbf{B}}_{\mathbf{r}}\in {\mathbb{R}}^{J\times L_r}$. Since the distribution of pixels in the space is not random, the assumption about the low rank of the approximate ${\mathbf{S}}_{\mathbf{r}}$ is justified. Thus, assuming that the abundance map is low-rank and reshapes Equation (8) in terms of a tensor, then the tensor ${\underline{Y}}_S$ is expressed as:

$${\underline{Y}}_S=\sum _{r=1}^R{\mathbf{S}}_{\mathbf{r}}\circ {\mathbf{c}}_{\mathbf{r}}$$

(10)

Setting rank $\left({\mathbf{S}}_{\mathbf{r}}\right)$ = $L_r\le min\left\{I,J\right\}$, Equation (10) could be rewritten as:

$${\underline{Y}}_S=\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}$$

(11)

The tensor model in Equation (11) is the LL1 model that this paper needs. The main task in the reconstruction of HRHS is to recover HRHS from MSI and HSI obtained in the same environment. Therefore, our purpose was how to evaluate the potential factors A, B, and C from MSI and HSI to reconstruct the HRHS by Equation (11).

3.2. Coupling LL1 Tensor Decomposition Modeling

The HSI and MSI can be denoted by the constants ${\underline{Y}}_H\in {\mathbb{R}}^{I_H\times {\mathrm{J}}_H\times K_H}$ and ${\underline{Y}}_M\in {\mathbb{R}}^{I_M\times {\mathrm{J}}_M\times K_M}$ of the third-order tensor. Respectively, where $I_H$, $J_H$, $I_M$, and $J_M$ denote the spatial dimensions, $K_H$ and $K_M$ indicates the spectral band amounts. Therefore, all the spectral images can be expressed in the form of “space × space × spectrum”. In general, MSI has finer spatial information than HSI due to the sensor characteristics, i.e., $I_H\ll I_M$, $J_H\ll J_M$ and $K_M\ll K_H$. We assume that both MSI and HSI are obtained via HRHS ${\underline{Y}}_S\in {\mathbb{R}}^{{}^{I_M\times {\mathrm{J}}_M\times K_H}}$ downsampling/degradation, where HRHS has the space resolution of MSI $(I_M\times J_M)$ and the spectral resolution of HSI $\left(K_H\right)$.

The LL1 model on HRHS is derived by assuming that the abundance map is a low-rank matrix.

$${\underline{Y}}_S=\sum _{\mathrm{r}=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}$$

(12)

where ${\mathbf{A}}_{\mathbf{r}}\in {\mathbb{R}}^{I_M\times L_r}$, ${\mathbf{B}}_{\mathbf{r}}\in {\mathbb{R}}^{J_M\times L_r}$, $\mathbf{C}=\left[{\mathbf{c}}_{\mathbf{1}},\cdots ,{\mathbf{c}}_{\mathbf{R}}\right]\in {\mathbb{R}}^{K_H\times R}$, $(\mathrm{r}=1,2,\cdots ,R)$. In turn, HSI can be interpreted as a space-blurred and down-sampled version of HRHS.

Therefore, it is modeled by multiplying the two fuzzy and compressed matrices via the row and column dimensions of each ${\underline{Y}}_S\left(:,:,k\right)$-component tensor, as follows [36].   

$${\underline{Y}}_H\left(:,:,k\right)={\mathbf{P}}_1{\underline{Y}}_S\left(:,:,k\right){\mathbf{P}}_{\mathbf{2}}^{\mathbf{T}}$$

(13)

$$k=1,2,\dots ,K_H$$

where ${\mathbf{P}}_{\mathbf{1}}\in {\mathbb{R}}^{I_H\times {\mathrm{I}}_M}$ and ${\mathbf{P}}_{\mathbf{2}}\in {\mathbb{R}}^{J_H\times {\mathrm{J}}_M}$ denote the upper spatial fuzzy and downsampling matrices along the two spatial dimensions, respectively. The process is shown in Figure 4.

If ${\underline{Y}}_S$ conforms to the LL1 model, as well as the space degeneracy pattern holds in Equation (13), the mathematical model of ${\underline{Y}}_H$ is shown below:

$${\underline{Y}}_H=\sum _{r=1}^R\left({\mathbf{P}}_{\mathbf{1}}{\mathbf{A}}_{\mathbf{r}}{\left({\mathbf{P}}_{\mathbf{2}}{\mathbf{B}}_{\mathbf{r}}\right)}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}$$

(14)

MSI can be expressed as the spectral band aggregation form of HRHS, and the mathematical model is shown below (see Figure 4 for the procedure):

$${\underline{Y}}_M\left(i,j,:\right)={\mathbf{P}}_{\mathbf{M}}{\underline{Y}}_S\left(i,j,:\right)$$

(15)

where ${\mathbf{P}}_{\mathbf{M}}\in {\mathbb{R}}^{K_M\times K_H}$ is the band aggregate matrix associated with sensor specification, and then Equation (15) is compactly expressed as follows:

$${\underline{Y}}_M=\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ {\mathbf{P}}_{\mathbf{M}}{\mathbf{c}}_{\mathbf{r}}$$

(16)

Under this degenerate model, the main primary goal of recovering ${\underline{Y}}_S$ is to identify ${\mathbf{c}}_{\mathbf{r}}$ (endmembers) from ${\underline{Y}}_H$ and the potential factor ${\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}$ (abundance map) from ${\underline{Y}}_M$. Once ${\left\{{\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right\}}_{r=1}^R$ and ${\mathbf{c}}_{\mathbf{r}}$ are identified, they can be reconstructed by using ${\underline{Y}}_S=\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}$.

In particular, it should be noted that the BTD model is more attractive than the CP and Tucker models in solving the MHF problem. This is because ${\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}$ and ${\mathbf{c}}_{\mathbf{r}}$ represent the abundance and endmembers, respectively, with physical significance such as non-negativity and spatial smoothness, which help to regulate the MHF criteria with a priori knowledge. The previous one represents the abundance information of the endmembers r, and the latter represents the rth endmember. Based on the description of problem (10) and the downsampling along the three dimensions, in the absence of noise, the MHF problem based on the coupled LL1 model is described as follows.

$$\underset{\mathbf{A},\mathbf{B},\mathbf{C}}{min}\frac{1}{2}{∥{\underline{Y}}_H-\sum _{r=1}^R\left({\mathbf{P}}_{\mathbf{1}}{\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}{\mathbf{P}}_{\mathbf{2}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}∥}_F^2+\frac{1}{2}{∥{\underline{Y}}_M-\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ \left({\mathbf{P}}_{\mathbf{M}}{\mathbf{c}}_r\right)∥}_F^2$$

(17)

$$s.t.\mathbf{A}\ge 0,\mathbf{B}\ge 0,\mathbf{C}\ge 0$$

3.3. Modeling

In the former two sections, we describe in detail the relationship between the LMM model and the LL1 model, together with a re-simulation of the MHF problem using the LL1 model. Through the above analysis, it can easily establish the fact that the LMM model and the LL1 model are consistent. Therefore, the potential factors of the LL1 model are physically interpretable, which allows us to fully exploit the prior information of endmembers ${\mathbf{c}}_{\mathbf{r}}$ and the abundance information of ${\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}$ to enhance the fusion effect in the presence of a large amount of noise. However, this does not mean that the fusion effect of the MHF problem can be optimized, and the $L_1$ parametrization and TV, which are effective means to describe sparsity and segmental smoothness. Introduce them into the model and add some rule terms and constraints in A, B, and C, to further improve the fusion effect. The following modified form is proposed for model (17).

$$\begin{array}{c}\hfill \underset{\mathbf{A},\mathbf{B},\mathbf{C}}{min}\frac{1}{2}{∥{\underline{Y}}_H-\sum _{r=1}^R\left({\mathbf{P}}_{\mathbf{1}}{\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}{\mathbf{P}}_{\mathbf{2}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}∥}_F^2+\frac{1}{2}{∥{\underline{Y}}_M-\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ \left({\mathbf{P}}_{\mathbf{M}}{\mathbf{c}}_{\mathbf{r}}\right)∥}_F^2\\ \hfill +{\lambda }_1{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{A}∥}_1+{\lambda }_2{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{B}∥}_1+{\lambda }_3{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{C}∥}_1\end{array}$$

(18)

$$s.t.\mathbf{A}\ge 0,\mathbf{B}\ge 0,\mathbf{C}\ge 0$$

where ${\lambda }_i>0\left(i=1,2,3\right)$ is the regularization parameter and ${\mathbf{D}}_{\mathbf{H}}$ is the limited differencing operator in the vertical and horizontal directions.

For the proposed new modification scheme, there are the following considerations: Firstly, the proposed model (17) is based on the noise-free case, in order to recover HRHS accurately, and when there is a lot of noise in MSI and HSI, the scheme will no longer be applicable, so it needs to impose constraints on it by adding some penalty terms to prevent the model from overfitting and limiting the complexity. Secondly, according to the properties of the LL1 model, where A, B, and C have good physical properties, it is easy to exploit the various prior information present in them, e.g., A, B, and C are non-negative [20,41]; MSI and HSI acquired in the same scenario, ${\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}$ and ${\mathbf{c}}_{\mathbf{r}}$ are coupled. Finally, considering sparsity and segmental smoothing, this paper defines two difference operators, ${\mathbf{D}}_{\mathbf{x}}$ and ${\mathbf{D}}_{\mathbf{y}}$, for the properties of A, B, and C (see Equations (19) and (20)), while adding regularization and imposing non-negative constraints on them, to effectively maintain edge information and preserve the smoothness of the local spectra.

$${\mathbf{D}}_x=\left[\begin{array}{cccccc}1& 0& 0& 0& \cdots & 1\\ -1& 1& 0& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& \cdots & ⋮\\ 0& 0& \cdots & 0& -1& 1\end{array}\right]$$

(19)

$${\mathbf{D}}_{\mathbf{y}}=\left[\begin{array}{cccccc}1& -1& 0& 0& \cdots & 0\\ 0& 1& -1& 0& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& \cdots & ⋮\\ 1& 0& \cdots & 0& 1& -1\end{array}\right]$$

(20)

The flow chart of the proposed algorithm in this paper is shown in Figure 4. Firstly, the low-resolution hyperspectral images (LR-HSI) and the high-resolution multispectral images (HR-MSI) are obtained by downsampling the Pavia University dataset. Secondly, the LR-HSI and HR-MSI are decomposed using BTD to obtain the endmember matrix C of LR-HSI and the abundance matrix A, B of HR-MSI, which are brought into the model (18). Finally, the alternating directional multiplier method (ADMM) is used to iteratively solve A, B, C, which is brought into Equation (12) to obtain the target image HRHS.

Next, this paper gives the efficient solution algorithm for the model (18). Since the problem is a nonconvex optimization problem, where A, B, and C are coupled variables, the optimization issue was convex for every variable, while keeping the other variables constant. Therefore, each variable is solved iteratively with the help of the proximal alternating optimization (PAO) [42,43] method.

Model (18) is solved by using PAO with the following procedure.

$$\begin{array}{c}\mathbf{A}=\underset{\mathbf{A}}{argmin}f\left(\mathbf{A},\mathbf{B},\mathbf{C}\right)+\beta {∥\mathbf{A}-{\mathbf{A}}^{\mathbf{pre}}∥}_F^2\\ \mathbf{B}=\underset{\mathbf{B}}{argmin}f\left(\mathbf{A},\mathbf{B},\mathbf{C}\right)+\beta {∥\mathbf{B}-{\mathbf{B}}^{\mathbf{pre}}∥}_F^2\\ \mathbf{C}=\underset{\mathbf{C}}{argmin}f\left(\mathbf{A},\mathbf{B},\mathbf{C}\right)+\beta {∥\mathbf{C}-{\mathbf{C}}^{\mathbf{pre}}∥}_F^2\end{array}$$

(21)

where $f\left(\mathbf{A},\mathbf{B},\mathbf{C}\right)$ is an implicit representation of model (18). ${\left(·\right)}^{\mathbf{pre}}$ and $\beta$ represent the result of the preceding iteration and a non-negative value, separately.

In the following, a detailed formulation of the A optimization problem in the model (18), and the B and C optimization problems are shown in Appendix B.

Optimization Problem on A

Fixing B, C, the optimization problem on A in the above equation is shown as follows:

$$\begin{array}{c}\hfill \underset{\mathbf{A}}{argmin}\frac{1}{2}{∥{\underline{Y}}_H-\sum _{r=1}^R\left({\mathbf{P}}_{\mathbf{1}}{\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}{\mathbf{P}}_{\mathbf{2}}^{\mathbf{T}}\right)\circ {\mathbf{c}}_{\mathbf{r}}∥}_F^2+\frac{1}{2}{∥{\underline{Y}}_M-\sum _{r=1}^R\left({\mathbf{A}}_{\mathbf{r}}{\mathbf{B}}_{\mathbf{r}}^{\mathbf{T}}\right)\circ \left({\mathbf{P}}_{\mathbf{M}}{\mathbf{c}}_{\mathbf{r}}\right)∥}_F^2\\ \hfill +{\lambda }_1{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{A}∥}_1+\beta {∥\mathbf{A}-{\mathbf{A}}^{\mathbf{pre}}∥}_F^2\end{array}$$

(22)

Via the mode-1 expansion, and through the introduction of the splitting variable ${\mathbf{U}}_{\mathbf{1}}={\mathbf{D}}_{\mathbf{H}}\mathbf{A}$, problem (22) can be formulated anew as:

$$\underset{\mathbf{A}}{argmin}\frac{1}{2}{∥{\mathbf{Y}}_{\mathbf{H}\mathbf{1}}-\mathbf{N}({\mathbf{P}}_{\mathbf{1}}\mathbf{A}{)}^{\mathbf{T}}∥}_F^2+\frac{1}{2}{∥{\mathbf{Y}}_{\mathbf{M}\mathbf{1}}-\mathbf{W}{\mathbf{A}}^{\mathbf{T}}∥}_F^2+{\lambda }_1{∥{\mathbf{U}}_{\mathbf{1}}∥}_1+\beta {∥\mathbf{A}-{\mathbf{A}}^{\mathbf{pre}}∥}_F^2$$

(23)

$$s.t.{\mathbf{U}}_{\mathbf{1}}={\mathbf{D}}_{\mathbf{H}}\mathbf{A}$$

where $\mathbf{N}\stackrel{\Delta }{=}\mathbf{C}\odot {\mathbf{P}}_{\mathbf{2}}\mathbf{B}$ and $\mathbf{W}\stackrel{\Delta }{=}{\mathbf{P}}_{\mathbf{M}}\mathbf{C}\odot \mathbf{B}$ is obtained by matrixing Equation (5) to obtain the augmented Lagrangian function for problem (23).

$$\begin{array}{c}\hfill f(\mathbf{A},{\mathbf{U}}_{\mathbf{1}},{\mathbf{L}}_{\mathbf{1}})=\frac{1}{2}{∥{\mathbf{Y}}_{\mathbf{H}\mathbf{1}}-\mathbf{N}({\mathbf{P}}_{\mathbf{1}}\mathbf{A}{)}^{\mathbf{T}}∥}_F^2+\frac{1}{2}{∥{\mathbf{Y}}_{\mathbf{M}\mathbf{1}}-\mathbf{W}{\mathbf{A}}^{\mathbf{T}}∥}_F^2\\ \hfill +{\lambda }_1{∥{\mathbf{U}}_{\mathbf{1}}∥}_1+\beta {∥\mathbf{A}-{\mathbf{A}}^{\mathbf{pre}}∥}_F^2+{\mu }_1{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{A}-{\mathbf{U}}_{\mathbf{1}}-{\mathbf{L}}_{\mathbf{1}}∥}_F^2\end{array}$$

(24)

where ${\mathbf{L}}_{\mathbf{1}}$ is the Lagrangian multiplier and ${\mu }_1$ is the positive penalized parameter.

For problem (24), the ADMM iterative equation is shown below:

$$\begin{array}{c}{\mathbf{A}}^{\left(\mathbf{k}+\mathbf{1}\right)}=\underset{\mathbf{A}}{argmin}\left(\mathbf{A},{\mathbf{U}}_{\mathbf{1}}^{\left(\mathbf{k}\right)}{,\mathbf{L}}_{\mathbf{1}}^{\left(\mathbf{k}\right)}\right)\hfill \\ {\mathbf{U}}_{\mathbf{1}}^{\left(\mathbf{k}+\mathbf{1}\right)}=\underset{{\mathbf{U}}_{\mathbf{1}}}{argmin}\left({\mathbf{A}}^{\left(\mathbf{k}+\mathbf{1}\right)},{\mathbf{U}}_{\mathbf{1}}{,\mathbf{L}}_{\mathbf{1}}^{\left(\mathbf{k}\right)}\right)\hfill \\ {\mathbf{L}}_{\mathbf{1}}^{\left(\mathbf{k}+\mathbf{1}\right)}=\underset{{\mathbf{L}}_{\mathbf{1}}}{argmin}\left({\mathbf{A}}^{\left(\mathbf{k}+\mathbf{1}\right)}{,\mathbf{U}}_{\mathbf{1}}^{\left(\mathbf{k}+\mathbf{1}\right)},{\mathbf{L}}_1\right)\hfill \end{array}$$

(25)

In the following, the solution procedure of problem (25) is given.

• Subproblem of A: for problem (24), the minimization objective function is as follows:

  $$\begin{array}{c}\hfill \underset{\mathbf{A}}{argmin}\frac{1}{2}{∥{\mathbf{Y}}_{\mathbf{H}\mathbf{1}}-{\mathbf{N}}^{\mathbf{k}}{({\mathbf{P}}_{\mathbf{1}}\mathbf{A})}^{\mathbf{T}}∥}_F^2+\frac{1}{2}{∥{\mathbf{Y}}_{\mathbf{M}\mathbf{1}}-{\mathbf{W}}^{\mathbf{k}}{\mathbf{A}}^{\mathbf{T}}∥}_F^2\\ \hfill +\beta {∥\mathbf{A}-{\mathbf{A}}^{\mathbf{pre}}∥}_F^2+{\mu }_1{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{A}-{\mathbf{U}}_{\mathbf{1}}-{\mathbf{L}}_1∥}_F^2\end{array}$$

  (26)
  Problem (26) is a quadratic optimization problem whose solution is unique. Compute Sylvester’s equation to solve A by calculating the following, i.e.:   

  $$\begin{array}{c}\hfill {\mathbf{P}}_{\mathbf{1}}^{\mathbf{T}}{\mathbf{P}}_{\mathbf{1}}\mathbf{A}{\mathbf{N}}^{\mathbf{kT}}{\mathbf{N}}^{\mathbf{k}}+\mathbf{A}\left({\mathbf{W}}^{\mathbf{kT}}{\mathbf{W}}^{\mathbf{k}}+\beta \mathbf{I}\right)+{\mu }_1{\mathbf{D}}_{\mathbf{H}}^{\mathbf{T}}{\mathbf{D}}_{\mathbf{H}}\mathbf{A}={\mathbf{P}}_{\mathbf{1}}^{\mathbf{T}}{\mathbf{Y}}_{\mathbf{H}\mathbf{1}}^{\mathbf{T}}{\mathbf{N}}^{\mathbf{k}}\\ \hfill +{\mathbf{Y}}_{\mathbf{M}\mathbf{1}}^{\mathbf{T}}{\mathbf{W}}^{\mathbf{k}}+\beta {\mathbf{A}}^{\mathbf{pre}}+{\mu }_1{\mathbf{D}}_{\mathbf{H}}^{\mathbf{T}}\left({\mathbf{U}}_{\mathbf{1}}+{\mathbf{L}}_{\mathbf{1}}\right)\end{array}$$

  (27)
  In this paper, this problem is effectively solved by using CG [44].
• Subproblem of ${\mathbf{U}}_{\mathbf{1}}$: For problem (24), the minimization objective function is as follows:

  $$\underset{{\mathbf{U}}_{\mathbf{1}}}{argmin}{\lambda }_1{∥{\mathbf{U}}_{\mathbf{1}}∥}_1+{\mu }_1{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{A}-{\mathbf{U}}_{\mathbf{1}}-{\mathbf{L}}_{\mathbf{1}}∥}_F^2$$

  (28)
  The solution of its ${\mathbf{U}}_{\mathbf{1}}$ is well known as a soft threshold, i.e.:

  $${\mathbf{U}}_{\mathbf{1}}=soft\left({\mathbf{D}}_{\mathbf{H}}\mathbf{A}-{\mathbf{L}}_{\mathbf{1}},\frac{{\lambda }_1}{2{\mu }_1}\right)$$

  (29)

  where $soft\left(u,a\right)=sign\left(u\right)\times max\left(\left|u\right|-a,0\right)$.
• Subproblem of ${\mathbf{L}}_{\mathbf{1}}$: For problem (24), the updated Lagrange multiplier ${\mathbf{L}}_{\mathbf{1}}$ is given by:

  $${\mathbf{L}}_{\mathbf{1}}={\mathbf{L}}_{\mathbf{1}}-\left({\mathbf{D}}_{\mathbf{H}}\mathbf{A}-{\mathbf{U}}_{\mathbf{1}}\right)$$

  (30)

The pseudo-code of the optimization algorithm on A is shown in Algorithm 1.

Algorithm 1 Solving A-Subproblem (22) via ADMM.

Input${\mathbf{Y}}_{\mathbf{H}\mathbf{1}}$, ${\mathbf{Y}}_{\mathbf{M}\mathbf{1}}$, ${\mathbf{P}}_{\mathbf{1}}$, ${\mathbf{P}}_{\mathbf{2}}$, ${\mathbf{P}}_{\mathbf{M}}$, ${\mathbf{A}}^{\mathbf{pre}}$, $\beta >0$, ${\lambda }_1>0$, ${\mu }_1>0$.

Output Dictionary matrix A

While unconverged do

Updating the matrix A by solving problem (24) with the CG method;

Updating the variable ${\mathbf{U}}_{\mathbf{1}}$ by $soft\left({\mathbf{D}}_{\mathbf{H}}\mathbf{A}-{\mathbf{L}}_{\mathbf{1}},{\lambda }_1/\phantom{{\lambda }_{1}2{\mu }_1}2{\mu }_1\right.$);

Updating the Lagrangian multiplier L1 by L1 − (DHA − U1);

end While;

4. Experiments and Analysis

4.1. Datasets

In this paper, two standard datasets and two local datasets are selected for experiments to verify the effectiveness of the proposed MHF fusion algorithm via numerical results.

4.1.1. Standard Datasets

The Pavia University dataset was obtained using the Italian Reflection Optical System Imaging Spectrometer (ROSIS) light sensor in the Pavia University downtown area. The ROSIS continuously images 115 bands in the wavelength range from 0.43 to 0.86 $\mathsf{\mu }$m, with a spatial resolution of 1.3 m and an image size of 610 × 340 × 115. After eliminating the water vapor bands, the band number decreased to 102. In view of the experimental requirements, a sub-image of size 256 × 256 × 102 is selected in the upper left part of the image as the reference image for testing.

The Indian Pine dataset was collected by the AVIRIS sensor in Indiana, with a spatial resolution of 20 m. The dataset has a large format of 144 × 144, with 224 bands. Bands 104–108, 150–163, and 220 are excluded because they cannot be reflected by water, leaving 200 bands as the object of study. For downsampling reasons, a subimage of size 144 × 144 × 200 in the upper left corner was selected as the reference image for the experiment.

4.1.2. Local Datasets

Referred to are the GF-1 and GF-5 Sand Lake datasets of Ningxia, China. In particular, the GF-1 Sand Lake dataset of Ningxia was taken by the GF-1 satellite carrying the WFV4 multispectral camera over the Sand Lake region of Ningxia on 31 July 2019, Beijing time, with each view image covering an area of about 1056.25 square kilometers, and the entire Sand Lake area accounting for about eight percent of the image. The multispectral camera continuously imaged three bands in the wavelength range of 0.45–0.89 $\mathsf{\mu }$m to obtain multispectral images, and the resulting images have a spatial resolution of 16 m and an image size of 12,000 × 13,400 × 3. The image of the Sand Lake area was selected as the MSI, which has a size of $254\times 254\times 3$. The GF-5 Sand Lake dataset of Ningxia was taken by the AHSI sensor-equipped GF-5 satellite over the Sand Lake region of Ningxia on 27 July 2019, Beijing time, with each view image covering an area of about 3600 square kilometers, and the entire Sand Lake region covering about three percent of the image. The sensor continuously imaged 330 bands in the wavelength range of 0.4–2.5 $\mathsf{\mu }$m to obtain hyperspectral images, and the resulting images have a spatial resolution of 30 m and an image size of $2774\times 2554\times 330$, with the number of bands reduced to 143 after excluding some noisy high and water vapor absorption bands. An HSI image of size $254\times 254\times 143$ was obtained by intercepting the GF-5 image at the place where the MSI image was acquired from the GF-1.

The UAV acquisition of Qingtongxia rice field dataset is a scene acquired by DJI UAV with the hyperspectral sensor Pika-L and the multispectral sensor Micasense Altum during an experimental rice field flight campaign in Qujing village, Qingtongxia, Ningxia, China, on 15 July 2021. The experimental area consisted of 19,992.71 m${}^2$ of paddy fields and 2134 m${}^2$ of houses, and the houses and courtyards accounted for 10 percent of the image in the whole image obtained. During the dataset acquisition, the UAV flew at a speed of 3 m/s, an altitude of 100 m, and a pitch angle of −90. Among them, the Pika-L hyperspectral sensor continuously imaged 150 bands in the wavelength range of 0.4–1.0 $\mathsf{\mu }$m to obtain hyperspectral images, and the spatial resolution of the generated images was 10 cm; the image size was $5376\times 1492\times 150$. After removing some noisy bands, the number of bands reduced to 60, from which a sub-image of $512\times 512\times 60$ containing information on houses, rice fields, courtyards, etc., was truncated as the HSI. The Micasense Altum multispectral sensor continuously imaged three bands in the wavelength range of 0.475–0.717 $\mathsf{\mu }$m to obtain a multispectral image with a spatial resolution of 5.2 cm and an image size of 16,103 × 9677. The image size is 16,103 × 9677 × 3. A sub-image of $512\times 512\times 3$ is intercepted at the same location as the MSI. The area of the intercepted image is about one-tenth of the whole image.

4.2. Compared Algorithms

Six algorithms, CNMF [16], Hysure [19], FUSE [20], SCOTT [23], STEREO [31], and BTD [32], were selected for comparison tests in this paper. In particular, SCOTT and STEREO are based on the Tucker and CPD models for solving MHF problems, respectively. All experiments were run on a PC with an Intel(R) Xeon(R) Gold 6154 CPU @3.00 GHz and 256 GB RAM. A Windows 10 × 64 operating system was used, on which the programming software was MATLAB R2017b. The result of all the experiments is the average of 10 independent experiments.

4.3. Test Indicators

In order to better evaluate the superiority of the algorithm, and to determine the quality of the fused images, numerical results analysis is an important measure, in addition to the direct observation of the fusion results. This paper uses test metrics that are widely used in the literature [2,15]. Its detailed definition is shown below.

Peak Signal To Noise Ratio (PSNR), which evaluates the level of image distortion or noise. The general situation of PSNR is between 30 dB and 50 dB. In the ideal case, it is expected to be as high as possible, with ∞ representing the optimal value. It should be especially noted that when the PSNR is more than 30 dB, it is difficult for the human eyes to see the difference between the fused image and the reference image. The definitions are as follows.

$$PSNR\left(\underline{Y},{\underline{Y}}_S\right)=\frac{1}{N_S}\sum _{i=1}^{N_S}PSNR\left({\underline{Y}}_i,{{\underline{Y}}_S}_i\right)$$

(31)

Structural Similarity (SSIM), which is used to calculate the structural similarity between the HRHS and the reference image in the spatial domain [45]. SSIM ranges between −1 to 1. When the two images are identical, the SSIM value is equal to 1. The definitions are as follows.

$$SSIM\left(\underline{Y},{\underline{Y}}_S\right)=\frac{1}{M}\sum _{i=1}^M\frac{\left(2{\overline{\underline{Y}}}_i,{\overline{{\underline{Y}}_S}}_i+c_1\right)\left(2{\sigma }_{{\overline{\underline{Y}}}_i,{\overline{{\underline{Y}}_S}}_i}+c_2\right)}{\left[{\left({\overline{\underline{Y}}}_i\right)}^2+{\left({\overline{{\underline{Y}}_S}}_i\right)}^2+c_1\right]\left({\sigma }_{\underline{Y}}^2+{\sigma }_{{\underline{Y}}_S}^2+c_2\right)}$$

(32)

Correlation Coefficient (CC), which describes the degrees of spectral likeness with respect to the objective image, as well as the fusion image. CC is based on a scale of −1 to 1. The closer the CC is to 1, the higher their positive correlation. The definitions are as follows.

$$CC\left(\underline{Y},{\underline{Y}}_S\right)=\frac{{\displaystyle \sum _{i=1}^{N_W}}{\displaystyle \sum _{j=1}^{N_H}}\left\{\left[\underline{Y}\left(i,j\right)-AV{E}_{\underline{Y}}\right]•\left[{\underline{Y}}_S\left(i,j\right)-AV{E}_{{\underline{Y}}_S}\right]\right\}}{\sqrt{{\displaystyle \sum _{i=1}^{N_W}}{\displaystyle \sum _{j=1}^{N_H}}{\left[\underline{Y}\left(i,j\right)-AV{E}_{\underline{Y}}\right]}^2•{\displaystyle \sum _{i=1}^{N_W}}{\displaystyle \sum _{j=1}^{N_H}}{\left[{\underline{Y}}_S\left(i,j\right)-AV{E}_{{\underline{Y}}_S}\right]}^2}}$$

(33)

Universal Image Quality Index (UIQI) [46], which is specifically used to detect the average correlation between the reference image and the fused image. If the fused image is closer to the reference image, the UIQI value is closer to the value 1, which means the better the reconstruction effect. The definitions are as follows.

$$UIQI\left(\underline{Y},{\underline{Y}}_S\right)=\frac{1}{M}\sum _{i=1}^M\frac{4{\sigma }_{{\overline{\underline{Y}}}_i{\overline{{\underline{Y}}_S}}_i}^2•{\overline{\underline{Y}}}_i,{\overline{{\underline{Y}}_S}}_i}{\left({\sigma }_{{\underline{Y}}_i}^2+{\sigma }_{{{\underline{Y}}_S}_i}^2\right)\left[{\left({\overline{\underline{Y}}}_i\right)}^2+{\left({\overline{{\underline{Y}}_S}}_i\right)}^2\right]}$$

(34)

Root Mean Squared Error (RMSE), which is an important index describing the root mean squared error between each band of the reference image and the fused image. If the fused image is closer to the reference image, the closer the RMSE value is to the value 0. The definitions are as follows.

$$RMSE\left(\underline{Y},{\underline{Y}}_S\right)=\sqrt{\frac{{∥\underline{Y},{\underline{Y}}_S∥}_F^2}{N_W{N}_H{N}_S}}$$

(35)

Erreur Relative Globale Adimensionnelle de Synthese (ERGAS) [47] is an important measure of the global quality and spectral variability of the reconstructed image. If ERGAS is closer to the value of 0, it means the better the reconstruction effect. The definitions are as follows.

$$ERGAS\left(\underline{Y},{\underline{Y}}_S\right)=\frac{100}{s}\sqrt{\frac{1}{N_S}\sum _{i=1}^{N_S}\frac{MSE\left(\underline{Y},{\underline{Y}}_S\right)}{MEAN\left({\underline{Y}}_S\right)}}$$

(36)

Spectral Angle Mapper (SAM) [48] is a metric used to measure the magnitude of the angle between the pixel spectra of the source image and the fused image. If the SAM value is closer to the value of 0, it means that the two images match more and the similarity is higher. The definitions are as follows.

$$SAM\left(\underline{Y},{\underline{Y}}_S\right)=\frac{1}{N_W{N}_H}\sum _{i=1}^{N_W{N}_H}arccos\frac{\left(\underline{Y},{\underline{Y}}_S\right)}{{∥{\underline{Y}}_i∥}_2•{∥{{\underline{Y}}_S}_i∥}_2}$$

(37)

where ${\underline{Y}}_i$ and ${{\underline{Y}}_S}_i$ the i-th band of the initial and fused images, $AV{E}_{\underline{Y}}$ and $AV{E}_{{\underline{Y}}_S}$ are the average pixel values of the initial and fusion images, respectively. ${\overline{\underline{Y}}}_i$, as well as ${\overline{{\underline{Y}}_S}}_i$ are the mean values of ${\underline{Y}}_i$ and ${{\underline{Y}}_S}_i$, respectively. ${\sigma }_i$, ${\sigma }_i^2$, and ${\sigma }_{i,j}^2$ are expressed as standard deviation, variance, and covariance, respectively. $c_i$ is a constant. Here, the larger the value of PSNR, the closer the SSIM, CC, and UIQI values are to 1, and the closer the RMSE, ERGAS, and SAM values are to 0, indicating a better fusion effect.

4.4. Degradation Model

For the standard dataset, a publicly available HSI dataset was used as the reference image following the approach in the literature [20,30,49]. The simulated HSI and MSI images used in the experiments were produced according to the Wald protocol. The degradation model of the reference image to HSI is as follows: the reference image is first blurred using a 9 × 9 Gaussian kernel, and then it is downsampled every four pixels along the spatial dimension. The degradation process of the reference image to the MSI is set according to the literature [31]. In addition, zero-mean Gaussian white noise is added to the MSI and HSI.

4.5. Parameter Discussion

There are many factors that affect the fusion effect of the algorithm, and parameters are one of the important influencing factors. In this paper, it needs to consider the effect of the regularization parameter ${\lambda }_i$, the penalty parameter ${\mu }_i$, the non-negative number $\beta$, and different signal-to-noise ratios (SNRs) on the algorithm, and how to choose the appropriate R and L to make the algorithm fusion effect reach the best result.

4.5.1. ${\lambda }_i$, ${\mu }_i$, and $\beta$ Parameter Selection

After conducting ablation experiments for the regularization parameter ${\lambda }_i$ and the penalty parameter ${\mu }_i$, respectively, it is found that ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ take approximately equal values, and ${\mu }_1$, ${\mu }_2$, and ${\mu }_3$ take approximately equal values. Therefore, the three parameters of ${\lambda }_i$, ${\mu }_i$, and $\beta$ are set as follows: ${\lambda }_1=\dots ={\lambda }_i\left(i=1,2,3\right)$, ${\mu }_1=\dots ={\mu }_i\left(i=1,2,3\right)$. With regard to the choice of parameter values, the optimal value of a parameter is found by fixing two parameters and changing one parameter. Here, we select the Pavia University dataset for experiments to analyze the effect of parameter value changes on the PSNR and to select the appropriate parameter values. The regularization parameter ${\lambda }_i$ is solved with a fixed penalty parameter ${\mu }_i$ and a non-negative number $\beta$. While the regularization parameter ${\lambda }_i$ grows at a rate of 10 times from 0.1, the change in the PSNR line chart is observed and the optimal value is selected. Similarly, the values of the penalty parameter ${\mu }_i$ and the non-negative number $\beta$ are obtained. As shown in Figure 5, the PSNR gradually stabilizes as the three parameters keep changing, and the parameter value corresponding to the time when the PSNR is maximum is selected. Therefore, the values of the three parameters are set as follows: ${\lambda }_i={10}^{-1}$, $\beta ={10}^{-2}$, ${\mu }_i={10}^{-2}$.

4.5.2. R and L Discussion

In the algorithm proposed in this paper, the matrix rank R and L of will be different for different datasets, and the variation of R and L will have a great impact on the performance of the fusion algorithm. Therefore, in order to make the algorithm achieve the best fusion effect on the Pavia University dataset and the Indian Pine dataset, the values of R and L need to be determined experimentally.

The PSNR values for different R and L in the Pavia University dataset are given in Figure 6. In the experiments, since the R and L values are uncertain, only rough estimates of R and L can be made initially, and then the experiments are repeated by choosing different values of R and L and finding the best values around the estimates. Thus, $R=25$ is set for an uncertain L value, resulting in Figure 6a. Similarly, $L=30$ is set for an uncertain value of R, resulting in Figure 6b, where the PSNR increases gradually when R is taken at $\left[5,25\right]$. At the value of $\left[25,40\right]$, the PSNR remains flat. The PSNR increases gradually when L is taken at $\left[5,30\right]$. The PSNR remains smooth when $\left[30,40\right]$ is taken. Therefore, this is set to: $R=25$, $L=30$.

The PSNR values for different R and L in the Indian Pine dataset are given in Figure 7. In the case of uncertainty in the value of L, set $R=10$, which yields Figure 7a. Similarly, set $L=20$ for an uncertain value of R, resulting in Figure 7b, where the PSNR increases gradually when R is taken at $\left[5,10\right]$. At $\left[10,20\right]$, the PSNR remains stable. The PSNR increases gradually when L is taken at $\left[5,20\right]$. The PSNR remains smooth at $\left[20,35\right]$. Therefore, R and L are set as follows: $R=10$, $L=20$.

4.5.3. SNRs Discussion

The Signal-to-Noise Ratios (SNRs) also affect the fusion effect of algorithms. In order to avoid the comparison of “optimal” and “non-optimal” effects between algorithms, it is necessary to select a suitable SNR value to ensure that all algorithms have achieved the best effect at that SNR value, and then discuss the superiority of the algorithms. The Pavia University dataset is selected for SNRs discussion in this paper. Before conducting the experiments, all parameters in the algorithm experiments have been tuned to be optimal. The SNRs were selected in the range of 15 dB–40 dB. Figure 8 plots the variation of evaluation metrics for all algorithms at different SNRs. From the figure, it can be seen that all algorithm curves level off after $SNR$ = 30 dB. Therefore, Pavia University dataset SNRs are set to $SNR$ = 30 dB.

4.6. Validity of the Regularization Terms

In the model (18) of the algorithm proposed in this paper, the smoothness and sparsity of the fusion algorithm are described using three regularization terms ${\lambda }_1{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{A}∥}_1$, ${\lambda }_2{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{B}∥}_1$, and ${\lambda }_3{∥{\mathbf{D}}_{\mathbf{H}}\mathbf{C}∥}_1$. In order to verify whether the algorithm in this paper is improved, experiments were conducted using the BTD algorithm and the proposed algorithm in this paper on the Pavia University dataset, the Indian Pine dataset, the GF-1 and GF-5 Sand Lake in Ningxia of China dataset, and the UAV Acquisition of the Qingtongxia Rice Field dataset, respectively. The numerical information derived from the experiments is shown in

Table 1. Comparison of BTD and 2D-CNBTD values.

Dataset		Pavia University		Indian Pine		Sand Lake		Rice Field
Algorithm		BTD	2D-CNBTD	BTD	2D-CNBTD	BTD	2D-CNBTD	BTD	2D-CNBTD
Method	PSNR(∞)	35.71	40.86	25.38	32.42	36.97	39.49	8.54	37.13
	SSIM(1)	0.8764	0.9632	0.7436	0.9328	0.9753	0.9434	0.0081	0.9911
	CC(1)	0.9705	0.9932	0.7091	0.8962	0.9960	0.9976	0.9409	0.9992
	UIQI(1)	0.8354	0.9321	0.4067	0.6949	0.9390	0.9634	0.0009	0.9269
	RMSE(0)	0.0222	0.0104	0.0306	0.0127	50.57	50.362	5.6012	0.0114
	ERGAS(0)	0.5454	0.2679	0.4431	0.1530	0.0882	0.0639	63.1562	0.0734
	SAM(0)	0.0957	0.0558	0.0850	0.0336	0.0436	0.0309	0.2995	0.0327
	Time(0)	80.94	89.68	7.26	11.62	64.79	73.44	40.67	49.39

. In which, the time unit is seconds, and the optimal value is noted as 0 s; bold in the table indicates better values; and PSNR, SSIM, and other test metrics are described in detail in Section 4.3. From Table 1, it can be seen that the algorithm in this paper has substantially improved SSIM and UIQI, which describe the spatial feature metrics, compared to before the improvement. The indicators CC, RMSE, and SAM about the spectral features are also significantly improved, but the time is slightly increased by 3 s–9 s. It shows that although the improved algorithm has a slightly longer running time, it can better retain the spatial and spectral information of the image during the image fusion process, which makes the fused image clearer.

4.7. Experimental Results for Standard Datasets

The fusion results of the seven tested methods on the Pavia University and Indian pine datasets are shown in this section.

4.7.1. Experimental Results of Pavia University Dataset

To demonstrate the spatial detail information, reconstruction effect, and fusion results, three bands (R: 61, G: 25, B: 13) are selected to synthesize pseudo-color images in this paper. Set $SNR$ = 30 dB. The fusion performances of all methods on the Pavia University dataset are plotted in Figure 9. Figure 9h shows the reference image, and the scene in the red box is shown in the upper left corner of the figure.

When the difference between HRHS and the reference images is small, it is difficult to find out the difference using the human eye. Therefore, some evaluation metrics are used to analyze the algorithm’s performance. These evaluations of the compared algorithms are shown in

Table 2. Performance for Pavia university.

Method	CNMF	Hysure	FUSE	SOCTT	STEREO	BTD	2D-CNBTD
PSNR(∞)	36.56	36.39	31.29	28.42	39.40	35.71	40.86
SSIM(1)	0.9627	0.9606	0.9240	0.7884	0.9545	0.8764	0.9632
CC(1)	0.9867	0.9853	0.9562	0.9458	0.9913	0.9705	0.9932
UIQI(1)	0.9343	0.9331	0.8818	0.7123	0.9168	0.8354	0.9321
RMSE(0)	0.0159	0.0169	0.0288	0.0314	0.0120	0.0222	0.0104
ERGAS(0)	0.4165	0.4395	0.7370	0.7415	0.3086	0.5454	0.2679
SAM(0)	0.0621	0.0684	0.0852	0.1051	0.0643	0.0957	0.0558
Time(0)	6.41	148.4	1.40	3.83	6.22	80.94	89.68

, and the better results are indicated in bold in the experiments. In terms of spatial features, both the algorithm in this paper and CNMF are closer to 1 in terms of the SSIM and UIQI values, but in comparison, the algorithm in this paper performs better in SSIM, which indicates that it is more similar to the reference image in terms of contrast, brightness, and spatial structure. The two algorithms have similar values in UIQI, which means that the fused image has the least loss of correlation information and is highly similar to the reference image. As far as the spectral features, the CC of this paper’s algorithm is closer to 1, and the RMSE, ERGAS, and SAM are closer to 0, indicating that the spectral distortion of the fused image of this paper’s algorithm is the smallest, which is consistent with the spectral features of the reference image. From the value of PSNR, the algorithm in this paper has the highest PSNR, which indicates that the algorithm in this paper can effectively reject the noise. The algorithms have moderate running times.

4.7.2. Experimental Results of the Indian Pine Dataset

To demonstrate the spatial detail information, reconstruction effect, and fusion results, three bands (R: 61, G: 25, B: 13) are selected to synthesize pseudo-color images in this paper. Set $SNR=35$ dB. The fusion performances of all methods on the Indian Pine dataset are plotted in Figure 10. Figure 10h shows the reference image, and the scene in the red box is shown in the upper left corner of the figure.

Here, the performance of the algorithm is analyzed using a selection of evaluation metrics. The evaluations of the compared algorithms are shown in

Table 3. Performance for Indian Pines.

Method	CNMF	Hysure	FUSE	SOCTT	STEREO	BTD	2D-CNBTD
PSNR(∞)	27.55	30.44	31.24	27.76	26.47	25.38	32.42
SSIM(1)	0.9356	0.9078	0.9259	0.8586	0.8166	0.7436	0.9328
CC(1)	0.8175	0.8768	0.8793	0.7727	0.7425	0.7091	0.8962
UIQI(1)	0.6437	0.6438	0.6650	0.4804	0.4604	0.4067	0.6949
RMSE(0)	0.0241	0.0196	0.0167	0.0215	0.0209	0.0306	0.0127
ERGAS(0)	0.2736	0.2160	0.1873	0.2707	0.3264	0.4431	0.1530
SAM(0)	0.0664	0.0496	0.0401	0.0537	0.0601	0.0850	0.0336
Time(0)	2.73	47.44	0.75	0.58	4.05	7.26	11.62

, and the better experimental results are indicated in bold. In terms of spatial characteristics, both the algorithm in this paper and CNMF are closer to 1 in terms of the SSIM and UIQI values, but in comparison, the paper algorithm performs better in UIQI, which indicates that the fused image has the least loss of relevant information and is highly similar to the reference image. The similar values of both algorithms on SSIM indicate that they are more similar to the reference image in terms of contrast, brightness, and spatial structure. In terms of the spectral features, the CC of the algorithm in this paper is closer to 1, and the RMSE, ERGAS, and SAM are closer to 0, suggesting that its fused image has the least spectral distortion and is consistent with the spectral features of the reference image. From the value of PSNR, the algorithm in this paper has the highest PSNR, which indicates that the algorithm in this paper can effectively eliminate noise. The running time of the algorithm is reasonable.

4.8. Experimental Results for Local Datasets

4.8.1. GF-1 and GF-5 Sand Lake in Ningxia of China

For the GF-1 and GF-5 Sand lake datasets in Ningxia, both datasets have been aligned and corrected usin ENVI 5.3. Due to the specificity of the satellite, the two datasets are treated as if they were taken at the same time. Regarding the three parameters ${\lambda }_i$, ${\mu }_i$, and $\beta$, the previous settings are followed. R and L are set via an ablation experiment: $R=30$, $L=40$. Here, the MSI and HSI image size ratio is 1:1, and the MSI and HSI of the same scene are directly fused without the spectral degradation and spatial degradation processes. To demonstrate the spatial detail information, reconstruction effect, and fusion results, three bands (R: 59, G: 38, B: 20) are selected to synthesize pseudo-color images in this paper. The fusion results of all algorithms on the GF-1 and GF-5 sand lake datasets of Ningxia, China, are shown in Figure 11. The original MSI and HSI images are shown in Figure 11a and Figure 11b, respectively. The scene in the top left corner of the figure is in the red box.

The evaluation of the compared algorithms is shown in

Table 4. Performance for GF-1 and GF-5 sand lake in Ningxia of China.

Method	CNMF	FUSE	SOCTT	STEREO	BTD	2D-CNBTD
PSNR(∞)	13.57	36.20	20.44	5.52	36.97	39.49
SSIM(1)	0.1649	0.9037	0.2383	0.2988	0.9753	0.9434
CC(1)	0.4650	0.9953	0.8690	0.8818	0.9960	0.9976
UIQI(1)	0.1773	0.9230	0.3617	0.3125	0.9390	0.9634
RMSE(0)	1067.9	114.44	374.84	922.06	50.57	50.362
ERGAS(0)	0.8250	0.0780	0.3282	0.8229	0.0882	0.0639
SAM(0)	0.2505	0.0702	0.1746	0.3341	0.0436	0.0309
Time(0)	27.54	1.91	8.26	19.82	64.79	73.44

, indicating the better experimental results in bold. In terms of spatial features, the algorithm in this paper is closer to 1 in terms of UIQI values, indicating that the fused image has the least loss of relevant information and is highly similar to the reference image. It is second only to BTD in SSIM, indicating that it is similar to the reference image in contrast, brightness, and spatial structure. In terms of spectral characteristics, the CC of this paper’s algorithm is closer to 1, and RMSE, ERGAS, and SAM are closer to 0, indicating that its fused image has the least spectral distortion and is consistent with the spectral characteristics of the reference image. From an analysis of PSNR values, the algorithm in this paper has the highest PSNR, indicating that the algorithm in this paper can effectively suppress noise. In terms of running time, the algorithm runs slightly longer, but with improved accuracy.

4.8.2. UAV Acquisition of the Qingtongxia Rice Field Dataset

For the UAV acquisition of the Qingtongxia rice field dataset, Both datasets have been preprocessed using ENVI 5.3. The three parameters ${\lambda }_i$, ${\mu }_i$, and $\beta$ follow the previous settings. R and L are set via ablation experiments: $R=20$, $L=30$. Here, the MSI and HSI image size ratio is 1:1, the spectral degradation and spatial degradation processes are not required, and the algorithm is directly used to fuse MSI and HSI in the same scene. To reveal the spatial detail information, reconstruction effect, and fusion results, three bands (R: 70, G: 44, B: 23) are selected to synthesize pseudo-color images in this paper. The fusion results of all algorithms on the UAV acquisition of the Qingtongxia rice field dataset are shown in Figure 12. The original MSI and HSI images are shown in Figure 12a and Figure 12b, respectively. The scene in the top left corner of the figure is in the red box.

The assessment of the compared algorithms is shown in

Table 5. Performance for UAV acquisition of Qingtongxia rice field.

Method	CNMF	FUSE	SOCTT	STEREO	BTD	2D-CNBTD
PSNR(∞)	20.57	35.23	26.82	14.69	8.54	37.13
SSIM(1)	0.7977	0.9898	0.8257	0.6628	0.0081	0.9911
CC(1)	0.9483	0.9988	0.9834	0.9497	0.9409	0.9992
UIQI(1)	0.1820	0.9330	0.2243	0.1832	0.0009	0.9269
RMSE(0)	0.0577	0.0018	0.0335	0.1886	5.6012	0.0114
ERGAS(0)	0.2926	0.0481	0.1778	1.2398	63.1562	0.0734
SAM(0)	0.0613	0.0318	0.0663	0.2032	0.2995	0.0327
Time(0)	66.85	4.80	3.07	57.43	40.67	49.39

, and the better experimental results are indicated in bold. In terms of the spatial spectral characteristics, the algorithms in this paper and FUSE are closer to 1 for SSIM, UIQI, and CC values, and closer to 0 for RMSE, ERGAS, and SAM, indicating that the fused images have minimal spectral distortion and better spatial structure. In comparison, the algorithm in this paper is better than FUSE. The algorithm in this paper has the highest PSNR, which indicates that the algorithm in this paper can eliminate noise well. The running time of the algorithm is slightly longer.

4.9. Discussion

The superiority and applicability of the fusion algorithm is mainly demonstrated by the results on different datasets.

As shown in Figure 9 and Figure 10, in the standard dataset test, CNMF lost part of the spatial details in Figure 9, and there was serious spectral distortion in Figure 10. Hysure, STEREO, and BTD algorithms had better fusion results in Figure 9, with a lower loss of spectral and spatial information, and only a small amount of blurred blocks existing in the images. In Figure 10, all three algorithms had poor fusion results, with a large number of blurred blocks in the images, indicating that in the Indian Pine dataset, the three algorithms lose a large amount of spectral and spatial information in the fusion process, and BTD is particularly serious. In Figure 9 and Figure 10, the FUSE algorithm has a small number of blurred blocks in the fusion results. In Figure 9 and Figure 10, SCOTT shows the worst fusion results, with a large number of blurred blocks and noise, and a serious loss of spatial–spectral information. In Figure 9 and Figure 10, the fusion results of the proposed algorithm in this paper are closest to the reference image, and the blurred blocks and noise in the images are small. In order to further verify the effectiveness of the fusion algorithm, the metrics for evaluating the spatial and spectral features were selected to describe the effectiveness of the algorithm (as shown in Table 2 and Table 3). The experimental results show that CNMF’s UIQI values in the Pavia University dataset and the SSIM values in the Indian Pine dataset are closer to the optimal values, indicating that CNMF is able to preserve the spatial information in the images. The SCOTT algorithm has poor evaluation metrics on both standard datasets, Hysure and FUSE have better evaluation metrics on both datasets, and STEREO and BTD have moderate performance on the Pavia University dataset but poor performance on the Indian Pine dataset; Except for the UIQI value of the Pavia University dataset and the SSIM value of the Indian pine dataset, the results of other metrics show that the algorithm in this paper is closer to the optimal value.

As shown in Figure 11 and Figure 12, in the local dataset test, CNMF has misalignment, fuzzy blocks, and spectral distortion in the fusion results in Figure 11, and larger fuzzy blocks in the fusion results in Figure 12. FUSE and BTD have clear fusion results in Figure 11 and smaller blurred blocks in the fusion results in Figure 12. SCOTT has serious noise, misalignment, and loss of spatial and spectral information in the fusion results in both Figure 11 and Figure 12. STEREO has severe spectral distortion and a large amount of noise in the fusion results in Figure 11, and large blurred blocks in the fusion results in Figure 12. The algorithm in this paper has clear spatial textures in the fusion results of Figure 11 and Figure 12, and there is no distortion, blurred blocks or other phenomena. In order to further verify the effectiveness of the fusion algorithm, the indexes for evaluating the spatial features and spectral features are selected to describe the effectiveness of the algorithm (as shown in Table 4 and Table 5). The experimental results show that the evaluation indexes of CNMF, SCOTT, STEREO, and BTD on the local dataset perform poorly in general. The evaluation indexes of FUSE on the local dataset perform moderately. The evaluation indexes of the algorithm in this paper on the local dataset are closer to the optimal value.

In summary, CNMF and STEREO show spectral distortion and misalignment in some dataset tests. The SCOTT, STEREO, and BTD algorithms show a large number of fuzzy blocks and noisy situations, indicating that CNMF, SCOTT, and other algorithms show a loss of spatial and spectral information in the process of fusion, and that the algorithms are less applicable scenes. The algorithm proposed in this paper performs better on both standard and local datasets, and the HRHS image texture information is clear, relatively smooth, and closer to the reference image, indicating that it can well retain the spectral information and spatial structure of the image in the fusion process, and that the algorithm can be applied to a variety of scenes.

5. Conclusions

A coupled non-negative block-term tensor decomposition model based on regularization is used to estimate HRHS. Considering the sparsity and smoothness of the endmember matrix and the abundance matrix, the $L_1$ norm is introduced to describe its sparsity. In addition, the difference operators in the horizontal and vertical directions are defined and introduced into the model to describe the segmental smoothness. Finally, the proposed model is solved using two optimization algorithms, ADMM and PAO. Experimental validation on real and local datasets shows that there is a slight difference with the CNMF algorithm in the spatial feature indexes UIQI and SSIM, but the overall performance is excellent in spectral information feature indexes. Further, it is confirmed that the spatial and spectral features of the fusion results of the algorithm in this paper remain consistent with the reference image.

The proposed algorithm in this paper has a minor limitation because the $L_1$ norm has a non-smooth feature, which is not the best choice for the sparse endmember matrix and abundance matrix, and the algorithm running time is slightly longer due to the tensor rank estimation. In future work, it can be extended in these two directions; firstly, by considering other means to promote the sparsity of the abundance and endmember matrices, and secondly, to reduce the algorithm running time without affecting the performance of the MHF algorithm.