# Hyperspectral and Multispectral Image Fusion Using Coupled Non-Negative Tucker Tensor Decomposition

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- The application of non-negativity priors to the Tucker tensor decomposition of LRHSI and HRMSI, to estimate spectral and spatial mode-dictionaries in a Tucker model, respectively. To the best of our knowledge, this is the first time that a non-negative Tucker decomposition is used to represent hyperspectral images in a HSI fusion framework.
- The preservation of spatio-spectral joint structures of HSIs without prior knowledge requirements and much lower information losses than matrix frameworks.
- The construction of an algorithm with lower-order complexity than the state-of-the-art.

## 2. Preliminaries on Tensors

## 3. HSI-MSI Fusion Problem Formulation

#### 3.1. Matrix Factorization-Based Fusion Scheme

#### 3.2. Tensor Decomposition-Based Fusion Scheme

## 4. Proposed CNTD Approach

#### 4.1. Updating Mode-Dictionary Matrices

#### 4.2. Updating Core Tensor

**H**and width

**W**are initialized using the latter. The spectral mode-dictionary $S$ is initialized using the simplex identification split augmented Lagrangian (SISAL) algorithm [50], which efficiently identifies a minimum volume simplex containing the LRHSI spectral vectors. The spatial mode-dictionary matrices

**W**and

**H**are initialized from the mode-one and -two matricization of the HRMSI, respectively, via dictionary update cycles of the KSVD method [51]. The core tensor $\mathcal{C}$ is initialized using the ADMM framework presented in [9].

Algorithm 1: The proposed coupled non-negative tensor decomposition method. |

Input: LRHSI (${\mathcal{Y}}_{h}$), HRMSI (${\mathcal{Y}}_{m}$). Output: HRHSI ($\mathcal{Z}$) |

Estimate PSF (${P}_{1}$, ${P}_{2}$), SRF (${P}_{3}$), using method from [19]. |

Initialize the core tensor ($\mathcal{C}$) via ADMM [9], and mode-dictionaries ($W,\text{}H,\text{}S)$ via DUC KSVD [51]. |

NTD for ${\mathcal{Y}}_{h}$ Initialize ${W}_{h}$, ${H}_{h}\text{}by$ (20), (21), respectively. Update ${W}_{h}$, ${H}_{h}$, $S$ and $\mathcal{C}$ alternately by (27), (29), (30) and (35), respectively until convergence of |

NTD for ${\mathcal{Y}}_{m}$ Initialize ${S}_{m}$ by (22) Update $W$, $H$, ${S}_{\mathit{m}}$ and $\mathcal{C}$ alternately by (36)–(39) until convergence of the objective function in (24). Using the estimated $W$, $H$, $S$ and $\mathcal{C}$ to calculate the HRHSI ($\mathcal{Z}$) via Tucker tensor decomposition (16). |

## 5. Computational Complexity

## 6. Experimental Observations and Results

#### 6.1. Data Sets

#### 6.2. Evaluation Criteria

#### 6.3. Evaluation of the Parameters

#### 6.4. Comparison with State of the Art Fusion Methods

## 7. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Dian, R.; Fang, L.; Li, S. Hyperspectral image super-resolution via non-local sparse tensor factorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Juan, PR, USA, 17–19 June 1997; pp. 5344–5353. [Google Scholar]
- Kwan, C.; Choi, J.H.; Chan, S.H.; Zhou, J.; Budavari, B. A super-resolution and fusion approach to enhancing hyperspectral images. Remote Sens.
**2018**, 10, 1416. [Google Scholar] [CrossRef] [Green Version] - Dian, R.; Li, S.; Fang, L.; Bioucas-Dias, J. Hyperspectral image super-resolution via local Low-rank and sparse representations. In Proceedings of the IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 22–27 July 2018; pp. 4003–4006. [Google Scholar]
- Dian, R.; Li, S.; Fang, L.; Wei, Q. Multispectral and hyperspectral image fusion with spatial-spectral sparse representation. Inf. Fusion
**2019**, 49, 262–270. [Google Scholar] [CrossRef] - Dong, W.; Fu, F.; Shi, G.; Cao, X.; Wu, J.; Li, G.; Li, X. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process.
**2016**, 25, 2337–2352. [Google Scholar] [CrossRef] - Ghasrodashti, E.; Karami, A.; Heylen, R.; Scheunders, P. Spatial resolution enhancement of hyperspectral images using spectral unmixing and bayesian sparse representation. Remote Sens.
**2017**, 9, 541. [Google Scholar] [CrossRef] [Green Version] - Joshi, M.V.; Upla, K.P. Multi-Resolution Image Fusion in Remote Sensing; Cambridge University Press: Cambridge, UK, 2019. [Google Scholar]
- Lanaras, C.; Baltsavias, E.; Schindler, K. Hyperspectral super-resolution with spectral unmixing constraints. Remote Sens.
**2017**, 9, 1196. [Google Scholar] [CrossRef] [Green Version] - Li, S.; Dian, R.; Fang, L.; Bioucas-Dias, J.M. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Trans. Image Process.
**2018**, 27, 4118–4130. [Google Scholar] [CrossRef] [PubMed] - Li, X.; Tian, L.; Zhao, X.; Chen, X. A super resolution approach for spectral unmixing of remote sensing images. Int. J. Remote Sens.
**2011**, 32, 6091–6107. [Google Scholar] [CrossRef] - Wei, Q.; Bioucas-Dias, J.; Dobigeon, N.; Tourneret, J.-Y. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Trans. Geosci. Remote Sens.
**2015**, 53, 3658–3668. [Google Scholar] [CrossRef] [Green Version] - Wei, Q.; Bioucas-Dias, J.; Dobigeon, N.; Tourneret, J.-Y.; Chen, M.; Godsill, S. Multiband image fusion based on spectral unmixing. IEEE Trans. Geosci. Remote Sens.
**2016**, 54, 7236–7249. [Google Scholar] [CrossRef] [Green Version] - Yang, J.; Li, Y.; Chan, J.; Shen, Q. Image fusion for spatial enhancement of hyperspectral image via pixel group based non-local sparse representation. Remote Sens.
**2017**, 9, 53. [Google Scholar] [CrossRef] [Green Version] - Akhtar, N.; Shafait, F.; Mian, A. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3631–3640. [Google Scholar]
- Zare, M.; Helfroush, M.S.; Kazemi, K. Fusing hyperspectral and multispectral images using smooth graph signal modelling. Int. J. Remote Sens.
**2020**, 41, 8610–8630. [Google Scholar] [CrossRef] - Hardie, R.C.; Eismann, M.T.; Wilson, G.L. MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor. IEEE Trans. Image Process.
**2004**, 13, 1174–1184. [Google Scholar] [CrossRef] - Nezhad, Z.H.; Karami, A.; Heylen, R.; Scheunders, P. Fusion of hyperspectral and multispectral images using spectral unmixing and sparse coding. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2016**, 9, 2377–2389. [Google Scholar] [CrossRef] - Wei, Q.; Dobigeon, N.; Tourneret, J.-Y. Bayesian fusion of multi-band images. IEEE J. Sel. Top. Signal Process.
**2015**, 9, 1117–1127. [Google Scholar] [CrossRef] [Green Version] - Simões, M.; Bioucas-Dias, J.; Almeida, L.B.; Chanussot, J. A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans. Geosci. Remote Sens.
**2014**, 53, 3373–3388. [Google Scholar] [CrossRef] [Green Version] - Veganzones, M.A.; Simoes, M.; Licciardi, G.; Yokoya, N.; Bioucas-Dias, J.M.; Chanussot, J. Hyperspectral super-resolution of locally low rank images from complementary multisource data. IEEE Trans. Image Process.
**2015**, 25, 274–288. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Yokoya, N.; Yairi, T.; Iwasaki, A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens.
**2012**, 50, 528–537. [Google Scholar] [CrossRef] - Dian, R.; Li, S.; Fang, L. Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst.
**2019**, 30, 2672–2683. [Google Scholar] [CrossRef] [PubMed] - Dian, R.; Li, S.; Fang, L.; Lu, T.; Bioucas-Dias, J.M. Nonlocal sparse tensor factorization for semiblind hyperspectral and multispectral image fusion. IEEE Trans. Cybern.
**2019**, 50, 4469–4480. [Google Scholar] [CrossRef] [PubMed] - Xu, Y.; Wu, Z.; Chanussot, J.; Comon, P.; Wei, Z. Nonlocal coupled tensor cp decomposition for hyperspectral and multispectral image fusion. IEEE Trans. Geosci. Remote Sens.
**2020**, 58, 348–362. [Google Scholar] [CrossRef] - Zhang, G.; Fu, X.; Huang, K.; Wang, J. Hyperspectral super-resolution: A coupled nonnegative block-term tensor decomposition approach. In Proceedings of the 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), Guadeloupe, France, 15–18 December 2019; pp. 470–474. [Google Scholar]
- Sun, W.; Ren, K.; Meng, X.; Xiao, C.; Yang, G.; Peng, J. A band divide-and-conquer multispectral and hyperspectral image fusion method. IEEE Trans. Geosci. Remote Sens.
**2021**. [Google Scholar] [CrossRef] - Yang, J.; Zhao, Y.-Q.; Chan, J.C.-W. Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network. Remote Sens.
**2018**, 10, 800. [Google Scholar] [CrossRef] [Green Version] - Zhang, L.; Nie, J.; Wei, W.; Li, Y.; Zhang, Y. Deep blind hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst.
**2021**. [Google Scholar] [CrossRef] - Loncan, L.; De Almeida, L.B.; Bioucas-Dias, J.M.; Briottet, X.; Chanussot, J.; Dobigeon, N.; Fabre, S.; Liao, W.; Licciardi, G.A.; Simoes, M. Hyperspectral pansharpening: A review. IEEE Geosci. Remote Sens. Mag.
**2015**, 3, 27–46. [Google Scholar] [CrossRef] [Green Version] - Lee, D.D.; Seung, H.S. Learning the parts of objects by non-negative matrix factorization. Nature
**1999**, 401, 788–791. [Google Scholar] [CrossRef] [PubMed] - Jouni, M.; Dalla Mura, M.; Comon, P. Hyperspectral Image Classification Based on Mathematical Morphology and Using Tensor CP Decomposition. Math. Morphol.Theory Appl.
**2019**, 1, 1–30. [Google Scholar] - Jouni, M.; Dalla Mura, M.; Comon, P. Classification of hyperspectral images as tensors using nonnegative CP decomposition. In Proceedings of the International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing, Saarbrücken, Germany, 8–10 July 2019; pp. 189–201. [Google Scholar]
- Zhao, G.; Tu, B.; Fei, H.; Li, N.; Yang, X. Spatial-spectral classification of hyperspectral image via group tensor decomposition. Neurocomputing
**2018**, 316, 68–77. [Google Scholar] [CrossRef] - Das, S. Hyperspectral image, video compression using sparse tucker tensor decomposition. IET Image Process.
**2021**, 15, 964–973. [Google Scholar] [CrossRef] - Fang, L.; He, N.; Lin, H. CP tensor-based compression of hyperspectral images. JOSA A
**2017**, 34, 252–258. [Google Scholar] [CrossRef] - Huang, F.; Yu, Y.; Feng, T. Hyperspectral remote sensing image change detection based on tensor and deep learning. J. Vis. Commun. Image Represent.
**2019**, 58, 233–244. [Google Scholar] [CrossRef] - Tan, J.; Zhang, J.; Zhang, Y. Target detection for polarized hyperspectral images based on tensor decomposition. IEEE Geosci. Remote Sens. Lett.
**2017**, 14, 674–678. [Google Scholar] [CrossRef] - Li, S.; Wang, W.; Qi, H.; Ayhan, B.; Kwan, C.; Vance, S. Low-rank tensor decomposition based anomaly detection for hyperspectral imagery. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 4525–4529. [Google Scholar]
- Fan, H.; Li, C.; Guo, Y.; Kuang, G.; Ma, J. Spatial–spectral total variation regularized low-rank tensor decomposition for hyperspectral image denoising. IEEE Trans. Geosci. Remote Sens.
**2018**, 56, 6196–6213. [Google Scholar] [CrossRef] - Zhang, H.; Liu, L.; He, W.; Zhang, L. Hyperspectral image denoising with total variation regularization and nonlocal low-rank tensor decomposition. IEEE Trans. Geosci. Remote Sens.
**2019**, 58, 3071–3084. [Google Scholar] [CrossRef] - Qian, Y.; Xiong, F.; Zeng, S.; Zhou, J.; Tang, Y.Y. Matrix-vector nonnegative tensor factorization for blind unmixing of hyperspectral imagery. IEEE Trans. Geosci. Remote Sens.
**2017**, 55, 1776–1792. [Google Scholar] [CrossRef] [Green Version] - Kanatsoulis, C.I.; Fu, X.; Sidiropoulos, N.D.; Ma, W.-K. Hyperspectral super-resolution: Combining low rank tensor and matrix structure. In Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 3318–3322. [Google Scholar]
- Kanatsoulis, C.I.; Fu, X.; Sidiropoulos, N.D.; Ma, W.-K. Hyperspectral super-resolution: A coupled tensor factorization approach. IEEE Trans. Signal Process.
**2018**, 66, 6503–6517. [Google Scholar] [CrossRef] [Green Version] - Cichocki, A.; Mandic, D.; De Lathauwer, L.; Zhou, G.; Zhao, Q.; Caiafa, C.; Phan, H.A. Tensor decompositions for signal processing applications: From two-way to multiway component analysis. IEEE Signal Process. Mag.
**2015**, 32, 145–163. [Google Scholar] [CrossRef] [Green Version] - Cichocki, A.; Zdunek, R.; Phan, A.H.; Amari, S.-I. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation; John Wiley & Sons: Hoboken, NJ, USA, 2009. [Google Scholar]
- Zhou, G.; Cichocki, A. Fast and unique Tucker decompositions via multiway blind source separation. Bull. Pol. Acad. Sci. Tech. Sci.
**2012**, 60, 389–405. [Google Scholar] [CrossRef] [Green Version] - Kim, Y.-D.; Choi, S. Nonnegative tucker decomposition. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8. [Google Scholar]
- Liu, X.; Xia, W.; Wang, B.; Zhang, L. An approach based on constrained nonnegative matrix factorization to unmix hyperspectral data. IEEE Trans. Geosci. Remote Sens.
**2010**, 49, 757–772. [Google Scholar] [CrossRef] - Burred, J.J. Detailed derivation of multiplicative update rules for NMF. Paris Fr.
**2014**. Available online: https://www.semanticscholar.org/paper/Detailed-derivation-of-multiplicative-update-rules-Burred/3376b4df752f2428c451e530f9c6e0ce3a3f05e4 (accessed on 25 May 2021). - Bioucas-Dias, J.M. A variable splitting augmented Lagrangian approach to linear spectral unmixing. In Proceedings of the 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, Grenoble, France, 26–28 August 2009; pp. 1–4. [Google Scholar]
- Smith, L.N.; Elad, M. Improving dictionary learning: Multiple dictionary updates and coefficient reuse. IEEE Signal Process. Lett.
**2012**, 20, 79–82. [Google Scholar] [CrossRef] - Dell’Acqua, F.; Gamba, P.; Ferrari, A.; Palmason, J.A.; Benediktsson, J.A.; Árnason, K. Exploiting spectral and spatial information in hyperspectral urban data with high resolution. IEEE Geosci. Remote Sens. Lett.
**2004**, 1, 322–326. [Google Scholar] [CrossRef] - Green, R.O.; Eastwood, M.L.; Sarture, C.M.; Chrien, T.G.; Aronsson, M.; Chippendale, B.J.; Faust, J.A.; Pavri, B.E.; Chovit, C.J.; Solis, M. Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (AVIRIS). Remote Sens. Environ.
**1998**, 65, 227–248. [Google Scholar] [CrossRef] - Wang, Z.; Bovik, A.C. A universal image quality index. IEEE Signal Process. Lett.
**2002**, 9, 81–84. [Google Scholar] [CrossRef]

**Figure 3.**Composite color images of (

**a**) HRMSI, (

**b**) LRHSI (

**c**) reference images of the Pavia University data set (top row) and Indian Pines data set (bottom row).

**Figure 4.**Spectral response functions; (

**a**) IKONOS like spectral response function, (

**b**) LANDSAT 7-like spectral response function.

**Figure 5.**The RMSE as function of the number of atoms ${n}_{w}$, ${n}_{h}$ and ${n}_{s}$ for the proposed CNTD approach: (

**a**) ${n}_{w}$; (

**b**) ${n}_{h}$; (

**c**) ${n}_{s}$.

**Figure 6.**Band 30 of the Pavia University (two top rows) and Indian Pines data sets (two bottom rows), respectively; the first and third rows (

**a**) LRHSI, (

**b**) CNMF, (

**c**) CSTF, (

**d**) NLSTF. The second and fourth rows (

**a**) reference image, (

**b**) CNN, (

**c**) STEREO, (

**d**) proposed CNTD.

**Figure 7.**The error images of the 30th band of Pavia University (two top rows) and Indian Pines data sets (two bottom rows), the first and third rows show the error image of (

**a**) LRHSI, (

**b**) CNMF, (

**c**) CSTF, and (

**d**) NLSTF. The second and fourth rows show the error images of (

**a**) reference error image, (

**b**) CNN, (

**c**) STEREO, and (

**d**) proposed CNTD method.

Notation | Description |
---|---|

$\mathcal{X}$ | Tensor |

X | Matrix |

$\U0001d4cd$ | Tensor element |

$\mathbf{\U0001d4cd}$ | Spectral vector of tensor |

X | Scaler |

${\times}_{n}$ | Mode-n product |

$\u2a02$ | Kronecker product |

$\u229b$ | Hadamard product |

${{\rm X}}_{\left(n\right)}$ | Mode-n matricization of tensor X |

${X}^{\left(n\right)}$ | n mode matrix in Tucker decomposition |

Method | Pavia University Data Set | ||||
---|---|---|---|---|---|

RMSE | SAM | DD | ERGAS | UIQI | |

Ideal value | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 |

CNMF [21] | 0.140 | 4.313 | 0.017 | 4.989 | 0.952 |

CSTF [9] | 2.160 | 2.390 | 1.055 | 1.230 | 0.991 |

NLSTF [23] | 1.452 | 0.964 | 0.846 | 0.520 | 0.993 |

CNN [27] | 0.016 | 2.203 | 0.103 | 1.447 | 0.976 |

STEREO [43] | 0.061 | 3.922 | 0.010 | 1.865 | 0.989 |

CNTD method | 0.008 | 1.963 | 0.005 | 1.169 | 0.996 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zare, M.; Helfroush, M.S.; Kazemi, K.; Scheunders, P.
Hyperspectral and Multispectral Image Fusion Using Coupled Non-Negative Tucker Tensor Decomposition. *Remote Sens.* **2021**, *13*, 2930.
https://doi.org/10.3390/rs13152930

**AMA Style**

Zare M, Helfroush MS, Kazemi K, Scheunders P.
Hyperspectral and Multispectral Image Fusion Using Coupled Non-Negative Tucker Tensor Decomposition. *Remote Sensing*. 2021; 13(15):2930.
https://doi.org/10.3390/rs13152930

**Chicago/Turabian Style**

Zare, Marzieh, Mohammad Sadegh Helfroush, Kamran Kazemi, and Paul Scheunders.
2021. "Hyperspectral and Multispectral Image Fusion Using Coupled Non-Negative Tucker Tensor Decomposition" *Remote Sensing* 13, no. 15: 2930.
https://doi.org/10.3390/rs13152930