# A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm

## Abstract

**:**

## 1. Introduction

## 2. Restoration-Based Approaches

## 3. Sparse Representation

#### 3.1. Sparse Image Fusion for Spatial-Spectral Fusion

#### 3.1.1. The SparseFI Family for Pan-Sharpening

#### 3.1.2. Hybrid SR-Based Approaches for Pan-Sharpening

#### 3.2. Sparse Image Fusion for Spatio-Temporal Fusion

## 4. Bayesian Approaches

## 5. Variational Approaches

## 6. Performance Comparisons

## 7. Conclusions

## Conflicts of Interest

## References

- Aiazzi, B.; Alparone, L.; Baronti, S.; Carlà, R.; Garzelli, A.; Santurri, L. Sensitivity of pan-sharpening methods to temporal and instrumental changes between multispectral and panchromatic datasets. IEEE Trans. Geosci. Remote Sens.
**2016**, submitted. [Google Scholar] - Aiazzi, B.; Baronti, S.; Selva, M. Improving component substitution pan-sharpening through multivariate regression of MS+Pan data. IEEE Trans. Geosci. Remote Sens.
**2007**, 45, 3230–3239. [Google Scholar] [CrossRef] - Tu, T.M.; Su, S.C.; Shyu, H.C.; Huang, P.S. A new look at IHS-like image fusion methods. Inf. Fusion
**2001**, 2, 177–186. [Google Scholar] [CrossRef] - Chavez, P.S., Jr.; Kwarteng, A.W. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogramm. Eng. Remote Sens.
**1989**, 55, 339–348. [Google Scholar] - Carper, W.; Lillesand, T.; Kiefer, R. The use of Intensity-Hue-Saturation transformations for merging SPOT panchromatic and multispectral image data. Photogramm. Eng. Remote Sens.
**1990**, 56, 459–467. [Google Scholar] - Shettigara, V.K. A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set. Photogramm. Eng. Remote Sens.
**1992**, 58, 561–567. [Google Scholar] - Laben, C.A.; Brower, B.V. Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-Sharpening. U.S. Patent 6,011,875, 4 January 2000. [Google Scholar]
- Aiazzi, B.; Baronti, S.; Lotti, F.; Selva, M. A comparison between global and context-adaptive pan-sharpening of multispectral images. IEEE Geosci. Remote Sens. Lett.
**2009**, 6, 302–306. [Google Scholar] [CrossRef] - Garzelli, A.; Nencini, F.; Capobianco, L. Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Trans. Geosci. Remote Sens.
**2008**, 46, 228–236. [Google Scholar] [CrossRef] - Garzelli, A. Pan-sharpening of multispectral images based on nonlocal parameter Optimization. IEEE Trans. Geosci. Remote Sens.
**2015**, 53, 2096–2107. [Google Scholar] [CrossRef] - Aiazzi, B.; Alparone, L.; Baronti, S.; Garzelli, A. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Trans. Geosci. Remote Sens.
**2002**, 40, 2300–2312. [Google Scholar] [CrossRef] - Núñez, J.; Otazu, X.; Fors, O.; Prades, A.; Palà, V.; Arbiol, R. Multiresolution-based image fusion with additive wavelet decomposition. IEEE Trans. Geosci. Remote Sens.
**1999**, 37, 1204–1211. [Google Scholar] [CrossRef] [Green Version] - González-Audícana, M.; Otazu, X.; Fors, O.; Seco, A. Comparison between Mallat’s and the “à trous” discrete wavelet transform based algorithms for the fusion of multispectral and panchromatic images. Int. J. Remote Sens.
**2005**, 26, 595–614. [Google Scholar] [CrossRef] - Schowengerdt, R.A. Remote Sensing: Models and Methods for Image Processing, 3rd ed.; Academic Press: San Diego, CA, USA, 2007. [Google Scholar]
- Liu, J.G. Smoothing filter based intensity modulation: A spectral preserve image fusion technique for improving spatial details. Int. J. Remote Sens.
**2000**, 21, 3461–3472. [Google Scholar] [CrossRef] - Aiazzi, B.; Alparone, L.; Baronti, S.; Garzelli, A.; Selva, M. MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogramm. Eng. Remote Sens.
**2006**, 72, 591–596. [Google Scholar] [CrossRef] - Garzelli, A.; Nencini, F. PAN-sharpening of very high resolution multispectral images using genetic algorithms. Int. J. Remote Sens.
**2006**, 27, 3273–3292. [Google Scholar] [CrossRef] - Garzelli, A.; Nencini, F. Panchromatic sharpening of remote sensing images using a multiscale Kalman filter. Pattern Recognit.
**2007**, 40, 3568–3577. [Google Scholar] [CrossRef] - Alparone, L.; Baronti, S.; Aiazzi, B.; Garzelli, A. Spatial methods for multispectral pan-sharpening: Multiresolution analysis demystified. IEEE Trans. Geosci. Remote Sens.
**2016**, 54, 2563–2576. [Google Scholar] [CrossRef] - Baronti, S.; Aiazzi, B.; Selva, M.; Garzelli, A.; Alparone, L. A theoretical analysis of the effects of aliasing and misregistration on pan-sharpened imagery. IEEE J. Sel. Top. Signal Process.
**2011**, 5, 446–453. [Google Scholar] [CrossRef] - Vivone, G.; Alparone, L.; Chanussot, J.; Dalla Mura, M.; Garzelli, A.; Restaino, R.; Licciardi, G.; Wald, L. A critical comparison among pan-sharpening algorithms. IEEE Trans. Geosci. Remote Sens.
**2015**, 53, 2565–2586. [Google Scholar] [CrossRef] - Yang, J.; Wright, J.; Huang, T.; Ma, Y. Images super-resolution via sparse representation. IEEE Trans. Image Process.
**2010**, 19, 2861–2873. [Google Scholar] [CrossRef] [PubMed] - Kundur, D.; Hatzinakos, D. Blind image deconvolution. IEEE Signal Process. Mag.
**1996**, 13, 43–64. [Google Scholar] [CrossRef] - Vivone, G.; Simões, M.; Mura, M.D.; Restaino, R.; Bioucas-Dias, J.M.; Licciardi, G.A.; Chanussot, J. Pan-sharpening based on semiblind deconvolution. IEEE Trans. Geosci. Remote Sens.
**2015**, 53, 1997–2010. [Google Scholar] [CrossRef] - Teng, Y.; Zhang, Y.; Chen, Y.; Ti, C. Adaptive morphological filtering method for structural fusion restoration of hyperspectral images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2016**, 9, 655–667. [Google Scholar] [CrossRef] - Alparone, L.; Garzelli, A.; Vivone, G. Interchannel calibration for MS pan-sharpening: Theoretical issues and practical solutions. IEEE Trans. Geosci. Remote Sens.
**2016**, submitted. [Google Scholar] - Zhang, Y.; De Backer, S.; Scheunders, P. Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images. IEEE Trans. Geosci. Remote Sens.
**2009**, 47, 3834–3843. [Google Scholar] [CrossRef] - Fasbender, D.; Radoux, J.; Bogaert, P. Bayesian data fusion for adaptable image pan-sharpening. IEEE Trans. Geosci. Remote Sens.
**2008**, 46, 1847–1857. [Google Scholar] [CrossRef] - Zhang, Y.; Duijster, A.; Scheunders, P. A Bayesian restoration approach for hyperspectral images. IEEE Trans. Geosci. Remote Sens.
**2012**, 50, 3453–3462. [Google Scholar] [CrossRef] - Palsson, F.; Sveinsson, J.R.; Ulfarsson, M.O. A new pan-sharpening algorithm based on total variation. IEEE Geosci. Remote Sens. Lett.
**2014**, 11, 318–322. [Google Scholar] [CrossRef] - Zhang, H.; Huang, B. A new look at image fusion methods from a bayesian perspective. Remote Sens.
**2015**, 7, 6828–6861. [Google Scholar] [CrossRef] - Palubinskas, G. Model-based view at multi-resolution image fusion methods and quality assessment measures. Int. J. Image Data Fusion
**2016**, 7, 203–218. [Google Scholar] [CrossRef] - Otazu, X.; González-Audícana, M.; Fors, O.; Núñez, J. Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Trans. Geosci. Remote Sens.
**2005**, 43, 2376–2385. [Google Scholar] [CrossRef] [Green Version] - Li, Z.; Leung, H. Fusion of multispectral and panchromatic images using a restoration-based method. IEEE Trans. Geosci. Remote Sens.
**2009**, 47, 1482–1491. [Google Scholar] - Donoho, D.L. Compressed sensing. IEEE Trans. Inf. Theory
**2006**, 52, 1289–1306. [Google Scholar] [CrossRef] - Li, S.; Yang, B. A new pan-sharpening method using a compressed sensing technique. IEEE Trans. Geosci. Remote Sens.
**2011**, 49, 738–746. [Google Scholar] [CrossRef] - Li, S.; Yin, H.; Fang, L. Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Trans. Geosci. Remote Sens.
**2013**, 51, 4779–4789. [Google Scholar] [CrossRef] - Zhu, X.X.; Bamler, R. A sparse image fusion algorithm with application to pan-sharpening. IEEE Trans. Geosci. Remote Sens.
**2013**, 51, 2827–2836. [Google Scholar] [CrossRef] - Cheng, M.; Wang, C.; Li, J. Sparse representation based pan-sharpening using trained dictionary. IEEE Geosci. Remote Sens. Lett.
**2014**, 11, 293–297. [Google Scholar] [CrossRef] - Huang, B.; Song, H. Spatiotemporal reflectance fusion via sparse representation. IEEE Trans. Geosci. Remote Sens.
**2012**, 50, 3707–3716. [Google Scholar] [CrossRef] - Song, H.; Huang, B. Spatiotemporal satellite image fusion through one-pair image learning. IEEE Trans. Geosci. Remote Sens.
**2013**, 51, 1883–1896. [Google Scholar] [CrossRef] - Huang, B.; Song, H.; Cui, H.; Peng, J.; Xu, Z. Spatial and spectral image fusion using sparse matrix factorization. IEEE Trans. Geosci. Remote Sens.
**2014**, 52, 1693–1704. [Google Scholar] [CrossRef] - Chen, S.; Donoho, D.L.; Saunders, M. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput.
**1998**, 20, 33–61. [Google Scholar] [CrossRef] - Aharon, M.; Elad, M.; Bruckstein, A. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process.
**2006**, 54, 4311–4322. [Google Scholar] [CrossRef] - Vicinanza, M.R.; Restaino, R.; Vivone, G.; Mura, M.D.; Chanussot, J. A pan-sharpening method based on the sparse representation of injected details. IEEE Geosci. Remote Sens. Lett.
**2015**, 12, 180–184. [Google Scholar] [CrossRef] - Zhu, X.X.; Grohnfeldt, C.; Bamler, R. Exploiting joint sparsity for pan-sharpening: The J-sparseFI algorithm. IEEE Trans. Geosci. Remote Sens.
**2016**, 54, 2664–2681. [Google Scholar] [CrossRef] - Alparone, L.; Aiazzi, B.; Baronti, S.; Garzelli, A.; Nencini, F.; Selva, M. Multispectral and panchromatic data fusion assessment without reference. Photogramm. Eng. Remote Sens.
**2008**, 74, 193–200. [Google Scholar] [CrossRef] - Wald, L.; Ranchin, T.; Mangolini, M. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogramm. Eng. Remote Sens.
**1997**, 63, 691–699. [Google Scholar] - Garzelli, A.; Nencini, F. Hypercomplex quality assessment of multi-/hyper-spectral images. IEEE Geosci. Remote Sens. Lett.
**2009**, 6, 662–665. [Google Scholar] [CrossRef]

**Figure 1.**Flowchart of a pan-sharpening algorithm based on compressed sensing [36].Pan-sharpening based on compressed sensing.

**Figure 2.**Block diagram of the sparse fusion of images (SparseFI) pan-sharpening method proposed in [38].

**Figure 3.**Predicting the Landsat image at date ${t}_{2}$ from Landsat images at dates ${t}_{1}$ and ${t}_{3}$ and MODIS images at all dates.

**Figure 4.**Block diagram of the spatio-temporal fusion proposed in Huang and Song [40].

**Figure 5.**WorldView-II image results: (

**a**) true-color composition of the original 4-m multispectral (MS) image, i.e., the reference data; (

**b**) input 4-m Pan image; (

**c**) input 16-m MS image; (

**b**,

**c**) are obtained by degrading the original Pan/MS resolutions by a factor of four; (

**d**) AWLP; (

**e**) Li algorithm; (

**f**) super-resolution-based details injection (SR-D) [45].

**Figure 6.**HySpex image results: (

**a**) true-color composition of the reference $0.75$-m MS image; (

**b**) input $0.75$-m Pan; (

**c**) input $1.5$-m MS; (

**d**) AWLP output; (

**e**) SparseFI output; (

**f**) J-SparseFI output [46].

AWLP [33] | SparseFI [38] | J-SparseFI [46] | Li et al. [37] | SR-D [45] | |
---|---|---|---|---|---|

${\mathrm{\Delta}}_{\mathrm{ERGAS}}$ (%) | $0\%$ | $-1.3\%$ | $-5.2\%$ | ${+4.6\%}$ | ${+13.6\%}$ |

${\mathrm{\Delta}}_{\mathrm{SAM}}$ (%) | $0\%$ | $-7.7\%$ | $-11.3\%$ | $-20.6\%$ | $-1.7\%$ |

${\mathrm{\Delta}}_{\mathrm{Q}2\mathrm{n}}$ (%) | $0\%$ | $+0.9\%$ | $+2.4\%$ | ${-5.1\%}$ | ${-0.8\%}$ |

${\mathrm{\Delta}}_{time}$ (s) | 0 | ${\sim +3000}$ | ${+2500}$ | ${+3000}$ | ${+8}$ |

**Table 2.**A synoptic view of recent remote sensing image fusion algorithms based on the super- resolution paradigm (FE: Filter Estimation; BDF: Bayesian Data Fusion; ASE: Adaptive Structuring Element; SPSTFM: sparse-representation-based spatio-temporal reflectance fusion model. SASFM: Spatial And Spectral Fusion Model).

Reference | Application Field | Complexity | Performances with Respect to Classical Methods | |
---|---|---|---|---|

SparseFI | [38] | Pan-sharpening | Huge | Comparable |

J-SparseFI | [46] | Pan-sharpening | Huge | Slightly better |

Li et al. | [37] | Pan-sharpening | Huge | Comparable |

SR-D | [45] | Pan-sharpening | Low | Comparable |

FE | [24] | Pan-sharpening | Very Low | Slightly better |

BDF | [28] | Pan-sharpening | Medium/High | Comparable |

Palsson et al. | [30] | Pan-sharpening | Low | Comparable |

ASE | [25] | Destriping | Low | Slightly better |

Zhang et al. | [27] | MS/HSFusion | High | Comparable |

Zhang et al. | [29] | MS/HS Fusion | High | Slightly better |

SPSTFM | [40] | Spatio-temporal Fusion | Medium/High | Comparable |

Song et al. | [41] | Spatio-temporal Fusion | High | Slightly better |

SASFM | [42] | Spatio-temporal Fusion | High | Slightly better |

© 2016 by the author; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Garzelli, A.
A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm. *Remote Sens.* **2016**, *8*, 797.
https://doi.org/10.3390/rs8100797

**AMA Style**

Garzelli A.
A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm. *Remote Sensing*. 2016; 8(10):797.
https://doi.org/10.3390/rs8100797

**Chicago/Turabian Style**

Garzelli, Andrea.
2016. "A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm" *Remote Sensing* 8, no. 10: 797.
https://doi.org/10.3390/rs8100797