# Rolling 3D Laplacian Pyramid Video Fusion

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Video Fusion

^{N+1}, where N is the number of pyramid levels. Similar to the 2D filtering situation, where each next level is obtained by decimation with factor 2, in the 3D case the number of frames is also decreased with factor 2 (Figure 1). The equivalent 3D Laplacian pyramid of a sequence is obtained in the same way as in the 2D case, using the Gaussian pyramid expansion and subtraction. The 3D pyramid fusion can then be performed using the same conventional methods of pyramid fusion used in image fusion. The final fused sequence is formed by reconstructing the 3D Laplace pyramid (Figure 1). Other methods of the static image fusion extended to the 3D fusion in this manner are 3D DWT [54], 3D DT CWT [55,56] and 3D Curvelets [16,17]. A related, advanced 3D fusion approach used to additionally achieve noise reduction is polyfusion [59], which performs the Laplace pyramid fusion of different 2D sections of the 3D pyramid (e.g., spatial only sections or spatio-dynamic sections involving lateral pyramid side (Figure 2). The final fused sequence is obtained by fusing these two fusion results, while taking care of the dynamic value range.

## 3. Dynamic Laplacian Rolling-Pyramid Fusion

_{i}

^{Va}(m,n,t) and L

_{i}

^{Vb}(m,n,t) which represent higher frequencies and, therefore finer details in the incoming multi-sensory sequences. Similar to the fusion of large-scale structures, the spatio-temporal energy approach based on a local neighborhood of M × N × T is also used here. The window size has been kept the same at 3.

#### Temporally Stable Fusion

## 4. Results

_{s}is the total number of observers that took part in the trial.

_{ST}with the structural similarity (SSIM) index and the perception characteristics of human visual system (HVS) [62]. First, for each frame, two sub-indices, i.e., the spatial fusion quality index and the temporal fusion quality index, are defined by the weighted local SSIM indices. Second, for the current frame, an individual-frame fusion quality measure is obtained by integrating the above two sub-indices. Last, the global video fusion metric is constructed as the weighted average of all the individual-frame fusion quality measures. In addition, according to the perception characteristics of HVS, some local and global spatial–temporal information, such as local variance, pixel movement, global contrast, background motion and so on, is employed to define the weights in the metric Q

_{ST}.

#### 4.1. Objective Evaluation

#### 4.2. Subjective Evaluation

#### 4.3. Computational Complexity

## 5. Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Pavlović, R.; Petrović, V. Objective evaluation and suppressing effects of noise in dynamic image fusion. Sci. Tech. Rev.
**2014**, 64, 21–29. [Google Scholar] - Du, Q.; Xu, H.; Ma, Y.; Huang, J.; Fan, F. Fusing infrared and visible images of different resolutions via total variation model. Sensors
**2018**, 18, 3827. [Google Scholar] [CrossRef] [PubMed] - Huang, Y.; Bi, D.; Wu, D. Infrared and visible image fusion based on different constraints in the non-subsampled shearlet transform domain. Sensors
**2018**, 18, 1169. [Google Scholar] [CrossRef] - Li, S.; Kang, X.; Fang, L.; Hu, J.; Yin, H. Pixel-level image fusion: A survey of the state of the art. Inf. Fusion
**2017**, 33, 100–112. [Google Scholar] [CrossRef] - Jin, X.; Jiang, Q.; Yao, S.; Zhou, D.; Nie, R.; Hai, J.; He, K. A survey of infrared and visual image fusion methods. Infrared Phys. Technol.
**2017**, 85, 478–501. [Google Scholar] [CrossRef] - Jiayi, M.A.; Yong, M.A.; Chang, L. Infrared and visible image fusion methods and applications: A survey. Inf. Fusion
**2019**, 45, 153–178. [Google Scholar] - Li, H.; Liu, L.; Huang, W.; Yue, C. An improved fusion algorithm for infrared and visible images based on multi-scale transform. Infrared Phys. Technol.
**2016**, 74, 28–37. [Google Scholar] [CrossRef] - Dogra, A.; Goyal, B.; Agrawal, S. From multi-scale decomposition to non-multi-scale decomposition methods: A comprehensive survey of image fusion techniques and its applications. IEEE Access
**2017**, 5, 16040–16067. [Google Scholar] [CrossRef] - Chang, L.; Feng, X.; Zhu, X.; Zhang, R.; He, R.; Xu, C. CT and MRI image fusion based on multiscale decomposition method and hybrid approach. IET Image Process.
**2018**, 13, 83–88. [Google Scholar] [CrossRef] - Bavirisetti, D.P.; Dhuli, R. Two-scale image fusion of visible and infrared images using saliency detection. Infrared Phys. Technol.
**2016**, 76, 52–64. [Google Scholar] [CrossRef] - Xing, C.; Wang, Z.; Meng, F.; Dong, C. Fusion of infrared and visible images with Gaussian smoothness and joint bilateral filtering iteration decomposition. IET Comput. Vis.
**2018**, 13, 44–52. [Google Scholar] [CrossRef] - Gan, W.; Wu, X.; Wu, W.; Yang, X.; Ren, C.; He, X.; Liu, K. Infrared and visible image fusion with the use of multi-scale edge-preserving decomposition and guided image filter. Infrared Phys. Technol.
**2015**, 72, 37–51. [Google Scholar] [CrossRef] - Toet, A.; Hogervorst, M.A. Multiscale image fusion through guided filtering. Proc. SPIE
**2016**, 9997, 99970J. [Google Scholar] - Cai, J.; Cheng, Q.; Peng, M.; Song, Y. Fusion of infrared and visible images based on nonsubsampled contourlet transform and sparse K-SVD dictionary learning. Infrared Physics & Technology
**2017**, 82, 85–95. [Google Scholar] - Liang, X.; Junping, D.; Zhenhong, Z. Infrared-visible video fusion based on motion-compensated wavelet transforms. IET Image Process.
**2015**, 9, 318–328. [Google Scholar] - Zhang, Q.; Yueling, C.; Long, W. Multisensor video fusion based on spatial–temporal salience detection. Signal Process.
**2013**, 93, 2485–2499. [Google Scholar] [CrossRef] - Zhang, Q.; Wang, Y.; Levine, M.D.; Yuan, X.; Wang, L. Multisensor video fusion based on higher order singular value decomposition. Inf. Fusion
**2015**, 24, 54–71. [Google Scholar] [CrossRef] - Gangapure, V.N.; Nanda, S.; Chowdhury, A.S. Superpixel-based causal multisensor video fusion. IEEE Trans. Circuits Syst. Video Technol.
**2018**, 28, 1263–1272. [Google Scholar] [CrossRef] - Hu, H.M.; Wu, J.; Li, B.; Guo, Q.; Zheng, J. An adaptive fusion algorithm for visible and infrared videos based on entropy and the cumulative distribution of gray levels. IEEE Trans. Multimed.
**2017**, 19, 2706–2719. [Google Scholar] [CrossRef] - Jiawei, W.; Hai-Miao, H.; Yuanyuan, G. A realtime fusion algorithm of visible and infrared videos based on spectrum characteristics. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP); IEEE: Piscataway, NJ, USA, 2016; pp. 3369–3373. [Google Scholar]
- Rockinger, O.; Fechner, T. Pixel-level image fusion: The case of image sequences. Proc. SPIE
**1998**, 3374, 378–388. [Google Scholar] - Li, J.; Nikolov, S.; Benton, C.; Scott-Samuel, N. Motion-based video fusion using optical flow information. In Proceedings of the 9th International Conference on Information Fusion, Florence, Italy, 10–13 July 2006; pp. 1–8. [Google Scholar]
- Blum Rick, S.; Zheng, L. Multi-Sensor Image Fusion and Its Applications; CRC Press: Boca Raton, FL, USA, 2005. [Google Scholar]
- Petrovic, V.; Cootes, T.; Pavlovic, R. Dynamic image fusion performance evaluation. In Proceedings of the 9th International Conference on Information Fusion, Québec, Canada, 9–12 July 2007; pp. 1–7. [Google Scholar]
- Vanmali, V.; Gadre, V.M. Visible and nir image fusion using weight-map-guided laplacian–gaussian pyramid for improving scene visibility. Sadhana
**2017**, 42, 1063–1082. [Google Scholar] - Xu, H.; Wang, Y.; Wu, Y.; Qian, Y. Infrared and multi-type images fusion algorithm based on contrast pyramid transform. Infrared Phys. Technol.
**2016**, 78, 133–146. [Google Scholar] [CrossRef] - Burt, P.; Adelson, E. The Laplacian pyramid as a compact image code. IEEE Trans. Commun.
**1983**, COM-31, 532–540. [Google Scholar] [CrossRef] - Chipman, L.J.; Orr, T.M.; Graham, L.N. Wavelets and image fusion. In Proceedings of the International Conference on Image Processing, Washington, DC, USA, 23–26 October 1995; pp. 248–251. [Google Scholar]
- Adu, J.; Gan, J.; Wang, Y.; Huang, J. Image fusion based on nonsubsampled contourlet transform for infrared and visible light image. Infrared Phys. Technol.
**2013**, 61, 94–100. [Google Scholar] [CrossRef] - Naidu, V. Novel image fusion techniques using dct. Int. J. Comput. Sci. Bus. Inf.
**2013**, 5, 1–18. [Google Scholar] - Kumar, B.S. Image fusion based on pixel significance using cross bilateral filter. Signal Image Video Process.
**2015**, 9, 1193–1204. [Google Scholar] [CrossRef] - Zhou, Z.; Wang, B.; Li, S.; Dong, M. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters. Inf. Fusion
**2016**, 30, 15–26. [Google Scholar] [CrossRef] - Li, S.; Kang, X.; Hu, J. Image fusion with guided filtering. IEEE Trans. Image Process.
**2013**, 22, 2864–2875. [Google Scholar] - Bavirisetti, D.P.; Dhuli, R. Fusion of infrared and visible sensor images based on anisotropic diffusion and karhunen-loeve transform. IEEE Sens. J.
**2016**, 16, 203–209. [Google Scholar] [CrossRef] - Liu, Y.; Wang, Z. Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Process.
**2014**, 9, 347–357. [Google Scholar] [CrossRef] - Liu, Y.; Liu, S.; Wang, Z. A general framework for image fusion based on multi-scale transform and sparse representation. Inf. Fusion
**2015**, 24, 147–164. [Google Scholar] [CrossRef] - Qu, X.; Hu, C.; Yan, J. Image fusion algorithm based on orientation information motivated pulse coupled neural networks. In Proceedings of the World Congress on Intelligent Control and Automation, Chongqing, China, 25–27 June 2008; pp. 2437–2441. [Google Scholar]
- Qu, X.B.; Yan, J.W.; Xiao, H.Z.; Zhu, Z.Q. Image fusion algorithm based on spatial frequency motivated pulse coupled neural networks in nonsubsampled contourlet transform domain. Acta Autom. Sin.
**2008**, 34, 1508–1514. [Google Scholar] [CrossRef] - Naidu, V. Hybrid ddct-pca based multi sensor image fusion. J. Opt.
**2014**, 43, 48–61. [Google Scholar] [CrossRef] - Bavirisetti, D.P.; Xiao, G.; Liu, G. Multi-sensor image fusion based on fourth order partial differential equations. In Proceedings of the 20th International Conference on Information Fusion, Xi’an, China, 10–13 July 2017; pp. 1–9. [Google Scholar]
- Zhang, Y.; Zhang, L.; Bai, X.; Zhang, L. Infrared and visual image fusion through infrared feature extraction and visual information preservation. Infrared Phys. Technol.
**2017**, 83, 227–237. [Google Scholar] [CrossRef] - Zhang, X.; Ma, Y.; Fan, F.; Zhang, Y.; Huang, J. Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition. JOSA A
**2017**, 34, 1400–1410. [Google Scholar] [CrossRef] - Ma, J.; Chen, C.; Li, C.; Huang, J. Infrared and visible image fusion via gradient transfer and total variation minimization. Inf. Fusion
**2016**, 31, 100–109. [Google Scholar] [CrossRef] - Luo, X.; Wang, S.; Yuan, D. Weber-aware weighted mutual information evaluation for infrared–visible image fusion. J. Appl. Remote Sens.
**2016**, 10, 045004. [Google Scholar] [CrossRef] - Toet, A. Image fusion by a ratio of low-pass pyramid. Pattern Recognit. Lett.
**1989**, 9, 245–253. [Google Scholar] [CrossRef] - Li, H.; Manjunath, B.S.; Mitra, S.K. Multisensor image fusion using the wavelet transform. Gr. Models Image Process.
**1995**, 57, 235–245. [Google Scholar] [CrossRef] - Zhan, L.; Zhuang, Y.; Huang, L. Infrared and visible images fusion method based on discrete wavelet transform. J. Comput.
**2017**, 28, 57–71. [Google Scholar] [CrossRef] - Madheswari, K.; Venkateswaran, N. Swarm intelligence based optimisation in thermal image fusion using dual tree discrete wavelet transform. Quant. Infrared Thermogr. J.
**2017**, 14, 24–43. [Google Scholar] [CrossRef] - Liu, S.; Piao, Y.; Tahir, M. Research on fusion technology based on low-light visible image and infrared image. Opt. Eng.
**2016**, 55, 123104. [Google Scholar] [CrossRef] [Green Version] - Candès, E.; Demanet, L.; Donoho, D.; Ying, L. Fast discrete curvelet transforms. Multiscale Model. Simul.
**2006**, 5, 861–899. [Google Scholar] [CrossRef] - Do, M.N.; Vetterli, M. The finite ridgelet transform for image representation. IEEE Trans. Image Process.
**2003**, 12, 16–28. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zuo, Y.; Liu, J.; Bai, G.; Wang, X.; Sun, M. Airborne infrared and visible image fusion combined with region segmentation. Sensors
**2017**, 17, 11–27. [Google Scholar] [CrossRef] [PubMed] - Masini, A.; Branchitta, F.; Diani, M.; Corsini, G. Sight enhancement through video fusion in a surveillance system. In Proceedings of the 14th International Conference on Image Analysis and Processing, Modena, Italy, 10–14 September 2007; pp. 554–559. [Google Scholar]
- Hill, R.; Achim, A.; Bull, D. Scalable video fusion. In Proceedings of the 2013 IEEE International Conference on Image Processing; IEEE: Piscataway, NJ, USA, 2013; pp. 1277–1281. [Google Scholar]
- Wang, Y.; Wang, I.; Selesnick, I.; Vetro, A. Video coding using 3D dual-tree wavelet transform. J. Image Video Process.
**2007**, 1, 1–15. [Google Scholar] - Hill, R.; Achim, A.; Bull, D. Scalable fusion using a 3D dual tree wavelet transform. In Proceedings of the Sensor Signal Processing for Defence (SSPD 2011), London, UK, 27–29 September 2011; p. 35. [Google Scholar]
- Hogervorst, M.A.; Toet, A. Improved Color Mapping Methods for Multiband Nighttime Image Fusion. J. Imaging
**2017**, 3, 36. [Google Scholar] [CrossRef] - Vlahovic, N.; Graovac, S. Sensibility Analysis of the Object Tracking Algorithms in Thermal Image. Scientific Technical Review
**2017**, 67, 13–20. [Google Scholar] [CrossRef] - Kai, Z.; Zhou, W. Polyview fusion: A strategy to enhance video denoising algorithms. IEEE Trans. Image Process.
**2012**, 21, 2324–2328. [Google Scholar] - Zheng, Y.; Blasch, E.; Liu, Z. Multispectral Image Fusion and Colorization; SPIE Press: Bellingham, WA, USA, 2018. [Google Scholar]
- Petrović, V. Subjective tests for image fusion evaluation and objective metric validation. Inf. Fusion
**2007**, 8, 208–216. [Google Scholar] [CrossRef] - Zhang, Q.; Wang, L.; Li, H.; Ma, Z. Video fusion performance evaluation based on structural similarity and human visual perception. Signal Process.
**2012**, 92, 912–925. [Google Scholar] [CrossRef] - Haghighat, M.B.A.; Aghagolzadeh, A.; Seyedarabi, H. A non-reference image fusion metric based on mutual information of image features. Comput. Electr. Eng.
**2011**, 37, 744–756. [Google Scholar] [CrossRef]

**Figure 6.**Video fusion performance of proposed local energy HF detail fusion (green) compared to conventional frame-by-frame select-max fusion (red) measured using objective fusion performance metric DQ.

**Figure 7.**Fused two successive frames (top images) and difference image obtained between these two frames (bottom image).

**Figure 8.**Results of objective measure DQ on proposed video fusion algorithm changing value of similarity threshold ξ from 0 to 1.

**Figure 9.**Pyramid fusion selection maps of the static Laplacian fusion (

**left**) and proposed fusion method (

**right**).

**Figure 12.**Fused images with Laplacian pyramid (

**left column**), the middle Shift invariant discrete wavelet (SIDWT) (

**middle column**) and proposed LAP-DIN fusion (

**right column**).

**Figure 16.**Comparing results of objective measure I on six fusion methods (static and dynamic) on database set.

LAP | SIWT | LAP-DIN | |
---|---|---|---|

Seq 1 | 0.23 | 0.23 | 0.26 |

Seq 2 | 0.26 | 0.25 | 0.30 |

Seq 3 | 0.20 | 0.22 | 0.23 |

Seq 4 | 0.26 | 0.26 | 0.29 |

Seq 5 | 0.23 | 0.26 | 0.28 |

Seq 6 | 0.19 | 0.21 | 0.23 |

Mean | 0.23 | 0.24 | 0.27 |

MCDWT | LAP 3D | LAP-DIN | |
---|---|---|---|

Seq 1 | 0.24 | 0.32 | 0.26 |

Seq 2 | 0.26 | 0.37 | 0.30 |

Seq 3 | 0.23 | 0.30 | 0.23 |

Seq 4 | 0.22 | 0.30 | 0.23 |

Seq 5 | 0.26 | 0.30 | 0.28 |

Seq 6 | 0.30 | 0.28 | 0.29 |

Mean | 0.25 | 0.31 | 0.27 |

LAP | SIDWT | MCDWT | LAP 3D | LAP-DIN | |
---|---|---|---|---|---|

DQ | 0.23 | 0.24 | 0.25 | 0.31 | 0.27 |

I | 11.18 | 11.30 | 11.50 | 11.87 | 11.70 |

Q_{ST} | 0.87009 | 0.870064 | 0.871903 | 0.878615 | 0.876745 |

FMI | 0.673145 | 0.641313 | 0.661495 | 0.692034 | 0.676197 |

LAP | SIDWT | MCDWT | LAP-3D | LAP-DIN | |
---|---|---|---|---|---|

Multiple of LAP | 1 | 1.6 | 1.8 | 1.75 | 1.3 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Pavlović, R.; Petrović, V.
Rolling 3D Laplacian Pyramid Video Fusion. *Electronics* **2019**, *8*, 447.
https://doi.org/10.3390/electronics8040447

**AMA Style**

Pavlović R, Petrović V.
Rolling 3D Laplacian Pyramid Video Fusion. *Electronics*. 2019; 8(4):447.
https://doi.org/10.3390/electronics8040447

**Chicago/Turabian Style**

Pavlović, Rade, and Vladimir Petrović.
2019. "Rolling 3D Laplacian Pyramid Video Fusion" *Electronics* 8, no. 4: 447.
https://doi.org/10.3390/electronics8040447