# Airborne Infrared and Visible Image Fusion Combined with Region Segmentation

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Fusion System on the Airborne Photoelectric Platform

## 3. Fast Registration of IR and Visible Images

## 4. IR-Visible Image Fusion Method Combined with Region Segmentation

^{l}

^{,1}, L

^{l}

^{,2}} and the high-frequency coefficients in six directions {H

^{l,θ}, θ = ±15°, ±45°, ±75°}, where l is the number of DTCWT decomposition layers. Low-frequency coefficients reflect the essential information of an image, while high-frequency coefficients show its detail information. Low-frequency components are fused in the ROI and background region under different rules, according to the image segmentation result. High-frequency components are fused under a rule that considers the weight assigned for every region by the richness of detail information. Section 4.1 of this paper introduces the region segmentation method based on ROI, while Section 4.2 and Section 4.3 describe low-frequency and high-frequency fusion rules, respectively.

#### 4.1. Image Region Segmentation Based on ROI

_{c}in the image I is defined as:

_{c}in the Lab space, indicating the color difference between I

_{c}and I

_{i}. Then, the Equation (1) can be rewritten as:

_{c}is the color value of pixel I

_{c}, n is the total number of colors contained in the image, and f

_{j}is the occurrence probability of a

_{j}in the image. This paper applies the above method to the significance extraction and segmentation of an IR image. In doing so, the Equation (2) is changed to calculate the significance of every gray level. The extraction of significance from an IR image is mainly to highlight the information on a hot target and to clearly distinguish it from background information. To increase the calculation speed, this paper proposes to change the gray values 0–255 to 0–15. In the Equation (2), there will be n = 16, showing that the gray difference of 16 pixels/scale can meet the requirement of hot-target significance detection in the IR image. The significance map quantified in this way may have some blemishes in the form of gray value jumping, but this can be improved through smoothening. Convert the significance value of every gray level into the weighted average significance value of similar gray levels. Choose the significance values of 4 adjacent gray levels to improve the significance of pixel a through the Equation (3):

_{i}. Thus, in the gray space, a bigger weight will be assigned to the gray level closer to a, and similar gray levels will be assigned to similar significance values. In this way, the flaws caused by quantification can be reduced. After obtaining the ROI through significance detection, we shall segment the region in the IR image with the help of ROI, take the initial segmentation result obtained after the binarization of significance map as the initial image, and conduct iterative operation through GrabCut [30] to further optimize the segmentation result. Every iterative operation is followed by corrosion and dilation operation to obtain a new target region, which, in turn, is followed by the next iteration. In this paper, no more than four iterations are allowed. The ROI-based region segmentation result in three different situations is shown in Figure 5. Quad is the infrared image of multiple targets in the urban environment, Airport is the real aerial image, and Noisy UNCamp is a complex field background infrared image and artificially added Gaussian noise.

#### 4.2. Low-Frequency Fusion Rule

_{ir}and I

_{vis}are IR image and visible image, respectively, in a size of M × N; $\overline{x}$ is average gray level of the image. The weights in Equation (5) are assigned as below:

#### 4.3. High-Frequency Fusion Rule

_{1}, r

_{2},…,r

_{n}}. The region weight Q is defined as:

_{i}(i = 1, 2…n), and is calculated through the following equation:

_{ri}in IR and visible images is chosen to calculate the fusion coefficient of region r

_{i}in the fusion image, and the average of high-frequency coefficients in the former region is taken as the high-frequency coefficient of region r

_{i}. To ensure that the edge details of every subregion in the fusion image are more remarkable and clearer, the binary edge matrix ${\mathrm{S}}^{l,\theta}$ is introduced as weight into the calculation of high-frequency coefficient. To ensure the edge continuity and integrity of every subregion, ${\mathrm{S}}^{l,\theta}$ shall be obtained through the dilation operation and 2D average filtering of the ROI-based image segmentation result, described in Section 2 of this paper. The high-frequency coefficient of the fusion image is rewritten as:

_{s}. To highlight the contrast of image details, a given value C (C > 1) is set to increase the high-frequency coefficient. But this simple method will amplify the remaining noise as well. To tackle this problem, we shall only increase the high-frequency coefficient for those pixel points capable of being connected into a large region, while defining a binary matrix ${M}^{l,\theta}$. When $\left|{H}_{F}^{l,\theta}\right|>\tau $, there will be ${M}^{l,\theta}$ = 1. Then the isolated points whose value is 1 are removed from the matrix ${M}^{l,\theta}$. Finally, the high-frequency coefficient of the fusion image becomes:

## 5. Analysis of Experimental Results

#### 5.1. Evaluation Indicators

^{AB/F}) separately. SD is the measure of dispersion of relative gray values. The bigger the SD, the better the image contrast and the higher the fusion quality. IE is the measure of amount of image information. The bigger the IE, the richer the information contained in the fusion image. AG is the measure of expressive ability of minor image details. The bigger the AG, the stronger their expressive ability. Q

^{AB/F}is the measure of how well the fusion image preserves the information on edge details of the source image. The bigger the Q

^{AB/F}, the higher the fusion quality.

#### 5.2. Experiment of Image Fusion Contrast

#### 5.2.1. Experiment 1: Quad Image Fusion Comparison

^{AB/F}). Judging from the evaluation indicators in Table 1, all the indicators of the fusion image obtained through the proposed method are higher than those of the other 4 methods, except for SD, which is slightly lower than that of IHS-PCNN. This is mainly because the fusion image obtained through IHS-PCNN is partly distorted, leading to contrast change in the distorted region, and an increase in image SD. Therefore, the evaluation result of the proposed method is the best in general.

#### 5.2.2. Experiment 2: Aviation IR-Visible Image Fusion Comparison

#### 5.2.3. Experiment 3: Anti-Noise Performance Experiment

## 6. Conclusions

## Author Contributions

## Conflicts of Interest

## References

- Huang, X.; Netravali, R.; Man, H.; Lawrence, V. Multi-sensor fusion of infrared and electro-optic signals for high resolution night images. Sensors
**2012**, 12, 10326–10338. [Google Scholar] [CrossRef] [PubMed] - Ma, Y.; Wu, X.; Yu, G.; Xu, Y.; Wang, Y. Pedestrian detection and tracking from low-resolution unmanned aerial vehicle thermal imagery. Sensors
**2016**, 16, 446. [Google Scholar] [CrossRef] [PubMed] - Jin, B.; Kim, G.; Cho, N.I. Wavelet-domain satellite image fusion based on a generalized fusion equation. J. Appl. Remote Sens.
**2014**, 8, 080599. [Google Scholar] [CrossRef] - Smeelen, M.A.; Schwering, P.B.W.; Toet, A.; Loog, M. Semi-hidden target recognition in gated viewer images fused with thermal IR images. Inf. Fusion
**2014**, 18, 131–147. [Google Scholar] [CrossRef] - Zitova, B.; Flusser, J. Image registration methods: A survey. Image Vis. Comput.
**2003**, 21, 977–1000. [Google Scholar] [CrossRef] - Wu, F.; Wang, B.; Yi, X.; Li, M.; Hao, H.; Zhou, H. Visible and infrared image registration based on visual salient features. J. Electron. Imaging
**2015**, 24, 053017. [Google Scholar] [CrossRef] - Guo, J.; Yang, F.; Tan, H.; Wang, J.; Liu, Z. Image matching using structural similarity and geometric constraint approaches on remote sensing images. J. Appl. Remote Sens.
**2016**, 10, 045007. [Google Scholar] [CrossRef] - Chen, Y. Multimodal Image Fusion and Its Applications. Ph.D. Thesis, The University of Michigan, Ann Arbor, MI, USA, 2016. [Google Scholar]
- Zhen, J. Image Registration and Image Completion: Similarity and Estimation Error Optimization. Ph.D. Thesis, University of Cincinnati, Cincinnati, OH, USA, 2014. [Google Scholar]
- Xu, H.; Wang, Y.; Wu, Y.; Qian, Y. Infrared and multi-type images fusion algorithm based on contrast pyramid transform. Infrared Phys. Technol.
**2016**, 17, 133–146. [Google Scholar] [CrossRef] - Mukane, S.M.; Ghodake, Y.S.; Khandagle, P.S. Image enhancement using fusion by wavelet transform and laplacian pyramid. Int. J. Comput. Sci. Issues
**2013**, 10, 122. [Google Scholar] - Zehua, Z.; Min, T. Infrared image and visible image fusion based on wavelet transform. Adv. Mater. Res.
**2014**, 756–759, 2850–2856. [Google Scholar] - Kingsbury, N.Q. The Dual-tree Complex Wavelet Transform: A New Technique for Shift Invariance and Directional Filters. In Proceedings of the 8th IEEE Digital Signal Processing Workshop, Bryce Canyon, UT, USA, 9–12 August 1998; pp. 120–131. [Google Scholar]
- Selesnick, W.; Baraniuk, R.G.; Kingsbury, N.Q. The dual-tree complex wavelet transform. IEEE Signal Process. Mag.
**2005**, 22, 123–151. [Google Scholar] [CrossRef] - Do, M.N.; Vetterli, M. The contourlet transform: An efficient directional multiresolution image representation. IEEE Trans. Image Process.
**2005**, 14, 2091–2106. [Google Scholar] [CrossRef] [PubMed] - Yang, L.; Guo, B.L.; Ni, W. Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform. Neurocomputing
**2008**, 72, 203–211. [Google Scholar] [CrossRef] - Starck, J.L.; Candes, E.J.; Donoho, D.L. The curvelet transform for image denoising. IEEE Trans. Image Process.
**2002**, 11, 670–684. [Google Scholar] [CrossRef] [PubMed] - Bhadauria, H.S.; Dewal, M.L. Medical image denoising using adaptive fusion of curvelet transform and total variation. Comput. Electr. Eng.
**2013**, 39, 1451–1460. [Google Scholar] [CrossRef] - Cunha, A.L.D.; Zhou, J.P.; Do, M.N. The nonsubsampled contourlet transform: Theory, design, and applications. IEEE Trans. Image Process.
**2006**, 15, 3089–3101. [Google Scholar] [CrossRef] [PubMed] - Xiang, T.; Yan, L.; Gao, R. A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain. Infrared Phys. Technol.
**2015**, 69, 53–61. [Google Scholar] [CrossRef] - Yang, Y.; Tong, S.; Huang, S.Y.; Lin, P. Multifocus Image Fusion Based on NSCT and Focused Area Detection. IEEE Sens. J.
**2015**, 15, 2824–2838. [Google Scholar] [CrossRef] - Li, H.; Liu, L.; Huang, W.; Yue, C. An improved fusion algorithm for infrared and visible images based on multi-scale transform. Infrared Phys. Technol.
**2016**, 74, 28–37. [Google Scholar] [CrossRef] - Shuo, L.; Yan, P.; Muhanmmad, T. Research on fusion technology based on low-light visible image and infrared image. Opt. Eng.
**2016**, 55, 123104. [Google Scholar] [CrossRef] - Li, S.; Kang, S.; Hu, J. Image fusion with guided filtering. IEEE Trans. Image Process.
**2013**, 22, 2864–2874. [Google Scholar] [CrossRef] [PubMed] - Montoya, M.D.G.; Gill, C.; Garcia, I. The load unbalancing problem for region growing image segmentation algorithm. J. Parallel Distrib. Comput.
**2003**, 63, 387–395. [Google Scholar] [CrossRef] - Dawoud, A.; Netchaev, A. Preserving objects in markov random fields region growing image segmentation. Pattern Anal. Appl.
**2012**, 15, 155–161. [Google Scholar] [CrossRef] - Chitade, A.Z.; Katiyar, D.S.K. Colour based image segmentation using K-means clustering. Int. J. Eng. Sci. Technol.
**2010**, 2, 5319–5325. [Google Scholar] - Sulaiman, S.N.; Isa, N.A.M. Adaptive fuzzy-K-means clustering algorithm for image segmentation. IEEE Trans. Consum. Electron.
**2010**, 56, 2661–2668. [Google Scholar] [CrossRef] - Chen, M.-M.; Zhang, G.; Mitra, N.J.; Huang, X.; Hu, S.-M. Global Contrast based salient region detection. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, 20–25 January 2011. [Google Scholar]
- Carten, R.; Vladimir, K.; Andrew, B. “Grabcut”—Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph.
**2004**, 23, 309–314. [Google Scholar] [CrossRef] - Loza, A.; Bull, D.; Canagarajah, N.; Achim, A. Non-Gaussian model based fusion of noisy images in the wavelet domain. Comput. Vis. Image Underst.
**2010**, 114, 54–65. [Google Scholar] [CrossRef]

**Figure 2.**Principle of star method. (

**a**) Visible star point; (

**b**) Infrared star point; (

**c**) Visible star points; (

**d**) Infrared star points.

**Figure 5.**Region segmentation result based on region of interest (ROI): (

**a**) infrared (IR) image; (

**b**) Significance map of the IR image; (

**c**) Region segmentation result based on ROI.

Image | Quad | Airport | Noisy UNCamp |
---|---|---|---|

Resolution | 640 × 496 | 640 × 436 | 360 × 270 |

Time/s | 0.036 | 0.029 | 0.019 |

Method | SD | IE | Q^{AB/F} | AG | Time/s |
---|---|---|---|---|---|

DWT [12] | 21.4123 | 5.8733 | 0.2108 | 2.6492 | 1.943 |

PCA-MST [22] | 31.2098 | 6.3987 | 0.2974 | 3.7382 | 1.019 |

GFF [24] | 32.0211 | 6.4520 | 0.2752 | 3.2129 | 1.373 |

IHS-PCNN [23] | 39.5745 | 6.6929 | 0.3269 | 3.6875 | 4.693 |

Proposed | 35.0284 | 6.8894 | 0.5645 | 5.3487 | 0.977 |

Method | SD | IE | Q^{AB/F} | AG | Time/s |
---|---|---|---|---|---|

DWT [12] | 40.3748 | 5.1037 | 0.2964 | 3.0552 | 1.526 |

PCA-MST [22] | 48.6327 | 6.3254 | 0.3978 | 3.7553 | 0.847 |

GFF [24] | 49.2637 | 7.0035 | 0.6755 | 3.3247 | 0.916 |

IHS-PCNN [23] | 45.9653 | 6.1276 | 0.4458 | 3.7514 | 3.932 |

Proposed | 53.3248 | 6.9054 | 0.6326 | 6.3072 | 0.852 |

Method | SD | IE | Q^{AB/F} | AG | Time/s |
---|---|---|---|---|---|

DWT [12] | 33.6973 | 7.0965 | 0.3575 | 16.4053 | 0.960 |

PCA-MST [22] | 37.3488 | 7.2314 | 0.3204 | 16.6699 | 0.499 |

GFF [24] | 37.9777 | 7.2664 | 0.3870 | 16.6331 | 0.693 |

IHS-PCNN [23] | 37.2536 | 7.1973 | 0.4544 | 11.2374 | 3.137 |

Proposed | 39.3742 | 7.2013 | 0.6447 | 15.2238 | 0.547 |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zuo, Y.; Liu, J.; Bai, G.; Wang, X.; Sun, M.
Airborne Infrared and Visible Image Fusion Combined with Region Segmentation. *Sensors* **2017**, *17*, 1127.
https://doi.org/10.3390/s17051127

**AMA Style**

Zuo Y, Liu J, Bai G, Wang X, Sun M.
Airborne Infrared and Visible Image Fusion Combined with Region Segmentation. *Sensors*. 2017; 17(5):1127.
https://doi.org/10.3390/s17051127

**Chicago/Turabian Style**

Zuo, Yujia, Jinghong Liu, Guanbing Bai, Xuan Wang, and Mingchao Sun.
2017. "Airborne Infrared and Visible Image Fusion Combined with Region Segmentation" *Sensors* 17, no. 5: 1127.
https://doi.org/10.3390/s17051127