Skip to Content
Remote SensingRemote Sensing
  • Article
  • Open Access

5 July 2022

22 Pages

MFST: Multi-Modal Feature Self-Adaptive Transformer for Infrared and Visible Image Fusion

,
,
,
,
and
1
School of Computer Science and Technology, Xidian University, Xi’an 710071, China
2
Guangzhou Institute of Technology, Xidian University, Xi’an 510555, China
3
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
4
School of Information of Science and Technology, Tibet University, Lhasa 850000, China

Abstract

Infrared and visible image fusion is to combine the information of thermal radiation and detailed texture from the two images into one informative fused image. Recently, deep learning methods have been widely applied in this task; however, those methods usually fuse multiple extracted features with the same fusion strategy, which ignores the differences in the representation of these features, resulting in the loss of information in the fusion process. To address this issue, we propose a novel method named multi-modal feature self-adaptive transformer (MFST) to preserve more significant information about the source images. Firstly, multi-modal features are extracted from the input images by a convolutional neural network (CNN). Then, these features are fused by the focal transformer blocks that can be trained through an adaptive fusion strategy according to the characteristics of different features. Finally, the fused features and saliency information of the infrared image are considered to obtain the fused image. The proposed fusion framework is evaluated on TNO, LLVIP, and FLIR datasets with various scenes. Experimental results demonstrate that our method outperforms several state-of-the-art methods in terms of subjective and objective evaluation.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.