Mamba-Based Infrared and Visible Images Fusion Method

Highlights

What are the main findings?

• A Mamba-based fusion framework is proposed, integrating Multi-Path Mamba (MPMamba) and Dual-path Mamba Attention Fusion (DMAF) modules, significantly enhancing information retention and structural consistency in fused images.
• Experiments on the MSRS dataset demonstrate that the proposed method outperforms state-of-the-art approaches in key metrics such as Information Entropy, Spatial Frequency, and Mutual Information, and exhibit strong generalization in object detection tasks.

What are the implications of the main findings?

• This study provides a novel paradigm for multimodal image fusion that combines global modeling capability with linear computational complexity, facilitating applications in autonomous driving and nighttime surveillance.
• Through dual-path feature decoupling and dynamic attention calibration, the method achieves a better balance between preserving thermal targets from infrared images and texture details from visible images, offering higher-quality inputs for downstream vision tasks.

Abstract

Visible-infrared image fusion is crucial for applications like autonomous driving and nighttime surveillance, yet it remains challenging due to the inherent limitations of existing deep learning models. Convolutional Neural Networks (CNNs) are constrained by their local receptive fields, while Transformers suffer from quadratic computational complexity. To address these issues, this paper investigates the application of the Mamba model—a novel State Space Model (SSM) with linear-complexity global modeling and selective scanning capabilities—to the task of visible-infrared image fusion. Building upon Mamba, we propose a novel fusion framework featuring two key designs: (1) A Multi-Path Mamba (MPMamba) module that orchestrates parallel Mamba blocks with convolutional streams to extract multi-scale, modality-specific features; and (2) a Dual-path Mamba Attention Fusion (DMAF) module that explicitly decouples and processes shared and complementary features via dual Mamba paths, followed by dynamic calibration with a Convolutional Block Attention Module (CBAM). Extensive experiments on the MSRS benchmark demonstrate that our framework achieves state-of-the-art performance, outperforming strong baselines such as U2Fusion and SwinFusion across key metrics including Information Entropy (EN), Spatial Frequency (SF), Mutual Information (MI), and edge-based fusion quality (Qabf). Visual results confirm its ability to produce fused images that saliently preserve thermal targets while retaining rich texture details.

1. Introduction

With growing military and civilian demands, the importance of visible and infrared image fusion in the field of computer vision has become increasingly prominent. Visible images provide rich texture details and contextual information, while infrared images possess all-weather operational capability, effectively capturing the thermal radiation characteristics of targets [1]. Fusing these two modalities can leverage their respective advantages, delivering more comprehensive visual information for applications such as intelligent surveillance, object detection [2], and military reconnaissance. However, significant differences in imaging mechanisms, resolution, and feature distribution between visible and infrared images pose considerable challenges for achieving efficient cross-modal feature fusion [3].

Traditional image fusion methods primarily rely on handcrafted feature extraction techniques, such as multi-scale transforms (e.g., wavelet and contourlet decomposition) [4,5,6] and sparse representation (e.g., fixed dictionary optimization) [7,8]. The representational capacity of these methods heavily depends on prior knowledge and parameter tuning, often struggling to capture global dependencies in complex scenarios, with fusion performance hinging on manual parameter adjustment. In recent years, deep learning-based fusion methods have made remarkable progress, particularly through the application of Convolutional Neural Networks (CNNs) [9,10,11] and Transformer models [12], significantly enhancing fusion performance. Nevertheless, CNNs are limited by their local receptive fields [13], making it difficult to model long-range dependencies. While Transformers can capture global information, their quadratic computational complexity restricts application efficiency on long-sequence data [14]. Consequently, there is an urgent need for a novel fusion approach that balances global modeling capacity with computational efficiency.

The State Space Model (SSM), a classical sequence modeling method, has demonstrated considerable potential in fields such as natural language processing and time series analysis. Among these, the Mamba model [15], by introducing a selective scanning mechanism and dynamic path planning, achieves efficient modeling of long sequences while maintaining linear computational complexity. This characteristic offers a new research direction for cross-modal image fusion: utilizing the global modeling capability of the Mamba model to capture long-range dependencies between visible and infrared images, and achieving real-time fusion through its efficient computational mechanism, thereby addressing the limitations of existing methods in global dependency modeling and computational efficiency.

The main contributions of this paper are as follows:

• A Multi-path Mamba module is designed to perform multi-scale characterization of texture details in visible images and thermal radiation features in infrared images, providing rich feature representations for cross-modal fusion.
• A Dual-path Mamba Attention Fusion (DMAF) module is proposed, which leverages the global modeling capability and dynamic feature selection mechanism of the Mamba model to separately process common and distinct features from visible and infrared images, and achieves adaptive fusion through the Convolutional Block Attention Module (CBAM).
• Experiments conducted on the MSRS dataset validate the superiority of the proposed method in both objective metrics and subjective quality.

This study provides a novel technical pathway for multimodal image fusion.

2. Materials

2.1. Deep Learning-Based Fusion

This approach establishes end-to-end learning frameworks to achieve collaborative representation of cross-modal information in the feature space. According to the evolution of network architectures, current research can be categorized into three technical pathways: (1) methods based on CNNs and autoencoders [16,17]; (2) methods based on generative models such as GANs [18,19] and diffusion models [20]; and (3) methods based on Transformers [21,22] and hybrid architectures.

Within the CNN and autoencoder paradigm, mainstream solutions employ multi-scale feature pyramids for hierarchical feature extraction, followed by image reconstruction via a decoder. However, a fundamental conflict exists between the strong local feature representation capability of this architecture and the requirement for global modeling: the local perceptive nature of convolution hinders the establishment of long-range cross-modal dependencies, making it difficult for the fused image to preserve both the thermal characteristics of infrared targets and the detail integrity of visible textures in dynamic scenes. Furthermore, encoder design faces an inherent trade-off: parameter sharing mechanisms can weaken the expression of modality-specific features, while heterogeneous encoders encounter difficulties in cross-modal feature alignment.

The application of generative models in image fusion primarily manifests through two technical paths: The GAN framework achieves modality distribution alignment by designing multi-discriminator structures [23], where discriminators learn the distribution characteristics of different modalities separately. However, the training process often suffers from gradient vanishing and challenges in reaching Nash equilibrium convergence. Diffusion models, conversely, utilize noise scheduling strategies to generate fused images progressively and theoretically offer more stable generation performance. Yet the iterative mechanism of forward noising and reverse denoising leads to exponentially increasing computational complexity, limiting their real-time application capabilities. Beyond these mainstream generative paradigms, other innovative architectures have emerged. For instance, FS-Diff [24] integrates semantic guidance to simultaneously address fusion and super-resolution, while AFDFusion [25] explores adaptive frequency decoupling to enhance fusion performance. These works underscore the ongoing exploration of more sophisticated fusion mechanisms beyond pixel-level blending.

In recent years, Transformer-based fusion methods have demonstrated significant advantages. They establish global long-range dependencies across modalities through self-attention modules, enhanced by multi-head mechanisms that increase the diversity of feature interactions. However, the sensitivity of positional encodings to geometric transformations and the computational redundancy of cross-attention create memory bottlenecks when processing high-resolution multimodal data. Notably, CNN-Transformer hybrid architectures attempt to combine local inductive biases with global modeling capabilities, but the cross-scale fusion mechanisms for inter-modal feature pyramids still require in-depth exploration. The core challenge of feature matching and alignment across modalities or views is also a focal point in related fields like cross-view geo-localization. For example, the Dual-branch transformer framework with gradient-aware weighting feature alignment [26] addresses robust feature matching under drastic viewpoint changes, and an efficient cross-view image fusion method based on selected state space and hashing [27] explores efficient cross-view perception. Although targeting different data (cross-view vs. cross-modal), these studies share the fundamental need for modeling long-range dependencies and aligning asymmetric features, providing valuable insights for designing efficient and robust fusion modules.

In summary, image fusion systems for practical applications must balance information fidelity with computational efficiency. Breaking through the performance bottlenecks of existing architectures and establishing superior paradigms for feature interaction remains the core challenge for advancing this field.

2.2. State Space Models

State Space Models (SSMs) originated from classical control theory, with their theoretical framework traceable to the dynamic system modeling methods proposed by Kalman in the 1960s. SSMs were initially used to describe the dynamic behavior of linear time-invariant systems through state variables. However, the application of traditional SSMs in deep learning was long hampered by challenges in parameter initialization and discretization efficiency, until a breakthrough came with the proposal of the Structured State Space Model (S4) [28]. S4 introduced the HiPPO (High-order Polynomial Projection Operator) framework, which utilizes orthogonal polynomial basis functions (such as Legendre polynomials) to project input sequences, thereby addressing the issues of information compression and state updates in long-range dependency modeling. This improvement enabled SSMs to efficiently handle global dependencies in sequential data, laying the technical foundation for subsequent models.

Building upon S4, researchers further proposed various optimization schemes. For instance, the S5 layer [29] achieved parallelized scanning by diagonalizing the state matrix, significantly enhancing computational efficiency. The H3 model [30], meanwhile, combined bidirectional state space mechanisms with positional embedding techniques, enabling SSMs to achieve performance comparable to Transformers in language modeling tasks. The Mamba model improved upon S4 by introducing a selection mechanism, allowing the model to selectively filter relevant information based on the input, alongside proposing a hardware-aware algorithm for efficient training and inference. Compared to Transformer models of similar scale, Mamba demonstrates higher inference speed, throughput, and overall performance.

The innovations of Mamba have promoted the application of SSMs in broader domains. For example, Vision Mamba (Vim) [31] integrated bidirectional SSM modules with the Vision Transformer architecture, achieving efficient visual representation learning through positional embeddings and cross-modal feature compression. VL-Mamba [32] further extended Mamba to multimodal tasks, utilizing a visual encoder and a multimodal connector to handle joint image-text reasoning tasks. While studies like “MambaOut” [33] suggest that long-sequence modeling may be superfluous for some vision classification tasks, we argue that it is crucial for multimodal image fusion. This task extends beyond recognizing local patterns to solving two global challenges: (1) cross-modal semantic alignment: identifying and associating complementary features (e.g., a thermal target in a dark background) that may be spatially distant within and across modalities; and (2) global structure preservation: ensuring the fused image maintains spatial coherence and natural gradients without artifacts. The selective state space model (SSM) within the Mamba architecture provides an efficient mechanism to capture such long-range dependencies, enabling our network to perform global context reasoning for effective fusion, which distinguishes our approach from local-operation-based or attention-inefficient fusion methods.

3. Methods

3.1. Background

State Space Models (SSMs) are traditionally defined as linear time-invariant (LTI) systems that map input sequences $\mathrm{x}(t)\in {ℝ}^N$ to output responses $\mathrm{y}(t)\in {ℝ}^N$ via a latent state $\mathrm{h}(t)\in {ℝ}^N$. The system can be mathematically described by a linear ordinary differential equation (ODE):

$${\mathrm{h}}^{\prime }(t)=A\mathrm{h}(t)+B\mathrm{x}(t)$$

(1)

$$\mathrm{y}(t)=C\mathrm{h}(t)$$

(2)

where $\mathrm{h}(t)\in {ℝ}^N$ represents the state vector of dimension $N$, $A\in {ℝ}^{N\times N}$ is the state transition matrix, $B\in {ℝ}^{N\times N}$ is the input projection matrix, and $C\in {ℝ}^{N\times N}$ is the output projection matrix.

As continuous-time models, SSMs face computational challenges when integrated into deep learning frameworks. Discretization is therefore required. Using the Zero-Order Hold (ZOH) method, Equation (1) can be discretized as follows:

$${\mathrm{h}}_t=\overline{A}{\mathrm{h}}_{t-1}+\overline{B}{\mathrm{x}}_t$$

(3)

$${\mathrm{y}}_t=C{\mathrm{h}}_t$$

(4)

The discretization parameters satisfy $\overline{A}=\exp (\Delta A)$ and $\overline{B}={(\Delta A)}^{-1}(\exp (\Delta A)-I)\cdot \Delta B$, where $\Delta$ is the discretization step size. However, the parameters $\overline{A}$, $\overline{B}$ and $\overline{C}$ in the LTI system described by Equations (3) and (4) are input-invariant, which limits the model’s expressive power.

Mamba overcomes this limitation by introducing a selective scanning mechanism. Its parameters $B$, $C$, and the most critical $\Delta$ are dynamically generated based on the input, enabling context-aware sequence modeling. Furthermore, by incorporating hardware-aware optimization algorithms, Mamba further enhances its potential. This innovation lays the foundation for the application of SSMs in complex tasks.

To explore the feasibility of the Mamba model in the field of visible and infrared image fusion, the Mamba modules referenced in this paper adopt the architecture proposed in Vision Mamba (Vim), the process of patch embedding and de-embedding has been added, and the architecture is shown in Figure 1. This structure employs parallel processing of bidirectional SSMs (forward and backward paths) to overcome the limitations of unidirectional modeling. For each direction, after extracting local features using 1D convolution, the state transition parameters ($\Delta$, $A$, $B$) are dynamically adjusted. Finally, contextual information from both directions is fused through a gating mechanism, enabling the model to capture both spatial causal and non-causal dependencies in the image. The algorithmic workflow of Vision Mamba Encoder is summarized in Algorithm 1.

Algorithm 1 $Vim Block Process$

$\mathbf{I}\mathbf{n}\mathbf{p}\mathbf{u}\mathbf{t}:token sequence {\mathbf{T}}_{l-1}:(B,M,D)$

$\mathbf{O}\mathbf{u}\mathbf{t}\mathbf{p}\mathbf{u}\mathbf{t}:token sequence {\mathbf{T}}_{l-1}:(B,M,D)$

1: $/* normalize the input sequence \mathbf{T}{\prime }_{l-1}\mathrm{*}/$

2: $\mathbf{T}{\mathbf{\prime }}_{l-1}\leftarrow \mathit{N}\mathit{o}\mathit{r}\mathit{m}\left({\mathbf{T}}_{l-1}\right)$

3: $\mathbf{x}:(B,M,E)\leftarrow {\mathit{L}\mathit{i}\mathit{n}\mathit{e}\mathit{a}\mathit{r}}^{\mathbf{x}}\left(\mathbf{T}{\prime }_{l-1}\right)$

4: $\mathbf{z}:(B,M,E)\leftarrow {\mathit{L}\mathit{i}\mathit{n}\mathit{e}\mathit{a}\mathit{r}}^{\mathbf{z}}\left(\mathbf{T}{\prime }_{l-1}\right)$

5: $/* process with different direction */$

6: $\mathbf{f}\mathbf{o}\mathbf{r} o\in \{forward,backward\} \mathbf{d}\mathbf{o}$

7: $\mathbf{x}{\prime }_o\leftarrow \mathit{S}\mathit{i}\mathit{L}\mathit{U}\left({\mathit{C}\mathit{o}\mathit{n}\mathit{v}\mathbf{1}\mathit{d}}_o\right(\mathbf{x}\left)\right)$

8: ${\mathbf{B}}_o\leftarrow {\mathit{L}\mathit{i}\mathit{n}\mathit{e}\mathit{a}\mathit{r}}_o^{\mathbf{B}}\left(\mathbf{x}{\prime }_o\right)$

9: ${\mathbf{C}}_o\leftarrow {\mathit{L}\mathit{i}\mathit{n}\mathit{e}\mathit{a}\mathit{r}}_o^{\mathbf{C}}\left(\mathbf{x}{\prime }_o\right)$

10: $/* softplus ensures positive {\mathsf{\Delta }}_o*/$

11: $({\mathsf{\Delta }}_o\leftarrow \log (1+\exp ({\mathit{L}\mathit{i}\mathit{n}\mathit{e}\mathit{a}\mathit{r}}_o^{\mathsf{\Delta }}(\mathbf{x}{\prime }_o)+{\mathbf{P}\mathbf{a}\mathbf{r}\mathbf{a}\mathbf{m}\mathbf{e}\mathbf{t}\mathbf{e}\mathbf{r}}_o^{\mathsf{\Delta }})))$

12: $/* shape of {\mathbf{P}\mathbf{a}\mathbf{r}\mathbf{a}\mathbf{m}\mathbf{e}\mathbf{t}\mathbf{e}\mathbf{r}}_o^{\mathbf{A}} is (E,N) */$

13: ${\overline{\mathbf{A}}}_o:(B,M,E,N)\leftarrow {\mathsf{\Delta }}_o\otimes {\mathbf{P}\mathbf{a}\mathbf{r}\mathbf{a}\mathbf{m}\mathbf{e}\mathbf{t}\mathbf{e}\mathbf{r}}_o^{\mathbf{A}}$

14: ${\overline{\mathbf{B}}}_o:(B,M,E,N)\leftarrow {\mathsf{\Delta }}_o\otimes {\mathbf{B}}_o$

15: $/* initialize {\mathbf{h}}_o and {\mathbf{y}}_o with 0 */$

16: ${\mathbf{h}}_o:(B,E,N)\leftarrow zeros(B,E,N)$

17: ${\mathbf{y}}_o:(B,M,E)\leftarrow zeros(B,M,E)$

18: $/* SSM recurrent */$

19: $\mathbf{f}\mathbf{o}\mathbf{r} i in \{0,\dots ,M-1\} \mathbf{d}\mathbf{o}$

20: ${\mathbf{h}}_o\leftarrow {\overline{\mathbf{A}}}_o[:,i,:,:]\odot {\mathbf{h}}_o+{\overline{\mathbf{B}}}_o[:,i,:,:]\odot \mathbf{x}{\prime }_o[:,i,:,None]$

21: ${\mathbf{y}}_o[:,i,:]\leftarrow {\mathbf{h}}_o\otimes {\mathbf{C}}_o[:,i,:]$

22: $\mathbf{e}\mathbf{n}\mathbf{d}\mathbf{f}\mathbf{o}\mathbf{r}$

23: $\mathbf{e}\mathbf{n}\mathbf{d}\mathbf{f}\mathbf{o}\mathbf{r}$

24: $/* get gated \mathbf{y} */$

25: $\mathbf{y}{\prime }_{forward}:(B,M,E)\leftarrow {\mathbf{y}}_{forward}\odot \mathit{S}\mathit{i}\mathit{L}\mathit{U}(\mathbf{z})$

26: $\mathbf{y}{\prime }_{backward}:(B,M,E)\leftarrow {\mathbf{y}}_{backward}\odot \mathit{S}\mathit{i}\mathit{L}\mathit{U}(\mathbf{z})$

27: $/\mathbf{*} \mathbf{r}\mathbf{e}\mathbf{s}\mathbf{i}\mathbf{d}\mathbf{u}\mathbf{a}\mathbf{l}\mathbf{c}\mathbf{o}\mathbf{n}\mathbf{n}\mathbf{e}\mathbf{c}\mathbf{t}\mathbf{i}\mathbf{o}\mathbf{n} \mathbf{*}/$

28: ${\mathbf{T}}_l\leftarrow {\mathit{L}\mathit{i}\mathit{n}\mathit{e}\mathit{a}\mathit{r}}^{\mathbf{T}}(\mathbf{y}{\prime }_{forward}+\mathbf{y}{\prime }_{backward})+{\mathbf{T}}_{l-1}$

29: $\mathbf{R}\mathbf{e}\mathbf{t}\mathbf{u}\mathbf{r}\mathbf{n}:{\mathbf{T}}_l$

In the Vision Transformer architecture, the Patch Embedding module plays a critical role in transforming multi-dimensional visual features into a representation suitable for sequential processing. In this study, we adopt this classic design concept and introduce adaptive improvements tailored for the image fusion task. Specifically, our input is not raw image data but rather intermediate feature representations extracted by a deep neural network. Given an input feature tensor of size H × W × C, where H and W denote the height and width of the feature map respectively, and C represents the number of feature channels, we first perform a spatial partitioning operation. This operation evenly divides the input feature map along the spatial dimension into N non-overlapping regular patches, each with a spatial size of P × P × C, where P is the predefined patch size parameter, uniformly set to 16 in our experiments. The total number of patches N can be calculated using the formula $N={\displaystyle \frac{H}{P}}\times {\displaystyle \frac{W}{P}}$, ensuring that the spatial information of the original features is fully preserved without any loss.

Upon completion of the spatial partitioning, each multi-dimensional feature patch is flattened into a one-dimensional feature vector, the length of which is determined by the total number of elements within the patch, namely P² × C. This flattening operation essentially transforms the joint spatial-channel features within each local receptive field into a standard vector representation. Subsequently, these flattened vectors undergo a dimensionality transformation via a learnable linear projection layer, which is parameterized by a weight matrix $W\in {ℝ}^{D\times (P^2\times C)}$ and a bias term $\mathrm{b}\in {ℝ}^D$, where D denotes the target latent space dimension, set to 64 in this work. This mapping process can be formally expressed as:

$${\mathrm{z}}_i=W•{\mathrm{x}}_i+\mathrm{b}$$

(5)

where ${\mathrm{x}}_i$ is the i-th flattened patch feature vector, and ${\mathrm{z}}_i$ is its corresponding D-dimensional latent space representation. The role of linear projection extends beyond mere dimensionality reduction or expansion; more importantly, it enhances the nonlinear transformation capability of the feature space through learnable parameters.

To compensate for potential loss of local spatial relationships caused by the patch partitioning operation, we introduce a learnable positional encoding mechanism. Specifically, we assign an independent D-dimensional positional vector to each spatial location, and these vectors collectively form a learnable parameter matrix $E_{pos}\in {ℝ}^{N\times D}$. The positional encoding is incorporated via element-wise addition:

$${\mathrm{z}}_i^{\prime }={\mathrm{z}}_i+{\mathrm{e}}_i$$

(6)

where ${\mathrm{e}}_i\in {ℝ}^D$ denotes the encoding vector corresponding to the i-th position. This design enables the model to explicitly perceive the absolute positional information of each feature patch within the original two-dimensional space, providing an important spatial prior for subsequent global relation modeling. Unlike the original Vision Transformer, given that the nature of the image fusion task places greater emphasis on pixel-level feature reconstruction rather than global image classification, we discard the additional learnable classification token, allowing the model to focus on the modeling and transformation of the features themselves.

The de-embedding process, serving as the inverse transformation of Patch Embedding, assumes the critical function of reconstructing the serialized representation back into a two-dimensional feature map. This process takes the N × D-dimensional sequence output by the Vision Mamba Encoder as input. First, each D-dimensional vector is mapped back to the original feature dimension via a linear back-projection layer. This back-projection layer is parameterized by a weight matrix $W^{\prime }\in {ℝ}^{(P^2\times C)\times D}$ and a bias term $\mathrm{b}\prime \in {ℝ}^{P^2\times C}$, and its mapping process can be expressed as:

$${\mathrm{x}}_i^{\prime }=W^{\prime }•{\mathrm{z}}_i^{″}+\mathrm{b}\prime$$

(7)

where ${\mathrm{z}}_i^{″}$ is the i-th D-dimensional vector output by the encoder, and ${\mathrm{x}}_i^{\prime }$ is the reconstructed flattened patch feature. Although this back-projection layer forms a symmetric structure with the projection layer from the embedding stage, its parameters are typically learned independently to achieve better reconstruction performance.

After the dimensionality restoration, each vector of length P² × C is reshaped into a three-dimensional feature block of size P × P × C. This reshaping operation strictly adheres to the partitioning strategy used in the embedding stage, ensuring each vector accurately recovers its corresponding local feature structure. Subsequently, all N reconstructed feature blocks are reassembled according to their original spatial arrangement order: adjacent blocks are first concatenated along the width direction to form feature rows; then these rows are stacked along the height direction, ultimately reconstructing the complete H × W × C two-dimensional feature map. This spatial reassembly process can be mathematically formulated as:

$$Y=reshape(permute(concat(x_1^{\prime },x_2^{\prime },\dots ,x_N^{\prime })))$$

(8)

where $Y\in {ℝ}^{H\times W\times C}$ is the reconstructed feature map.

The entire inverse embedding process not only accomplishes the transformation from a sequence to a two-dimensional structure but, more importantly, preserves the global contextual information learned by the Vision Mamba Encoder. The reconstructed feature map integrates local detail features with global semantic information. Channel count C remains consistent with the input features, and the spatial dimensions H × W are fully restored to their original size, thereby ensuring seamless integration with subsequent network layers.

3.2. Overall Architecture

The model consists of three stages: feature extraction, feature fusion, and image reconstruction. The complete workflow is illustrated in Figure 2. Given a pair of input visible and infrared images, denoted as $I_{vis}$ and $I_{\inf }$, the goal is to generate the fused image $I_{fused}$.

In the initial feature extraction stage, the input images are projected into a shared feature space via convolutional layers C1 and C2. However, due to the limited receptive field of convolutional layers, capturing global features remains challenging. Therefore, the MPMamba module is employed to extract high-level, multi-scale features, generating modality-specific features.

Table 1. Kernel size, stride, output channels and activation function of C1, C2, C3 and C4.

Layer	Size	Stride	Channel (Input)	Channel (Output)	Activation
C1	3	1	1	16	Leaky RELU
C2	3	1	16	32	Leaky RELU
C3	3	1	32	16	Leaky RELU
C4	3	1	16	1	Leaky RELU

details the parameter configurations of convolutional layers C1 and C2, including kernel size, output channels, and activation functions. The specific parameters of the MPMamba module will be elaborated in subsequent sections.

Subsequently, in the feature fusion stage, we introduce a dual-path Mamba attention fusion module. Leveraging the complementary characteristics of visible and infrared feature maps, this module directs the shared features of both modalities into the common-path Mamba module, while routing their unique features to the distinct-path Mamba module. The outputs of these two Mamba paths are concatenated and then dynamically calibrated by the Convolutional Block Attention Module (CBAM), ultimately producing the fused modality features suitable for reconstruction.

Finally, in the image reconstruction stage (which is the inverse process of feature extraction), the fused features are processed by the Vim module, followed by the application of convolutional layers C3 and C4 to generate the fusion result. The specific configurations of C3 and C4 are listed in Table 1. As is well known, information loss is a critical issue in image fusion. Therefore, all convolutional layers in this network adopt the same padding strategy with a stride of 1. Consequently, no downsampling operation is introduced in our network, and the size of the fused image remains consistent with that of the source images.

3.3. MPMamba Module

Convolutional layers excel in early visual processing, offering stronger optimization stability and superior processing performance. While Convolutional Neural Networks (CNNs) are constrained by limited receptive fields when extracting deep features, Transformers face the challenge of quadratic complexity. Based on this, the Mamba module emerges as an ideal choice for extracting high-level modality-specific features. The shallow features extracted by C1 and C2 are subsequently fed into the MPMamba module for global context modeling. This module is an improved version of the Multi-Path Vision Transformer (MPViT) [34], where the Transformer encoders are replaced with Mamba modules. Its structure is illustrated in Figure 3.

In the MPMamba module, the input features are first fed in parallel into three convolutional layers with different kernel sizes. The distinct physical receptive fields of these varying kernel sizes enable precise focus on the core features of infrared and visible light images. The 3 × 3 small kernel is employed to extract high-frequency texture details (such as edges and gradients) from visible light images, effectively compensating for the detail loss caused by thermal radiation blurring in infrared images. Local information in visible light images (e.g., building outlines, vegetation textures) can be accurately captured by the small kernel. The 5 × 5 medium kernel strikes a balance between local and contextual information, enhancing the representation of medium-scale objects (e.g., human posture, vehicle contours). It is particularly suitable for feature fusion in transition areas between thermal targets in infrared images and the visible background. The 7 × 7 large kernel can capture the global thermal radiation distribution in infrared images (e.g., vehicle engine heat sources) while establishing spatial contextual relationships in visible light scenes (e.g., road continuity, building layout).

This multi-scale convolutional combination effectively addresses the limitations of using a single kernel size: relying solely on 3 × 3 kernels may overlook large thermal targets, while using only 7 × 7 kernels would weaken texture details. To reduce the number of parameters, all convolutions within the MPMamba module employ Depthwise Separable Convolutions (DWConv). Specifically, two consecutive 3 × 3 convolutions can replace a 5 × 5 convolution, and three consecutive 3 × 3 convolutions can replace a 7 × 7 convolution, thereby further streamlining the parameters.

While the Mamba module can capture long-range dependencies (i.e., global context), it is likely to overlook the structural information and local relationships within each patch. In contrast, CNNs can leverage the local connectivity inherent in translation invariance, enabling them to efficiently extract local features from images, such as edges and textures. Therefore, MPMamba combines CNNs and Mamba in a complementary manner. To represent local features, a ConvBlock is adopted, which includes a 1 × 1 convolution, a 3 × 3 depthwise convolution, another 1 × 1 convolution, and a residual connection. Finally, the output of the convolutional block is concatenated with the outputs of the scale-specific Mamba modules, followed by a 1 × 1 convolution to fuse local and global features.

Table 2. The parameter configurations of each layer in the MPMamba module.

Layer	Size	Stride	Groups	Channel (Input)	Channel (Output)	Activation
DWConv (3 × 3 Conv)	3	1	32	32	32	Hardswish
PWConv (3 × 3 Conv)	1	1	1	32	32
DWConv (5 × 5 Conv)	3	1	32	32	32	Hardswish
DWConv (5 × 5 Conv)	3	1	32	32	32
PWConv (5 × 5 Conv)	1	1	1	32	32
DWConv (7 × 7 Conv)	3	1	32	32	32	Hardswish
DWConv (7 × 7 Conv)	3	1	32	32	32
DWConv (7 × 7 Conv)	3	1	32	32	32
PWConv (7 × 7 Conv)	1	1	1	32	32
1 × 1 Conv (ConvBlock)	1	1		32	16	Hardswish
DWConv (ConvBlock)	3	1	16	16	16
1 × 1 Conv (ConvBlock)	1	1		16	32
1 × 1 Conv	1	1		128	32	Hardswish

details the parameter configurations of each layer in the MPMamba module.

The cascaded design of shallow convolutions and deep Mamba enables the synergistic optimization of local and global features. The fine-grained features extracted by the convolutional layers provide high signal-to-noise ratio input for the Mamba module, while Mamba’s global modeling capability compensates for the limited receptive fields of the convolutional layers. The multi-scale architecture of MPMamba adapts to targets of different sizes: the small-scale Mamba path focuses on texture details, while the large-scale Mamba path is responsible for modeling thermal radiation distribution.

3.4. Dual-Path Mamba Attention Fusion Module

Visible feature $F_{vis}$ and infrared features $F_{\inf }$ can be represented by the following shared and complementary components:

$$F_{vis}={\displaystyle \frac{F_{vis}+F_{vis}}{2}}+{\displaystyle \frac{F_{\inf }-F_{\inf }}{2}}={\displaystyle \frac{F_{vis}+F_{\inf }}{2}}+{\displaystyle \frac{F_{vis}-F_{\inf }}{2}}$$

(9)

$$F_{\inf }={\displaystyle \frac{F_{\inf }+F_{\inf }}{2}}+{\displaystyle \frac{F_{vis}-F_{vis}}{2}}={\displaystyle \frac{F_{vis}+F_{\inf }}{2}}-{\displaystyle \frac{F_{vis}-F_{\inf }}{2}}$$

(10)

Here, the shared components represent the features common across different modalities, while the complementary components reflect the unique features that complement each other between modalities, analogous to the principle of a differential amplifier circuit. Based on the shared and complementary features of visible and infrared feature maps, this paper proposes a Dual-path Mamba Attention Fusion module, whose architecture is shown in Figure 4, ”+” and “−” represent element-wise addition and element-wise subtraction, respectively.

The Dual-path Mamba Attention Fusion Module adopts a design philosophy of dual-path parallel processing and attention-guided fusion, aiming to effectively integrate multimodal feature information from visible (VI) and infrared (IR) images. This module independently processes common features and differential features and utilizes channel-spatial attention mechanisms for adaptive feature fusion, thereby enhancing the model’s representational capacity for multi-source information.

The input visible features $F_{vis}$ and infrared features $F_{\inf }$ first undergo a feature decomposition operation, separating them into common features and differential features:

$$F_{common}={\displaystyle \frac{F_{vis}+F_{\inf }}{2}}$$

(11)

$$F_{diff}={\displaystyle \frac{F_{vis}-F_{\inf }}{2}}$$

(12)

The decomposed features then enter two independent but structurally symmetric processing paths:

(1)

Common Feature Path

The common features first pass through a residual block for basic feature transformation: the block consists of two 3 × 3 convolutional layers, each followed by InstanceNorm and ReLU activation functions, with residual connections to ensure gradient flow and alleviate the vanishing gradient problem in deep networks. Subsequently, the features are fed into a Mamba module, which captures global context information by leveraging the long-range dependency modeling capability of the State Space Model (SSM) and the bidirectional scanning mechanism.

(2)

Differential Feature Path

The differential features undergo exactly the same processing flow as the common path: the same residual block structure (with independently learned parameters) and the same Mamba module (but with independent parameters). The parameters of the convolutional layers are listed in

Table 3. The parameter configurations of convolutional layer in the DMAF module.

Layer	Size	Stride	Channel (Input)	Channel (Output)	Activation
conv1	3	1	32	16	LeakyRELU
conv2	3	1	16	32	LeakyRELU
conv3	3	1	32	16	LeakyRELU
conv4	3	1	16	32	LeakyRELU
1 × 1 conv	1	1	64	32	LeakyRELU

.

The symmetric design of the two paths ensures that common features and differential features are processed at the same level of abstraction, and modality-specific information is treated equally, facilitating subsequent feature integration operations.

The output feature maps from the two paths are concatenated along the channel dimension, increasing the feature dimension from 32 to 64 channels while maintaining the spatial resolution of the original input. This operation explicitly organizes the shared and complementary information between modalities within the same feature tensor. The concatenated features are then adaptively recalibrated by the Convolutional Block Attention Module (CBAM), whose network architecture is shown in Figure 5.

(1)

Channel Attention Module

First, the input feature map undergoes both max pooling and average pooling simultaneously, compressing the spatial dimensions to 1 × 1 to obtain channel-wise global statistics. Then, these two pooled results are processed by a shared Multi-Layer Perceptron (MLP), which consists of two fully connected layers (implemented with 1 × 1 convolutions), with a ReLU activation function introducing nonlinearity in between. The two outputs of the MLP are added together and then passed through a Sigmoid activation function to generate the channel attention weight map of size C × 1 × 1. Finally, this weight map is multiplied channel-wise with the original input feature map, completing the feature recalibration in the channel dimension. The network structure is shown in Figure 6.

(2)

Spatial Attention Module

The feature map enhanced by channel attention undergoes max pooling and average pooling along the channel dimension, respectively, resulting in two spatial feature maps of size 1 × H × W. These two feature maps are concatenated along the channel dimension to form a 2-channel feature map. Then, a 7 × 7 convolutional layer is used to fuse these two features, followed by a Sigmoid activation function to generate the spatial attention weight map of size 1 × H × W. The network structure is shown in Figure 6. Finally, this weight map is multiplied element-wise with the output of the channel attention, completing the feature recalibration in the spatial dimension. The network structure is shown in Figure 7.

The dimensional changes in the whole processing flow of CBAM module are shown in

Table 4. The dimensional changes of CBAM.

Layer	Input Dimension	Output Dimension
Input	[B,64,H,W]
Channel Attention Mechanism
MaxPool	[B,64,H,W]	[B,64,1,1]
AvgPool	[B,64,H,W]	[B,64,1,1]
Conv2d (64 → 4)	[B,64,1,1]	[B,4,1,1]
ReLU	[B,64,1,1]	[B,4,1,1]
Conv2d (4 → 64)	[B,4,1,1]	[B,64,1,1]
addition & sigmoid	[B,64,1,1] × 2	[B,64,1,1]
channel weighting	[B,64,H,W] × [B,64,1,1]	[B,64,H,W]
Spatial Attention Mechanism
Channel MaxPool	[B,64,H,W]	[B,1,H,W]
Channel AvgPool	[B,64,H,W]	[B,1,H,W]
concatenation	[B,1,H,W] × 2	[B,2,H,W]
7 × 7 conv	[B,2,H,W]	[B,1,H,W]
sigmoid	[B,1,H,W]	[B,1,H,W]
space weighting	[B,64,H,W] × [B,64,1,1]	[B,64,H,W]
Output		[B,64,H,W]

.

Finally, the attention-enhanced features are integrated through a 1 × 1 convolutional layer to obtain the fused features for image reconstruction. The parameters of the 1 × 1 convolutional layer are listed in Table 3.

4. Results

4.1. Training

To achieve decoupled optimization of cross-modal feature representation learning and dynamic fusion weights, this paper adopts a two-stage progressive training framework.

The first stage focuses on pre-training the foundational reconstruction capability of the encoder-decoder. Using a simplified architecture where the DMAF module is excluded, the model is optimized with a joint loss combining Structural Similarity (SSIM) loss and pixel-wise loss:

$$L_{ssim}=1-ssim(O,I)$$

(13)

$$L_{pixel}={‖O-I‖}_2$$

(14)

$$L_1=\lambda L_{ssim}+L_{pixel}$$

(15)

where $O$ and $I$ represent the output and input images, respectively.

The second stage introduces a dynamic fusion mechanism. The parameters of the pre-trained encoder-decoder are frozen, and the DMAF module is separately trained. Building upon the SSIM loss, an intensity loss and a gradient loss are incorporated.

The Intensity Loss, guided by the $\max (I_{vis},I_{\inf })$ fusion rule, ensures that the most salient features from both input images are preserved. Concurrently, the Gradient Loss leverages the Sobel operator to emphasize edge information and suppress artifacts. These three losses are combined with weighting factors to form a multi-granularity constraint that operates on three distinct levels: overall structural integrity (SSIM), pixel-wise intensity fidelity (Intensity), and local edge sharpness (Gradient).

$$L_{ssim}=0.5\cdot (1-ssim(I_{fused},I_{vis}))+0.5\cdot (1-ssim(I_{fused},I_{\inf }))$$

(16)

$$L_{\mathrm{int}ensity}={\displaystyle \frac{1}{HW}}{‖I_{fused}-\max (I_{vis},I_{\inf })‖}_1$$

(17)

$$L_{gradient}={\displaystyle \frac{1}{HW}}{‖\left|Sobel(I_{fused})\right|-\max (\left|sobel(I_{vis})\right|,\left|sobel(I_{\inf })\right|)‖}_1$$

(18)

$$L_2={\lambda }_1{L}_{ssim}+{\lambda }_2{L}_{\mathrm{int}ensity}+{\lambda }_3{L}_{gradient}$$

(19)

4.2. Dataset

This study employs a two-stage training strategy. In the first stage, the encoder-decoder network is trained using the MS-COCO dataset. MS-COCO is one of the most influential and widely used large-scale datasets in the field of computer vision. The acronym stands for Microsoft Common Objects in Context. Unlike datasets focused on single-object classification (e.g., ImageNet), the core philosophy of COCO is “scene understanding.” It primarily consists of complex everyday scene images where objects are placed in their natural environments. It emphasizes the recognition and localization of multiple objects within an image and understanding the relationships between them. A total of 80,000 images are selected to form the training set; these images are converted to grayscale and resized to 256 × 256 pixels. Figure 8 displays some examples from the MS-COCO dataset.

The second stage utilizes an enhanced version of the Multi-Spectral Road Scenarios (MSRS) dataset. This dataset is constructed by applying infrared image contrast enhancement (using the dark channel prior algorithm) and sample alignment optimization to the MFNet dataset, resulting in a collection of 1444 high-quality infrared-visible image pairs. The training set consists of 1083 pairs (75%), covering both day (715 pairs) and night (368 pairs) scenes, as well as multi-object interaction environments involving roads, pedestrians, and vehicles. The test set comprises 361 pairs (25%), specifically designed to include challenging scenarios such as low illumination (moonlight/streetlights), dynamic occlusions (trees/buildings), and extreme weather conditions (haze/rain/snow). All samples are annotated with pixel-level semantic segmentation labels (8 obstacle categories) to support joint optimization for downstream tasks. Figure 9 displays some examples from the MSRS dataset.

4.3. Experimental Setup

The experimental procedures were supported by a high-performance computing system specifically configured for deep learning tasks, as shown in

Table 5. Configuration and training environment.

Parameter	Configuration
CPU model	Intel(R) Core(TM) i9-13900K
GPU model	NVIDIA GeForce RTX 4080
Operating system	Windows 10
Deep learning frame	Pytorch1.10.0
GPU accelerator	CUDA12.7
Integrated development environment	Pycharm
Scripting language	Python3.8
Neural network accelerator	CUDNN8.2.0

. The hardware core consisted of an Intel Core i9-13900K CPU, with a hybrid architecture of 24 cores and 32 threads, and an NVIDIA GeForce RTX 4080 GPU, which facilitated accelerated parallel processing for model training. The operating system was Windows 10. Our software stack was built on Python 3.8 within the PyCharm 2024.3.6 integrated development environment. For deep learning, we utilized PyTorch 1.10.0, which was accelerated using CUDA 12.7 and the cuDNN 8.2.0 library to optimize computational performance for neural network operations.

Furthermore, the configuration of the training parameters is shown in

Table 6. Hyperparametric configuration.

Parameter	Configuration
Neural network optimizer	Adam
Learning rate	0.001
Stage 1 Training epochs	10
Stage 2 Training epochs	50
Momentum β1	0.9
Momentum β2	0.999
Batch size	4
λ	100
λ1	2
λ2	2
λ3	5

. The hyperparameters for the model training were meticulously configured to optimize performance and ensure stable convergence. Considering GPU memory constraints, the batch size was set to 4. The first and second stage training process were conducted for 10 and 50 epochs, respectively. This total number of epochs, which was empirically determined, ensured that the model reached convergence. We employed the Adam optimizer with its default momentum parameters (β₁ = 0.9, β₂ = 0.999). A dynamic learning rate scheduler was adopted, starting with an initial rate of 1 × 10⁻³. Furthermore, a staged loss weighting strategy was implemented: in the first stage, the weight λ was set to 100, while in the second stage, the respective weights λ₁, λ₂, and λ₃ were assigned values of 2, 2, and 5 to balance the contribution of different loss components.

For gray visible and infrared images fusion, the input channel of visible images is 1. For RGB and infrared image fusion, we first convert visible images to the YCbCr color space, processing is applied exclusively to the luminance channel (Y), while the Cb/Cr chrominance channels remain unaltered, so the input channel of visible images is 1, too, the fused image can be converted into the RGB color space via combining with Cb and Cr channels of visible images.

4.4. Evaluation Metric

To avoid a mere list of metrics, we logically categorize them into four groups to systematically evaluate different aspects of fusion performance:

(a)

Information Richness

Entropy (EN): Measures the average amount of information contained in the fused image. A higher EN value indicates richer information and more abundant tonality. Its calculation is provided in Appendix A.

Mutual Information (MI): Quantifies the total amount of information transferred from the source images to the fused image. A higher MI value indicates more sufficient information transfer. The formula is detailed in Appendix A.

(b)

Structural Fidelity

Structural Similarity Index Measure (SSIM): Evaluates the similarity in structural information between the fused image and the source images from the perspective of the human visual system. A higher SSIM value implies better structural preservation capability. The corresponding formula can be found in Appendix A.

Visual Information Fidelity (VIF): Measures the amount of visual information shared between the fused and reference images, being highly correlated with human subjective perception. Its computation follows the method referenced in Appendix A.

(c)

Detail and Clarity

Spatial Frequency (SF): Reflects the overall clarity and texture detail level of an image by calculating its gradient activity. A higher SF value indicates a sharper image. The calculation is provided in Appendix A.

Quality Assessment of Blended Features (Qabf): A metric specifically designed to evaluate the extent to which edge information is extracted from the source images into the fused image. Qabf is a key indicator for assessing detail preservation capability, with its formula listed in Appendix A.

(d)

Contrast and Saliency

Standard Deviation (SD): Reflects the dispersion of pixel values relative to the mean. A larger SD value typically indicates higher image contrast and more vivid visual effects. The formula is detailed in Appendix A.

Sum of Correlation Differences (SCD): Calculates the sum of the correlations between the fused image and each source image. A higher SCD value suggests that the fused image maintains a good association with both source images, effectively balancing information from both sides. The computation is provided in Appendix A.

4.5. Comparison with Existing Methods

This section presents a comparative analysis between the proposed network and the following state-of-the-art methods: the CNN and AE-based group: IFCNN and U2Fusion; the GAN-based group: RCGAN and MFEIF; and the Transformer-based group: SwinFusion.

Quantitative Comparison: The results are quantitatively evaluated using the eight metrics mentioned in Section 4.4. The results are summarized in

Table 7. Quantitative Results for the VIF Task on the MSRS dataset. Bold values indicate the best performance.

	Datasets: MSRS Fusion Dataset
	EN	SD	SF	MI	SCD	VIF	Qabf	SSIM
IFCNN	6.18	26.61	4.73	1.73	1.33	0.73	0.54	0.48
U2Fusion	5.22	22.8	3.57	1.37	1.15	0.51	0.38	0.38
RCGAN	5.90	23.8	3.49	1.60	1.16	0.55	0.40	0.37
MFEIF	5.91	33.58	3.55	2.06	1.50	0.72	0.54	0.34
SwinFusion	6.62	42.99	5.01	3.30	1.69	0.93	0.66	0.51
Ours	6.65	38.98	5.69	3.35	1.50	0.89	0.68	0.48

. The comprehensive quantitative evaluation demonstrates the superior performance of the proposed network across multiple key metrics. Our method achieves top scores in Information Entropy (EN), Spatial Frequency (SF), Mutual Information (MI), and Qabf, indicating its exceptional capability in preserving source information and generating high-quality fusion outputs. Furthermore, it secures highly competitive, second-best results in metrics including Standard Deviation (SD), Structural Consistency (SCD), Visual Information Fidelity (VIF), and Structural Similarity (SSIM). This compellingly shows that our network not only excels in information fusion but also maintains a balanced and robust performance across a wide spectrum of image attributes, outperforming other state-of-the-art methods in overall comparative analysis.

Qualitative Comparison: Figure 10, Figure 11 and Figure 12 presents the visual comparison results. The proposed network effectively integrates thermal radiation information from infrared images with textural and illumination details from visible images. Its advantages are demonstrated in the following aspects:

Enhancement of Low-Light Targets: Targets in dark environments (e.g., pedestrians) are significantly highlighted, resulting in a clear foreground-background distinction.

Clarification of Background Details: Previously blurred background features under low illumination (e.g., building contours, vegetation) exhibit sharp edges and rich structural information. Furthermore, the multi-scale mechanism inherent in the MPMamba module ensures effective fusion of targets at various scales within the image (e.g., pedestrians at different distances), thereby significantly enhancing scene comprehension capabilities.

4.6. Generalization Experiment

To validate the model’s adaptability in unseen scenarios, we conducted generalization testing by training the network on the MSRS dataset and evaluating it on two independent datasets: TNO and RoadScene. The visual comparisons of different algorithms are presented in Figure 13, Figure 14, Figure 15 and Figure 16. As shown in the green box in Figure 13 and the red box in Figure 14, fusion results from IFCNN, U2Fusion, MFEIF, and RCGAN exhibited issues such as edge artifacts or weakened thermal target intensity. Furthermore, as indicated by the red boxes in Figure 15 and Figure 16, all compared methods except ours and SwinFusion suffered from either edge artifacts or the loss of fine-grained texture details from visible images in certain scenes. In contrast, the images generated by our method effectively avoided significant target degradation and spectral contamination in background regions. While maintaining the saliency of infrared thermal targets, our approach better preserved clear edges and rich texture details from the visible images, demonstrating superior cross-modal information integration capability.

Quantitative evaluation results are summarized in

Table 8. Quantitative Results for the VIF Task on the TNO & RoadScene dataset. Bold values indicate the best performance.

	Datasets: TNO & RoadScene Dataset
	EN	SD	SF	MI	SCD	VIF	Qabf	SSIM
IFCNN	6.5946	31.402	7.2917	1.6544	1.7135	0.5852	0.483	0.4258
U2Fusion	6.5208	27.1557	6.4177	1.2312	1.7328	0.5468	0.4221	0.5017
RCGAN	6.4169	26.4547	4.9966	1.485	1.5423	0.5404	0.4177	0.4385
MFEIF	6.5388	30.5476	4.0907	1.8147	1.7716	0.6194	0.4073	0.5051
SwinFusion	6.6812	37.8414	6.3565	2.2236	1.7496	0.6589	0.4963	0.5064
Ours	7.1021	40.4312	7.7482	2.0288	1.7142	0.5716	0.4599	0.5093

,which further corroborate the aforementioned advantages. Our method achieved leading performance across multiple core metrics: EN (7.1021), SD (40.4312), SF (7.7482), and SSIM (0.5093) all ranked first among all methods. This indicates that the fused images possess higher information entropy, contrast, and spatial activity, while maximally preserving the structural information of the source images, particularly the visible images. Performance on other metrics was only slightly lower than that of individual alternative methods.

In summary, both qualitative and quantitative results consistently demonstrate that our method exhibits excellent generalization performance on datasets not involved in training. It can adaptively fuse complementary information from infrared and visible images, effectively suppressing artifacts while preserving background textures, all while enhancing target saliency.

4.7. Efficiency Comparison

To comprehensively evaluate the efficiency of our proposed method, we compare it with current mainstream image fusion approaches in terms of model size, computational complexity, and inference speed. As shown in

Table 9. Computational efficiency comparison.

Method	IFCNN	U2Fusion	SwinFusion	MFEIF	RCGAN	Ours
Size (M)	0.0836	0.6592	0.9737	0.1581	0.1137	0.1261
FLOPs (G)	39.93	405.17	292.53	25.36	45.60	50.88
Time (s)	0.201	0.361	1.467	1.423	0.238	0.392

, our proposed model maintains a lightweight design with a parameter count of 0.1261 M, which is significantly lower than that of U2Fusion (0.6592 M) and SwinFusion (0.9737 M), and is on par with IFCNN (0.0836 M) and RCGAN (0.1137 M), reflecting the compactness of our model architecture. In terms of computational complexity, the FLOPs of our method (50.88 G) are slightly higher than those of IFCNN (39.93 G) and MFEIF (25.36 G) but substantially lower than U2Fusion (405.17 G) and SwinFusion (292.53 G), highlighting our balanced consideration between global modeling capability and computational efficiency. Inference time tests demonstrate that PIAFusion (0.392 s) maintains real-time processing capability while ensuring fusion quality, with its speed notably superior to SwinFusion (1.467 s) and MFEIF (1.423 s) and comparable to U2Fusion (0.361 s). These results fully validate the linear complexity advantage of the Mamba module in handling long-range dependencies. Collectively, these efficiency metrics indicate that our method excels in model lightweighting, computational efficiency, and practical deployment feasibility, offering an efficient and practical solution for image fusion tasks.

4.8. Ablation Study

Ablation studies were conducted to systematically validate the contribution of the key designs and modules. The eight metrics mentioned in Section 4.4 were employed for the quantitative evaluation of fusion performance, with the results summarized in

Table 10. Ablation study results on the MSRS test set. Bold values indicate the best performance.

	EN	SD	SF	MI	SCD	VIF	Qabf	SSIM
Feature Extraction
I. Use Mamba block instead of MPMamba block	6.54	38.65	5.59	3.28	1.47	0.87	0.57	0.41
Feature Fusion
II. Additive strategy	6.37	35.89	5.18	2.87	1.35	0.69	0.61	0.42
III. Maximum value strategy	6.26	37.48	5.45	3.16	1.27	0.76	0.65	0.45
Ours	6.65	38.98	5.69	3.35	1.50	0.89	0.68	0.48

. In addition, typical visual comparisons (as shown in Figure 16) are provided to qualitatively analyze the fusion results under different configurations.

The ablation experiments focus on two critical stages: the feature extraction stage and the feature fusion stage.

In the feature extraction stage, the proposed Multi-Path Mamba (MPMamba) block was replaced with a Mamba block.

In the feature fusion stage, a simple element-wise addition strategy and an element-wise maximum strategy were tested as alternatives to the proposed dual-path attention fusion module.

The results indicate that the fusion performance degrades in both modified scenarios compared to the complete proposed architecture. This demonstrates the effectiveness and necessity of the specifically designed MPMamba block for multi-scale feature extraction, as well as the dedicated fusion module for integrating cross-modal features. Visual comparisons (as shown in Figure 17) further reveal that using only Mamba block tends to lose fine-grained texture details, while simple fusion strategies often cause feature confusion, manifesting as blurry details or reduced contrast. In contrast, our complete method preserves the detail and structural information of the source images more naturally. The underlying reasons are as follows:

(1)

The complementary design of multi-path convolutions and the Mamba block in MPMamba captures both local details and global context, overcoming the limitations of a single scale or a single operator’s receptive field;

(2)

The dual-path Mamba attention fusion module explicitly separates and integrates shared and complementary cross-modal features, followed by adaptive enhancement through channel-spatial attention, thereby achieving precise and robust information fusion. Consequently, our method exhibits superior comprehensive performance in both quantitative metrics and visual perception.

4.9. Object Detection Performance Evaluation

To objectively evaluate the contribution of image fusion methods to downstream visual tasks, we employed object detection as a validation task and conducted a systematic comparison between fused images and original source images. We selected eighty pairs of infrared and visible images from the MSRS dataset as test sets and annotated them with people and cars.

The experiment utilized the YOLOv5 model pre-trained on the MS-COCO dataset, with fixed model parameters, focusing specifically on the detection performance of “person” and “car” categories.

We adopted Average Precision (AP) as the core evaluation metric, calculating detection accuracy at IoU thresholds of 0.5 and 0.9 to comprehensively assess localization accuracy and recall capability. IoU = 0.5 represents conventional detection standards, while IoU = 0.9 imposes extremely high requirements on bounding box localization precision. The results are summarized in

Table 11. The object detection performance (mAP) of visible, infrared, and fused images obtained from different methods on the MSRS dataset. Bold values indicate the best performance.

		AP@0.5			AP@0.9
	Person	Car	All	Person	Car	All
Infrared	0.955	0.661	0.808	0.89	0.538	0.714
Visible	0.679	0.943	0.811	0.617	0.902	0.76
IFCNN	0.949	0.919	0.934	0.887	0.873	0.88
U2Fusion	0.959	0.931	0.945	0.876	0.898	0.887
SwinFusion	0.934	0.927	0.931	0.867	0.889	0.878
MFEIF	0.934	0.926	0.93	0.861	0.881	0.871
RCGAN	0.935	0.918	0.926	0.856	0.875	0.866
Ours	0.96	0.928	0.946	0.894	0.900	0.889

, Typical test result comparisons are shown in Figure 18.

The results from the downstream detection task clearly validate the complementary characteristics of different imaging modalities. Specifically for different categories, we observed that infrared images performed excellently in person detection but showed limitations in vehicle detection, while visible images exhibited the opposite characteristics. All of the fused images successfully overcame the inherent deficiencies of single modalities, achieving balanced and superior detection performance for both target categories, validating the practical value of fusion methods in real-world applications.

For the fusion method discussed in the paper, our proposed approach demonstrates exceptional performance in the ‘person’ category, achieving an AP@0.5 score of 0.96 and an AP@0.9 score of 0.894, ranking among the top performers across all input fused images. This indicates that the fused images of our method provide an optimal representation for detecting and precisely localizing human figures, effectively combining the thermal saliency from infrared images with the textual details from visible images. The proposed technical solution demonstrates exceptional overall performance, as evidenced by its ‘ALL’ category scores of 0.946 in AP@0.5 and 0.889 in AP@0.9. These leading results confirm the robustness and comprehensive superiority of our fusion method in enhancing downstream object detection tasks.

As for the ‘car’ category, the highest detection accuracy was consistently secured by the original visible images, with AP@0.5 of 0.943 and AP@0.9 of 0.902. We attribute this phenomenon to several factors. Primarily, the object detector was pre-trained on the visible-light-dominant MS-COCO dataset, making it highly attuned to the textual and structural features inherent in visible images. For vehicles, which possess abundant and discriminative features in the visible spectrum, the introduction of infrared thermal signatures through fusion may not provide substantial complementary benefits for the detection task itself. Instead, it could potentially dilute the model’s familiar feature representations or introduce subtle artifacts at the object boundaries, thereby marginally compromising the localization precision, especially under the stringent AP@0.9 metric. This finding underscores that the advantage of image fusion is task-dependent; it is most beneficial when the complementary modality (e.g., infrared for pedestrians) provides crucial information that is absent or unreliable in the primary modality (e.g., visible light).

In summary, through quantitative evaluation using downstream object detection tasks, we have confirmed that the proposed fusion method can generate high-quality fusion results, providing more informative input data for high-level visual applications.

5. Discussion

This study presents a Mamba-based framework for infrared and visible image fusion, demonstrating superior performance on key metrics and visual quality. The following discussion interprets these results within the context of existing literature, elaborates on the methodological innovations, acknowledges limitations, and suggests future directions.

5.1. Interpretation of Results and Methodological Innovation

The core achievement of this work lies in successfully adapting the selective state space model (SSM), specifically Mamba, to address the pivotal challenges in cross-modal image fusion: global dependency modeling and computational efficiency. While CNNs are limited by local receptive fields and Transformers by quadratic complexity, our framework leverages Mamba’s linear-complexity global scanning. The quantitative results (Table 7), showing top-tier performance in EN, SF, MI, and Qabf, confirm that this architecture effectively enriches information content while preserving structural and edge details from both modalities. Visually, the fused images saliently retain thermal targets from infrared inputs and rich textures from visible inputs, validating the design premise.

The Multi-path Mamba (MPMamba) module’s effectiveness stems from its hybrid local-global design. The parallel convolutional streams with different kernel sizes explicitly capture multi-scale local features—crucial for textures and varying target sizes. The subsequent Mamba paths then model *global* contextual relationships across these extracted features. This synergistic design overcomes the limitation of using either pure convolutions (which lack global context) or a single-scale Mamba block (which may overlook fine-grained details), as evidenced by the ablation study.

The Dual-path Mamba Attention Fusion (DMAF)module introduces a structured approach to cross-modal feature matching and alignment, a critical aspect in multimodal fusion. By explicitly decomposing features into common and differential components and processing them through dual Mamba paths, the network learns to align semantically similar structures while preserving modality-specific information. This design philosophy—tackling feature decoupling and alignment—resonates with a broader trend in advanced fusion and cross-view perception research, which seeks to establish robust correspondences between heterogeneous data sources. Our approach, utilizing the parallel modeling capacity of state-space models followed by attention-based calibration, offers a novel pathway within this important research direction.

5.2. Implications and Broader Context

The superior performance in the downstream object detection task has significant implications. The fused images, particularly from our method, enhanced “person” detection by combining thermal saliency with textual detail, demonstrating the practical value of fusion for perception in adverse conditions. The finding that pure visible images sometimes excelled in “car” detection underscores a key insight: **the utility of fusion is task- and target-dependent**. Fusion provides maximum benefit when the complementary modality supplies information critically missing or weak in the primary modality. This argues for future fusion systems that are dynamically adaptive to the target scene and the downstream task’s requirements.

Computationally, our model strikes a favorable balance. It maintains a compact size and significantly lower FLOPs than Transformer-based counterparts, while achieving competitive inference speed. This efficiency, inherent to the Mamba architecture, enhances the feasibility for real-time applications such as autonomous driving and surveillance, addressing a major practical limitation of many high-performing fusion models.

5.3. Limitations and Future Research Directions

Despite its strengths, this work has limitations that open avenues for future research:

• Dynamic and Adaptive Fusion Mechanisms: The current fusion strategy employs a fixed architecture. Future work could explore more dynamic mechanisms, such as content-adaptive weighting or gating networks, to decide “what” and “how” to fuse at different regions or feature levels. This aligns with the growing focus on developing adaptive fusion strategies that can handle varying scene complexities and modality-specific information strengths.
• Advanced Feature Alignment Techniques: While the DMAF module provides a foundational decoupling-and-alignment strategy, the challenge of robust feature matching under significant modality gaps persists. Future improvements could investigate more sophisticated alignment techniques inspired by recent progress in multimodal and multi-view learning, such as employing gradient-aware alignment losses or exploring non-local correlation modules, to further enhance the precision of cross-modal feature integration.
• Extension to More Modalities and Tasks: The Mamba-based architecture is promising for fusing more than two modalities. Exploring its capacity for unified multimodal (e.g., visible, infrared, depth) representation learning and for other downstream perception tasks like segmentation is a logical next step.
• End-to-End Task-Specific Optimization: While we evaluated on detection, the fusion network was not trained end-to-end with the task loss. Joint optimization could further tailor the fused features to maximize performance for specific applications, creating a tighter synergy between fusion and perception.

6. Conclusions

This paper proposed a novel and efficient framework for infrared and visible image fusion based on the Mamba model. The introduced Multi-path Mamba (MPMamba) module effectively captures multi-scale and global features, while the Dual-path Mamba Attention Fusion (DMAF) module provides a principled approach for aligning and integrating cross-modal information through explicit feature decoupling and attention-based calibration. Comprehensive experiments demonstrate that the proposed method has good performance on standard metrics, produces visually superior fusion results, enhances downstream object detection capability, and maintains favorable computational efficiency. The work not only validates Mamba as a powerful backbone for multimodal fusion tasks but also contributes to the ongoing exploration of effective feature alignment strategies. Future work will focus on developing more adaptive, task-oriented, and extensible fusion systems based on this foundational architecture.