Multi-Perspective Information Fusion Network for Remote Sensing Segmentation

Highlights

What are the main findings?

• The MPIFNet integrates multiple perspectives to achieve multi-scale semantic feature extraction of complex land cover objects.
• The GLMBSA integrates global and local information, while DBHWT extracts multi-level and frequency-domain features, jointly enhancing the model’s feature representation.

What are the implications of the main findings?

• The MPIFNet provides an effective solution for semantic segmentation of complex remote sensing images, improving the recognition of objects at different scales.
• The modular design demonstrates the potential of feature enhancement, providing a basis for lightweight, efficient remote sensing segmentation models.

Abstract

Remote sensing acquires Earth surface information without physical contact through sensors operating at diverse spatial, spectral, and temporal resolutions. In high-resolution remote sensing imagery, objects often exhibit large scale variation, complex spatial distributions, and strong inter-class similarity, posing persistent challenges for accurate semantic segmentation. Existing methods still struggle to simultaneously preserve fine boundary details and model long-range spatial dependencies, and lack explicit mechanisms to decouple low-frequency semantic context from high-frequency structural information. To address these limitations, we propose the Multi-Perspective Information Fusion Network (MPIFNet) for remote sensing semantic segmentation, motivated by the need to integrate global context, local structures, and multi-frequency information into a unified framework. MPIFNet employs a Global and Local Mamba Block Self-Attention (GLMBSA) module to capture long-range dependencies while preserving local details, and a Double-Branch Haar Wavelet Transform (DBHWT) module to separate and enhance low- and high-frequency features. By fusing spatial, hierarchical, and frequency representations, MPIFNet learns more discriminative and robust features. Evaluations on the Vaihingen, Potsdam, and LoveDA datasets through ablation and comparative studies highlight the strong generalization of our model, yielding mIoU results of 86.03%, 88.36%, and 55.76%.

1. Introduction

Remote sensing semantic segmentation plays a crucial role in a wide range of applications, including urban planning, environmental monitoring, disaster assessment, and precision agriculture [1,2,3,4]. Unlike natural images, remote sensing imagery exhibits several distinctive challenges, such as large variations in object scales, complex and irregular object boundaries, and highly heterogeneous land-cover distributions. These characteristics make it difficult to achieve accurate and robust pixel-level classification, especially in densely built-up and geographically diverse scenes.

Early remote sensing image segmentation approaches mainly relied on handcrafted features and traditional image processing techniques, such as thresholding [5] and region growing [6]. Although these methods exploited low-level cues including color, texture, and shape, they were highly sensitive to parameter settings and scene variability, and often failed to generalize to complex real-world scenarios.

With the advent of deep learning, convolutional neural networks (CNNs) significantly advanced remote sensing semantic segmentation by enabling hierarchical feature learning. Fully Convolutional Networks (FCNs) [7] pioneered end-to-end dense prediction, while U-Net [8] introduced encoder–decoder structures to recover spatial details through skip connections. Subsequent studies further enhanced performance by incorporating multiscale feature representations and pyramid structures [9,10,11]. However, due to the inherently local receptive fields of convolutions, CNN-based models often struggle to capture long-range spatial dependencies, which are critical for modeling large-scale objects and contextual relationships in remote sensing scenes. Moreover, as ground object information becomes increasingly complex and dispersed, CNNs alone are insufficient to model global context. To mitigate this limitation, attention mechanisms have been introduced [12,13,14], enabling networks to selectively focus on relevant regions and better capture long-range dependencies. Building on this idea, Transformer [15] architectures have recently emerged as powerful alternatives by leveraging self-attention mechanisms to establish global dependencies. Vision Transformer (ViT) [16] and its derivatives [17,18,19] have demonstrated strong performance in semantic segmentation by modeling long-range contextual interactions. Nevertheless, excessive reliance on global self-attention may lead to the loss of fine-grained local details and blurred object boundaries, which are particularly detrimental for remote sensing tasks that require precise delineation of complex structures such as building edges and road networks.

To balance global context modeling and local detail preservation, hybrid CNN–Transformer frameworks have been proposed. Representative methods such as FTransUNet [20] and A TransUNet [21] integrate convolutional encoders with Transformer modules to combine local texture modeling and global context reasoning. While these hybrid approaches improve performance to some extent, they still face challenges in effectively modeling multiscale structures and preserving fine boundary details under highly heterogeneous remote sensing scenarios.

To address these issues, advanced spatial sequence modeling techniques have recently gained momentum. State Space Models (SSMs) [22] and Mamba-based frameworks offer efficient long-range dependency modeling with linear complexity. Existing Mamba-based methods, such as CPSSNet [23], employ dual-branch SSM structures to separately capture spatial and channel dependencies but suffer from limited inter-branch semantic fusion. PPMamba [24], built upon SS2D, further improves spatial context modeling through directional state-space scanning; however, its modeling remains predominantly spatial and lacks mechanisms to incorporate fine texture cues or frequency-aware information. In parallel, frequency-domain methods such as Fourier operations [25] and wavelet-based approaches including SFFNet [26] decompose features into low- and high-frequency components to strengthen structural details and boundary sharpness. Nevertheless, they focus mainly on frequency representation and neglect complementary long-range spatial dependencies.

Based on these insights, we design the Multi-Perspective Information Fusion Network (MPIFNet) to explicitly model complementary feature representations from multiple viewpoints. As illustrated in Figure 1, remote sensing images exhibit distinct characteristics when analyzed at different levels. From a spatial-scale perspective, large-scale structures provide stable semantic cues, while fine-scale details preserve object boundaries and local texture patterns. From a frequency-domain perspective, low-frequency components encode global layout and category-level information, whereas high-frequency components capture edges and structural details. These observations motivate our multi-perspective modeling framework and provide the foundation for the design of MPIFNet. The primary contributions of this work are summarized as follows:

Figure 1. Characteristic information under different perspectives. (a,b) represent the feature information under different scale perspectives. (c,d) represent the feature information under the low frequency and high frequency perspectives.

• We introduce MPIFNet, a network designed to integrate global and local perspectives, multi-level features, and frequency-domain information, enabling comprehensive capture of multiscale semantic cues and effective extraction of complex features.
• We propose a Global and Local Mamba Block Self-Attention mechanism that enhances multiscale feature representation by simultaneously capturing global context from large-scale regions and local details from fine-grained areas. This dual-perspective design enables the model to effectively integrate semantic cues across different object scales, resulting in more accurate and comprehensive segmentation performance.
• Block self-attention, as a part of the integrated GLMBSA mechanism, is capable of capturing long-range dependencies from global, local, and combined views. By extending attention beyond local blocks, it preserves fine details while enhancing global context, effectively balancing local precision and global understanding.
• To mitigate feature loss and enrich semantics, we design the DBHWT module, which integrates multi-level features and employs the Haar Wavelet Transform to extract both high- and low-frequency components, enhancing semantic discrimination across scales.

2. Related Work

2.1. CNN-Based Semantic Segmentation

Fully Convolutional Networks (FCNs) [7] pioneered end-to-end semantic segmentation using convolutional and deconvolutional layers, but their simplicity often leads to the loss of spatial detail. U-Net [8] addressed this with symmetric encoder–decoder structures and skip connections, preserving fine-grained features, yet it remains limited in handling large scale variations. DeepLab [9,10,11] introduced Atrous Spatial Pyramid Pooling (ASPP) to capture multiscale context via dilated convolutions. Despite these improvements, CNN-based approaches still struggle under complex backgrounds and with small or irregular objects, highlighting the need for richer semantic modeling and better global context reasoning.

2.2. Transformer-Based Semantic Segmentation

Attention mechanisms were proposed to mitigate CNNs’ limitations in capturing global dependencies. SENet [27] and DANet [28] enhanced feature representation via channel and spatial–channel attention, while ECANet [29] improved efficiency through local channel interactions. Transformers [15], leveraging self-attention, overcome spatial locality by modeling long-range interactions, with variants such as Longformer [30] and BigBird [31] improving computational efficiency. However, these models often require high computation, complex architectures, and careful hyperparameter tuning, which can limit scalability. Moreover, excessive reliance on global attention may reduce local structural accuracy, a critical factor for remote sensing tasks with fine-grained boundaries.

2.3. Mamba-Based Semantic Segmentation

Mamba [32] introduces a selective state-space mechanism for long-sequence modeling with linear complexity, offering computationally efficient alternatives to standard self-attention. Extensions such as VMamba [33] and VSSM [34] enable bidirectional and bidimensional context modeling, enhancing spatial perception. While Mamba-based methods improve efficiency and global context capture, existing approaches often lack effective fusion with local or frequency-domain cues, limiting fine-scale boundary preservation and multiscale semantic integration in remote sensing imagery.

2.4. Frequency Domain-Based Semantic Segmentation

Frequency-domain methods complement spatial modeling by capturing information at multiple scales. Wavelet Transform (WT) [35] supports multiresolution analysis, preserving both global and fine-grained details. Wavelet-CNN [36] and later works [37,38] fuse spectral cues to enhance object representation. SFFNet [26] demonstrates that Haar wavelet decomposition can improve boundary sharpness and semantic discrimination. Despite these benefits, most frequency-based approaches overlook long-range spatial dependencies, which limits their performance on complex remote sensing scenes.

2.5. Research Gap and Motivation for This Work

Although methods based on CNN, Transformer, Mamba, and frequency-domain have advanced remote sensing segmentation, existing approaches typically address either local structure, global context, or multiscale and frequency cues in isolation. Very few methods integrate these complementary perspectives simultaneously, which is essential for accurately segmenting heterogeneous landscapes with complex object boundaries. Motivated by this gap, we propose the Multi-Perspective Information Fusion Network (MPIFNet), which unifies spatial–frequency modeling with Mamba-based global–local feature interaction. Unlike prior methods, MPIFNet effectively fuses local and global dependencies while preserving frequency-domain cues, enabling more accurate and robust segmentation in multiscale and heterogeneous remote sensing scenes.

3. Methodology

3.1. Overall Architecture

The overall framework is illustrated in Figure 2. We adopt the classic U-Net architecture [8], with ConvNeXt [39] as the encoder backbone to extract spatial features, where the number of channels increases progressively from 96 to 768. The model is trained with a hybrid loss function that combines cross-entropy and Dice losses. To enrich semantic information, we introduce a Double-Branch Haar Wavelet Transform (DBHWT) in the skip connections, enabling the extraction of multi-level and multi-frequency features. In the decoder, we design a plug-and-play Global and Local Mamba Block Self-Attention (GLMBSA) module, inserted after each upsampling layer. This module incorporates a State Space Model (SSM) based Mamba block [32] and jointly applies Global Mamba and Local Mamba mechanisms to capture both global contextual dependencies and fine-grained local details across scales. Finally, Block Self-Attention is applied to enhance both global and local semantic interactions, yielding more complete and precise feature representations for semantic segmentation.

Figure 2. The overall structure of MPIFNet. The U-Net network structure is used, with a Double-Branch Haar Wavelet Transform applied in the skip connection part. After each decoder, Global and Local Mamba Block Self-Attention is introduced.

3.2. Global and Local Mamba Block Self-Attention

We introduced the State Spatial Model (SSM), which is the core of the Mamba approach, and the specific computation details can be referenced in the original paper [32].

To extend the one-dimensional state space model (SSM) to two-dimensional feature maps $X\in {\mathbb{R}}^{C\times H\times W}$, we adopt 2D Selective Scanning (SS2D) formulation. Specifically, the input feature map is scanned along both the horizontal and vertical directions, where one-dimensional SSMs are applied separately. Formally, the SSM in 1D is defined as:

$$h_t=A{h}_{t-1}+B{x}_t,y_t=C{h}_t$$

(1)

where $x_t$ is the input sequence, $h_t$ the hidden state, and $y_t$ the output. Mamba further introduces an input-dependent gating mechanism:

$$h_t=\sigma \left(v_t\right)\odot (A{h}_{t-1}+B{u}_t),y_t=C{h}_t$$

(2)

where $u_t$ and $v_t$ are projections of $x_t$, and $\sigma (·)$ is the sigmoid function.

For the two-dimensional case, the SS2D module applies SSMs along the horizontal (W) and vertical (H) axes independently:

$$Y^{\left(h\right)}={\mathrm{SSM}}_h\left(X\right),Y^{\left(v\right)}={\mathrm{SSM}}_v\left(X\right)$$

(3)

where ${\mathrm{SSM}}_h$ and ${\mathrm{SSM}}_v$ denote selective scanning along the horizontal and vertical dimensions, respectively. The final output is obtained by fusing the two directions:

$$Y=Y^{\left(h\right)}+Y^{\left(v\right)}$$

(4)

This design enables SS2D to capture long-range dependencies in both spatial dimensions while preserving computational efficiency, which is particularly beneficial for dense prediction tasks such as semantic segmentation.

However, for the Mamba block we adopt, although its scanning mechanism enables the capture of global information across the entire feature map, it may to some extent overlook local detailed information. Therefore, we proposed a Global and Local Mamba Block Self-Attention, as shown in Figure 2. To capture localized information and compensate for deficiencies in local detail, the input features $X\in {\mathbb{R}}^{C\times H\times W}$ is first processed globally to extract contextual information. The feature map is then partitioned along the height H and width W dimensions into four spatial subregions: top left($I_{ul}$), top-right($I_{ur}$), bottom left($I_{dl}$), and bottom-right($I_{dr}$). The computation is formally expressed as follows:

$$I_{ul}=X(0:{\displaystyle \frac{H}{2}},0:{\displaystyle \frac{W}{2}})$$

(5)

$$I_{dl}=X({\displaystyle \frac{H}{2}}:H,0:{\displaystyle \frac{W}{2}})$$

(6)

$$I_{ur}=X(0:{\displaystyle \frac{H}{2}},{\displaystyle \frac{W}{2}}:W)$$

(7)

$$I_{dr}=X({\displaystyle \frac{H}{2}}:H,{\displaystyle \frac{W}{2}}:W)$$

(8)

Each subregion is then independently processed by two layers of Mamba blocks to extract localized contextual representations:

$$M_{ul}=Mamba\left(I_{ul}\right)$$

(9)

$$M_{dl}=Mamba\left(I_{dl}\right)$$

(10)

$$M_{ur}=Mamba\left(I_{ur}\right)$$

(11)

$$M_{dr}=Mamba\left(I_{dr}\right)$$

(12)

where, $M_{ul}$, $M_{dl}$, $M_{ur}$ and $M_{dr}$ represent local scans in different positions.

Finally, the enhanced local features are hierarchically concatenated to reconstruct the full feature map. First, features are concatenated along the height dimension:

$$M_{left}=Conca{t}_H(M_{ul},M_{dl})$$

(13)

$$M_{right}=Conca{t}_H(M_{ur},M_{dr})$$

(14)

Then, the two halves are concatenated along the width dimension to form the final Local Mamba feature map:

$$LMamba=Conca{t}_W(M_{left},M_{right})$$

(15)

This hierarchical procedure effectively integrates local features from different quadrants while preserving fine-grained details and global context.

After that, to enable effective fusion of Global Mamba and Local Mamba features, a Block Self-Attention (BSA) mechanism is incorporated into the GLMBSA framework, as illustrated in Figure 3. The corresponding pseudo-code is provided in Algorithm 1. In this context, Local Mamba focuses on capturing fine-grained spatial details within confined windows, preserving boundaries and small-scale structures, while Global Mamba models long-range dependencies across the entire feature map to maintain contextual consistency for large-scale objects. This mechanism facilitates detailed interaction between local and global features: the feature representations from the Mamba modules replace the conventional Query(Q) and Key(K), while the original input X is used as Value(V). In this way, the locally derived Q and globally contextualized K jointly guide the attention process, whereas the unaltered values (V) retain the original spatial details. This complementary interaction allows the network not only to capture high-level contextual semantics but also to refine fine-grained textures, thereby achieving more effective feature fusion than simple concatenation or direct mixing.

Algorithm 1 Block Self-Attention in GLMBSA

Require:  Feature maps $Q,K,V\in {\mathbb{R}}^{B\times C\times H\times W}$ Ensure:  Output $O\in {\mathbb{R}}^{B\times C\times H\times W}$   1: Compute $N=H\times W$ and set chunk size $S=H$   2: Reshape $\widehat{Q},\widehat{K},\widehat{V}\in {\mathbb{R}}^{B\times N\times C}$   3: Initialize $O^{\prime }\in {\mathbb{R}}^{B\times N\times C}$ as zeros   4: for  $i=0$   to   $N-1$   step  S  do   5:        $s=i$, $e=min(i+S,N)$   6:        Extract chunk: ${\widehat{Q}}_c=\widehat{Q}[:,s:e,:]$, ${\widehat{K}}_c=\widehat{K}[:,s:e,:]$, ${\widehat{V}}_c=\widehat{V}[:,s:e,:]$   7:        Compute attention scores: $A={\displaystyle \frac{{\widehat{Q}}_c{\widehat{K}}_c^{\top }}{\sqrt{C}}}$   8:        Apply softmax: $\alpha =\mathrm{softmax}\left(A\right)$   9:        Weighted sum: $O^{\prime }[:,s:e,:]=\alpha {\widehat{V}}_c$ 10: end for 11: Reshape $O^{\prime }\to O\in {\mathbb{R}}^{B\times C\times H\times W}$

Figure 3. Illustration of Block Self-Attention. The module first divides three components—Global Mamba, Local Mamba, and original features—into blocks. After performing self-attention computations, it aggregates all processed features.

To preserve local details while aggregating long-range context, a block-wise partitioning strategy is introduced. Specifically, the input query tensor $Q\in {\mathbb{R}}^{C\times N}$, where $N=H\times W$, is divided into n non-overlapping spatial blocks $(Q_1,Q_2,\dots ,Q_n)$, each of length H. The same partitioning is applied to K and V, yielding $(K_1,K_2,\dots ,K_n)$ and $(V_1,V_2,\dots ,V_n)$. Attention is computed independently within each block, allowing the network to focus on both local correlations and contextual dependencies.

By integrating block-wise attention with feature fusion, the BSA module balances the contribution of local and global information. This design preserves fine-grained spatial details while aggregating long-range dependencies, thereby enhancing semantic consistency, boundary precision, and overall segmentation accuracy. The detailed computation process is illustrated as follows:

$$\begin{array}{cc}\hfill Q_1,Q_2,\dots ,Q_n& =\mathrm{Chunk}\left(Q\right)\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill K_1,K_2,\dots ,K_n& =\mathrm{Chunk}\left(K\right)\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill V_1,V_2,\dots ,V_n& =\mathrm{Chunk}\left(V\right)\hfill \end{array}$$

(18)

where, Chunk operation is performed in N dimensions and is divided into n pieces, each with a length of H, and $n=H$.

Each block is then processed with self-attention, and the feature information after self-attention encompasses the contextual information within the corresponding region. The process is as follows:

$$\begin{array}{cc}\hfill S{A}_1(Q_1,K_1,V_1)& =softmax\left({\displaystyle \frac{Q_1{K}_1^T}{\sqrt{d_k}}}\right)V_1\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill S{A}_2(Q_2,K_2,V_2)& =softmax\left({\displaystyle \frac{Q_2{K}_2^T}{\sqrt{d_k}}}\right)V_2\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill S{A}_n(Q_n,K_n,V_n)& =softmax\left({\displaystyle \frac{Q_n{K}_n^T}{\sqrt{d_k}}}\right)V_n\hfill \end{array}$$

(21)

Finally, these processed blocks are concatenated to form a complete feature map with richer information. The computation is as follows:

$$\begin{array}{cc}\hfill BSA(Q,K,V)& =Concatenate(S{A}_1,S{A}_2,\dots ,S{A}_n)\hfill \end{array}$$

(22)

The above approach extends computation from block-level processing to whole-area processing, thereby enabling interaction between global and local contextual information. The entire GLMBSA process for the input feature $X\in {\mathbb{R}}^{C\times H\times W}$ can be expressed as:

$$\begin{array}{cc}\hfill X_G& =Mamba\left(X\right)\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill X_L& =LMamba\left(X\right)\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill Q& =Reshape\left(X_L\right)\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill K& =Reshape\left(X_G\right)\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill V& =Reshape\left(X\right)\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill X_{out}& =BSA(Q,K,V)+X_G+X_L\hfill \end{array}$$

(28)

where $X_G$ is the global feature map, $X_L$ is the local feature map, and $X_{out}$ is the feature map produced by their integration.

3.3. Double-Branch Haar Wavelet Transform

Traditional skip connection models typically propagate only the feature information from the current layer, which may result in the loss of critical semantic cues. To capture richer semantic information and enhance the interaction between features across different levels, we propose a Double-Branch model structure. Combining high- and low-level features, this architecture enhances the completeness and expressiveness of semantic representations. Furthermore, we incorporate the Haar Wavelet Transform to extract semantic information at varying frequency scales. By fusing semantic features from multiple perspectives, this design helps improve the model’s segmentation performance. We refer to this structure as the Double-Branch Haar Wavelet Transform (DBHWT), as illustrated in Figure 4.

Figure 4. Illustration of Double-Branch Haar Wavelet Transform. The double-branch structure originates from different skip connection layers. Each branch processes four frequency components extracted via Haar wavelet transform, with final feature integration achieved through additive fusion followed by channel concatenation.

Although CNNs progressively abstract visual information, high-level feature maps, such as those in the skip connections of our U-Net, still retain structured spatial and frequency characteristics. The low-frequency responses primarily capture global semantic context, whereas the high-frequency responses emphasize fine-grained details including edges, boundaries, and textures. The proposed DBHWT module leverages this property by decomposing CNN-derived features into low- and high-frequency sub-bands. Through this decomposition, DBHWT explicitly separates semantic-rich low-frequency components from detail-enhanced high-frequency components, thereby strengthening multi-scale feature representation. Moreover, the double-branch design provides complementary information extraction: one branch performs element-wise addition to emphasize semantic consistency, while the other performs channel concatenation to preserve discriminative details. Both branches undergo Haar wavelet decomposition and frequency-wise fusion, resulting in multi-scale enriched feature maps that capture both global semantic context and fine-grained structural details.

To implement this double-branch fusion, the lower-layer feature map is first padded and transformed to align with the spatial and channel dimensions of the current-layer feature map. One branch performs channel concatenation, while the other performs element-wise addition, enabling complementary feature interaction. The resulting combined feature maps are then decomposed via Haar wavelet transform into four frequency components—LL for low-frequency structures, LH for vertical details, HL for horizontal details, and HH for diagonal details. These frequency-wise features are fused to produce semantically enriched, multi-scale representations, which are then delivered through the skip connections to guide feature reconstruction in the decoder.

The wavelet transformation is first applied along the rows and subsequently decomposes high-level feature maps into low- and high-frequency components. The low-frequency component preserves global structures, while the high-frequency component emphasizes edges, textures, and fine-grained details. This multi-scale decomposition facilitates effective integration of spatial context with local information, thereby enhancing the network’s representational capacity for downstream tasks. For each input channel $X_c\in {\mathbb{R}}^{H\times W}$, the wavelet transformation begins with a row-wise operation, as formulated below:

$$\begin{array}{cc}\hfill L(i,j)& ={\displaystyle \frac{x_c(i,2j-1)+x_c(i,2j)}{2}}\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill H(i,j)& ={\displaystyle \frac{x_c(i,2j-1)-x_c(i,2j)}{2}}\hfill \end{array}$$

(30)

where $L(i,j)$ represents low-frequency information and $H(i,j)$ represents high-frequency information. The row index i varies from 1 to H (height), while the column index j spans 1 to $W/2$ (width) of the feature map.

The subsequent column-wise calculation is performed as follows:

$$\begin{array}{cc}\hfill LL(i,j)& ={\displaystyle \frac{L(2i-1,j)+L(2i,j)}{2}}\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill LH(i,j)& ={\displaystyle \frac{L(2i-1,j)-L(2i,j)}{2}}\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill HL(i,j)& ={\displaystyle \frac{H(2i-1,j)+H(2i,j)}{2}}\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill HH(i,j)& ={\displaystyle \frac{H(2i-1,j)-H(2i,j)}{2}}\hfill \end{array}$$

(34)

where $LL(i,j)$ represents low-frequency information, $LH(i,j)$ captures vertical high-frequency features, $HL(i,j)$ corresponds to horizontal high-frequency features, and $HH(i,j)$ encodes diagonal high-frequency features. Indices i and j run from 1 to $H/2$ along the rows and columns, respectively.

The four different high and low-frequency components obtained from the wavelet transform of the two branches are then summed together, with components of the same frequency combined. Finally, the four components are subsequently concatenated along the channel dimension to form the feature representation, and then linear interpolation and convolution are carried out to restore them to the original size, and then through the residual network, the multi-information feature map covering the contextual semantic information is finally obtained.

The DBHWT model captures semantic information at multiple levels, decomposes spatial features into high-frequency components to retain local details and low-frequency components to preserve global context, and performs cross-frequency extraction and fusion, ultimately producing comprehensive and enriched feature representations.

3.4. Loss Function

The loss function integrates cross-entropy with Dice loss to guide model optimization. This hybrid loss function aims to leverage the strengths of both metrics, with cross-entropy focusing on pixel-wise classification accuracy and Dice loss improving the model’s performance on imbalanced datasets by emphasizing the correspondence between predicted and true regions. It is formulated as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{ce}}& =-{\displaystyle \frac{1}{N}}\sum _{n=1}^N\sum _{k=1}^K{y}_k^{\left(n\right)}log{\widehat{y}}_k^{\left(n\right)}\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{dice}}& =1-{\displaystyle \frac{2}{N}}\sum _{n=1}^N\sum _{k=1}^K{\displaystyle \frac{{\widehat{y}}_k^{\left(n\right)}{y}_k^{\left(n\right)}}{{\widehat{y}}_k^{\left(n\right)}+y_k^{\left(n\right)}}}\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill \mathcal{L}& ={\mathcal{L}}_{\mathrm{ce}}+{\mathcal{L}}_{\mathrm{dice}}\hfill \end{array}$$

(37)

where N and K represent the number of samples and the number of classes. $y_k^{\left(n\right)}$ and ${\widehat{y}}_k^{\left(n\right)}$ represent the one-hot encoding of the true semantic labels and the corresponding softmax output of the network, where $n\in [1,\dots ,N]$. Specifically, ${\widehat{y}}_k^{\left(n\right)}$ indicates the confidence that sample n belongs to class k.

4. Experimental and Analysis

4.1. Datasets

Vaihingen Dataset: This dataset was obtained using aerial sensors and covers the medium-sized city of Vaihingen in southern Germany. The images were captured from an aerial platform and have high spatial resolution. The entire dataset consists of 33 images, with each image containing RGB and near-infrared bands. The image content includes 6 main categories, which are: Buildings (red annotation), Cars (cyan annotation), Trees (green annotation), Ground (white annotation), Low vegetation (yellow annotation), and Miscellaneous objects/Background (blue annotation).

Potsdam Dataset: The dataset, supplied by ISPRS, contains high-resolution aerial imagery and serves as a standard benchmark for semantic segmentation in remote sensing applications. The dataset focuses on the Potsdam region in Germany and is primarily used for evaluating land cover classification in urban areas. Similar to the Vaihingen dataset, the image content covers 6 main categories. Meanwhile, this dataset includes five-band multispectral images, including RGB bands, near-infrared bands, and a DHM (Digital Height Model). The DHM provides height information for terrain and buildings, which helps with object segmentation.

LoveDA Dataset: Sourced from Wuhan University, this dataset was constructed and made publicly available by researchers specializing in remote sensing image processing. Its uniqueness lies in its coverage of both urban and rural scenes, which enables it to address the complex issues arising from the differences in land cover between urban and rural areas. The dataset provides high-resolution remote sensing images with a resolution of 0.3 meters per pixel, allowing for the capture of fine details of surface objects. In total, the dataset includes 5987 images, divided into two categories: urban and rural. Specifically, it contains 2522 urban area images and 3465 rural area images. The distinct urban and rural scene variations in this dataset provide valuable generalization for evaluating semantic segmentation algorithms.

4.2. Experimental Details

Experiments were conducted within the PyTorch 2.3 framework on a single NVIDIA A40 GPU. The AdamW optimizer was chosen, with weight decay serving to improve generalization performance, prevent overfitting, and accelerate the training process. A cosine learning rate scheduling strategy is applied, with the learning rate initialized to $6\times {10}^{-4}$ and weight decay applied at 0.01. Since the original images on the Vaihingen and Potsdam datasets are relatively large and cannot be input directly into the model, we crop the images to 1024 × 1024. The LoveDA dataset does not require random cropping of the images. Training consisted of 105 epochs for Vaihingen and Potsdam and 45 epochs for LoveDA, reflecting the characteristics of the datasets: Vaihingen and Potsdam are relatively small, while LoveDA contains more diverse scenes and categories. These epoch settings are consistent with most baseline studies, ensuring fair comparison and the validity of our experiments. Standard data augmentation techniques were employed, including random scaling (0.5, 0.75, 1.0, 1.25, 1.5), horizontal and vertical flipping, and cropping to 512 × 512. During testing, Test Time Augmentation (TTA) with rotations and flips was applied to enhance model robustness and accuracy.

4.3. Evaluation Metrics

Evaluation of the proposed approach was conducted using three widely adopted metrics in remote sensing segmentation: Mean Intersection over Union ($mIoU$), Overall Accuracy ($OA$), and the F1 score. Their definitions are as follows:

$$mIoU={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{T{P}_i}{T{P}_i+F{P}_i+F{N}_i}}$$

(38)

where $T{P}_i$ represents the true positives for class i, $T{N}_i$ represents the true negatives for class i, $F{P}_i$ presents the false positives for class i, $F{N}_i$ represents the false negatives for class i, and N is the number of classes.

$$OA={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(39)

Computation of the F1 score requires first understand precision and recall, defined as follows:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(40)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(41)

where, $TP$, $FP$, $FN$, and $TN$ represent the numbers of true positive, false positive, false negative, and true negative predictions, respectively.

The calculation formula of F1 score is as follows:

$$F1=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(42)

4.4. Comparative Experiments

4.4.1. Comparative Experiments on the Vaihingen Dataset

The experimental comparison on the Vaihingen dataset is summarized in Table 1, showing that MPIFNet attained 92.32% for mF1, 94.05% for OA, and 86.03% for mIoU. Compared to SFFNet [26] using the Haar wavelet transform, our MPIFNet outperforms it by 0.65%, 0.54% and 1.12% in mF1, OA and mIoU, respectively. Compared to RS3Mamba [40] using the Mamba module, we achieve improvements of 1.29%, 0.63% and 2.15% in mF1, OA and mIoU, respectively. Furthermore, compared to ConvLSR-Net [41] using the same backbone network, our experimental results show improvements of 0.75%, 0.43%, and 1.27%. The results show that MPIFNet outperforms existing models in terms of overall performance. In Figure 5, we show the visualized segmentation results of the comparative experiments. To more clearly highlight the segmentation capability of our model, we performed a comparison with the original images. As seen in Figure 6a, compared to the label maps, our model better preserves detailed structures from the original images, particularly improving the segmentation of the car and impervious surface classes, while minimizing misclassification over large homogeneous areas. This also confirms that feature information from different perspectives can effectively assist the network in segmentation, thereby improving its segmentation capability. Figure 7 presents the overall segmentation maps for ID35 and ID38.

Table 1. Comparison of quantitative results on the Vaihingen dataset between our method and SOTA approaches. The best values in each column are highlighted in bold and underlined, while the second-best values are highlighted in bold. All metrics are reported as percentages (%).

Method	F1/IoU					mF1	OA	mIoU
	Imp.Surf.	Building	Low Veg.	Tree	Car
U-Net [8]	95.79/91.93	92.77/86.52	81.91/69.36	88.69/79.68	82.40/70.06	88.31	91.51	79.51
UNetFormer [42]	96.58/93.38	95.10/90.66	84.17/72.66	90.03/81.87	85.77/75.09	90.33	92.93	82.73
FT-UNetFormer [42]	97.04/94.25	96.17/92.63	84.96/73.85	90.42/82.51	87.81/78.28	91.28	93.62	84.30
A2-FPN [43]	96.78/93.76	95.56/91.51	84.63/73.36	90.08/81.95	88.47/79.32	91.10	93.26	83.98
MANet [44]	96.74/93.68	95.46/91.32	84.61/73.32	89.72/81.36	88.74/79.77	91.05	93.07	83.89
EIGNet [45]	96.59/93.42	95.26/90.95	84.17/72.67	89.84/81.55	87.36/77.57	90.64	92.94	83.23
CMTFNet [17]	96.61/93.45	95.15/90.76	84.93/73.80	89.87/81.61	86.36/76.00	90.58	93.04	83.12
CGGLNet [46]	96.87/93.93	95.53/91.45	84.25/72.79	90.26/82.26	88.31/79.07	91.04	93.28	83.90
ConvLSR-Net [41]	97.02/94.22	96.00/92.31	85.31/74.38	90.72/83.02	88.82/79.88	91.57	93.62	84.76
SFFNet [26]	97.06/94.30	95.75/91.84	85.06/74.00	90.14/82.05	90.33/82.36	91.67	93.51	84.91
E-PyramidMamba [47]	96.82/93.84	95.44/91.29	84.80/73.62	90.24/82.21	86.93/76.89	90.85	93.29	83.57
RS3Mamba [40]	96.83/93.86	95.79/91.92	85.04/73.98	90.28/82.29	87.22/77.35	91.03	93.42	83.88
PPMamba [24]	97.00/94.19	96.08/92.47	85.54/74.73	90.48/82.62	88.77/79.81	91.57	93.68	84.76
CPSSNet [23]	97.00/94.18	95.84/92.03	84.86/73.70	90.39/82.47	89.68/81.29	91.55	93.51	84.73
MPIFNet (ours)	97.35/94.84	96.49/93.21	85.89/75.27	90.79/83.14	91.11/83.58	92.32	94.05	86.03

Figure 5. Visualization of segmentation results on the Vaihingen dataset, where GT represents ground truth. The red square marks the area to be compared.

Figure 6. Comparison with the real label: (a) an image from the Vaihingen dataset; (b) an image from the Potsdam dataset. The red square marks the region for comparison, and the arrow points to its enlarged visualization. The colored blocks represent different semantic classes: white for impervious surface, blue for building, cyan for low vegetation, green for tree, yellow for car, and red for background.

Figure 7. Overall segmentation maps for ID 35 and ID 38 are shown in the figure.

4.4.2. Comparison of Parameter Count and Complexity

To examine model efficiency, we compared the number of parameters, computational cost, and inference speed of our approach with current state-of-the-art (SOTA) models. The results summarized in Table 2, show that our model achieves superior segmentation performance while maintaining a reasonable balance between complexity and speed. Although MPIFNet has more parameters than some lightweight methods, its performance gains clearly justify the added complexity, and it does not have the highest parameter count among the SOTA models. In terms of computational cost, MPIFNet remains highly competitive. It does not have the highest FLOPs among the compared models, and its inference time and FPS demonstrate a favorable balance between speed and accuracy. Compared with CPSSNet, which has the longest inference time of 77.1 ms and the lowest FPS of 13.0, MPIFNet is considerably faster with an inference time of 57.6 ms and an FPS of 17.4, while also achieving higher segmentation accuracy with an mIoU of 86.03% compared to 84.73%. Although lightweight models such as A²-FPN or E-PyramidMamba achieve higher FPS due to lower computational demand, MPIFNet provides substantially better segmentation performance. Overall, the increases in parameter count, FLOPs, and inference time are justified by the improvements in accuracy, demonstrating that MPIFNet maintains an effective balance between computational efficiency and segmentation performance across multiple evaluation metrics.

Table 2. Comparison of parameters and computation of different networks. The minimum values of Params, FLOPs and Inference Time are highlighted in bold and underline, as smaller values indicate better efficiency.

Method	Backbone	Params (M)	Flops (G)	Infernce Time (ms)	FPS	mIoU
U-Net [8]	ResNet18	31.0	193.4	16.0	62.3	79.51
UNetFormer [42]	ResNet18	11.7	12.0	5.9	170.3	82.73
FT-UNetFormer [42]	Swin-S	96.0	128.4	26.1	38.2	84.30
A2-FPN [43]	ResNet18	12.2	13.6	3.4	295.0	83.98
MANet [44]	ResNet18	35.9	54.5	10.7	93.4	83.89
EIGNet [45]	ResNet18	34.5	45.1	17.4	57.6	83.23
CMTFNet [17]	ResNet50	30.1	34.6	10.9	92.1	83.12
CGGLNet [46]	ResNet50	36.9	88.5	21.1	47.3	83.90
ConvLSR-Net [41]	ConvNeXt-S	68.1	71.1	32.3	30.9	84.76
SFFNet [26]	ConvNeXt-T	34.2	52.1	16.3	59.9	84.91
E-PyramidMamba [47]	ResNet18	28.8	19.1	3.8	264.3	83.57
RS3Mamba [40]	ResNet18	43.3	39.6	24.7	40.5	83.88
PPMamba [24]	ResNet18	21.7	23.1	22.1	45.3	84.76
CPSSNet [23]	ConvNeXt-T	31.9	28.8	77.1	13.0	84.73
MPIFNet (ours)	ConvNeXt-S	62.9	88.3	57.6	17.4	86.03

4.4.3. Comparative Experiments on the Potsdam Dataset

The comparative experimental outcomes on the ISPRS Potsdam dataset are summarized in Table 3, showing that mF1, OA, and mIoU reached 93.69%, 92.53%, and 88.36%, respectively. Similarly, compared to models using the same backbone network, our model outperforms them. When compared to similar approaches, our model also shows better performance. Our model outperforms existing SOTA methods according to mF1, OA, and mIoU metrics. Figure 8 presents the visual segmentation results from the comparative experiments. Figure 6b demonstrates the comparison with the ground truth label maps for this dataset. Similarly, our model captures more detailed information from the original images while minimizing misclassification over large areas. These observations indicate the model’s robust generalization ability. Figure 9 shows the overall segmentation results for ID 3_13 and 3_14.

Table 3. Comparison of quantitative results on the Potsdam dataset between our method and SOTA approaches. The best values in each column are highlighted in bold and underlined, while the second-best values are highlighted in bold. All metrics are reported as percentages (%).

Method	F1/IoU					mF1	OA	mIoU
	Imp.Surf.	Building	Low Veg.	Tree	Car
U-Net [8]	92.25/85.61	94.09/88.84	86.27/75.86	87.34/77.53	95.71/91.77	91.13	89.38	83.92
UNetFormer [42]	94.05/88.77	96.58/93.40	87.45/77.70	88.82/79.89	96.79/93.78	92.74	91.39	86.71
FT-UNetFormer [42]	94.43/89.44	96.96/94.11	88.41/79.24	89.77/81.43	96.49/93.23	93.21	91.97	87.49
A2-FPN [43]	93.97/88.62	96.37/93.00	87.49/77.76	88.62/79.56	96.21/92.71	92.53	91.31	86.33
MANet [44]	93.77/88.28	96.44/93.14	87.63/77.99	89.16/80.44	96.40/93.06	92.68	91.42	86.58
EIGNet [45]	93.68/88.11	96.35/92.97	88.29/79.03	89.30/80.66	96.71/93.64	92.86	91.41	86.88
CMTFNet [17]	93.86/88.43	96.15/92.58	88.13/78.78	89.13/80.39	96.63/93.48	92.78	91.49	86.73
CGGLNet [46]	94.21/89.06	96.90/93.99	88.08/78.70	89.21/80.52	96.73/93.66	93.02	91.80	87.19
ConvLSR-Net [41]	94.86/90.22	97.18/94.52	88.21/78.91	89.45/80.92	97.00/94.17	93.34	92.16	87.75
SFFNet [26]	94.62/89.80	96.90/93.99	88.24/78.95	89.22/80.54	96.93/94.05	93.18	91.97	87.47
E-PyramidMamba [47]	93.83/88.37	96.20/92.68	87.90/78.42	89.16/80.44	96.47/93.19	92.71	91.43	86.62
RS3Mamba [40]	94.49/89.56	96.62/93.46	87.90/78.41	89.17/80.46	96.65/93.52	92.97	91.79	87.08
PPMamba [24]	94.54/89.65	96.95/94.08	88.29/79.03	89.48/80.97	96.47/93.18	93.15	92.06	87.38
CPSSNet [23]	94.31/89.24	96.71/93.63	88.02/78.61	89.53/81.04	96.85/93.89	93.08	91.74	87.28
MPIFNet (ours)	95.09/90.64	97.49/95.11	88.77/79.80	90.05/81.90	97.08/94.33	93.69	92.53	88.36

Figure 8. Visualization of segmentation results on the Potsdam dataset, where GT represents ground truth. The red square marks the area to be compared.

Figure 9. Overall segmentation maps for ID $3\text{\_}13$ and ID $3\text{\_}14$ are shown in the figure.

4.4.4. Comparative Experiments on the LoveDA Dataset

The performance of MPIFNet was further validated using the LoveDA dataset. The results in Table 4 show an mIoU of 55.76%, outperforming previously reported methods. High segmentation accuracy was observed for the Building and Forest classes, with IoUs of 66.37% and 46.38%, respectively, and other classes also yielded satisfactory results. Overall, our method achieves the best average performance metrics. Visualization results are provided in Figure 10.

Table 4. Comparison of quantitative results on the LoveDA dataset between our method and SOTA approaches. The best values in each column are highlighted in bold and underlined, while the second-best values are highlighted in bold. All metrics are reported as percentages (%).

Method	IoU							mF1	OA	mIoU
	Background	Building	Road	Water	Barren	Forest	Agriculture
U-Net [8]	50.50	56.72	47.80	52.63	30.43	40.66	45.69	62.90	65.14	46.35
UNetFormer [42]	50.33	55.66	55.47	68.03	24.91	42.08	50.31	65.27	67.17	49.54
FT-UNetFormer [42]	55.48	62.10	56.03	72.26	33.34	43.12	57.00	69.51	72.00	54.19
A2-FPN [43]	52.25	56.15	52.04	70.14	34.08	42.43	51.60	67.13	68.89	51.24
MANet [44]	51.41	60.12	54.91	54.44	29.12	41.06	49.55	64.83	66.98	48.66
EIGNet [45]	52.39	58.29	52.40	62.85	38.83	38.40	48.89	66.49	67.96	50.29
CMTFNet [17]	53.79	62.67	54.89	70.14	30.80	41.34	53.86	67.99	70.39	52.50
CGGLNet [46]	54.33	65.95	54.08	69.00	31.74	38.90	52.10	67.78	70.35	52.30
ConvLSR-Net [41]	54.49	64.03	57.32	72.14	34.12	44.18	56.63	69.97	71.81	54.70
SFFNet [26]	54.37	64.22	56.56	68.68	32.95	44.66	52.02	68.87	70.71	53.35
E-PyramidMamba [47]	54.37	60.54	50.92	63.21	33.42	39.92	55.01	66.99	70.07	51.06
RS3Mamba [40]	53.31	57.73	52.50	66.40	37.39	38.31	53.62	67.29	69.45	51.33
PPMamba [24]	54.15	63.28	55.25	68.36	34.96	40.86	51.30	68.25	69.95	52.59
CPSSNet [23]	54.62	64.79	57.58	70.08	35.86	43.74	54.85	69.90	71.51	54.50
MPIFNet (ours)	55.56	66.37	56.89	71.99	37.04	46.38	56.11	70.97	72.43	55.76

Figure 10. Visualization of segmentation results on the LoveDA dataset, where GT represents ground truth. The red and blue square marks the area to be compared.

4.5. Ablation Experiments

4.5.1. Ablation Experiments of MPIFNet Components

To assess the effectiveness of the DBHWT and GLMBSA modules, ablation studies were performed by adding each module separately to the baseline, yielding two experimental configurations: baseline + DBHWT and baseline + GLMBSA. The corresponding results are presented in Table 5. Due to the plug-and-play nature of GLMBSA, it can be removed without affecting the structure after integrating DBHWT. When only the DBHWT module was applied, the model achieved performance gains on the Vaihingen dataset of 0.93% (mF1), 0.49% (OA), and 1.51% (mIoU), and on the Potsdam dataset of 0.55%, 0.60%, and 0.95%, respectively. This confirms the effectiveness of DBHWT in enhancing feature representation through multi-frequency information fusion. In another configuration, the DBHWT module was removed and replaced with standard skip connections, while GLMBSA was embedded. The model achieved even greater improvements: 1.00% (mF1), 0.46% (OA), and 1.62% (mIoU) on Vaihingen, and 0.96%, 1.08%, and 1.06% on Potsdam, respectively. These results demonstrate that GLMBSA significantly improves segmentation precision by effectively capturing global-local contextual dependencies.Although the introduction of both modules increases parameter count and computational cost, their combined use yields the most substantial performance gains. Specifically, the integration of both DBHWT and GLMBSA resulted in improvements of 1.37% (mF1), 0.69% (OA), and 2.27% (mIoU) on Vaihingen, and 1.00%, 1.25%, and 1.75% on Potsdam. These results demonstrate that the two modules complement each other and that their combined use improves the model’s robustness and generalization in semantic segmentation.

Table 5. Ablation Experiments on the Vaihingen and Potsdam datasets. Best values and ours in columns are shown in bold and underlined. All scores are expressed as a percentage (%).

Datasets	Method	Params (M)	Flops (G)	mF1	OA	mIoU
Vaihingen	Baseline	58.4	68.9	90.95	93.36	83.76
	Baseline + DBHWT	59.6	71.9	91.88	93.85	85.27
	Baseline + GLMBSA	61.7	85.3	91.95	93.82	85.38
	Baseline + GLMBSA + DBHWT	62.9	88.3	92.32	94.05	86.03
Potsdam	Baseline	58.4	68.9	92.69	91.28	86.61
	Baseline + DBHWT	59.6	71.9	93.24	91.88	87.56
	Baseline + GLMBSA	61.7	85.3	93.65	92.36	87.67
	Baseline + GLMBSA + DBHWT	62.9	88.3	93.69	92.53	88.36

4.5.2. Ablation of the GLMBSA Module

Ablation experiments were conducted to evaluate the capability of GLMBSA in capturing multiscale features. First, we removed the local feature extraction component and retained only global modeling, referred to as Global Mamba (GMamba). Second, we retained only the local perspective, denoted as Local Mamba (LMamba). Third, we removed the Block Self-Attention mechanism and directly fused global and local features from both branches, termed Global and Local Mamba (GLMamba). To assess multiscale recognition capability, we focused on segmentation performance for low vegetation (representing large-scale regions) and cars (representing small-scale objects). The results, summarized in Table 6, reveal distinct performance characteristics. As shown in the second row, LMamba improves car segmentation but degrades accuracy for larger regions like low vegetation, indicating limited global context. Conversely, the third row shows that simple fusion of global and local features (GLMamba) improves performance on large-scale categories, suggesting partial alleviation of scale-related trade-offs. Finally, the fourth row—embedding the Block Self-Attention mechanism to explicitly model interactions between global and local features—achieves the best results on both object types. This confirms that GLMBSA enhances multiscale representation and contributes to comprehensive performance gains.

Table 6. Effectiveness of the GLMBSA module. Best values are shown in bold and underlined.

Method	Params (M)	Flops (G)	Low Veg.	Car	mIoU
Baseline + GMamba	61.7	76.8	74.56	80.19	84.74
Baseline + LMamba	61.7	76.8	73.92	81.23	84.90
Baseline + GLMamba	61.7	84.7	75.15	80.82	85.13
Baseline + GLMBSA	61.7	85.3	75.24	81.66	85.38

4.5.3. Ablation of the DBHWT Module

Ablation experiments were performed by modifying the network structure to assess the effectiveness of the proposed Double-Branch Haar Wavelet Transform (DBHWT) module in feature extraction. Several structural variants were designed for comparison. When HWT is applied to a single-branch skip connection, the structure is termed One-Branch Haar Wavelet Transform (OBHWT). Conversely, when HWT is removed but the dual-branch structure is preserved, with two feature maps combined through element-wise addition, the variant is referred to as Double Branch (DB). The corresponding results are reported in Table 7. The DB configuration facilitates the integration of multi-level semantic features, enriching contextual representation and improving feature discrimination. Meanwhile, the incorporation of HWT enables frequency decomposition, allowing for more compact and informative encoding of structural details. Through the combined integration of these features, the DBHWT module captures additional semantic cues, leading to enhanced segmentation outcomes.

Table 7. Effectiveness of the DBHWT module. Best values are shown in bold and underlined.

Method	Params (M)	Flops (G)	mF1	OA	mIoU
Baseline	58.4	68.9	90.95	93.36	83.76
Baseline + DB	59.6	69.1	91.57	93.60	84.75
Baseline + OBHWT	59.2	70.7	91.61	93.69	84.83
Baseline + DBHWT	59.6	71.9	91.88	93.85	85.27

4.5.4. The Impact of the GLMBSA Module’s Position Embedding

Given the plug-and-play nature of the GLMBSA module, we explored its effectiveness when embedded at different stages of the network—specifically in the encoder, decoder, and both. In the encoder-only setting, the module was inserted after each block to evaluate its impact on high-level feature extraction, while a hybrid configuration placed the module in both the encoder and decoder. The corresponding results are shown in Table 8. As indicated by the second and third rows, inserting GLMBSA solely in the encoder yields limited performance gains, likely because the encoder focuses on abstract representation learning, where capturing long-range dependencies is less critical. In contrast, integrating GLMBSA into the decoder significantly improves performance, as the decoder is responsible for spatial reconstruction, where global-local context modeling proves more beneficial. Decoder-only placement achieves the highest mIoU (85.38%) with lower Params (61.7 M) and Flops (85.3 G), while encoder-only placement yields slightly lower mIoU (84.78%) with higher computational cost (70.6 M Params, 89.7 G Flops), highlighting the trade-off between segmentation performance and model complexity and underscoring the clear advantage of decoder placement. Interestingly, embedding the module in both stages leads to a slight performance drop, possibly due to increased model complexity causing overfitting, and redundant feature modeling introducing interference. Overall, these results suggest that placing GLMBSA in the decoder achieves the most effective balance between accuracy and efficiency, enhancing spatial detail recovery without incurring unnecessary overhead.

Table 8. Impact of GLMBSA embedding placement on model performance: encoder, decoder, and combined stages. Best values are shown in bold and underlined. Check mark indicates the module is included, whereas the hyphen indicates it is excluded.

Encoder	Decoder	Params (M)	Flops (G)	mIoU
-	-	58.4	68.9	83.76
✓	-	70.6	89.7	84.78
-	✓	61.7	85.3	85.38
✓	✓	73.9	106.1	84.58

4.5.5. Ablation of Mamba Depths

To investigate how the depth of the Mamba module affects model performance, we designed five configurations (Mamba × 0, × 1, × 2, × 4, and × 8), while keeping the remaining architecture consistent. Experiments were conducted on the Vaihingen dataset. Table 9 and Figure 11 show that both the parameter count and computational complexity (FLOPs) rose steadily with increasing module depth, from 59.6 M and 72.4 G for Mamba × 0 to 72.7 M and 135.9 G for Mamba × 8. However, performance metrics showed non-monotonic trends. Among all configurations, Mamba × 2 achieved the best results, with a mean F1 of 92.32%, OA of 94.05%, and mIoU of 86.03%, significantly outperforming the others. This can be attributed to the intrinsic properties of the Mamba block: its selective state space mechanism efficiently captures long-range dependencies, but excessive stacking introduces redundant global context and increases optimization difficulty, leading to diminished generalization. In contrast, Mamba × 2 provides sufficient depth to exploit the advantages of Mamba in modeling global dependencies while avoiding redundancy and overfitting, thereby striking the optimal balance between accuracy and efficiency.

Table 9. Comparison of computations for mamba blocks of different depths. Best values are shown in bold and underlined.

Depths	Params (M)	Flops (G)	mF1	OA	mIoU
Mamba × 0	59.6	72.4	91.99	93.88	85.45
Mamba × 1	61.2	80.4	92.18	93.95	85.76
Mamba × 2	62.9	88.3	92.32	94.05	86.03
Mamba × 4	66.2	104.2	92.10	93.90	85.65
Mamba × 8	72.7	135.9	92.16	93.91	85.75

Figure 11. Effect of Mamba Depths and Chunk Sizes on Model Performance.

4.5.6. Ablation of Self-Attention Chunk Sizes

To investigate how chunk size affects Block Self-Attention performance, we tested five configurations—H, H × 2, H × 4, H × 8, and H × 16—keeping the parameter count fixed at 62.9 M. As shown in Table 10 and Figure 11, testing on the Vaihingen dataset reveals that more partitions increase FLOPs slightly (88.3 G to 96.2 G) and reduce segmentation accuracy progressively. Specifically, the mean F1 score dropped from 92.32% (H) to 92.03% (H × 16), while mIoU decreased from 86.03% to 85.53%. These results suggest that excessive chunking compromises the model’s ability to capture long-range dependencies, thereby limiting contextual representation and semantic discrimination. In contrast, the configuration with chunk size H achieved the best trade-off between accuracy and efficiency, confirming the advantage of smaller chunk sizes for preserving global context and maintaining segmentation quality.

Table 10. Comparison of computations with different Self-Attention chunking size. Best values are shown in bold and underlined.

Sizes	Params (M)	Flops (G)	mF1	OA	mIoU
H	62.9	88.3	92.32	94.05	86.03
H × 2	62.9	88.8	92.14	93.97	85.72
H × 4	62.9	89.9	92.12	93.84	85.69
H × 8	62.9	92.0	92.07	93.88	85.60
H × 16	62.9	96.2	92.03	93.94	85.53

4.5.7. Ablation of Self-Attention Computation

To further explore the influence of different feature modeling strategies within Block Self-Attention on semantic segmentation performance, we conducted comparative experiments by applying Global Mamba (GMamba), Local Mamba (LMamba), or retaining the original features (denoted as X) to the Query (Q), Key (K), and Value (V) components. The results on the Vaihingen dataset, presented in Table 11, demonstrate that the choice of modeling strategy markedly affects performance, with mIoU values ranging from 85.48% to 86.03% across configurations. The best performance (mIoU of 86.03%) was achieved when Q employed LMamba, K adopted GMamba, and V remained unchanged (X). This suggests that using LMamba for Q enhances local spatial discrimination, while incorporating GMamba into K improves global contextual understanding. Retaining the original features in V appears beneficial for preserving stable semantic representations. Overall, these findings highlight that an appropriate combination of local and global modeling strategies can further enhance the effectiveness of Block Self-Attention, providing valuable guidance for future architectural design.

Table 11. Comparison of Self-Attention calculation for different combinations. Best values are shown in bold and underlined.

Query	Key	Value	mIoU
X	LMamba	GMamba	85.72
X	GMamba	LMamba	85.48
GMamba	X	LMamba	85.65
LMamba	X	GMamba	85.54
GMamba	LMamba	X	85.61
LMamba	GMamba	X	86.03

4.5.8. Ablation of Different Wavelet Bases

In the ablation study of different wavelet bases, we compared Haar with Daubechies, Symlet, Coiflet, and Biorthogonal wavelets. As shown in Table 12, the overall performance across different bases is relatively close, which demonstrates that our framework is not overly sensitive to the specific choice of wavelet. Nevertheless, Haar achieves the highest results, with an OA of 94.05% and mIoU of 86.03%. This improvement can be explained by the characteristics of the Haar wavelet. Its piecewise constant basis function is highly sensitive to abrupt changes in intensity, allowing it to better capture edges and sharp boundaries that are critical in remote sensing semantic segmentation tasks. In addition, the simplicity of Haar avoids the excessive smoothing effect that often occurs in smoother wavelets such as Daubechies, Symlet, or Coiflet, thereby preserving more discriminative high-frequency details. Finally, Haar provides complementary information to convolutional networks, since convolutional layers tend to emphasize smooth and low-frequency components, whereas Haar strengthens high-frequency edge features. These combined advantages explain why Haar delivers superior segmentation performance compared with other wavelet bases in our experiments.

Table 12. Comparison of computations with different Wavelet Bases. Best values are shown in bold and underlined.

Wavelet Bases	Params (M)	Flops (G)	mF1	OA	mIoU
Daubechies	62.9	88.3	92.16	93.86	85.76
Symlet	62.9	88.3	92.13	93.92	85.69
Coiflet	62.9	88.3	92.18	93.93	85.78
Biorthogonal	62.9	88.3	92.08	93.84	85.62
Haar	62.9	88.3	92.32	94.05	86.03

In addition to the quantitative results, we provide a qualitative comparison of the low-frequency components extracted by different wavelet bases, as shown in Figure 12. The figure visualizes the decomposed wavelet components corresponding to Daubechies, Symlet, Coiflet, Biorthogonal, and Haar. It can be observed that Haar preserves sharper and more distinct structural details, such as building edges and linear features, compared with the smoother wavelets, which produce more diffused low-frequency patterns. This visual evidence confirms that Haar not only captures critical edge information more effectively but also retains low-frequency texture details, complementing convolutional feature maps that tend to emphasize smooth, low-frequency components.

Figure 12. Visualization of Wavelet Low-Frequency Components.

4.5.9. Ablation of Different Backbone

To evaluate the effect of backbone architectures, we conducted experiments with ResNet, Swin, and ConvNeXt families. The experimental results are shown in Table 13. The lightweight ResNet18 (18.6 M/52.5 G) achieves the lowest performance with 91.45% F1 and 84.58% mIoU, while ResNet50 (31.3 M/66.6 G) improves the results to 92.07% F1 and 85.57% mIoU, confirming the benefit of increased network depth. Under comparable computational costs, modern architectures exhibit clear advantages: ConvNeXt-Tiny (41.2 M/66.2 G) reaches 92.14% F1 and 85.71% mIoU, outperforming ResNet50 despite similar complexity, and Swin-T (40.9 M/68.4 G) achieves a comparable 91.99% F1 and 85.46% mIoU, highlighting the effectiveness of attention-based designs. Among all tested models, ConvNeXt-Small (62.9 M/88.3 G) delivers the best overall results with 92.32% F1, 94.05% OA, and 86.03% mIoU, establishing a new performance benchmark. These findings demonstrate that our framework can effectively exploit the hierarchical features of modern high-capacity backbones, with ConvNeXt emerging as the most competitive choice under similar computational budgets.

Table 13. Comparison of computations with different Backbone. Best values are shown in bold and underlined.

Backbone	Params (M)	Flops (G)	mF1	OA	mIoU
ResNet18	18.6	52.5	91.45	93.55	84.58
ResNet50	31.3	66.6	92.07	93.72	85.57
Swin-Tiny	40.9	68.4	91.99	93.82	85.46
Swin-Small	62.2	92.2	92.00	93.87	85.47
ConvNext-Tiny	41.2	66.2	92.14	93.91	85.71
ConvNext-Small	62.9	88.3	92.32	94.05	86.03

4.5.10. Ablation of Partitioning Strategies in Local Mamba

To evaluate the effectiveness of different partitioning strategies in the local Mamba branch, we compared the default block-based partitioning with several sliding window configurations. As summarized in Table 14, sliding windows with sizes and strides of 32 × 32/16, 64 × 64/16, and 128 × 128/64 lead to increased computational costs, with FLOPs rising up to 101.8G, while providing only marginal improvements in accuracy. For instance, the best sliding window setting, 32 × 32/16, achieves an mIoU of 85.85%, which remains lower than the 86.03% obtained by the original block-based design. In contrast, our proposed method achieves the highest overall accuracy, with an mIoU of 86.03% and an OA of 94.05%, while maintaining a lower computational cost of 88.3 G FLOPs. These results indicate that the direct use of sliding windows does not substantially enhance segmentation performance and may compromise efficiency. By comparison, the tailored GLMBSA partitioning offers a more favorable trade-off between local detail extraction and computational efficiency, thereby validating the effectiveness of our design.

Table 14. Comparison of Computational Efficiency and Performance under Different Local Mamba Partitioning Strategies. Best values are shown in bold and underlined.

Partitioning Strategies	Window	Stride	Params (M)	Flops (G)	mF1	OA	mIoU
Sliding Window	32 × 32	12	69.2	97.7	92.21	93.94	85.85
Sliding Window	32 × 32	16	69.2	88.3	92.18	93.93	85.79
Sliding Window	64 × 64	16	69.2	101.8	92.10	93.89	85.65
Sliding Window	64 × 64	32	69.2	89.7	92.15	93.99	85.73
Sliding Window	64 × 64	64	69.2	85.9	92.19	94.01	85.80
Sliding Window	128 × 128	64	69.2	83.4	92.18	93.99	85.79
Block-based	-	-	62.9	88.3	92.32	94.05	86.03

4.5.11. Ablation of Different Noise Levels

To further assess the robustness of MPIFNet under noisy conditions, we conducted experiments by injecting Gaussian noise with different intensities, namely $\sigma$ = 0.01, 0.03, and 0.05, into the input data. We compared MPIFNet with two representative baselines, ConvLSRNet and CPSSNet, under identical settings. As summarized in Table 15, all models experience performance degradation as the noise level increases. Nevertheless, MPIFNet consistently outperforms the compared methods across all noise intensities in terms of F1, OA, and mIoU. For instance, under strong noise with $\sigma$ = 0.05, MPIFNet achieves an mIoU of 74.15%, exceeding the 71.34% of ConvLSRNet and the 71.55% of CPSSNet. Regarding the relative performance decline, MPIFNet shows a drop of 13.8%, which is smaller than the 15.8% observed in ConvLSRNet and the 15.6% in CPSSNet. These results confirm that the proposed multi-perspective feature integration effectively mitigates noise-induced degradation and ensures stable performance in practical scenarios.

Table 15. Comparison of computations with different Noise levels. Best values are shown in bold and underlined.

Method	Noise	Paramsb (M)	Flopsb (G)	mF1	OA	mIoU
MPIFNet (our)	0.00	62.9	88.3	92.32	94.05	86.03
	0.01	62.9	88.3	85.56	89.34	75.48
	0.03	62.9	88.3	85.54	89.93	75.52
	0.05	62.9	88.3	84.63	88.85	74.15
ConvLSR-Net	0.00	68.1	71.1	91.57	93.62	84.76
	0.01	68.1	71.1	83.62	87.54	72.61
	0.03	68.1	71.1	83.99	89.00	73.40
	0.05	68.1	71.1	82.60	87.34	71.34
CPSSNet	0.00	31.9	28.8	91.55	93.51	84.73
	0.01	31.9	28.8	83.73	87.15	72.51
	0.03	31.9	28.8	83.88	87.24	72.95
	0.05	31.9	28.8	82.83	87.65	71.55

5. Discussion

From a remote sensing perspective, MPIFNet contributes to existing research by integrating context-aware and frequency-aware modeling. The GLMBSA module captures both long-range contextual relationships and localized spatial details. Mamba-based architectures, including RS3Mamba, have shown strong ability to maintain semantic consistency across large-scale structures, such as urban blocks and road networks, but may struggle with boundary localization or small, sparse objects. Conversely, ConvLSR-Net emphasizes localized spatial refinement to mitigate such issues. Our findings suggest that combining long-range contextual awareness with mechanisms that preserve local continuity achieves more stable segmentation across object scales, particularly in heterogeneous, multiscale landscapes.

Complementing this, the DBHWT module introduces explicit frequency-domain decomposition, a widely adopted approach in remote sensing for separating coarse semantic structures from fine-grained boundary and texture information. Prior studies, such as SFFNet, demonstrate that emphasizing high-frequency components enhances boundary sharpness, particularly for objects with intricate outlines. Consistent with these findings, our results indicate that distinguishing low-frequency semantic structures from high-frequency details improves the discriminability of classes with subtle spectral differences or complex boundaries, such as low vegetation and impervious surfaces. By situating MPIFNet within this context, we demonstrate that it not only aligns with established practices but also addresses limitations observed in prior models, providing a balanced approach for diverse remote sensing scenes.

In terms of class-wise segmentation performance, MPIFNet benefits categories characterized by distinct structural patterns or rich textures. Extremely small or rare classes (e.g., narrow roads, scattered vegetation patches) remain challenging due to inherent sample imbalance, a common issue in remote sensing segmentation. Importantly, even modest improvements in IoU (1–3%) can have practical significance in operational scenarios. For instance, a 1% increase in road segmentation accuracy can substantially improve topological continuity, ensuring that road junctions and narrow linear features are correctly connected. This observation is consistent with findings reported in recent studies, such as UX-Net [48], which emphasize the importance of feature continuity for sparse linear structures. To support reproducibility and practical applicability, all training protocols, preprocessing steps, and evaluation metrics follow widely accepted standards in the remote sensing community, and the model consistently performs across Vaihingen, Potsdam, and LoveDA datasets, demonstrating adaptability to variations in spatial resolution, scene composition, and urban–rural environments.

Despite its strengths, several limitations remain. The relatively deep hierarchical structure and multi-branch design increase computational overhead, potentially limiting deployment on real-time or resource-constrained platforms. In addition, MPIFNet has not been thoroughly evaluated on extremely high-resolution images (e.g., sub-meter resolution), and its robustness under seasonal variations and across different sensors remains unclear. Standard benchmark datasets alone are insufficient to examine these practical challenges.

To provide a preliminary assessment beyond standardized benchmarks, we tested MPIFNet on a single remote sensing image obtained from publicly available online sources, which better represents operational scenarios. By partitioning the image into subregions, we illustrated both overall segmentation results and fine-grained local patterns, as shown in Figure 13. Since ground-truth annotations are unavailable, quantitative evaluation is not possible; nevertheless, full-resolution inference shows that MPIFNet maintains stable boundary delineation and coherent class predictions, demonstrating strong robustness and generalization.

Figure 13. MPIFNet segmentation results on a real-world remote sensing image, illustrating overall and local structure. The colored blocks represent different semantic classes: white for impervious surface, blue for building, cyan for low vegetation, green for tree, yellow for car, and red for background.

Future work will address both model and application-level limitations. On the model side, we plan to develop lightweight MPIFNet variants to reduce computational costs, incorporate class-balanced learning and synthetic data augmentation to improve performance on underrepresented categories, and explore domain adaptation and multimodal fusion to enhance cross-season and cross-sensor generalization. On the application side, we will establish evaluation pipelines combining large-area batch inference, semi-automatic annotation, and scenario-driven simulation to systematically assess runtime performance, scalability, and deployment stability, bridging the gap between benchmark evaluation and real-world operational scenarios.

6. Conclusions

In this work, we revisit the long-standing challenge of jointly capturing global contextual relations and preserving fine-grained structural details in high-resolution remote sensing semantic segmentation. While existing transformer- and CNN-based approaches address these aspects separately, they often struggle to achieve a balanced representation capable of handling large object scale variations and subtle boundary transitions. Our proposed MPIFNet addresses this by integrating spatial dependency modeling and multi-frequency feature decoupling within a unified architecture. Experimental results across the Vaihingen, Potsdam, and LoveDA datasets demonstrate that the synergy between the GLMBSA and DBHWT modules effectively enhances both semantic coherence and structural precision. Beyond performance gains, our findings underscore the importance of jointly leveraging spatial, hierarchical, and frequency information to construct a more comprehensive feature representation for complex urban scenes. The consistent improvements observed in both ablation studies and benchmark comparisons further validate the robustness and generalizability of our design. Moreover, this study provides insight into how spatial state models and wavelet-based frequency decomposition can be synergistically embedded into modern segmentation networks—a paradigm that may inspire future research on multi-perspective representations, particularly for tasks involving highly heterogeneous or visually ambiguous regions.