Link Aggregation for Skip Connection–Mamba: Remote Sensing Image Segmentation Network Based on Link Aggregation Mamba

Abstract

The semantic segmentation of satellite and UAV remote sensing imagery is pivotal for address exploration, change detection, quantitative analysis and urban planning. Recent advancements have seen an influx of segmentation networks utilizing convolutional neural networks and transformers. However, the intricate geographical features and varied land cover boundary interferences in remote sensing imagery still challenge conventional segmentation networks’ spatial representation and long-range dependency capabilities. This paper introduces a novel U-Net-like network for UAV image segmentation. We developed a link aggregation Mamba at the critical skip connection stage of UNetFormer. This approach maps and aggregates multi-scale features from different stages into a unified linear dimension through four Mamba branches containing state-space models (SSMs), ultimately decoupling and fusing these features to restore the contextual relationships in the mask. Moreover, the Mix-Mamba module is incorporated, leveraging a parallel self-attention mechanism with SSMs to merge the advantages of a global receptive field and reduce modeling complexity. This module facilitates nonlinear modeling across different channels and spaces through multipath activation, catering to international and local long-range dependencies. Evaluations on public remote sensing datasets like LovaDA, UAVid and Vaihingen underscore the state-of-the-art performance of our approach.

1. Introduction

With the continuous advancements in sensor technology, urban remote sensing imagery captured by satellites and unmanned aerial vehicles (UAVs) now exhibits higher resolution and greater detail. The classification of geographic features in remote sensing images, namely pixel-level semantic segmentation, has extensively influenced various urban planning applications such as change detection [1,2,3], environmental monitoring [4,5,6,7], cartography [8,9] and traffic surveillance [10,11]. Recent studies have predominantly focused on deep learning frameworks based on convolutional neural networks [12,13,14,15,16,17] and transformers [18,19,20,21,22,23,24], employing extensive datasets for supervised learning to generalize applications to real-world scenarios.

Compared to traditional image segmentation algorithms such as region growing [25,26], fuzzy set theory [27] and conditional random fields [28,29], deep-learning-based methods excel in capturing the global and local context of input images through progressively expanding receptive fields. They utilize data-driven approaches to delve deeply into expert prior knowledge, forming an end-to-end segmentation paradigm. The primary frameworks are divided into CNNs and attention mechanisms. CNN-based segmentation networks continually stack two-dimensional convolutional kernels to build fully convolutional feature reconstruction networks that map images to masks, exemplified by architectures like U-Net [15] and PSPNet [16]. Attention mechanisms capture the long-range dependencies of images globally or locally to reconstitute the relationships between spatial and channel contexts, thus focusing on semantic responses, as seen in models like Segmenter [30]. Recent studies indicate that segmentation networks combining CNNs with self-attention mechanisms leverage the strengths of both, serializing the representational outcomes of CNNs and using transformers to construct spatial relationships between tokens for more precise semantic localization and boundary delineation, with ViT [31] and UnetFormer [32] being the most representative models.

Remote sensing images, due to their complex and variable land cover scenes, multispectral channels and high spatiotemporal resolution, pose significant challenges to the representational capacity of segmentation models. Pure CNN-based approaches can capture fine-grained local context but are constrained by their limited receptive fields, making it difficult to learn complex representations. This limitation prevents them from fully capturing cross-regional semantic information in irregular large-scale land scenes. Although some methods employ global pooling modules to supplement missing panoramic class information, these modules can interfere with the recognition of fine-grained local details. Transformer models, through self-attention mechanisms, compute sequential feature responses and construct long-range dependencies, enabling the accurate mapping of cross-class geographical positions. However, as the input sequence length or network depth increases, the fully connected multi-head self-attention mechanism incurs higher computational complexity, greatly limiting both model efficiency and depth. This constraint becomes especially problematic when processing high-resolution remote sensing images, as computational resources and efficiency are strained. Recent studies have combined CNNs and transformers in hybrid architectures [33,34,35] to mitigate the limitations of individual modules, but regardless of whether these modules are sequentially or interleaved, the computational cost remains high, and the generalization and robustness on small-scale datasets or sparse features are limited. On the other hand, cross-scale connections, which fuse low-level spatial details from the encoder with high-level semantic information from the decoder, enable better handling of objects with complex boundaries, facilitating both localization and detail refinement. As a result, they are widely used in remote sensing and medical segmentation. However, current connections simply concatenate features at matching scales, which introduces low-dimensional textures into high-dimensional features, placing additional pressure on the decoder’s representational capacity.

The state-space model (SSM) establishes long-range dependencies through state transitions and utilizes convolutional computations to perform these transitions, achieving near-linear computational complexity. Mamba [36] enhances efficiency during training and inference by incorporating time-varying parameters and constructing standard SSM modules. Vim [37] and VMamba [38] extend Mamba technology from processing one-dimensional sequential data to two-dimensional visual domains, where they have been proven to outperform transformer architectures in performance and various downstream tasks. Jamba [39] has innovatively restructured the hybrid expert architecture by interweaving transformers with Mamba layers, enabling flexible parameter management tailored to various large-scale foundational language tasks. HC-Mamba [40] integrates depthwise separable convolution and dilated convolution into Mamba, creating the hc-ssm module to capture a broader range of contextual information, thereby processing large-scale medical image data at a lower computational cost. MxT [41] enhances pixel-level interactive learning through the complementary synergy of Mamba and transformers, improving the model’s performance in high-quality image reconstruction.

This paper proposes a U-Net-like network for remote sensing image segmentation. The overall framework integrates two efficient state-space feature processing models, LASC-Mamba and Mix-Mamba, within an architecture based on a ResNet18 encoder and a symmetric decoder with multi-head attention mechanisms. Specifically, LASC-Mamba serializes the output from the ResNet18 feature layers, applies positional encoding and forms global modeling through shared parameters. It adaptively facilitates cross-resolution interactions of land cover features via cascading, thereby avoiding the limitations of the direct concatenation of single-scale features. The Mix-Mamba module reconstructs semantics by combining local convolutional features, global semantic multi-head attention and Mamba branches with the corresponding scale’s encoder outputs. This enhances the decoder’s ability to integrate multi-dimensional information from both the LASC-Mamba output and the previous feature outputs, capturing information invariant to diverse land covers. The two Mamba modules are inserted into the network’s fusion and decoding layers, achieving more complex mask mapping through cross-scale and multi-path representations.

Our principal contributions are as follows:

• We propose a SSM-based link aggregation method, LASC-Mamba, to facilitate semantic exchange between the encoder and decoder. This approach enhances the cross-scale representation capability during the skip-layer stages in remote sensing segmentation networks.
• During the decoding phase, a dual-pathway Mix-Mamba is employed to activate insensitive information across different spatial and sequential dimensions, thereby augmenting the capability for high-level image understanding.
• Comparative and ablation experiments are conducted on two public remote sensing image segmentation datasets. The results demonstrate that our method outperforms traditional CNN- and transformer-based segmentation approaches.

2. Related Work

In recent years, the advancement of semantic segmentation in remote sensing imagery has benefited from iterative updates to transformer architectures and U-net-like networks. Recent studies have introduced Mamba architecture to the field of computer vision.

2.1. U-Net-like Network

The U-Net network, with its symmetrical encoder–decoder architecture, was introduced by [15] to extract multilevel features by progressively downsampling spatial resolution during the encoding phase and gradually restoring resolution through skip connections in the decoding phase, and has been widely used for high-precision semantic segmentation. Several works, such as Res-UNet [42], Dense-UNet [43], UNet++ [44] and UNet3+ [45], have improved U-Net performance. MFU-Net [46] integrates a new multi-scale feature extraction module that captures shallow information across different scales. TreeUNet [47] utilizes a confusion matrix to construct tree–CNN blocks that merge multi-scale features and adaptively enhance pixel-level classification accuracy. Reunite-a [42] expands the U-Net baseline with a rich mix of optimization strategies, including residual connections, pyramid scene parsing, pooling and multitask inference. However, U-Net based on convolutional neural networks is limited by the convolutional kernel’s receptive field and lacks the capability for holistic image modeling. Many researchers have turned to transformers. Chen et al. [20] introduced TransUNet, which replaces CNNs with a multilayer transformer at the encoding layer but retains a CNN architecture at the decoding layer. Cao et al. [19] proposed Swin-Unet, a purely transformer-based U-Net-like network. Wang et al. [32] introduced UNetFormer, employing it as a decoder to mine image information and implementing an efficient global–local attention mechanism. Although transformers enhance the capacity to handle long-range dependencies, they possess quadratic complexity, which is costly for image segmentation tasks that typically feature high resolution.

2.2. Hybrid Architecture of CNN and Transformer

To address the global semantic deficiencies of CNNs and reduce the computational cost of transformers, recent research has relied on hybrid architectures combining CNNs with transformers. For instance, DETR [48] utilizes CNN-extracted low-resolution feature maps, reshaping them into feature sequences and applying positional encoding. This allows the transformer to linearly represent the output sequence, reducing the input size to the transformer while enhancing model learning speed and overall performance. However, compared to region-based feature extraction models, DETR, based on self-attention mechanisms, is less sensitive to small object details. In contrast, ViT-FRCNN [49] serially integrates faster R-CNN [50] with ViT, reconstructing the object detection network by reconfiguring ViT’s linear output into a two-dimensional feature input for the convolutional residual layers in the detection phase, enabling image object detection with transformers. MobileViT [51] and MobileViTV2 [35] achieve an alternating integration of CNNs and transformers through the design of lightweight MobileViT modules and inverted residual blocks, leveraging inductive bias to share sequential and spatial positional information of image features. STT [34] constructs a dual-path transformer architecture that uses a sparse token sampler to learn long-range dependencies in both spatial and channel dimensions in parallel, improving building extraction accuracy while reducing time costs. However, the effectiveness of this model on remote sensing datasets is constrained by the number of sparse token samplers and requires extensive experimentation to accumulate tuning knowledge. These approaches rely on various mixed modes of convolutional and self-attention mechanisms but still do not address the transformer’s insensitivity to local information in high-dimensional semantic decoding stages. Additionally, compared to pure convolutional models, these hybrid models exhibit limited generalization performance on small-scale datasets and sparse information.

2.3. Mamba Structure

The state-space model (SSM) is a foundational scientific model in modern control theory of linear definite systems that maps a one-dimensional function or sequence $x\left(t\right)\in \mathbb{R}$ to an output $y\left(t\right)\in \mathbb{R}$, and can be represented by the following linear ordinary differential equation (ODE):

$$h^{\prime }\left(t\right)=Ah\left(t\right)+Bx\left(t\right)$$

(1)

$$y\left(t\right)=Ch\left(t\right)$$

(2)

where $y\left(t\right)\in \mathbb{R}$ is from input signal $x\left(t\right)\in \mathbb{R}$ and implicit latent state $h\in {\mathbb{R}}^N$. $A\in {\mathbb{R}}^{N\times N}$ denotes the state transition matrix, and $B\in {\mathbb{R}}^{N\times 1}$ and $C\in {\mathbb{R}}^{N\times 1}$ are the projection matrix, representing its parameters. However, the method has prohibitive computational and memory requirements compared to the equivalent CNNs, which limits its application to deep learning tasks. Structured state-space sequence models (S4) [52] utilize a high-order polynomial projection operator (HIPPO) [53] to impose structured forms on state transition matrix A, building a deep sequence model with efficient remote reasoning capabilities. Recent work, Mamba [36], employs an input-dependent selection mechanism to filter non-valuable information from the input and develops a hardware-aware algorithm, increasing speed. In addition, Mamba demonstrates state-of-the-art performance and significant computational efficiency by merging SSM blocks with linear layers.

Overall, MAMBA and S4 implemented the continuous system described in Equations (1) and (2) in a discrete form and integrated it into a deep learning approach. A and B are discretized by a zeroth-order hold (ZOH) with the time scale parameter $\Delta$. The process is shown below:

$$\overline{\mathbf{A}}=exp(\Delta \mathbf{A})$$

(3)

$$\overline{\mathbf{B}}={(\Delta \mathbf{A})}^{-1}(exp(\Delta \mathbf{A})-\mathbf{I})·\Delta \mathbf{B}$$

(4)

After discretization, the state-space model (SSM) can be further computed in two ways: a linear recurrence relation (Equations (5) and (6)) and a global convolution operation (Equations (7) and (8)).

$$h^{\prime }\left(t\right)=\overline{\mathbf{A}}h\left(t\right)+\overline{\mathbf{B}}x\left(t\right)$$

(5)

$$y\left(t\right)=\mathbf{C}h\left(t\right)$$

(6)

$$\overline{K}=(\mathbf{C}\overline{\mathbf{B}},\mathbf{C}\overline{\mathbf{AB}},\dots ,\mathbf{C}{\overline{\mathbf{A}}}^{L-1}\overline{\mathbf{B}})$$

(7)

$$y=x\ast \overline{\mathbf{K}}$$

(8)

where $\overline{\mathbf{K}}\in {\mathbb{R}}^L$ denotes a structured convolutional kernel and L is the length of the input sequence.

Recent research has applied Mamba to various downstream tasks in remote sensing imagery. Xiao [54] designed a frequency-assisted Mamba framework (FMSR) that integrates a frequency selection module, a visual state-space module and a hybrid gate module. This framework complements the global and local dependencies of low-resolution features in super-resolution tasks while reducing computational cost and complexity. CDMamba [55] introduced the scaled residual ConvMamba (SRCM) to address Mamba’s shortcomings in remote sensing change detection, such as the lack of detail and challenges in fine detection, dynamically enabling dual-temporal feature guidance. Zhu [56] replaced the multi-head self-attention mechanism in ViT with the Mamba-based Samba module as an encoder structure, combining it with UperNet’s decoder for efficient multi-level semantic information extraction in remote sensing imagery.

3. Method

In this section, we present the design of a U-net-like network for remote sensing image segmentation, incorporating two Mamba modules. Section 3.1 outlines the overall architecture of the proposed network, while Section 3.2 and Section 3.3 provide detailed descriptions of the implementation processes and underlying principles of LASC-Mamba and Mix-Mamba, respectively.

3.1. Network Architectures

The overall structure of the remote sensing image segmentation network developed in this paper is depicted in Figure 1. In the encoder phase, we initially utilize the ResNet18 network as the feature extraction layer, constructing convolutional outputs at varying scales through four downsampling stages. Subsequently, in the skip connection stage of the UnetFormer architecture, outputs from four different scale intermediate layers are linearly reconstructed using the Mamba architecture, and the reconstructed results are aggregated and linked to the corresponding fusion layer of the encoder. Finally, in the decoding phase, Mamba modules are configured in parallel on each global–local transformer module, merging the two structures to share dependencies.

3.2. Link Aggregation for Skip Connection Mamba (LASC-Mamba)

Mamba has achieved unprecedented results in the serialization of discrete data sequences. Given that an image represents two-dimensional discrete data sampled from a continuous signal, the experience with transformers suggests that two-dimensional input data can be transformed into one-dimensional sequences using encoded embeddings. Therefore, we capitalize on Mamba’s linear scaling advantage to enhance UnetFormer’s ability to perform contextual reconstruction during skip connection processes. The LASC-Mamba skip connection structure, constructed between the CNN encoder and decoder, is illustrated in Figure 2. The architecture comprises multi-stage Mamba state-space encoding, link aggregation and feature restoration layers.

In the structure discussed in this paper, Mamba extends two-dimensional image features into one-dimensional sequential features while employing a dual-pathway linear encoding structure to capture long-range dependencies. As shown in the first branch of Figure 2, given the output features $I\in {\mathbb{R}}^{B\times C\times H\times W}$ from a convolutional layer, we directly flatten these into a vector $I\in {\mathbb{R}}^{B\times L\times C}$, where B denotes the batch size, L represents the feature length, C indicates the channel count and $L=H\times W$. After layer normalization and processing through linear layers in two parallel branches, the feature dimensions expand to (B, 2L, C). In the first branch, the output of the linear layer undergoes processing through a one-dimensional convolutional layer, a SiLU activation function [57] and an SSM layer, while the second branch utilizes only the SiLU activation function. Subsequently, the features from both branches are combined via the Hadamard product and dimensionally reduced in the final linear layer. Ultimately, the features are reshaped to the input dimensions (B, C, H, W). The Mamba module constructed in this section can be described as follows:

$$Mamba\left(I\right)=\mathcal{L}\left(SSM\right(\sigma \left(Conv\right(\mathcal{L}\left(I\right)\left)\right))\circ \sigma (\mathcal{L}\left(I\right)\left)\right)$$

(9)

where $\sigma$ denotes the SiLU activation function, $SSM$ represents the state-space model and ∘ denotes the Hadamard product operation performed between the former and the latter. $\mathcal{L}$ denotes the linear layer and $Conv$ denotes the one-dimensional convolutional layer.

In the UnetFormer, the link aggregation structure aggregates Mamba output features at four scales, corresponding to multilevel resolutions, allowing each stage’s path output to share global information from other links. The approach involves maintaining the same B and L while altering the channel count for stacking. To ensure that the final aggregated dimensions are multiples of the original stage dimensions, outputs at one-eighth size are duplicated. The final dimensions of the link aggregation layer are (B, L/16, C × 32). Channel reduction is performed through convolution at each stage to (B, L, C), followed by a linear transformation to restore the features to two-dimensional form (B, C, H, W). In order to mitigate potential noise disturbances and incomplete information from a single Mamba architecture, the original skip connections in UnetFormer are preserved, and the link aggregate output for each stage is represented as follows:

$$O_i=I_i+\delta (Conv(\sum _{i=1}^{4}Mamba(LN(I_i))))$$

(10)

where $O_i$ denotes the skip connection output at the i-th stage, $I_i$ denotes the feature input at the i-th stage, ${\sum }_{i=1}^4$ denotes link aggregation from the first to the fourth stage outputs, $\delta$ denotes linear scale transformation and $Conv$ refers to a one-dimensional convolution with a kernel size of 1 and a stride of 1.

3.3. Mix-Mamba

Our mix module implements a parallel Mamba module design based on the global–local transformer self-attention mechanism. It integrates global window semantic information, local two-dimensional feature information and state-space sequence information. This design addresses limited self-attention receptive fields and suboptimal modeling capabilities. The overall structure of mix mamba is shown in Figure 3.

In the architectural details, convolutional layers execute the local attention module parallel to kernel sizes 1 and 3. This configuration considers that larger kernel sizes could impede computational efficiency. Subsequently, batch normalization is applied, and the output results are summed. The structure can be represented as follows:

$$LB\left(I\right)=\beta \left(Con{\nu }_1\left(I\right)\right)+\beta \left(Con{\nu }_3\left(I\right)\right)$$

(11)

where $\beta$ denotes the batch normalization operation while $Con{v}_1$ and $Con{v}_2$ represent two-dimensional convolutions with kernel sizes of 1 and 3, respectively.

On the other hand, the global attention mechanism module constructs the consistency of the overall remote sensing scene through a window-based multi-head self-attention mechanism. Contextual information from two-dimensional windows is computed and fused separately via a cross-window context interaction module. The structure is as follows:

$$GB(I)=avepoo{l}_1(\delta ({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}})V)+avepoo{l}_2(\delta ({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}})V)$$

(12)

where $avepoo{l}_1$ and $avepoo{l}_2$ denote average pooling across two different dimensions, while $\delta$ denotes the $softmax$ function.

The Mamba module used in the Mamba branch is consistent with the previous section, featuring excellent spatial localization capabilities. Through the SSM, this architecture selectively compresses two-dimensional features to address content awareness of the input information. This parallel and hierarchical design enables the segmentation network’s decoder to capture global and local contexts across multiple scales while maintaining high efficiency. Consequently, the output of Mix-Mamba can be represented as follows:

$$Mix\left(I\right)=LB\left(I\right)+GB\left(I\right)+Mamba\left(I\right)$$

(13)

The Mix-Mamba module introduced in this section incorporates a multi-head self-attention mechanism, convolutional local feature extraction and Mamba global state-space mapping. Combining these three components’ advantages, it autonomously shares the necessary sparse features when addressing complex and variable geographical environments. These complementary sparse features restore essential contextual information, producing refined geographical mapping results.

4. Experiments

We conducted comparative and ablation experiments on the LoveDA and UAVid datasets to validate the effectiveness of LASC-Mamba and Mix-Mamba. Initially, we described the characteristics and training details of these two public datasets. Subsequently, we compared our model with advanced semantic segmentation algorithms from recent years to evaluate its performance. Finally, ablation studies were performed on the two Mamba modules to substantiate our theoretical claims.

4.1. Dataset

UAVid: The UAVid dataset [58] is a fine-resolution unmanned aerial vehicle semantic segmentation dataset for urban street scenes. The dataset comprises 42 sequences, totaling 420 individual images, across eight semantic categories. The images are divided into three subsets: a training set with 200 images, a validation set with 70 images and an official test set with 150 images for benchmarking purposes. Following [32], we preprocess each image from the dataset by padding it and cropping it into eight patches with 1024 × 1024 pixels.

LoveDA: The LoveDA dataset [59] contains high-quality optical remote sensing imagery (GDS 0.3 m) of three cities, divided into rural and urban scenarios covering seven landcover classes: background, building, road, water, barren, forest and agricultural. The dataset consists of 5987 images with a resolution of 1024 × 1024. The train set comprises 2522 images, the validation set comprises 1669 images and the test set is provided by the official government and contains 1796 images.

Vaihingen: The Vaihingen dataset consists of 33 TOP image tiles with a very fine spatial resolution, mainly depicting urban scenes. However, the images have three bands NIR-R-G with an approximate spatial resolution of 8 cm. We follow Unetformer, and selected the designated 16 images for training and the remaining 17 for testing.

We processed the UAVid dataset into 1024 × 1024 input images and applied data augmentation during training with random vertical flips, horizontal flips and brightness adjustments. During testing, test-time augmentation strategies such as vertical and horizontal flips were employed.

For the LoveDA and Vaihingen datasets, images were randomly cropped to 512 × 512 patches. Training involved random scaling with ratios of 0.5, 0.75, 1.0, 1.25 and 1.5, along with data augmentation techniques such as random vertical flips, horizontal flips and rotations. In the testing phase, multi-scale and random flip augmentations were utilized.

4.2. Training Detail

We implemented the LASC-Mamba and Mix-Mamba modules on the UNetformer baseline using the PyTorch framework. All experiments were conducted on a single NVIDIA RTX 4090 GPU. The proposed method utilized the AdamW optimizer [60] with a weight decay of 0.01, a batch size of 8 and 100 epochs. The learning rate was adjusted using the CosineAnnealingLR strategy to prevent overfitting during training. The initial learning rate for the pre-trained ResNet18 backbone was set to $6e^{-5}$, while other modules started at $6e^{-4}$ to optimize more quickly to the same gradient as the backbone. The final learning rate was gradually reduced to its minimum at the max epoch, ensuring smooth parameter updates and improved model generalization.

4.3. Comparison with Other Methods

To validate the efficacy of the proposed method, we conducted both qualitative comparisons and quantitative evaluations. We compared the approach described in this paper with various methods and generated visualizations for analysis. The comparative methods can be broadly categorized into three groups: CNN-based semantic segmentation networks such as Banet [61], ABCNet [62] and MANet [63], and transformer-based networks like TransUNet [20], Segmenter [30], CMTFNet [64], DC-Swin [21] and UnetFormer [32].

Banet [61]: Building on the foundation of visual attention mechanisms, BANet incorporates both forward and backward attention modules. By constructing two feature extraction pathways and employing a feature aggregation module, BANet effectively integrates local and global information, thereby resolving ambiguities.

ABCNet [62]: ABCNet constructs an attention-based dual-path lightweight network with spatial and contextual pathways, thereby reducing the high computational cost associated with fine-grained spatial details and large receptive fields.

MANet [63]: MANet employs an inter- and intra-region refinement (IIRR) method to reduce feature redundancy caused by fusion while constructing a multi-scale collaborative learning (MCL) framework to enhance the diversity of multi-scale feature representations. This approach leverages the correlations between different scales to effectively learn multi-scale features.

TransUNet [20]: TransUNet enhances the U-Net model by introducing a hybrid encoder that combines CNNs and transformers. This integration addresses the limitations of traditional convolutional neural networks in modeling long-range dependencies and processing large-scale images, thereby improving the handling of long-range dependencies, capturing semantic information in images and enhancing the model’s representational capacity and generalization performance.

Segmenter [30]: This network builds on the latest vision transformer (ViT) research, segments images into patches and maps them to a sequence of linear embeddings, which are then encoded by the encoder. The mask transformer decodes the outputs from the encoder and class embeddings, and, after upsampling, applies Argmax to classify each pixel, producing the final pixel segmentation map.

CMTFNet [64]: CMTFNet constructs a transformer decoder based on a multi-scale multi-head self-attention module to extract rich multi-scale global context and channel information. On one hand, the transformer block introduces an efficient feed-forward network (E-FFN) to enhance information interaction across different feature channels. On the other hand, the multi-scale attention fusion (MAF) module effectively integrates feature information from different levels.

DC-Swin [21]: DC-Swin incorporates the Swin-transformer as its encoder and features a densely connected convolutional decoder designed for high-resolution remote sensing image segmentation.

UNetFormer [32]: UNetFormer employs the lightweight ResNet18 as its encoder and has developed an effective global–local attention mechanism to model both global and local information within the decoder.

CM-Unet [65]: CM-UNet integrates the CSMamba module, which is based on spatial and channel attention activation, with a multi-scale attention aggregation module. This effectively captures long-range dependencies and multi-scale global context information in large-scale remote sensing images.

RS3Mamba [66]: RS3Mamba leverages the visual state-space (VSS) model to construct an auxiliary branch and introduces a collaborative completion module (CCM) to enhance and fuse features from dual encoders.

Quantitative Results: The quantitative results of the comparative experiments constructed in this paper are presented in

Table 1. Quantitative comparison results on the LoveDA test set with recent research methodologies. The best values in each column are highlighted in bold.

Method	Background	Building	Road	Water	Barren	Forest	Agriculture	mIoU
ABCNet	41.8	56.6	50.7	77.1	14.9	45.2	54.2	48.6
MANet	41.8	55.1	53.4	75.5	14.6	44.4	55.1	48.5
BANet	43.7	51.5	51.1	76.9	16.6	44.9	62.5	49.6
TransUNet	43.0	56.1	53.7	78.0	9.3	44.9	56.9	48.9
Segmenter	38.0	50.7	48.7	77.4	13.3	43.5	58.2	47.1
A2FPN	42.7	57.6	54.3	78.0	14.1	45.0	54.9	49.5
DC-Swin	43.3	54.3	54.3	78.7	14.9	45.3	59.6	50.0
UNetFormer	44.7	58.8	54.9	79.6	20.1	46.0	62.5	52.4
CM-Unet	54.6	64.1	55.5	68.1	29.6	42.9	50.4	52.2
RS3Mamba	39.7	58.8	57.9	61.0	37.2	39.7	34.0	50.9
Ours	46.1	58.4	55.6	79.9	19.2	47.6	63.2	52.9

,

Table 2. Quantitative comparison results on the UAVid set with recent research methodologies. The best values in each column are highlighted in bold.

Method	Building	Road	Tree	LowVeg.	Mov Car	Static Car	Human	Clutter	mIoU	mF1	OA
BANet	90.04	75.66	78.11	68.42	70.17	64.12	42.68	62.27	68.93	80.88	86.95
DC-Swin	92.67	78.95	77.73	67.08	69.27	64.59	37.27	65.91	69.2	80.80	87.90
A2FPN	90.83	77.45	77.97	68.66	67.07	64.9	46.91	63.21	69.62	81.48	87.3
LSKNet-T	91.80	73.83	79.09	69.47	75.85	69.43	46.85	60.72	70.89	82.32	87.34
MANet	91.77	78.10	78.59	69.14	72.20	69.48	48.50	65.28	71.63	82.92	87.99
UNetFormer	90.64	76.45	77.52	67.76	71.22	67.38	46.62	62.70	70.00	81.64	87.21
Ours	92.45	82.94	82.35	58.56	79.84	70.47	50.64	63.74	72.63	84.63	88.55

and

Table 3. Quantitative comparison results on the Vaihingen with recent research methodologies. The best values in each column are highlighted in bold.

Method	Imp. Surf.	Building	Low.veg.	Tree	Car	mF1	mIoU	OA
DANet	90.0	93.9	82.2	87.3	44.5	79.6	69.4	88.2
ABCNet	92.7	95.2	84.5	89.7	85.3	89.5	81.3	90.7
BANet	92.2	95.2	83.8	89.9	86.8	89.6	81.4	90.5
Segmenter	89.8	93.0	81.2	88.9	67.6	84.1	73.6	88.1
ESDINet	92.7	95.5	84.5	90.0	87.2	90.0	82.0	90.9
UNetFormer	92.7	95.3	84.9	90.6	88.5	90.4	82.7	90.4
CMTFNet	90.6	94.2	81.9	87.6	82.8	87.4	78.0	88.7
RS3Mamba	96.7	95.5	84.4	90.0	86.9	90.7	83.3	93.2
Ours	96.7	96.0	86.1	89.5	84.7	90.6	83.2	93.5

. On the LoveDA remote sensing image dataset, the method achieved the best results in five categories for the IoU metric and surpassed other CNN- and transformer-based methods in terms of mean IoU, indicating superior average performance. Analysis of the categories where the method excelled reveals that LASC-Mamba and Mix-Mamba provide more accurate semantic information in extensive scene contexts, particularly for the forest and agriculture categories. This suggests that LASC-Mamba introduces local texture information while incorporating lower-resolution contextual details. However, in the Building category, the IoU performance did not significantly outperform UNetFormer. The main reason for poor performance in the Barren category is the class imbalance in the LoveDA dataset, leading to unstable network training, with all methods showing either very poor results or irregular IoU distributions in this category. Ultimately, our method demonstrates a 0.5% improvement over the UNetFormer network. Qualitative experiments on the UAVid dataset demonstrate that our method surpasses the baseline UNetFormer in most categories, particularly in the small-object classes of moving car and static car, where we significantly outperform the baseline. However, our approach underperforms in the Low Vegetation category, a shortcoming we explore in the Further Analysis section. Nonetheless, our method achieves the best results in the mF1 score and overall accuracy (OA), improving by 2.99% and 1.34% over UNetFormer, respectively. The visualization results on the Vaihingen dataset indicate that our approach produces outcomes comparable to RS3Mamba, which also utilizes the Mamba architecture, but with superior OA scores. We achieved a notable 4% improvement in the impervious surface category compared to UNetFormer. Consequently, the Tree and Car categories, which are spatially enclosed by impervious surfaces, saw a decrease in performance. Although the improvement in the class-balanced mIoU metric was modest, the classification gain in impervious surface pixels led to a 3.1% increase in OA compared to UNetFormer, exhibiting classification characteristics similar to RS3Mamba.

Qualitative Results: The visualization results are shown in Figure 4, Figure 5 and Figure 6. Our method outperforms the CNN-based network BANet and the purely transformer-based architecture DC-Swin on the LoveDA dataset. Our approach effectively minimizes interference from pixels of different categories in dynamically changing geographical scenes, especially where different category scenes intersect or contain each other, such as the buildings in red. Our method distinctly identifies background and some road information within clusters of buildings. In large-scene distributions such as vegetation, the network model based on the Mamba architecture exhibits superior performance to the UNetFormer network, primarily by maintaining connectivity between the same categories of ground and objects, thus preventing severe visual erosion caused by various types of disturbances. Furthermore, compared to other segmentation results, our method presents smoother mask boundaries and contains fewer noise artifacts, indicating that the contextual information extracted by Mamba is effectively utilized during the encoder phase. The visualization results on the UAVid dataset demonstrate that our method maintains correct semantic relationships in categories with regular shapes. For instance, the logical connections and appropriate depth positioning between vehicles, between vehicles and roads and between trees and low vegetation are accurately preserved, thereby avoiding the boundary confusion seen in UNetFormer, which arises from its reliance on large-scale global information. In contrast, the Vaihingen dataset features more defined contours and shapes across categories compared to UAVid and LoveDA. This results in a more coherent relationship between impervious surfaces and other categories, particularly with an improved encapsulation of small objects, leading to a higher degree of individual object separation. This characteristic is especially evident in the Low Vegetation category.

4.4. Ablation Study

We compared the qualitative and quantitative outcomes of two Mamba architectures under various configurations.

Table 4. Quantitative results of ablation experiments conducted on the UAVid dataset. The best values in each column are displayed in bold.

UNetFormer	LASC-Mamba	Mix-Mamba	Building	Road	Tree	LowVeg.	Mov Car	Static Car	Human	Clutter	mIoU
✔	-	-	90.64	76.45	77.52	67.76	71.22	67.38	46.62	62.70	70.0
✔	✔	-	91.18	78.18	78.13	68.92	72.50	68.96	47.57	64.18	71.20
✔	-	✔	91.35	78.79	78.37	68.24	73.01	70.29	47.99	65.20	71.66
✔	✔	✔	92.45	82.94	82.35	58.56	79.84	70.47	50.64	63.74	72.63

demonstrates that LASC-Mamba and Mix-Mamba significantly enhance semantic precision across most categories, particularly in cross-resolution geographical classes such as buildings and roads. Mix-Mamba effectively synthesizes global information from remote sensing imagery, ensuring the integrity of the final mask output. This is also evident in the visual representations in Figure 7, where the segmentation of buildings (represented in red) shows a substantial reduction in misclassification within the structures, resulting in more cohesive boundaries. On the other hand, the LASC-Mamba architecture refines the representation for irregular categories and fine-grained local features, incorporating multi-level basic texture information through feature link aggregation for categories like Low Vegetation and Clutter, which lack distinct boundaries, thereby enhancing the accuracy and robustness of edge segmentation. Figure 8 shows that the hybrid architecture achieves a more balanced performance across categories compared to the individual modules, particularly in the Building, Barren and Forest classes. This indicates that the final fusion effectively adapts and allocates global and local information in these categories. Although the mIoU improvement after fusion is modest, it still achieves state-of-the-art results. The visual representation in Figure 8 illustrates how the final fused model integrates the strengths of LASC-Mamba and Mix-Mamba, particularly in restoring cross-category geographical information and complementing localized details.

Notable, the results in

Table 5. Quantitative results of ablation experiments conducted on the LoveDA dataset. The best values in each column are displayed in bold.

UNetFormer	LASC-Mamba	Mix-Mamba	Background	Building	Road	Water	Barren	Forest	Agriculture	mIoU
✔	-	-	44.7	58.8	54.9	79.6	20.1	46.0	62.5	52.4
✔	✔	-	46.7	57.8	57.3	80.9	15.8	47.5	63.7	52.8
✔	-	✔	46.1	58.3	58.5	79.6	18.5	46.9	61.5	52.8
✔	✔	✔	46.1	58.4	55.5	79.9	19.2	47.6	63.2	52.9

indicating the integrated final structure incorporating both modules’ achieved optimal results across the majority of categories and the ultimate metric, mean intersection over union (mIoU), demonstrating the efficacy of our approach in utilizing the advantages of the two distinct Mamba modules.

4.5. Further Analysis

Limitations and Failure Cases: We present the failure cases in Figure 9. Some visual results on the Vaihingen dataset indicate that, in complex and irregular large-scale scenes, our method may produce semantic errors compared to baseline methods. These errors are primarily concentrated in the Forest, Barren and Agriculture categories, leading to the loss of information for these high-coverage features. This may be due to the additional parallel Mamba framework in the Mix-Mamba module indirectly reducing the adaptive weight allocation of the multi-head self-attention mechanism, which generates global information, thereby leading to class confusion in large receptive fields. On the other hand, while the LASC-Mamba module effectively smooths cross-scale information, particularly enhancing fine-grained representation and accurate edge segmentation in the final output layer, the allocation of high-dimensional semantic information is limited due to feature dimensions within the linkage, which could also contribute to these failures.

Model Complexity:

Table 6. Measurements of the computational complexity analysis on a single NVIDIA 3090 GPU.

Method	FLOPs (G)	Param. (M)
ABCNet	7.81	13.39
CMTFNet	17.14	30.07
UNetformer	5.87	11.69
RS3Mamba	31.65	43.32
CM-UNet	6.01	12.89
Ours	7.89	15.63

illustrates the comparison of model complexity using two metrics: floating-point operations (FLOPs) and model parameters. Due to the addition of extra modules, our approach does not hold an advantage in terms of parameter count. Compared to the baseline method, our training parameters increased by 33.7%, incurring some computational cost to achieve performance improvements.

5. Discussion

The semantic segmentation of UAV and satellite imagery using deep learning is a crucial technique for feature detection. In recent years, researchers have consistently improved the performance and metrics of remote sensing image classification, detection and segmentation tasks by flexibly combining convolutional neural networks with transformer modules. Most hybrid architectures effectively balance the detailed local features extracted by convolution with the broad global information captured by transformers. Some approaches also reduce input dimensions through downsampling to save computational costs for transformers.

In the field of remote sensing image segmentation, these methods are typically implemented in encoder–decoder structures, particularly in U-Net-like networks, where researchers often directly incorporate hierarchical feature outputs from hybrid or pretrained encoders into corresponding decoder stages. However, this approach overlooks the spatial representation capabilities of cross-scale outputs. This paper aims to enhance the spatiotemporal representation during fusion and decoding stages using the Mamba module. The LASC-Mamba aggregates mixed scales across multiple links, reconstructs cross-scale spatiotemporal relationships and feeds the fused output back into each decoder stage. During decoding, the Mix-Mamba module processes local convolutional features, global self-attention features and discrete SSM representations in parallel to achieve more comprehensive feature reconstruction and semantic information extraction. This multimodal feature fusion method not only enhances the model’s sensitivity to complex feature characteristics but also significantly improves the accuracy and robustness of remote sensing image segmentation. We compared the reconstructed hybrid model with baseline methods and other hybrid architectures in this field, outperforming them across various metrics on multiple standard datasets, demonstrating the model’s robustness and generalization capabilities as an effective universal architecture.

Although our work has improved the metrics and accuracy of feature classification through the construction of LASC-Mamba and Mix-Mamba, there are still some limitations. While the additional Mamba modules have enhanced the baseline method’s performance in terms of IoU and accuracy for various classes, they have also increased the model’s complexity, slightly reducing training and inference speed. Additionally, although our proposed model has shown robustness and versatility in most scenarios, it still underperforms in recognizing certain feature categories, such as the “Building” and “Barren” classes in the LoveDA dataset. This limitation may stem from the model’s insensitivity to certain dataset categories and the feature extraction bottlenecks inherent in the encoder backbone framework. Therefore, further research into underexplored feature semantics will be a focus of our future work.

6. Conclusions

This paper proposes a novel U-Net-like network for UAV image segmentation, addressing the limitations of traditional CNN–transformer networks in capturing fine-grained feature details. By constructing a hybrid model based on the Mamba architecture, we enhance the UNetFormer base model with a multi-scale fusion link aggregation Mamba. This module maps shallow semantic information into a unified state space, selectively sharing long-sequence linear information to address scale allocation issues in cross-layer connections, and reconstructs all outputs into a unified scale space. On the other hand, the Mix-Mamba module combines global–local self-attention’s long-range semantic perception with sequence length linear scaling, constructing a linear modeling mechanism across different channels and spatial dimensions through multi-path interactions. Additionally, the Mix-Mamba module can serve as a plug-and-play component for other general architectures. Experimental evaluations on two public remote sensing image segmentation datasets demonstrate that the proposed method outperforms other CNN- and transformer-based semantic segmentation algorithms, highlighting its significant potential in the field of remote sensing image segmentation.