NestedMambaUNet: A Direction-Aware State Space Network for Landslide Mapping from Remote Sensing Images

Highlights

What are the main findings?

• NestedMambaUNet improves landslide extraction accuracy by combining nested multi-scale feature fusion, coordinate-enhanced shallow feature learning, and direction-aware state-space modeling.
• The proposed DASS2D mechanism captures long-range dependencies along horizontal, vertical, and diagonal directions, leading to better boundary continuity and structural integrity of irregular landslides.

What are the implications of the main findings?

• Direction-aware state-space modeling provides an effective alternative to CNN- and Transformer-based methods for complex landslide mapping, balancing global context modeling with computational efficiency.
• The improved recall, F1-score, and robustness across different datasets suggest strong potential for large-scale, time-sensitive post-disaster landslide mapping and emergency response applications.

Abstract

The rapid and accurate extraction of landslide areas from remote sensing imagery is critical for post-disaster emergency response and rescue operations. However, landslides exhibit complex morphologies and irregular orientations and are easily confused with background features such as bare ground and roads. However, accurately modeling long-range spatial dependencies, effective multi-scale feature fusion, and precise boundary delineation for complex landslide scenarios remains challenging. To address these challenges, we propose NestedMambaUNet, a nested state-space network specifically designed for landslide extraction in remote sensing imagery. Building upon the dense skip-connection architecture of UNet++, the model first introduces a coordinate-enhanced convolution module called CoordConvBlock to explicitly encode spatial positional information during shallow feature extraction, thereby improving the modeling of spatial relationships between landslides and surrounding terrain. It further incorporates a 2D direction-adaptive selective scanning mechanism (DASS2D), which adaptively aggregates scanning results from four directions (horizontal, vertical, main diagonal, and anti-diagonal) to capture long-range spatial dependencies aligned with the irregular structures of landslides. Additionally, a direction-adaptive selective scanning fusion block (DASS Fusion Block) is designed to enhance multi-scale feature integration and improve boundary continuity by combining attention-based gating for skip connections with direction-adaptive state-space modeling. The experimental results on two public datasets, LMHLD and HR-GLDD, demonstrate that the proposed method outperforms competing approaches across multiple evaluation metrics. Specifically, the IoU reaches 71.23% and 58.84%, representing improvements of 1.17 and 5.65 percentage points over the second-best method, respectively, while the recall increases by 4.05 and 12.01 percentage points, respectively. Although the proposed method exhibits a slight reduction in Precision on certain datasets, it achieves the best overall F1-score, indicating a favorable balance between missed detection reduction and false positive control for landslide extraction tasks. These results indicate that NestedMambaUNet effectively improves the structural integrity of landslides and enhances boundary delineation, while exhibiting good robustness across different data distributions and geographic scenarios. In addition, the proposed method achieves a favorable balance between segmentation accuracy and computational efficiency, demonstrating its potential for time-sensitive large-scale landslide mapping applications.

1. Introduction

Landslides are geological phenomena in which rock and soil masses undergo overall or localized movement along a distinct sliding surface or weak structural plane, primarily driven by gravity [1]. They can be triggered by various natural and anthropogenic factors, including rainstorms [2], earthquakes [3,4], volcanic eruptions [5], glacial melting [6], and engineering activities [7], among other environmental disturbances [8]. Owing to their high frequency, wide spatial distribution, and sudden onset, landslides pose a significant threat to human life, property, and critical infrastructure [9,10], thereby necessitating rapid and accurate landslide mapping for effective emergency response and subsequent rescue and relief operations [11]. Traditional landslide investigation primarily relies on field surveys, in situ monitoring, and visual interpretation [12], which, despite their reliability, are often time-consuming, limited in spatial coverage, and difficult to deploy over large-scale disaster zones or in high-risk areas inaccessible to personnel. To address these limitations, remote sensing technology, with its wide spatial coverage, high acquisition efficiency, strong revisit capability, and relatively low cost, has become an effective means for rapid large-area landslide identification and a critical tool for landslide monitoring and emergency response [13,14].

Landslide extraction has witnessed a transition from manual visual interpretation to traditional machine learning and, more recently, deep learning approaches [15]. Although visual interpretation can yield relatively reliable results when guided by expert knowledge, it remains labor-intensive, lacks automation, and is ill-suited for time-critical disaster response scenarios [16,17]. Traditional machine learning methods offer improved automation; however, their performance is heavily dependent on handcrafted features, which limits their generalization across complex land cover conditions, heterogeneous sensor characteristics, and diverse geographic regions [18]. In recent years, deep learning has substantially advanced landslide extraction by enabling end-to-end feature learning, thereby improving both accuracy and efficiency under complex terrain conditions [19].

Convolutional neural networks (CNNs) and their variants have been widely applied to landslide extraction tasks [20]. Prakash et al. [21] proposed a progressive CNN training strategy based on a merged inventory list, leveraging multi-region event-based landslide data to improve model generalization. Li et al. [22] developed a two-stage landslide detection framework using satellite imagery, in which Faster R-CNN is first employed to localize landslide regions, followed by an enhanced U-Net for fine-grained segmentation, yielding improved performance in both detection and segmentation accuracy. Ghorbanzadeh et al. [23] systematically evaluated the transferability of U-Net and ResU-Net, demonstrating that ResU-Net trained on a Japanese landslide dataset can still achieve strong F1 scores when applied to other regions, thereby exhibiting robust cross-region generalization capability. These studies demonstrate the effectiveness of CNN-based methods for landslide extraction. However, owing to the inherent locality of convolution operations, CNN-based approaches remain limited in their ability to model global contextual information and capture long-range spatial dependencies.

To overcome the limitations of CNNs in global feature modeling, recent studies have introduced Transformer architectures, which have proven highly effective in general remote sensing segmentation, into landslide extraction tasks [24,25]. Widely adopted models such as Swin Transformer [26] and SegFormer [27] have demonstrated strong performance by capturing long-range dependencies, and their success has inspired landslide-specific adaptations. For instance, Li et al. [28], for instance, designed CTHNet, which incorporates a multi-scale lightweight Transformer module for global context modeling and achieves a 3.81% improvement in the F1 score for landslide extraction. Zhao et al. [29] proposed CTLGNet, which demonstrates strong performance in landslide susceptibility mapping across the Three Gorges Reservoir area and the Jiuzhaigou region, attaining AUC values of 0.9817 and 0.9693, respectively, and exhibiting robust predictive capability and cross-region generalization. Despite their advantages in global context modeling, Transformer-based models suffer from high computational complexity, as the computational cost of self-attention grows quadratically with input resolution. When applied to large-scale, high-resolution remote sensing imagery, these models often incur substantial memory consumption and reduced inference efficiency, rendering them less suitable for time-critical regional landslide emergency interpretation tasks.

Recently emerging state-space models (SSMs), particularly selective state-space sequence models represented by Mamba [30], have attracted increasing attention due to their ability to achieve global receptive fields with linear computational complexity, offering a promising solution to the trade-off between accuracy and efficiency. Building on this foundation, VMamba [31] extends the Mamba architecture from one-dimensional sequence modeling to vision tasks by introducing a two-dimensional selective scanning mechanism, thereby enabling effective global context modeling of images while preserving linear complexity. Zhao et al. [32] applied RS3Mamba and Ultralight VM-Unet to landslide datasets, demonstrating the effectiveness of Mamba-based architectures for remote sensing landslide mapping and reporting superior performance compared with Transformer-based models. SCGC-Net [33] further incorporates a hierarchical context interaction and modulation mechanism (HMIE) that combines contextual modulation with progressive calibration, thereby enhancing feature representation in both CNN and Mamba branches, reducing reliance on large-scale annotated data, and improving detection efficiency and stability in previously unseen regions.

Although the above studies have demonstrated the potential of state-space models for landslide interpretation, existing methods still suffer from several limitations. First, existing two-dimensional selective scanning mechanisms typically propagate states along fixed horizontal and vertical directions. However, landslides in natural scenes are often distributed along slope directions, drainage channels, or terrain structures, exhibiting pronounced non-axis-aligned characteristics. Consequently, such fixed-direction scanning strategies are insufficient to accurately capture the true spatial morphology and directional variability of landslides. Second, conventional multi-scale feature fusion methods, such as dense skip connections or direct channel concatenation, lack effective feature selection mechanisms. As a result, shallow-layer background textures and irrelevant noise from the encoder are easily propagated into the decoder through skip connections, interfering with precise boundary delineation and detection stability. Third, a gap persists between long-range dependency modeling and local spatial detail representation in state-space models, which limits their performance in small-scale landslide detection and boundary refinement. In addition, pixel-level landslide extraction from remote sensing imagery often suffers from severe class imbalance, where non-landslide background pixels substantially outnumber landslide pixels. This characteristic increases the difficulty of accurately detecting small-scale or boundary-blurred landslides.

To address these limitations, we propose NestedMambaUNet, a nested state-space network designed for landslide extraction from remote sensing imagery. The main contributions of this work are summarized as follows:

• A nested state-space architecture, NestedMambaUNet, is proposed for landslide extraction. Built upon the dense skip-connection structure of UNet++ [34], the model incorporates a coordinate-enhanced convolution module, CoordConvBlock, and a directional adaptive selective scan fusion block (DASS Fusion Block), systematically integrating the global modeling capability of state-space models into multi-scale feature extraction and fusion. This design enables precise boundary delineation and structurally consistent landslide representation under complex terrain conditions.
• A 2D directional adaptive selective scan mechanism (DASS2D) is designed. It employs a lightweight directional weighting branch to adaptively fuse scanning outputs from four directions—horizontal, vertical, main diagonal, and anti-diagonal—enabling the model to capture direction-specific spatial dependencies and improve the continuity of landslide boundaries.
• CoordConvBlock is introduced to explicitly encode spatial positional information during shallow feature extraction. This design enhances the model’s ability to distinguish landslides from spectrally similar land-cover types and provides spatially discriminative feature representations for subsequent global context modeling.

2. Materials and Methods

2.1. Datasets

Datasets serve as the foundation for evaluating model performance and generalization, where data quality, scale, and scene diversity directly affect the objectivity and reliability of experimental results. In landslide extraction tasks, the diversity of remote sensing data sources and the spatiotemporal heterogeneity of landslide events pose additional challenges to model robustness. To comprehensively evaluate the proposed method under diverse conditions, two representative public landslide datasets, LMHLD [35] and HR-GLDD [36], are adopted in this study. These datasets differ in terms of geographic coverage, imaging conditions, triggering mechanisms, and landslide types, enabling a thorough assessment of the generalization capability of NestedMambaUNet across multi-source and cross-regional scenarios.

The LMHLD dataset covers landslide events triggered by multiple factors and is collected from five different satellite sensors across seven regions worldwide, with spatial resolutions ranging from 0.8 m to 10 m. It contains 25,365 image patches and 32,296 landslide instances, encompassing diverse scales, morphologies, and terrain conditions. The dataset is divided into training, validation, and test sets with a ratio of 7:2:1.

The HR-GLDD dataset is a high-resolution landslide dataset with a spatial resolution of 3 m, covering ten geographic regions across South Asia, Southeast Asia, East Asia, South America, and Central America. It includes ten landslide events, consisting of five rainfall-triggered and five earthquake-triggered cases, spanning diverse geomorphological settings. The dataset is split into training, validation, and test sets with a ratio of 6:2:2.

Each image patch in both datasets contains four spectral bands (red, green, blue, and near-infrared (NIR)) along with corresponding binary masks for landslide detection. For consistency and fair comparison, all experiments are conducted using a unified input size of 128 × 128 pixels.

Figure 1 provides an overview of the LMHLD and HR-GLDD datasets. Spanning multiple landslide-prone regions across Asia, the Americas, Europe, Africa, and Oceania, these datasets exhibit substantial diversity in terrain conditions, climatic settings, and triggering mechanisms. Such broad coverage ensures strong representativeness across diverse geological and environmental contexts, offering a comprehensive basis for evaluating the cross-scenario generalization capability of the proposed method.

Figure 1. Global distribution of the study areas.

To illustrate the visual characteristics and annotation quality of the two datasets, Figure 2 presents representative samples from the LMHLD and HR-GLDD datasets. Each sample group includes an RGB image, a NIR grayscale image, and the corresponding binary landslide mask, covering landslide instances of varying scales and diverse terrain and vegetation conditions.

Figure 2. Visualization of dataset samples: (a,d) RGB images; (b,e) NIR images; (c,f) ground truth masks.

2.2. Methods

2.2.1. Overall Architecture of NestedMambaUNet

Landslides in remote sensing imagery exhibit pronounced multi-scale characteristics, ranging from large-scale contiguous bare regions to small, fragmented failures with blurred boundaries. Consequently, an effective landslide extraction network should be capable of capturing both fine-grained local details and global morphological patterns. To this end, we adopt UNet++ as the baseline architecture and enhance its feature extraction and fusion capabilities by integrating a state-space model for long-range dependency modeling, along with coordinate convolution and attention-based gating mechanisms to improve boundary representation and structural detail preservation.

As illustrated in Figure 3, NestedMambaUNet follows an encoder–decoder architecture. The input RGB & NIR imagery is first processed by CoordConvBlocks in the initial four encoder stages (L1, L2, L3 and L4) for hierarchical feature extraction and downsampling, generating multi-scale feature representations with explicit spatial positional encoding. At the bottleneck, a directional adaptive selective scan block (DASS Block) performs direction-adaptive global scanning on the compressed high-level semantic features, enabling effective modeling of long-range spatial dependencies. In the decoder, skip-connected features from each encoder stage are progressively fused with upsampled features through DASS Fusion Blocks, where an attention-based gating mechanism first suppresses irrelevant information, followed by a direction-adaptive selective scanning operation that enhances the spatial context of the fused features. After progressive multi-level fusion and upsampling, the network ultimately produces a binary segmentation mask of the landslide region.

Figure 3. Overall architecture of the proposed NestedMambaUNet.

2.2.2. CoordConvBlock

The development and spatial distribution of landslides are typically controlled by factors such as slope aspect, gully orientation, and local geomorphological structures, exhibiting strong directional patterns and spatial neighborhood dependencies. Standard convolutional operations, however, rely on local receptive fields with shared weights and consequently lack explicit modeling of relative spatial positions between pixels. To address this limitation, we propose a CoordConvBlock inspired by CoordConv [37], which explicitly encodes spatial coordinate information into convolutional feature representations and incorporates a gating mechanism to adaptively emphasize informative regions.

As shown in Figure 4, the input features are first processed by a standard convolutional layer followed by a GELU activation and then split into a residual branch and a main branch. The residual branch propagates features directly after channel adjustment via a 1 × 1 convolution, ensuring stable information flow. The main branch performs spatial structure enhancement through sequential coordinate convolution, gated weighting, and nonlinear transformation. In the coordinate convolution layer, following the CoordConv strategy, normalized x- and y-coordinate maps are generated based on the spatial dimensions of the input feature map and concatenated with the original features along the channel dimension, increasing the channel count from N to N + 2 before the subsequent convolution. By explicitly embedding spatial coordinate information, this design provides positional cues to convolutional kernels, thereby enhancing their capacity to model landslide boundaries, morphological structures, and spatial distribution patterns.

Figure 4. Schematic of the CoordConvBlock architecture and the coordinate convolution principle.

The output of the coordinate convolution is first passed through batch normalization and then fed into a gating branch. This branch consists of a 1 × 1 convolution followed by a Sigmoid activation, which generates a spatial gating weight map. The resulting weight map is applied to the main-branch features via element-wise multiplication, enabling adaptive modulation of informative responses and suppression of irrelevant activations. The gated features are subsequently processed by a GELU activation and a second convolutional layer to obtain more discriminative representations. Finally, the refined main-branch features are fused with the residual branch via element-wise addition to produce the module output. Overall, the CoordConvBlock integrates residual learning, explicit spatial positional encoding, and adaptive feature modulation, thereby enhancing the representation of landslide spatial structures while improving robustness to complex background noise.

2.2.3. Direction-Adaptive Selective Scan Block

After shallow features with enhanced spatial representations are extracted by the CoordConvBlocks in the first four encoder stages, global context modeling is required on the highly compressed semantic features at the bottleneck layer. To this end, we propose a Direction-Adaptive Selective Scan Block (DASS Block), as illustrated in Figure 5a. While maintaining low computational complexity, the proposed module introduces a multi-directional state-space modeling mechanism to enhance the network’s capacity to capture complex spatial structures and long-range dependencies in landslide regions.

Figure 5. Schematic of the DASS Block and DASS Fusion Block architectures: (a) DASS Block; (b) DASS Fusion Block.

In the implementation, the input features are first processed by a CoordConvBlock for local spatial enhancement and explicit positional encoding. A residual connection is employed to preserve the original information, thereby improving spatial representation while stabilizing feature propagation. The resulting features are then normalized and linearly projected to double the channel dimension. The expanded features are subsequently split along the channel dimension into a feature branch and a gating branch: the gating branch generates dynamic modulation weights, while the feature branch is used for subsequent state-space modeling.

In the feature branch, the input features are first refined through a depthwise separable convolution followed by a SiLU activation to enhance boundary and texture representations. The refined features are then fed into the DASS2D to model long-range spatial dependencies across multiple directions. Previous studies have demonstrated that diagonal scanning can improve structural perception in remote sensing imagery [38]. Motivated by this insight, DASS2D unfolds the 2D feature maps into four 1D sequences along the horizontal, vertical, main diagonal, and anti-diagonal directions, and processes each sequence independently using a selective state-space model for directional scanning and feature modeling. Given an input feature $x\in {\mathbb{R}}^{H\times W\times D}$, this process can be formulated as:

$$Y_1,Y_2,Y_3,Y_4=S6\left(\left[S_1\left(x\right),S_2\left(x\right),S_3\left(x\right),S_4\left(x\right)\right]\right)$$

(1)

where $S6$ denotes the selective scan mechanism proposed in Mamba, Transform $S_{\mathrm{i}}$ represents the operation that unfolds the 2D feature map into a 1D sequence along a specific direction, and $Y_{\mathrm{i}}$ denotes the output features of the corresponding scanning path.

Although the four-direction scanning strategy covers multiple spatial orientations, simply aggregating the outputs with equal weights may be insufficient to accommodate the variability of landslide distribution directions across different scenes. To address this limitation, DASS2D further introduces a direction-adaptive weighting mechanism. Specifically, a lightweight directional weight generation branch is employed prior to the scanning operation. This branch consists of two 1 × 1 convolutional layers followed by a SiLU activation and produces a four-dimensional directional weight vector. The resulting weights are passed through a $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}$ function and shifted by a constant bias to obtain the final fusion weights $w_i$ for each scanning direction. The formulation is given as:

$$w_i=1.0+\alpha ·\tanh \left(Conv\left(SiLU\left(Conv\left(x\right)\right)\right)\right)$$

(2)

The parameter α controls the modulation strength and is empirically set to 0.3 in this study. The generated directional weights are spatially adaptive pixel-level weights, enabling different spatial locations to dynamically emphasize different scanning directions according to local structural characteristics. The four directional weights are not normalized using softmax. Instead, the proposed formulation adopts a residual-style modulation strategy centered around 1.0. Since the tanh activation constrains the weights within a limited range, excessive feature amplification is effectively avoided while maintaining stable adaptive directional enhancement. The final output of DASS2D is obtained by aggregating features from all scanning paths using their corresponding fusion weights, formulated as:

$$Y_{fused}=w_1·Y_1+w_2·Y_2+w_3·Y_3+w_4·Y_4$$

(3)

The proposed direction-adaptive weighting mechanism enables DASS2D to adaptively modulate the contribution of each scanning path based on the spatial structure of the input features, thereby improving the modeling of directional spatial dependencies in landslides. The weighted and fused sequence features are then mapped back to a 2D feature representation via the inverse transformation $S^{-1}$:

$$Y=S^{-1}\left(Y_{fused}\right)$$

(4)

where $S^{-1}$ denotes the inverse operation of scan unfolding, which maps the fused 1D sequences back to the original 2D spatial layout. The reconstructed 2D features are first normalized and then modulated via element-wise multiplication with the output of the gating branch activated by a SiLU function. Subsequently, the features are linearly projected to restore the original channel dimension, and a residual connection with the module input is applied to produce the final output. Through this design, the DASS Block captures the directional spatial dependencies of landslides, providing global contextual representations that facilitate multi-scale feature fusion in the subsequent decoding stage.

2.2.4. Direction-Adaptive Selective Scan Fusion Block

In the decoding stage, skip-connected features from the encoder and upsampled features from deeper layers are fused to progressively recover spatial details and refine landslide boundaries. However, conventional fusion strategies based on direct concatenation lack selective feature modeling, allowing shallow-layer noise, background textures, and irrelevant responses to propagate into the decoder and degrade boundary accuracy. To address this limitation, we propose a Direction-Adaptive Selective Scan Fusion Block (DASS Fusion Block), as illustrated in Figure 5b. This module incorporates an attention-based gating mechanism to adaptively filter skip-connected features and integrates a direction-adaptive state-space scanning strategy for enhanced multi-source feature fusion.

The DASS Fusion Block consists of three components: an attention gating branch, a feature fusion branch, and a DASS2D module. First, the skip-connected features $x_{skip}$ from the encoder and the upsampled features $x_{up}$ from deeper layers are each processed by a 1 × 1 convolution followed by batch normalization for channel alignment and distribution normalization and are projected into a shared intermediate feature space. The two feature streams are then fused via element-wise addition and activated by a ReLU function. The resulting features are fed into an attention generation branch composed of a 1 × 1 convolution and a Sigmoid activation to produce a spatial attention map $\psi$. This process can be formulated as:

$$\psi =\sigma \left(Conv\right(ReLU(W_g\left(x_{up}\right)+W_x\left(x_{skip}\right))\left)\right)$$

(5)

where $W_g$ and $W_x$ denote the linear transformations applied to the upsampled features and skip-connected features, respectively, and $\sigma (·)$ denotes the Sigmoid activation function. The attention coefficient map $\psi \in [0,1]$ encodes the spatial importance distribution, assigning higher weights to salient regions.

The attention coefficient map is then applied to the skip-connected features via element-wise multiplication to suppress background responses and irrelevant textures. The weighted skip features are subsequently concatenated with the upsampled features along the channel dimension and compressed to the target dimensionality via a 1 × 1 convolution, achieving the initial fusion of multi-scale features.

The fused features are first normalized and linearly projected to expand the channel dimension and then split along the channel dimension into a feature branch and a gating branch. In the feature branch, the features are refined via a depthwise separable convolution followed by a SiLU activation and subsequently processed by DASS2D for multi-directional selective scanning and direction-adaptive weighted fusion. The gating branch generates dynamic modulation weights through a SiLU-based transformation, which are then applied to the DASS2D-enhanced features via element-wise multiplication. Finally, the modulated features are added to the pre-fusion input through a residual connection to produce the module output. Overall, the DASS Fusion Block enables adaptive filtering of skip-connected features during multi-scale fusion and leverages the multi-directional global modeling capability of DASS2D to enhance the spatial structure representation and boundary continuity of landslides.

2.3. Evaluation Metrics

Five widely used quantitative metrics in semantic segmentation are adopted to comprehensively evaluate model performance: overall accuracy (OA), precision (P), recall (R), F1-score (F1), and intersection over union (IoU). These metrics assess the model’s interpretation capability from multiple perspectives, including pixel-wise classification accuracy and the trade-off between false positives and false negatives. To further evaluate the fine-grained quality of landslide boundary extraction, Boundary IoU (BIoU) [39] is introduced as a complementary metric. BIoU computes the intersection-over-union ratio within a narrow boundary band between the prediction and the ground truth, providing a more sensitive measure of the model’s boundary delineation accuracy and addressing the limited sensitivity of standard IoU to boundary errors.

$$OA={\displaystyle \frac{TP+TN}{TP+TN+FN+FP}}$$

(6)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(7)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(8)

$$F1={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(9)

$$IoU={\displaystyle \frac{TP}{TP+FP+FN}}={\displaystyle \frac{\left|G\cap P\right|}{\left|G\cup P\right|}}$$

(10)

$$Boundary IoU={\displaystyle \frac{\left|\left(G_d\cap G\right)\cap \left(P_d\cap P\right)\right|}{\left|\left(G_d\cap G\right)\cup \left(P_d\cap P\right)\right|}}$$

(11)

where $TP$ denotes the number of pixels correctly predicted as landslide, $FP$ denotes the number of non-landslide pixels misclassified as landslide, $TN$ denotes the number of pixels correctly identified as non-landslide, and $FN$ denotes the number of landslide pixels missed by the model. $G$ and $P$ represent the ground-truth binary mask and the predicted mask, respectively. $G_d$ denotes the boundary band obtained by dilating the ground truth mask $G$ with a radius of $d$ pixels, and $P_d$ denotes the corresponding boundary band derived from the predicted mask $P$ through the same dilation operation. In this study, the boundary dilation radius $d$ is determined using a dilation ratio of 0.05 with respect to the image diagonal length.

Recall and IoU are widely used metrics for evaluating landslide extraction performance. Recall is used to evaluate the completeness of landslide detection and reflects the model’s ability to reduce missed detections, which is particularly important in emergency landslide mapping scenarios. Precision measures the reliability of predicted landslide regions and reflects the degree of false positives. The F1-score provides a balanced evaluation of Precision and Recall, thereby reflecting the overall detection performance of the model. IoU is adopted as a standard segmentation metric to evaluate the overlap quality between predicted results and ground truth. In addition, Boundary IoU (BIoU) is introduced to further assess boundary localization accuracy and structural continuity, which are critical for landslide delineation tasks with complex and irregular boundaries. Therefore, the combined use of these metrics enables a more comprehensive and task-oriented evaluation of landslide extraction performance.

3. Results and Discussion

3.1. Experimental Configuration

To ensure fair and reliable comparisons, all compared methods were trained and evaluated under the same experimental settings, including identical data splits, input modalities, preprocessing procedures, data augmentation strategies, optimizer settings, and evaluation metrics. All experiments were conducted on a Windows 11 Professional system equipped with an NVIDIA GeForce RTX 5090 GPU (32 GB). The model was implemented using the PyTorch 2.9.0 framework. The input data consist of four-channel tensors constructed by concatenating RGB optical imagery with the NIR band along the channel dimension. All models were trained from scratch without using pre-trained weights, since the input data consist of four spectral bands (RGB + NIR), which are not directly compatible with standard RGB-based pre-trained backbones. During training, random scaling followed by random cropping to 128 × 128 pixels was employed as part of the data augmentation strategy. Additional augmentation operations, including random horizontal flipping and vertical flipping, were applied consistently to both the RGB and NIR bands to preserve spatial alignment. The model was trained using the AdamW optimizer with an initial learning rate of 2 × 10⁻⁴ and a weight decay of 0.01, and a batch size of 16. The binary cross-entropy loss was employed, and the model was trained for 100 epochs. No learning rate scheduler or early stopping strategy was used during training. All experiments were conducted using a fixed random seed, and the reported results correspond to a single training run.

3.2. Comparative Experiment

To comprehensively evaluate the performance of NestedMambaUNet, we select five widely used semantic segmentation models for comparison, categorized into three groups based on their architectural paradigms. The first group includes CNN-based methods, namely Attention U-Net [40] and UNet++, the latter serving as the baseline model in this study. The second group consists of Transformer-based architectures, including SegFormer and TransUNet [41]. The third group comprises Mamba-based approaches, represented by VM-UNet [42]. Together, these models span the major architectural paradigms in landslide mapping, providing a solid basis for assessing the effectiveness of the proposed method across different architectural paradigms.

Table 1 reports the quantitative results on the LMHLD validation set. Overall, NestedMambaUNet achieves superior performance across most evaluation metrics. Compared with the baseline model UNet++, the proposed method improves IoU and BIoU by 1.84 and 3.58 percentage points, respectively, preliminarily validating the effectiveness of CoordConvBlock, DASS2D, and DASS Fusion Block.

Table 1. Quantitative comparison of different methods on the LMHLD validation set.

Method	LMHLD (Val)
	OA (%)	P (%)	R (%)	F1 (%)	IoU (%)	BIoU (%)
AttUnet	95.49	87.36	75.37	80.92	67.96	49.24
TransUnet	95.32	88.02	73.14	79.89	66.52	47.32
VMUnet	95.74	85.56	79.93	82.65	70.43	50.74
SegFormer	95.14	82.25	78.73	80.45	67.29	48.26
Unet++	95.76	85.44	80.30	82.79	70.64	49.68
Ours	95.86	82.48	85.67	84.05	72.48	53.26

Table 2 reports the performance of the proposed model on the LMHLD test set. NestedMambaUNet achieves superior results across key metrics, including Recall, F1-score, IoU, and BIoU, demonstrating its effectiveness in landslide region detection and boundary delineation. Specifically, the recall reaches 85.21%, improving by 4.05 percentage points over the second-best model, UNet++, indicating enhanced completeness in landslide detection. This improvement can be largely attributed to DASS2D, which models multi-directional spatial dependencies and strengthens long-range contextual representation, thereby reducing missed detections. Although the proposed method achieves relatively lower Precision compared with several competing methods, this characteristic indicates that the model is more inclined to reduce missed detections. In practical applications such as emergency landslide mapping and post-disaster assessment, missed detections may lead to incomplete hazard identification and potentially increase disaster risks. Therefore, higher Recall is of greater practical significance in such scenarios. Moreover, the proposed method still achieves the highest F1-score among all compared models. F1-score provides a balanced evaluation of Precision and Recall; this result demonstrates that the proposed method maintains the best overall detection performance despite the slight decrease in Precision. In terms of boundary accuracy, the proposed model achieves a BIoU of 55.81%, exceeding the second-best model, VM-UNet, by 5.33 percentage points. This result highlights the model’s strong capability in boundary localization and structural continuity. These gains stem from the synergistic design of the network, in which dense skip connections facilitate multi-level feature propagation and DASS2D enhances directional contextual modeling during feature fusion.

Table 2. Quantitative comparison of different methods on the LMHLD test set.

Method	LMHLD (Test)
	OA (%)	P (%)	R (%)	F1 (%)	IoU (%)	BIoU (%)
AttUnet	95.26	85.38	75.20	79.97	66.62	49.13
TransUnet	95.23	86.97	73.00	79.37	65.80	47.09
VMUnet	95.56	84.33	79.45	81.82	69.23	50.48
SegFormer	94.72	80.33	76.89	78.57	64.71	47.33
Unet++	95.64	83.66	81.16	82.39	70.06	49.51
Ours	95.67	81.27	85.21	83.19	71.23	55.81

To qualitatively compare segmentation performance, several representative samples from the LMHLD test set are selected, as shown in Figure 6. Overall, NestedMambaUNet produces more complete landslide structures with improved regional consistency, consistent with the quantitative results. In the third row, where the landslide exhibits a large-scale contiguous pattern, methods such as AttUNet and TransUNet capture only partial regions, resulting in fragmented predictions and structural discontinuities. In contrast, NestedMambaUNet better preserves the global connectivity of the landslide region, indicating improved capability in modeling large-scale spatial structures. This improvement can be attributed to the multi-directional selective scanning mechanism, which strengthens the modeling of long-range spatial dependencies. In the complex scenario shown in the fifth row, where landslide areas are intermixed with bare ground, several methods exhibit noticeable missed detections. By comparison, NestedMambaUNet identifies a larger proportion of the landslide region, consistent with its higher Recall values. However, some false positives are still observed, where non-landslide regions are incorrectly classified as landslides. From a practical perspective, missed detections may pose a higher risk in emergency applications, while false positives can often be mitigated through post-processing or manual verification. Furthermore, the proposed method produces smoother and more continuous boundaries, whereas other methods tend to generate fragmented or blurred boundary predictions.

Figure 6. Visual comparison of segmentation results on the LMHLD test set. The results are color-coded based on the confusion matrix: red denotes false negatives (missed detections), and blue denotes false positives (false alarms).

Table 3 and Table 4 report the quantitative results of all models on the HR-GLDD validation and test sets, respectively. Overall, NestedMambaUNet achieves the best performance across most evaluation metrics, further demonstrating its robustness under different data distributions and geographic conditions. As shown in the test set results, despite significant differences between HR-GLDD and LMHLD in terms of land cover composition, imaging conditions, and landslide morphology, the proposed method maintains consistent advantages across multiple metrics. Specifically, Recall reaches 76.61%, outperforming all compared methods and exceeding the second-best model, TransUNet, by 12.01 percentage points, indicating enhanced detection capability in complex scenarios with reduced missed detections. IoU and F1-score reach 58.84% and 74.09%, respectively, both surpassing the competing methods, demonstrating stable overall segmentation performance under cross-domain settings. Although the proposed method does not achieve the best OA on the HR-GLDD validation set and shows slightly lower Precision on the test set, it still maintains competitive performance on key metrics including Recall, F1-score, IoU, and BIoU, indicating its overall effectiveness. In terms of boundary accuracy, BIoU reaches 51.28%, improving by 7.65 percentage points over TransUNet, suggesting that the proposed method maintains relatively stable boundary delineation capability under varying data distributions. This improvement can be attributed to the complementary effects of CoordConv-based spatial encoding and direction-adaptive modeling in DASS2D, which together enhance boundary continuity and structural consistency.

Table 3. Quantitative comparison of different methods on the HR-GLDD validation set.

Method	HR-GLDD (Val)
	OA (%)	P (%)	R (%)	F1 (%)	IoU (%)	BIoU (%)
AttUnet	94.39	77.85	62.67	69.44	53.19	41.13
TransUnet	93.95	72.11	66.01	68.93	52.59	43.45
VMUnet	93.98	73.10	64.55	68.56	52.16	42.79
SegFormer	92.95	67.46	58.49	62.65	45.62	38.47
Unet++	94.23	75.23	64.50	69.45	53.20	43.70
Ours	94.08	68.32	78.14	72.90	57.36	48.52

Table 4. Quantitative comparison of different methods on the HR-GLDD test set.

Method	HR-GLDD (Test)
	OA (%)	P (%)	R (%)	F1 (%)	IoU (%)	BIoU (%)
AttUnet	93.94	78.42	61.35	68.84	52.49	41.41
TransUnet	93.77	74.90	64.60	69.37	53.10	43.63
VMUnet	93.64	74.24	64.01	68.05	52.38	44.23
SegFormer	92.95	71.81	58.38	64.00	47.49	39.66
Unet++	94.01	79.25	61.16	69.04	52.72	39.91
Ours	94.15	71.73	76.61	74.09	58.84	51.28

Figure 7 presents representative visualization results from the HR-GLDD test set. Overall, NestedMambaUNet better preserves the spatial boundaries of landslide regions, which is consistent with the higher BIoU scores reported in the quantitative results. In the third and fourth rows, where low visibility and complex background interference weaken the spectral and textural characteristics of landslide areas, the discrimination task becomes more challenging. Under these conditions, both UNet++ and VM-UNet exhibit varying degrees of missed detections, with parts of the landslide regions only partially identified and the resulting predictions appearing fragmented. In contrast, the proposed method produces more complete and spatially coherent predictions, maintaining relatively stable structural representation even in complex scenes. This demonstrates enhanced capability in modeling long-range spatial dependencies, which can be attributed to the DASS2D.

Figure 7. Visual comparison of segmentation results on the HR-GLDD test set. The results are color-coded based on the confusion matrix: red denotes false negatives (missed detections), and blue denotes false positives (false alarms).

3.3. Computational Efficiency Analysis

In landslide emergency response scenarios, a model is expected to deliver not only high segmentation accuracy but also efficient inference to meet real-time processing requirements. Excessive computational complexity may hinder the practical deployment of a model in large-area remote sensing interpretation tasks. Table 5 compares the computational cost of different models.

Table 5. Comparison of Computational Efficiency Across Different Models.

Method	MACs	Params
AttUnet	16.667 G	34.879 M
TransUnet	8.215 G	66.821 M
VMUnet	1.884 G	60.275 M
SegFormer	819.839 M	7.715 M
Unet++	8.731 G	9.164 M
Ours	4.269 G	13.652 M

As shown in Table 5, NestedMambaUNet maintains a moderate computational cost while preserving strong feature representation capability. Compared with TransUNet and UNet++, the proposed method achieves lower computational complexity while delivering the best segmentation accuracy. Although SegFormer exhibits lower computational cost, it shows limited capability in capturing fine-grained structures such as landslide boundaries. Overall, NestedMambaUNet strikes a favorable balance between accuracy and efficiency, making it well-suited for time-sensitive landslide interpretation tasks. The reported MACs are evaluated under the unified input size of 128 × 128, which is consistent with the patch size adopted in the experimental datasets. Computational costs are expected to further increase under larger input resolutions in practical applications.

4. Discussion

4.1. Ablation Experiments

To evaluate the effectiveness of each component in NestedMambaUNet and validate the network design, ablation experiments are conducted on the HR-GLDD dataset, focusing on different scanning strategies and the contributions of key modules. In landslide extraction tasks, Recall and IoU are commonly used to assess detection capability and overall segmentation accuracy, while BIoU specifically evaluates boundary delineation quality. Accordingly, Recall, IoU, and BIoU are adopted to comprehensively evaluate model performance in the ablation study.

Table 6 summarizes the ablation results for each component. “Coord” denotes the proposed CoordConvBlock, and “Fusion” denotes the proposed DASS Fusion Block. “Baseline” refers to UNet++. Exp 6 incorporates the original equal-weight four-direction scanning strategy from VMamba, serving as a reference for evaluating the effectiveness of the direction-adaptive weighting mechanism in DASS2D.

Table 6. Quantitative ablation analysis of the proposed NestedMambaUNet.

Exp ID	Coord	DASS2D	Fusion	R (%)	IoU (%)	BIoU (%)	MACs	Params
Baseline				65.40	54.49	44.95	8.731 G	9.164 M
1	✓			70.28	55.47	49.61	9.619 G	12.848 M
2		✓		67.92	56.06	48.30	3.055 G	10.167 M
3			✓	68.66	55.83	43.98	2.189 G	5.289 M
4		✓	✓	72.78	58.23	49.78	3.388 G	10.097 M
5	✓	✓		71.88	57.09	51.15	3.919 G	13.459 M
6	✓		✓	71.81	56.63	49.46	4.153 G	13.322 M
Ours	✓	✓	✓	76.61	58.84	51.28	4.269 G	13.652 M

Overall, progressively integrating each component into the baseline UNet++ consistently improves segmentation performance across all evaluation metrics, validating the effectiveness of the proposed design. In Exp 1, introducing CoordConvBlock significantly improves Recall (65.40% → 70.28%) and BIoU (44.95% → 49.61%), indicating that explicit spatial coordinate encoding enhances the model’s capacity to capture structural and boundary information. In Exp 2, incorporating DASS2D raises IoU to 56.06% while maintaining competitive computational efficiency, suggesting that multi-directional scanning contributes to enhanced regional consistency in segmentation. In Exp 3, using only the DASS Fusion Block leads to a slight decline in boundary accuracy, indicating that its effectiveness in boundary refinement is more pronounced when combined with global context modeling. In Exp 4, jointly integrating DASS2D and the DASS Fusion Block further improves IoU (58.23%) and Recall (72.78%), demonstrating the complementary effects between global spatial modeling and adaptive feature fusion. In Exp 5, combining CoordConvBlock with DASS2D achieves the highest BIoU (51.15%), highlighting the benefit of coupling spatial positional encoding with directional context modeling. In Exp 6, replacing the direction-adaptive weighting strategy with the original equal-weight four-direction scanning of VMamba leads to degraded performance, confirming the importance of adaptive directional weighting in capturing spatial structure. Overall, the full NestedMambaUNet achieves the best performance among all variants, demonstrating that the integration of all components yields a more effective balance between feature representation and structural modeling.

To qualitatively analyze the attention behavior of different ablation variants in landslide regions, Grad-CAM is employed for visualization. Grad-CAM generates class-discriminative heatmaps via gradient backpropagation, where warmer colors indicate regions with higher contribution to the model’s prediction. The visualization is generated using the output feature maps of the final layer, and the activation maps are normalized to the range of [0, 1] for visualization. Figure 8 presents representative results across all ablation settings. The baseline model produces relatively broad but unfocused responses, with limited attention to small-scale landslide regions, indicating insufficient fine-grained representation capability. After introducing CoordConvBlock, attention becomes more concentrated on small landslide regions, while background interference is partially suppressed, suggesting enhanced spatial localization awareness. With the incorporation of DASS2D, the activation maps exhibit more continuous spatial distributions, indicating improved modeling of long-range structural dependencies within landslide regions. In Exp 3, where only the fusion mechanism is employed, the attention remains relatively scattered, with incomplete coverage of landslide structures. In Exp 4 and Exp 5, combining different components yields progressively more coherent and complete attention regions, with clearer emphasis along landslide boundaries, reflecting improved boundary sensitivity. In Exp 6, the equal-weight scanning strategy produces relatively uniform but less targeted responses in structurally complex areas, suggesting limited adaptability to spatially heterogeneous patterns. In contrast, the full model produces more focused and structurally consistent activation patterns, with clear responses along both the main landslide body and boundary regions. These results qualitatively confirm the effectiveness of each component in enhancing spatial representation and boundary-aware feature learning.

Figure 8. Visualization of ablation heatmaps for NestedMambaUNet on the HR-GLDD test set.

4.2. Limitations and Future Work

Although the proposed method achieves strong landslide extraction performance across multi-regional and multi-sensor datasets, several limitations remain. First, the NIR information is still relatively limited. At present, NIR data are simply concatenated with RGB channels as input, without explicit modeling of their physical significance or task-specific contribution. In addition, the relative spectral contribution and complementarity between RGB and NIR bands have not been systematically analyzed in the current study. Consequently, the potential of NIR information for improving landslide discrimination has not yet been fully realized.

Second, the model still produces a certain degree of false positives. While higher recall helps reduce missed detections, excessive false alarms may increase the burden of subsequent manual verification and compromise the overall efficiency of practical interpretation workflows. As shown in Figure 9, some regions with spectral and texture characteristics similar to landslides, such as bare soil, quarries, riverbanks, and road cut slopes, are prone to being incorrectly identified as landslide areas. In addition, although the proposed method demonstrates stable performance across multi-sensor datasets, its robustness under more diverse sensor characteristics and unseen geographic regions has not yet been systematically evaluated. Potential domain gaps caused by differences in terrain conditions, climate environments, and imaging characteristics may still affect the generalization capability of the model in practical deployment.

Figure 9. Visualization of typical failure cases of the proposed method.

To address these limitations, future work will focus on two directions. First, more effective utilization and interpretation of NIR information will be explored, including spectral contribution analysis and the incorporation of vegetation-related indices such as the normalized difference vegetation index (NDVI) to enrich spectral and physical feature representation. Second, more targeted false positive suppression strategies and imbalance-aware loss functions will be investigated to better distinguish landslides from visually similar land-cover types, such as bare soil, quarries, and road cut slopes, thereby improving the robustness and practical applicability of the model while maintaining high recall.

5. Conclusions

To address the challenges of incomplete landslide structure representation, imprecise boundary delineation, and insufficient global context modeling in remote sensing-based landslide interpretation, we propose NestedMambaUNet, a state-space model-based network for landslide detection. By integrating spatial structure perception and long-range contextual modeling capabilities, the proposed method improves the structural integrity and boundary representation of landslide regions. The experimental results on two public datasets, LMHLD and HR-GLDD, demonstrate that the proposed method achieves superior performance across multiple evaluation metrics. Specifically, the IoU and BIoU reach 71.23% and 55.81% on the LMHLD test set, and 58.84% and 51.28% on the HR-GLDD test set, respectively. In addition, the proposed method achieves strong Recall performance, effectively reducing missed detections while maintaining more continuous and complete landslide boundaries. Although the Precision is relatively lower compared with some competing methods, the overall F1-score remains higher, indicating that the proposed method achieves a better balance between Precision and Recall. These advantages indicate the potential of the proposed method for practical applications such as emergency landslide mapping and post-disaster remote sensing interpretation. Future work will further explore more effective utilization of spectral information, false positive suppression strategies, and model generalization under more complex scenarios to improve the robustness and practical applicability of the model.