VFM-MoME: A Remote Sensing Landslide Image Segmentation Network Guided by a Visual Foundation Model and a Mixture of Mamba Experts

Highlights

What are the main findings?

• We provide the VFM-MoME, which achieves efficient remote sensing landslide image segmentation under extreme conditions such as low resolution and abnormal lighting.
• We provide a dual-branch joint encoding architecture that utilizes the FAWB as the primary encoding branch and the visual foundation model fusion as the auxiliary branch, thereby addressing the limitation of generalizable features in specific landslide study areas.
• We develop the MoMEB to enable the decoder to process both global contextual information and local fine-grained features in landslides.
• The BUE is introduced to guide the model in analyzing difficult samples, further enhancing the model’s ability to handle ambiguous features.

What are the implications of the main findings?

• A new remote sensing method for landslide segmentation is proposed.
• Enhance the model’s ability to distinguish ambiguous boundaries in landslides.

Abstract

The linear computational complexity embraced by Mamba has demonstrated significant application potential in context modeling for the landslide segmentation tasks from remote sensing images. However, existing methods show deficiencies in terms of discrimination and generalization when applied to extreme remote sensing landslide scenarios, such as low resolution and abnormal lighting. To confront these challenges, we propose a remote sensing image landslide segmentation network (VFM-MoME) jointly guided by a vision foundation model and a mixture of Mamba experts. Specifically, we first design a dual-branch joint encoding architecture that integrates a frequency-aware wavelet block as the main encoding branch with the visual foundation model fusion as the auxiliary branch, thereby mitigating the issue of insufficient generalized features in specific landslide study areas. We also construct a mixture of Mamba expert block to enable the decoder to process both global context and local fine-grained features of landslides, addressing the shortcoming of simple serial Mamba in capturing local details and balancing between global semantic relationships and the edges and textural details of objects. Furthermore, we bring in a binary uncertainty enhancement module to guide the model in exploring challenging samples, thus enhancing the model’s ability to handle ambiguous features. Test results on the publicly available datasets of Landslide4Sense and GVLM demonstrate that our method achieves competitive performance.

1. Introduction

Landslides are common sudden geological disasters in mountainous areas. Characterized by their widespread distribution, hidden nature, and extensive destructive power, they are often triggered by factors such as heavy rainfall, seismic activity, and construction excavation, thereby posing a serious threat to transportation infrastructure, urban areas, the ecological environment, and the safety of people’s lives and property. In emergency response to landslide disasters, risk evaluation, and long-term monitoring, the rapid and precise automatic identification, segmentation, and detailed mapping of landslide areas from remote sensing images (RSI) serve as a key technological foundation for enhancing disaster prevention and mitigation capabilities [1].

In recent years, significant advancements in high-resolution remote sensing and aerial photography technologies have provided a vast source of detailed data for large-scale landslide surveys. However, traditional landslide interpretation methods mainly rely on manual visual interpretation or shallow machine learning, which suffer from deficiencies such as low efficiency, high subjectivity, and poor generalizability. Early deep learning-based interpretation of landslides in RSI was primarily dominated by convolutional neural networks (CNNs), with U-Net and its variants being widely applied to landslide semantic segmentation [2]. To improve adaptability to multiscale and complex scenarios, researchers have introduced multiscale fusion, dilated convolutions, and attention mechanisms; however, these strategies still struggle to overcome the limitations of local features and effectively model large-scale contextual dependencies. Transformer-based models have been shown to enhance global modeling capabilities through self-attention [1], yet they face challenges such as high computational complexity and deployment difficulties [3,4].

As a novel architecture based on the selective state space model (SSM), Mamba enables efficient modeling of long sequences with linear computational complexity, combining global modeling capabilities with efficient inference. It thus offers a new direction for the intelligent interpretation of landslides in RSI [5]. Focusing on landslide detection, segmentation, and high-resolution mapping tasks [6], many researchers have proposed a series of Mamba-based improved models, achieving significant progress in aspects such as multi-scale feature fusion for landslides [7] and fine-grained boundary and domain generalization [8,9].

Many of the aforementioned methods are contingent upon large amounts of training data for model training, but the performance of these existing models experiences a precipitous decline when applied to complex study areas with insufficient data [3]. Particularly in extreme scenarios such as low resolution and abnormal lighting conditions (severe overexposure or underexposure), the texture and contextual semantic information in images are severely compromised [1,10], preventing traditional methods from accurately distinguishing between landslide foregrounds and backgrounds at a fine-grained level. Furthermore, the presence of uncertain boundaries or ambiguous areas in real-world remote sensing landslide scenarios, such as areas where relevant objects are heavily obscured, can also result in misclassifications or missed detections [11].

To address these challenges, we propose a remote sensing landslide image segmentation network guided by a visual foundation model and a mixture of Mamba experts (VFM-MoME), which is designed for extreme scenarios such as low resolution and abnormal lighting conditions. To be specific, firstly, to efficiently extract fine-grained features and contextual ambiguity information from remote sensing landslide images, we design a frequency-aware wavelet block (FAWB) that combines image frequency information and spatial information for feature encoding within the encoder. We also leverage the advantages of Mamba in efficiently capturing long-range spatial context dependencies and the ability of a mixture of expert branches to flexibly integrate convolutional and locally aware units, thereby addressing the shortcomings of simple serial Mamba in capturing local details and simultaneously balancing between global semantic relationships and the edges and textural details of objects. Thus, we develop a mixture of Mamba expert block (MoMEB) to enable the decoder to process the global contextual information and local fine-grained features of landslides. Secondly, we introduce a globally guided visual foundation model fusion branch to assist model encoding, and adopt the pre-trained prior knowledge from the visual foundation model to address the lack of generalizable features in specific study areas. Finally, to tackle the uncertainty issue caused by ambiguous areas in remote sensing landslide images, such as areas with severe occlusion or blurred boundaries, we bring in a binary uncertainty enhancement module (BUE). This module uses structural optimization and loss functions to guide the model in analyzing difficult samples, thereby enhancing the model’s ability to handle ambiguous features. Test results on the publicly available datasets of Landslide4Sense [10] and GVLM [12] indicate that our method achieves competitive performance. The main contributions of this study are summarized as follows:

(1)

We provide the VFM-MoME, which achieves efficient remote sensing landslide image segmentation under extreme conditions such as low resolution and abnormal lighting.

(2)

We provide a dual-branch joint encoding architecture that utilizes the FAWB as the primary encoding branch and the visual foundation model fusion as the auxiliary branch, thereby addressing the limitation of generalizable features in specific landslide study areas.

(3)

We develop the MoMEB to enable the decoder to process both global contextual information and local fine-grained features in landslides, which addresses the shortcoming of simple serial Mamba in capturing local details while simultaneously balancing between global semantic relationships and the edges and textural details of objects.

(4)

The BUE is introduced to guide the model in analyzing difficult samples, further enhancing the model’s ability to handle ambiguous features.

2. Related Work

2.1. Mamba-Based Remote Sensing Landslide Image Segmentation Methods

In recent years, there has been a significant advancement in the field of Mamba models for remote sensing landslide image interpretation. These methods aim to enhance landslide segmentation performance through technical approaches including architectural design, feature fusion, causal learning, domain adaptation, and boundary optimization. Among these, attention U-Mamba [5] is designed as a concise U-shaped Mamba-based landslide segmentation method. This method adopts the U-Net framework as its foundation and also replaces the encoder’s convolutional module with the Mamba architecture to enhance context awareness of landslide areas with its global modeling capability. Additionally, it incorporates an attention mechanism to highlight key features and suppress background noise. For landslide detection tasks from very-high-resolution RSI, some researchers propose lightweight Mamba detection models, constructing an architecture that employs Mamba as the core encoder and a Multi-Layer Perceptron (MLP) as the lightweight decoder [4]. Certain studies also develop fine-grained landslide mapping methods that combine causal learning with Mamba. Among these, CSLMamba-LM [8] integrates Mamba with causal self-contrastive learning for fine-grained landslide mapping from high-resolution aerial images, and this model uses causal learning to mitigate spurious correlations caused by factors such as lighting and terrains. To address the domain drift issue in landslide remote sensing data across multiple study areas, CRLMDG-LM [9] is devised with a multi-target domain generalization framework guided by causal representation learning. This model utilizes an enhanced Mamba network as its backbone and learns domain-invariant features through causal intervention learning, significantly improving the model’s generalization performance in unknown target domains. MSCG-Net [7] is provided with a dual-branch, multi-scale fusion Mamba landslide detection network, which utilizes Mamba to model global context and CNN to extract local textures and boundary details from complex landslide images. Building on this foundation, DFmamba [13] is further introduced deformable convolutions, enabling convolution kernels to adaptively conform to irregular geometric shapes while enhancing multiscale recognition and boundary refinement capabilities. CS-Mamba [6] leverages an attention-guided cross-scale contextual semantic integration mechanism and constructs a multi-scale Mamba detection framework to address the challenges posed by the wide variation in landslide scales and complex contextual relationships. It enhances multi-scale feature representations in the encoder, implements bidirectional information flow fusion in the neck, and uses attention to guide multi-level feature alignment. This effectively addresses the challenges of large-scale variations in landslides and complex contextual relationships, maintaining high detection and segmentation accuracy even in complex scenarios.

Overall, existing Mamba-based remote sensing landslide segmentation methods generally employ hybrid architectures that combine Mamba with CNNs or attention mechanisms, balancing global modeling with local details. These methods also attempt to settle problems such as complexity in real-world environments by optimizing multi-scale fusion, handling irregular boundaries, and improving fine-grained mapping and domain generalization capabilities, thereby enhancing segmentation performance. However, a considerable number of these methods rely on large amounts of training data for model training. When faced with data scarcity in complex study areas, the direct application of these existing models frequently engenders a substantial decline in performance, which is an issue often overlooked in many studies.

2.2. Large-Model-Based Remote Sensing Landslide Image Segmentation Methods

The emergence of visual foundation models has provided a novel technical paradigm for landslide detection. Among these, the Segment Anything Model (SAM) has become a focal point in recent landslide detection studies due to its powerful general visual representation capability [3]. To address issues such as the wide variation in the scale of landslide objects and blurred boundaries in RSI, Yang et al. [14] design a multi-dimensional feature fusion strategy based on the SAM architecture, effectively integrating the spectral and spatial features of RSI to improve the accuracy and robustness of landslide detection. Yu et al. [15], on the other hand, construct the adaptive visual foundation model Landslidenet, integrating residual blocks, attention modules, and multiscale fusion operations. This model improves the effectiveness and robustness of landslide detection in complex scenes by suppressing redundant information and enhancing its focus on key areas. Tang et al. [16] focus on the specific scenario of earthquake-induced landslides. By introducing a frequency prompt mechanism based on SAM, they design LS-FPSAM, which enhances the model’s ability to identify landslide objects in complex post-earthquake terrain. Other studies also introduce transfer learning techniques to address the challenges of scarce labeled data and insufficient model generalization in landslide detection. For example, Hou et al. [17] propose a transfer learning scheme based on a visual foundation model, designing an adaptive transfer learning architecture that enables the segmentation capability of the visual foundation model to be transferred to the landslide semantic segmentation task with minimal training of parameters. Fu et al. [18] further optimize transfer learning strategies by designing a three-stage transfer learning framework within the proposed LSDSAM, enabling rapid cross-regional landslide detection in scenarios with limited data. In addition to improvements in visual foundation models, the optimization of specific segmentation networks is another important branch of landslide detection studies. Taking the work from Wang et al. [19] as a case in point, the dual-encoder segmentation network DS Net fully leverages the prior knowledge of large models to enhance the ability to extract and distinguish landslide features, thereby achieving intelligent and precise extraction of landslide areas.

These large models, when applied to the remote sensing landslide image segmentation task, strengthen the automation and accuracy of landslide detection through the implementation of strategies such as leveraging a visual foundation model as their core, being supported by transfer learning, and optimizing specific networks. However, many existing methods demonstrate suboptimal performance in extreme scenarios, such as those involving low resolution or abnormal lighting conditions.

Different from existing methods, the VFM-MoME proposed in this study employs a dual-branch joint encoding structure, with FAWB serving as the main encoding branch and visual foundation model fusion as the auxiliary branch, thereby addressing the issue of insufficient generalization in specific landslide study areas. The MoMEB is developed to enable the decoder to process both the global contextual information and local fine-grained features of landslides. It addresses the shortcomings of simple serial Mamba models, which lack sufficient local details, while also balancing global semantic relationships with the edges and textural details of objects. Furthermore, the BUE module is introduced to guide the model in exploring challenging samples, thereby augmenting the model’s capacity to process ambiguous features.

3. Methods

3.1. Overall Architecture

As shown in Figure 1, the encoder of VFM-MoME is designed as a dual-branch joint encoding architecture, while the decoder utilizes MoMEB to process the global contextual information and local fine-grained features of landslides. This architecture is a hierarchical multi-stage encoder–decoder network, aiming to deeply integrate the powerful semantic priors of the frozen visual foundation model with the linear-complexity global modeling capability of the SSM. To be specific, during feature processing, an input image $X\in R^{B\times C\times H\times W}$ is given. The model then extracts multi-level features through a convolution patch embedding backbone and successive downsampling stages. This module performs spatial downsampling and channel dimension mapping using two cascaded 3 × 3 convolutional layers (with a stride of 2 and padding of 1). Group normalization (GN) and the GELU activation function are applied after each convolution operation. After processing by this patch embedding module, the input image is progressively downsampled to one-quarter of its original resolution and projected into a high-dimensional feature space $F_{stem}\in R^{C\times {\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}}$.

In addition, the FAWB is introduced in Encoder1 to decouple and refine the low-frequency and high-frequency feature components separately, with the objective of preserving the global structure and fine textural features within the complex background of remote sensing landslide images. The FAWBs from various layers of the encoder are processed downward via convolutional downsampling. Specifically, the input feature map first undergoes layer normalization (LN). This operation stabilizes the feature distribution across both channel and spatial dimensions without relying on the input batch size, thereby effectively mitigating internal covariance shifts. Subsequently, the network performs spatial downsampling using a 2 × 2 kernel with a stride of 2. This operation precisely halves the spatial resolution of the feature map (such as from 56 × 56 to 28 × 28) while proportionally expanding the channel dimension (such as from 16 dimensions to 64 dimensions).

To fully exploit the representational potential of the input image, an additional Encoder2 is introduced to facilitate encoding. In this study, a global-guided FiLM fusion module is proposed, which utilizes a frozen DINOv2 foundation model and a global semantic token to dynamically modulate local dense features.

We design the MoMEB within the decoder to drive feature encoding, in order to address the spatial heterogeneity present in complex remote sensing landslide scenes across multi-source study areas. This module combines a direction-aware 2D selective scanning (SS2D) expert with a shared large-kernel convolutional path to capture omnidirectional long-range dependencies. Then, the decoder is employed to progressively reconstruct spatial resolution and output a final high-precision segmentation map by PixelShuffle upsampling, skip connections, and BUE. In detail, at each PixelShuffle upsampling level, the deep feature is first subjected to a 3 × 3 convolutional layer for channel feature alignment and spatial smoothing, followed by a PixelShuffle operation with a scaling factor of r = 2, which periodically rearranges the channel-dimension feature into the spatial dimension. This process achieves a lossless doubling of spatial resolution while proportionally reducing the number of channels (such as from C = 256 to C = 64). Each rearrangement is immediately followed by GN and the GELU activation function. Thus, $F_{input}\in R^{C\times H\times W},F_{output}\in R^{C/4\times 2H\times 2W}$ is present in the first two upsampling stages, PS1 and PS2. In particular, in the final upsampling module (PS3), the network restores the feature resolution from one-quarter of the original input (16 × 56 × 56) directly to the full-size resolution of 1 × 224 × 224 through two consecutive subpixel rearrangement operations, and outputs a single-channel segmentation response map. Subsequent sections will provide a detailed introduction to the key modules in the architecture.

3.2. FAWB

We aim for the model to preserve overall structural information while providing sharp boundary details during the remote sensing landslide image segmentation task [20]. As shown in Figure 1a, a two-dimensional Haar discrete wavelet transform (2D-DWT) is introduced for feature enhancement. The 2D-DWT decomposes middle features into low-frequency (LL) and high-frequency (LH, HL, HH) sub-bands in a lossless manner. Low-frequency context aggregation is performed using the LL sub-band. The principle of this process is to utilize the LL sub-band to capture the fundamental structure of the image and refine it through a global context attention (GCA) mechanism, thereby effectively suppressing background noise. Specifically, given the low-frequency feature $F_{LL}$ separated by the wavelet transform, the GCA module first performs spatial dimension squeeze via two-dimensional adaptive global average pooling, aggregating the global receptive fields of each channel into a scalar descriptor. Subsequently, a 1 × 1 convolution is applied to squeeze the channel dimension by a compression ratio of r = 4, with nonlinearity introduced via the GELU activation function. Then, the second 1 × 1 convolution precisely restores the number of channels to the original count. Finally, channel-level attention weights ranging from [0, 1] are generated using the Sigmoid activation function. The aforementioned process can be described by Equation (1).

$$L{L}_{out}=F_{LL}\oplus GCA(F_{LL})$$

(1)

To enhance high-frequency edge refinement, as high-frequency subbands can encode critical boundary information, we extract an edge-aware mask from the LL components using a lightweight edge attention gating network. This $M_{edge}$ mask spatially guides the feature enhancement of the high-frequency components. The final processing of the high-frequency components is described by Equations (2)–(5).

$$M_{edge}=Sigmoid(BN(Conv(LL)))$$

(2)

$$H{F}_{gated}=HF\odot M_{edge}$$

(3)

$$H{F}_{out}=HF\oplus SFRM(H{F}_{gated})$$

(4)

$$Refine(H{F}_{gated})=Sigmoid(DSC(H{F}_{gated}))$$

(5)

Here, DSC stands for the depth-wise separable convolution, and BN denotes batch normalization.

Subsequently, the inverse discrete wavelet transform (IDWT) is applied to reconstruct spatial features, ensuring that the boundary sharpness of landslide features is explicitly enhanced in the feature space. In practice, the optimized components $L{L}_{out}$ and $H{F}_{out}$ (re-decomposed into LH, HL, and HH) are fed into the IDWT. The IDWT perfectly reconstructs the decoupled frequency-domain signals into a single high-resolution spatial-domain response map.

To eliminate blocky artifacts that may result from the frequency-domain transformation and to smooth adjacent pixels, the reconstructed features undergo a Conv + GroupNorm + GELU operation (CGG) through a local feature fusion layer. In detail, CGG sequentially applies a 3 × 3 convolution, GN, and the GELU activation function. Additionally, we introduce global residual connections around the module, so the final output of the FAWB module can be expressed as Equation (6).

$$F_{FAWB}^i=CGG(IDWT(Chunk(H{F}_{out}),L{L}_{out}))\oplus F_{FAWB}^{i-1},(i=1,2)$$

(6)

3.3. Global-Guided Visual Foundation Model Fusion

Although baseline models such as DINOv2 provide extremely powerful frozen representations, directly concatenating features often leads to poor feature alignment [21]. To address this issue, in the Encoder2 shown in Figure 1, we simultaneously extract dense spatial patch tokens F_patch and global category token F_cls from the frozen DINOv2 model, where $F_{patch}\in R^{L\times 16\times 16},F_{cls}\in R^{1\times L}$. When dinov2_vitb14 is used, L = 768; while dinov2_vits14 is used, L = 384.

In the Encoder2 presented in Figure 1, to facilitate the efficient fusion of different features with minimal computational overhead and to promote the consistent alignment of large-scale background features in remote sensing landslide images guided by global semantic scaling, we do not simply use DINOv2 as the feature extraction backbone network [21,22], which is unlike many existing studies that directly use foundation models to assist in feature encoding. Rather, we introduce a fusion strategy based on feature-wise linear modulation (FiLM) to process the features output by DINOv2. The global token F_cls is projected to generate the scaling parameter $(\gamma )$ and the bias parameter $(\beta )$, as shown in Equation (7).

$$\left[\gamma ,\beta \right]=MLP(F_{cls})$$

(7)

$MLP(F_{cls})$: The input features pass through the first linear fully connected layer and the GELU activation function, and are projected into the target feature channel dimension. Then, the second fully connected layer further maps them into a parameter vector with twice the number of target channels. Finally, this high-dimensional vector is evenly split into two parts along the channel dimension, which serve as the spatial scaling factor $(Scale,\gamma )$ and translation factor $(Shift,\beta )$, respectively, for subsequent modulation operations. These parameters are broadcast across the spatial dimensions to modulate the dimension-reduced local features F_local, as indicated in Equation (8).

$$F_{local}=project(F_{patch})$$

(8)

$project(F_{patch})$: For the input F_patch, the module first applies a 1 × 1 point-wise convolution to aggregate information across channels and reduce the dimensionality, aligning it with the target feature dimension of the network (such as 256 dimensions). Then, GN and the GELU activation function are applied in sequence to stabilize the distribution of deep-layer features and endow the network with sufficient nonlinear representation power, as shown in Equation (9).

$$F_{modulated}=F_{local}\odot (1+\gamma )+\beta$$

(9)

This mechanism ensures that the global semantic context serves as a dynamic prior, guiding local spatial features to focus on task-relevant areas. This is particularly crucial for distinguishing highly similar complex backgrounds from foreground objects in RSI.

Following the dynamic modulation of local features by global semantics (FiLM), we introduce a spatial feature refinement module (SFRM) at the end of the fusion module to further enhance the local contextual coherence between adjacent pixels. Specifically, the modulated feature map is fed into a 3 × 3 local perceptual convolutional layer (with padding set to 1 to maintain constant spatial resolution). Next, the subsequent GN and GELU activation function effectively stabilize the distribution of deep-layer features while enhancing the network’s nonlinear representation power.

3.4. MoMEB

In recent years, Mamba-based architectures have demonstrated significant potential in the field of remote sensing image processing due to their ability to model long-range context while maintaining linear computational complexity. However, existing Mamba methods typically have certain limitations. Firstly, Mamba’s inherent sequential modeling paradigm disrupts the original two-dimensional spatial structure of images, leading to a loss of spatial coherence. Secondly, although multi-directional scanning is commonly used to enhance feature diversity, existing methods often treat all directions equally, ignoring the significant directional heterogeneity among different land cover types (such as roads and rivers) in remote sensing landslide scenes.

To address the aforementioned limitations, inspired by the mixture of experts (MoE) paradigm [11,23], we introduce a novel MoMEB to tackle with the spatial heterogeneity of multi-source complex remote sensing landslide scenes. The module structure is shown in Figure 1b. MoMEB adopts a structure similar to that of a standard Transformer block, with its core innovation being the replacement of the original attention mechanism with a shared large-kernel convolutional MoME (SLCMoME) module. Specifically, the normalization is performed via an LN layer, followed by processing via the SLCMoME module, as illustrated in Figure 1(b1). This module primarily consists of two core branches: the space and route selection expert path (SRSEP), which dynamically captures heterogeneous directional dependencies, and the shared large-kernel convolutional modulation path (SLCMP), which anchors local two-dimensional structures. Then, the output is further refined through an LN layer and an MLP layer, and is then enhanced again via residual addition.

3.4.1. SRSEP

SRSEP consists of multiple Mamba experts configured with different scanning directions, each of which uses a specialized SSM to capture structural information in a specific direction from the input. As shown in Figure 1(b3), we employ four standardized scanning trajectories: horizontal, vertical, S-shaped, and diagonal.

For a given input feature $X\in R^{B\times C\times H\times W}$, the j-th expert flattens the two-dimensional feature into a one-dimensional sequence fj according to its predefined scanning path. The t-th element of this sequence is denoted as $f_j^t$. The sequence is then fed into the SSM, where the state shift $h_j^t$ and the output $y_j^t$ are computed as shown in Equations (10) and (11).

$$h_j^t=\overline{A}{h}_j^{t-1}+\overline{B}{f}_j^t$$

(10)

$$y_j^t=C{h}_j^t+f_j^t$$

(11)

$\overline{A},\overline{B},C$ refers to trainable parameter matrixes. In this way, each expert is able to capture a unique spatial topology based on its specified scanning direction.

To enable the model to adaptively capture spatial context based on heterogeneous objects, we introduce a lightweight routing network for dynamic expert assignment. This router generates a set of weight coefficients from the input features using adaptive global pooling and a linear projection layer, as shown in Equation (12).

$$w_j=softmax(Router(x))$$

(12)

During the training phase, all experts participate in the forward pass to ensure diversity in feature representation. During the inference phase, to improve computational efficiency while maintaining classification performance, we leverage the sparsity of the routing mechanism to activate only the Top-k (such as k = 3) experts with the highest weights for computation. The final output of SRSEP $F_{SRSEP}$ is a weighted combination of the outputs from these activated experts, as shown in Equation (13).

$$F_{SRSEP}={\displaystyle \sum _{j\in Top-k}{w}_j•E_j(X)}$$

(13)

This selective inference strategy enables the model to focus computational resources on the most relevant directional features, effectively balancing performance and efficiency.

3.4.2. SLCMP and Adaptive Fusion

Although SRSEP offers robust dynamic direction sensitivity, the discrete routing and one-dimensional scanning mechanism of MoE inevitably lead to the fragmentation of the receptive field in two-dimensional space. To address this issue, we design a parallel SLCMP.

SLCMP combines 5 × 5 local DSC with 7 × 7 dilated convolutions (with a dilation factor of 3), thereby providing a massive continuous receptive field equivalent to 21 × 21 with an extremely low number of parameters. This shared branch acts as a “structural anchor” within the network, providing a smooth, coherent two-dimensional spatial foundation for the discrete Mamba expert features.

Next, to achieve optimal synergy between discrete-sequence features and continuous-space features, we deploy a joint router (Gate Mamba) at the end of the module. This router dynamically evaluates the relative importance of SRSEP and SLCMP features and outputs a pair of normalized fusion weights $(\psi ,\zeta )$. The final feature map of SLCMoME $F_{SLCMoME}$ computed as shown in Equation (14).

$$F_{SLCMoME}=\psi F_{SRSEP}\oplus \zeta F_{SLCMP}$$

(14)

This design not only preserves the Mamba network’s ability to capture complex, long-range directional dependencies but also ensures the local spatial consistency of features during the reconstruction process, significantly enhancing the model’s robustness in the landslide segmentation task from RSI.

3.5. BUE

Given that the blurred areas often found in many remote sensing landslide images tend to introduce uncertainties into predictions, we bring in an auxiliary prediction head at the middle stage of the decoder (as shown in Figure 1c) to generate a coarse probability map $p\in [0,1]$. We quantify the pixel-level epistemic uncertainty U as in Equation (15).

$$U=1-\left|2p-1\right|$$

(15)

As p → 0.5, U → 1 (extremely uncertain); as p → 0 or 1, U → 0 (high confidence). The uncertainty map is then normalized to $\widehat{U}$ using the Min-Max method, as shown in Equation (16).

$$\widehat{U}={\displaystyle \frac{U-U_{\min }}{U_{\max }-U_{\min }+\epsilon }}$$

(16)

$\epsilon ={10}^{-6}$ is used to prevent Arithmetic Exception.

After obtaining the normalized uncertainty map $\widehat{U}$, if it is directly multiplied by the feature $F_{dec}$ as a mask, it can easily cause abrupt skips in feature values at the boundaries between high and low uncertainty areas, thereby disrupting spatial continuity. To address this, we design a lightweight adaptive frequency-spatial attention mechanism for DSC. As shown in Figure 1(c1), we first perform element-wise multiplication between the main feature $F_{dec}$ and the uncertainty map $\widehat{U}$, which is $F_{dec}\odot \widehat{U}$, and feed the result as the prior input into the attention network. This network includes a 3 × 3 depth-wise convolution $D{W}_{3\times 3}$ to smooth the local spatial receptive field and a 1 × 1 pointwise convolution $P{C}_{1\times 1}$ for cross-channel information interaction. Finally, a Sigmoid activation is applied to generate attention weights ${\omega }_{att}$ with high nonlinear representation power, as provided in Equations (17) and (18).

$${\omega }_{att}=Sigmid(P{C}_{1\times 1}(D{W}_{3\times 3}(F_{dec}\odot \widehat{U})))$$

(17)

$$F_{enhanced}={\omega }_{att}•F_{dec}\oplus F_{dec}$$

(18)

This mechanism compels the network to explicitly allocate augmented representational capacity to the areas containing highly challenging and uncertain pixels prior to the final pixel-level upsampling stage.

3.6. Loss Functions

To simultaneously ensure the pixel-level classification accuracy and the spatial structural integrity in the landslide segmentation task from RSI, we design a multi-dimensional hybrid loss function strategy for end-to-end network optimization, so as to effectively address the severe class imbalance in the study area’s data and the interference caused by challenging samples (such as blurred boundaries and small-size landslides). This loss function consists of three components: a weighted combination of the base loss for the fundamental segmentation task $L_{Bce-Dice}$, a gradient penalty term $L_{FT}$, and an uncertainty-aware refinement loss (UARL) for handling challenging samples $L_{UAR}$.

To address the significant challenges posed by irregular geometries and varying feature scales in complex landslide remote sensing scenarios, using only a single cross-entropy loss $L_{Bce}$ makes it highly likely that small-scale landslide features will be overlooked by the model. However, the introduction of $L_{Dice}$ can effectively enhance the model’s sensitivity to small-scale landslide features. Therefore, $L_{Bce-Dice}$ is designed to be composed of $L_{Bce}$ and $L_{Dice}$, where the former calculates the cross-entropy between the predicted probability and the ground-truth label on a pixel-wise basis, while the latter evaluates the overall segmentation performance by calculating the overlap between the predicted result and the ground-truth label. The final loss, $L_{Bce-Dice}$, is defined as the weighted sum of the two, as indicated in Equation (19).

$$L_{Bce-Dice}=L_{Bce}+L_{Dice}$$

(19)

Next, in remote sensing scenarios where there are a large number of small or indistinguishable objects (such as small landslide branches), conventional loss functions often struggle to assign sufficient gradient penalties. To address this, we introduce the focal Tversky loss $L_{FT}$, as shown in Equation (20).

$$L_{FT}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B{(1-T_b)}^{\eta }}$$

(20)

B is the batch size, and T_b signifies the Tversky index for different batch sizes. η is the focus adjustment factor, which is used to guide the model to focus on capturing those highly challenging, hard-to-distinguish samples during the later stages of training.

To refine the model’s ability to handle challenging samples, we propose a novel $L_{UAR}$ that combines a binary uncertainty enhancement mechanism in the decoding stage. This mechanism utilizes the uncertainty map generated by the model itself to perform the spatial-level masking and clipping of backpropagated gradients.

The true label is $Y\in R^{H\times W}$, the probability map with auxiliary prediction is $P_{aux}\in R^{H\times W}$, and the normalized uncertainty response map is $\widehat{U}\in R^{H\times W}$. The network first binarizes the uncertainty map using a predefined threshold (set to 0.5) to extract a set of pixels that are extremely ambiguous or very difficult to classify, thereby generating a mask M, as shown in Equation (21).

$$M=I(\widehat{U}>0.5)$$

(21)

Subsequently, the basic binary cross-entropy loss matrix is computed for all pixels, and the mask M is used to filter out all high-confidence (easily classified) pixels. The loss is only computed for difficult pixels, as shown in Equation (22).

$$L_{AUX}={\displaystyle \frac{{\displaystyle \sum (L_{pixel}\odot M)}}{{\displaystyle \sum M+\epsilon }}}$$

(22)

This strategy forces the model to perform fine-grained local correction at the decision boundary in the feature space, significantly enhancing the model’s robustness against interference in the complex environment.

The final loss function in total is denoted as Equation (23), where λ₁, λ₂, and λ₃ is the weight parameter for each loss function, respectively.

$$L_{ALL}={\lambda }_1{L}_{Bce-Dice}+{\lambda }_2{L}_{FT}+{\lambda }_3{L}_{AUX}$$

(23)

4. Experiments

4.1. Dataset Description

We select two publicly available multi-source remote sensing landslide datasets, Landslide4Sense [10] and GVLM [12], to validate the performance of the model we proposed. The Landslide4Sense dataset is derived from optical images captured by the Sentinel-2 satellite and contains 3799 training samples with a spatial resolution of 10 m. The dataset covers four regions globally most prone to landslides, which are the Iburi-Tobu area of Hokkaido, the Kodagu district of Karnataka, the Rasuwa district of Bagmati, and the western Taitung County, respectively. We train all comparison methods using the Landslide4Sense dataset across different semantic segmentation models and then evaluate their segmentation performance. We divide the dataset into training and testing sets at a 7:3 ratio with random seeds to ensure the reproducibility of experimental results. In addition, to meet the requirements of the model architecture, we resized the images to 224 × 224 pixels for the experiments. Figure 2 displays several images selected from the dataset that exhibit different morphological features and their corresponding labels. As shown in the figure, the dataset exhibits a relatively low pixel resolution, is significantly affected by extreme conditions such as lighting, and contains complex and ambiguous information. Thus, the sample data is highly diverse and challenging.

The GVLM [12] is a dataset suitable for landslide change detection and image segmentation tasks. In this study, we employ post-landslide images and labels for validation. For the dataset division, we use the original division ratio of the GVLM dataset: 5861 for the training set and 733 for the test set.

4.2. Experimental Settings

4.2.1. Evaluation Metrics

In the quantitative evaluation presented in this study, four performance metrics are used, which are precision (PE), recall (RE), intersection over union (IOU), and F1 score (F1). These metrics are calculated based on true positives (TP), false positives (FP), and false negatives (FN), as delineated in Equations (24)–(27). In addition to these performance metrics, Paras (m) and Giga Floating-point Operations Per Second (GFLOPs) are employed to evaluate the computational complexity of all models compared.

$$PE={\displaystyle \frac{TP}{TP+FP}}$$

(24)

$$RE={\displaystyle \frac{TP}{TP+FN}}$$

(25)

$$IOU={\displaystyle \frac{TP}{TP+FP+FN}}$$

(26)

$$F_1={\displaystyle \frac{2\times PE\times RE}{PE+RE}}={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(27)

4.2.2. Baseline Methods

To comprehensively evaluate the advantages of our method, we select representative Mamba-based models designed specifically for remote sensing image segmentation tasks over the past two years, including PPMamba [24], UMFormer [25], AfaMamba [26], and BS-Mamba [27]. We also select representative remote sensing landslide mapping methods published in the past three years, including SCDUNet++ [28], CResU-Net [29], SAM-CFFNet [3], Trans-Unet [30], ResUNet-BFA [31], and GeoNeXt [1]. These methods employ hybrid designs combining CNN, Transformer, or Mamba network architectures and integrate key techniques such as attention mechanisms and multiscale feature processing to model local features or global information.

4.2.3. Implementation Details

In the experiments, we use C_Adamw as the training optimizer, with a momentum factor of 0.9 and a weight decay of 1 × 10⁻⁴. For all methods compared, the learning rate is set to 1 × 10⁻⁴, and a cosine annealing learning rate scheduler is used, allowing the learning rate to gradually decay from the maximum value to the minimum value of 1 × 10⁻⁷ in a cosine-like manner over each epoch. The batch size is set to 8, and the number of epochs is set to 50. The values of the loss function weights λ₁, λ₂, and λ₃ are taken as 1, 0.5, and 1, respectively. All image samples in the test dataset are resized to 224 × 224 pixels. All experiments for the methods compared are implemented using PyTorch 2.1.0 and Python 3.10 (Ubuntu 22.04) on an NVIDIA GeForce RTX 4090 GPUmanufactured by NVIDIA Corporation (Santa Clara, CA, USA).

4.3. Experimental Results

4.3.1. Quantitative Evaluation

The quantitative results in

Table 1. The quantitative results of the performance metrics, PE (%), RE (%), IoU (%), and F1 (%), and computational complexity for all methods compared on Landslide4Sense are shown. (The best result is marked in bold, and the second-best result is underlined. The same applies below.)

Methods	PE	RE	IOU	F1	GFLOPs	Paras (m)
PPMamba [24]	66.65	70.35	52.04	68.45	14.22	21.70
UMFormer [25]	65.60	63.61	47.70	64.59	2.27	12.37
AfaMamba [26]	66.08	70.41	51.72	68.18	5.36	13.48
BSMamba [27]	68.99	65.34	50.51	67.12	14.96	7.24
SCDUNet++ [28]	69.56	67.54	52.14	68.54	22.47	4.74
CResU-Net [29]	67.68	67.17	50.86	67.43	2.48	1.17
SAM-CFFNet [3]	66.71	65.42	49.32	66.06	129.27	307.70
Trans-Unet [30]	59.21	61.78	43.34	60.47	23.88	59.62
ResUNet–BFA [31]	67.47	67.78	51.09	67.62	44.66	102.59
GeoNeXt [1]	65.95	71.28	52.10	68.51	32.30	32.21
VFM-MoME-B	70.76	69.97	54.27	70.36	22.71	90.98
VFM-MoME-S	68.67	70.49	53.34	69.57	6.25	26.26

demonstrate that our method achieves the best performance on both IoU and F1 on the test set of the Landslide4Sense dataset. Here, VFM-MoME-B and VFM-MoME-S represent the evaluation results of DINOv2 using different versions of ViT-B/S. A comparison of the VFM-MoME-B method with the best-performing method among the other baselines reveals its superiority, with an increase of 2.07% and 1.77% in IoU and F1, respectively. It is worth noting that while our method gains only the second-best performance on PE and RE, the gap between our results and the best is relatively minor. Similarly, our method also achieves the best performance on the GVLM dataset as shown in

Table 2. The quantitative results of the performance metrics, PE (%), RE (%), IoU (%), and F1 (%) for all methods compared on GVLM are displayed.

Methods	PE	RE	IOU	F1
PPMamba [24]	82.25	86.55	72.93	84.35
UMFormer [25]	80.97	84.4	70.43	82.65
AfaMamba [26]	78.81	86.06	69.89	82.27
BSMamba [27]	81.77	86.07	72.21	83.87
SCDUNet++ [28]	83.87	87.12	74.62	85.47
CResU-Net [29]	83.00	86.43	73.43	84.68
SAM-CFFNet [3]	83.36	85.21	72.82	84.27
Trans-Unet [30]	81.27	85.46	71.4	83.31
ResUNet–BFA [31]	77.65	86.67	69.36	81.91
GeoNeXt [1]	81.87	87.12	73.03	84.41
VFM-MoME-B	87.58	88.54	78.66	88.06

, with performance gaps of 3.71%, 1.42%, 4.04%, and 2.59% over the second-place method across the four evaluation metrics. Overall, these results demonstrate that our method exhibits enhanced robustness and a more significant capability for fine-grained semantic segmentation of landslides in the study area under complex environmental conditions.

In addition, we analyze the results by combining model performance metrics with complexity metrics. As shown by the Flops (G) and Paras (M) values in Table 1, under different implementation schemes, the Paras (M) of VFM-MoME-B is comparable to that of ResUNet–BFA [30], but its GFLOPs are only about half as high. Similarly, while VFM-MoME-S has a slightly higher Paras (m) than PPMamba [23], BSMamba [26], and SCDUNet++ [27], its GFLOPs are less than half those of these models. These results reveal that our method offers significant advantages in both performance and complexity.

4.3.2. Qualitative Evaluation

To further analyze the performance advantages of our method, Figure 3 presents a qualitative evaluation result using a subset of representative samples randomly selected from the Landslide4Sense test set. As illustrated in row (13) of the figure, these test samples include a wide variety of complex backgrounds, with objects of varying shapes and sizes, and substantial variations in lighting conditions. The visual results reveal that many comparison methods exhibit varying degrees of FN and FP (that is, a large number of blue and red areas appearing in the mapped region), whereas our method exhibits relatively fewer. As shown in column (d) of Figure 3, most comparison methods present FN. This is because the landslide areas in the test samples contain a small amount of pale green texture similar to the background, causing many models to misclassify these areas as background. In column (g) of Figure 3, most comparison methods exhibit FP. This is because the visual features of the background rivers in the test samples are quite similar to those of the landslide areas, causing many models to misclassify the background rivers as landslide areas. Thus, there are a large number of red regions. However, overall, whether dealing with large landslides or smaller, fragmented landslide branches, our method demonstrates a clear advantage, indicating that it exhibits strong adaptability even in extreme conditions such as low resolution and abnormal lighting.

4.4. Experimental Analyses

4.4.1. Ablation Study

To further demonstrate the effectiveness and validity of our method, we sequentially incorporate the smallest components designed in VFM-MoME-B and VFM-MoME-S, which are FAWB, MoMEB, Encoder2, and $L_{AUX}$, to evaluate and compare their performance. As displayed in the ablation results in

Table 3. Ablation results for the performance metrics, PE (%), RE (%), IoU (%), and F1 (%), and computational complexity of the various functional modules of VFM-MoME-B on the test dataset Landslide4Sense are provided.

Baseline	FAWB	MoMEB	Encoder2 (ViT-B)	${\mathit{L}}_{\mathit{A}\mathit{U}\mathit{X}}$	PE	RE	IOU	F1	GFLOPs	Paras (m)	Train. Paras (m)
√					51.84	65.05	40.55	57.70	0.17	0.90	0.90
√	√				60.16	65.70	45.78	62.81	0.39	2.87	2.87
√	√	√			60.58	67.15	46.73	63.70	0.55	3.32	3.32
√	√	√	√		68.18	70.69	53.15	69.41	22.71	90.98	4.40
√	√	√	√	√	70.76	69.97	54.27	70.36	22.71	90.98	4.40

and

Table 4. Ablation results for the performance metrics, PE (%), RE (%), IoU (%), and F1 (%), and computational complexity of the various functional modules of VFM-MoME-S on the test dataset Landslide4Sense are provided.

Baseline	FAWB	MoMEB	Encoder2 (ViT-S)	${\mathit{L}}_{\mathit{A}\mathit{U}\mathit{X}}$	PE	RE	IOU	F1	GFLOPs	Paras (m)	Train. Paras (m)
√					51.84	65.05	40.55	57.70	0.17	0.90	0.90
√	√				60.16	65.70	45.78	62.81	0.39	2.87	2.87
√	√	√			60.58	67.15	46.73	63.70	0.55	3.32	3.32
√	√	√	√		67.36	69.94	52.24	68.63	6.25	26.26	4.20
√	√	√	√	√	68.67	70.49	53.34	69.57	6.25	26.26	4.20

, its performance improves to varying degrees after sequentially adding these modules, with a particularly significant improvement observed after adding Encoder2. In addition, the changes in FLOPS (G) and parameter (m) during the inference stage reflect that the total number of trainable parameters is relatively small. However, after introducing the foundation model, although the number of trainable parameters (Train. Paras (m)) does not undergo significant change, the total number of parameters (Paras (m)) increases dramatically. This is because the visual foundation model contains a large number of pre-trained parameters.

4.4.2. Parameter Analysis

To further illustrate the rationale behind selecting the specific value of k in the Top-k set with the highest weights in the design shown in Figure 1(b2), we test values of k equal to 1, 2, 3, and 4, respectively, as described in

Table 5. Evaluation results of performance metrics under different values of k in Top-k with the maximum weight.

Top-k = ?	PE	RE	IOU	F1
Top-1	69.06	70.12	53.68	69.86
Top-2	69.63	70.76	54.07	70.19
Top-3	70.76	69.97	54.27	70.36
Top-4	69.09	71.52	54.18	70.28

. Based on the performance evaluation metrics, the overall performance is optimal when k is set to 3, demonstrating that our choice of k = 3 is reasonable.

To further illustrate the rationale behind the values of the loss function weight parameters λ₁, λ₂, and λ₃ in Equation (23), multiple sets of parameter combinations are designed based on the valid ranges of each component loss for ablation testing. As displayed in

Table 6. Evaluation results of performance metrics under different combination values of λ₁, λ₂, and λ₃.

${\mathit{\lambda }}_1$	${\mathit{\lambda }}_2$	${\mathit{\lambda }}_3$	PE	RE	IOU	F1
1	1	1	68.22	72.55	54.22	70.32
1	1	0.5	69.46	70.73	53.95	70.09
1	0.5	1	70.76	69.97	54.27	70.36
1	0.5	0.5	69.50	71.14	54.21	70.31
0.5	0.5	1	69.03	70.51	54.14	70.25
0.5	0.5	0.5	68.16	72.06	53.91	70.06
0.5	0.5	0.3	68.70	71.25	53.79	69.96

, the best overall performance across all metrics is achieved when λ₁, λ₂, and λ₃ are set to 1, 0.5, and 1, respectively.

4.4.3. Ablation Visualization Analysis

We conduct a qualitative visual analysis of the ablation studies presented in Table 3 to gain a deeper understanding of how the design of each component affects the model’s ability to distinguish landslides at the pixel level. As depicted in the visual results in Figure 4, the addition of each functional module leads to varying degrees of improvements in FP and FN in the test samples. In particular, in the top two rows of Figure 4, as modules are progressively added, the FP and FN in the depicted landslide areas are significantly reduced. Furthermore, as can be seen in the bottom two rows, the presence of thick cloud cover and exposed rocks in the input samples lead to varying degrees of FP in the baseline model and previous improvements; however, the final model shows improvements across the board. These visual results further substantiate the validity of the design of each functional module in VFM-MoME-B.

4.4.4. Visualization Analysis of Feature Maps

To further illustrate the advantages of the proposed model in feature extraction and its interpretability, we employ Gard-CAM to visualize the feature maps from the final layer of the convolutional network of all comparison models listed in Table 1. As shown in the visualization results corresponding to the second and fourth rows on the right side of Figure 5, compared to other methods, the highlighted regions in the feature maps generated by our method are more concentrated in the local areas where landslides occur. In conjunction with the refined landslide mapping evaluation shown in the first and third rows on the right, it indicates that the key features in these regions play a significant role in the model’s decision-making process, further demonstrating the merits of our method.

5. Discussion

The findings of the aforementioned quantitative and qualitative analyses illustrate the merits of the proposed method, which is guided by a visual foundation model and a mixture of Mamba experts, in the low-resolution remote sensing landslide image segmentation. The ablation studies in Table 5 and Table 6 also validate the rationality of the design of each component, and overall, outstanding results are achieved. However, as analyzed in Section 4.4.1, the introduction of the visual foundation model yields significant performance improvements, but it also results in a sharp increase in the model’s total data volume. This poses challenges for future deployment of the model on resource-constrained platforms.

Moreover, while the integration of a visual foundation model has led to a substantial enhancement in performance, the feature processing steps within the DINOv2 component increase the model’s complexity. Although we have introduced feature map visualization in Figure 5 to further analyze which key areas the model focuses on during feature processing, thereby enhancing the model’s interpretability, we have not yet incorporated interpretability techniques into the model design [6,8,9]. Consequently, subsequent research endeavors will investigate lightweight techniques, including model distillation, parameter pruning, and quantization, to enhance the design of the visual foundation model [32]. The utilization of interpretable artificial intelligence methods, such as feature map visualization and causal analysis, will facilitate enhanced model analysis, thereby enhancing the model’s interpretability and generalization robustness.

6. Conclusions

We provide VFM-MoME to address the challenge of landslide segmentation in remote sensing images under extreme conditions such as low resolution and abnormal lighting. Specifically, we first design a dual-branch joint encoding architecture that employs FAWB as the main encoding branch and the visual foundation model fusion as the auxiliary branch, thereby mitigating the issue of insufficient generalized features in specific landslide study areas. We also construct the MoMEB to enable the decoder to process both global context and local fine-grained features of landslides, addressing the shortcoming of simple serial Mamba in capturing local details and balancing between global semantic relationships and the edges and textural details of objects. Additionally, we bring in a BUE module to guide the model in exploring challenging samples, thereby enhancing the model’s ability to handle ambiguous features. The tests on the publicly available Landslide4Sense dataset reveal that, compared to the best-performing method among all methods compared, the VFM-MoME-B method achieves 2.07% and 1.77% higher on the metrics IoU and F1, respectively. It also achieves the best performance on the publicly available GVLM dataset, with lead margins of 3.71%, 1.42%, 4.04%, and 2.59% over the runner-up across the four evaluation metrics. These results demonstrate that our method offers competitive performance advantages in landslide segmentation from RSI under extreme conditions such as low resolution and abnormal lighting.