SGH-Net: An Efficient Hierarchical Fusion Network with Spectrally Guided Attention for Multi-Modal Landslide Segmentation

Highlights

What are the main findings?

• A lightweight multi-modal landslide segmentation framework is developed that avoids naive input-level concatenation and instead injects multispectral band and terrain guidance into RGB features across multiple encoder stages.
• The proposed Guided Attention Block improves boundary delineation and reduces background confusion by selectively recalibrating RGB representations using multispectral and DEM-derived cues.

What are the implications of the main findings?

• This study indicates that modality-guided hierarchical fusion is more effective for landslide segmentation than the straightforward early fusion of heterogeneous inputs.
• The results indicate that physically grounded modality guidance can systematically mitigate background confusion, improving robustness to spectral confusion and fragmented target structure.

Abstract

Accurate landslide segmentation from remote sensing imagery is important for geohazard assessment and emergency response, yet it remains challenging because landslide regions are often spectrally confused with bare soil, riverbeds, shadows, and disturbed surfaces while also suffering from severe foreground–background imbalance. To address these issues, we propose an Efficient Spectrally Guided Hierarchical Fusion Network (SGH-Net) for multi-modal landslide segmentation. Instead of directly concatenating heterogeneous inputs at the image level, SGH-Net adopts an asymmetric encoder–decoder design in which a pretrained EfficientNet-B4 extracts RGB features, while two lightweight guidance encoders capture complementary multispectral band and DEM-derived terrain cues. These guidance features are progressively injected into the RGB backbone through multi-stage Guided Attention Blocks, enabling selective feature recalibration and reducing cross-modal interference. In addition, a hybrid Dice–Focal loss is used to alleviate class imbalance. Experiments on the Landslide4Sense dataset show that SGH-Net achieves the best overall performance among the compared methods under the adopted evaluation protocol, reaching 81.15% IoU and a 77.86% F1-score. Compared with representative multi-modal baselines, the proposed method delivers more accurate boundary delineation and fewer false alarms while maintaining favorable model complexity. These results indicate that modality-guided hierarchical fusion is an effective and efficient strategy for multi-modal landslide segmentation.

1. Introduction

Landslides are among the most destructive geological disasters worldwide, posing significant threats to human lives, infrastructure, and the natural environment [1,2,3,4]. The accurate and timely mapping of landslide-affected areas is therefore essential for disaster prevention, emergency response, and long-term environmental management [5,6]. Remote sensing, with its capability for large-scale and rapid data acquisition, has become the primary tool for landslide monitoring [7,8]. However, accurate landslide segmentation is a significant challenge. This difficulty stems from the inherent complexity of landslide processes. Landslides often exhibit high spatial heterogeneity, with ambiguous boundaries that are difficult to delineate, while their spectral signatures are frequently confused with those of other land cover types [9].

Early landslide segmentation approaches relied on manual visual interpretation or classical machine learning algorithms based on handcrafted features. These methods were often time-consuming, subjective, and difficult to scale across large regions [10]. With the advent of deep learning, convolutional neural networks (CNNs), particularly encoder–decoder architectures such as U-Net and DeepLabv3+, have revolutionized semantic segmentation by automatically learning hierarchical and discriminative features from imagery [11,12,13]. While these models have achieved remarkable success in remote sensing applications, most existing methods rely solely on RGB imagery, overlooking valuable information contained in multispectral data and Digital Elevation Models (DEMs).

Nevertheless, segmentation from optical images alone remains inherently difficult. Landslides in optical imagery often exhibit spectral ambiguity, with reflectance patterns similar to those of bare soil, dry riverbeds, or anthropogenic disturbances [14]. In addition, the spatial patterns of landslide deposits create challenges. Landslides often have irregular shapes and fragmented boundaries. Partial vegetation cover can further obscure them. These characteristics complicate the recognition process [15]. Terrain also causes variations in illumination. This leads to shadows and inconsistent surface reflectance. Such lighting effects can hide true landslide features. These challenges demonstrate the limits of approaches that rely solely on RGB imagery. Achieving reliable segmentation in diverse landscapes requires leveraging complementary data modalities [16,17,18].

To overcome these limitations, multi-modal data fusion has emerged as a promising direction [19,20,21]. Multispectral bands provide complementary information about material composition and vegetation condition, while DEM-derived terrain attributes offer useful topographic constraints related to slope instability [22,23]. However, existing multi-modal fusion strategies differ substantially in how they handle heterogeneous sources. Early fusion methods concatenate all channels at the input and process them with a single encoder, which is simple but often ineffective because RGB texture, multispectral reflectance, and terrain attributes have different physical meanings and statistical properties. Such shallow fusion may therefore introduce feature interference and fail to resolve spectral confusion in challenging scenes. To better preserve modality-specific information, recent methods adopt dual-encoder or multi-branch architectures and perform feature fusion at intermediate stages. These designs generally improve complementarity, but they may also introduce redundant computation and still lack explicit stage-wise alignment between auxiliary modalities and the main representation stream. Transformer-based cross-modal interaction modules can further enhance long-range feature exchange, yet their increased parameter count and inference cost reduce their appeal for efficient landslide mapping. Therefore, a key open problem is how to incorporate multispectral and terrain priors into RGB feature learning in a selective, hierarchical, and lightweight manner.

To address this problem, we propose SGH-Net, an efficient multi-modal landslide segmentation framework based on an asymmetric encoder–decoder architecture. A pretrained EfficientNet-B4 serves as the primary RGB pathway, while two lightweight guidance encoders extract complementary multispectral band and DEM-derived features. Instead of treating all modalities as equally mixed raw inputs, the auxiliary modalities are distilled into guidance priors and injected into multiple backbone stages through Guided Attention Blocks. This design allows the network to recalibrate RGB features using physically meaningful cues, thereby reducing background confusion and improving boundary delineation. To further address the pronounced foreground–background imbalance in landslide segmentation, we adopt a hybrid Dice–Focal loss. We demonstrate on a dataset that the proposed design achieves the best overall performance among the compared methods under the adopted protocol while maintaining a favorable trade-off between accuracy and computational efficiency.

2. Related Work

2.1. Landslide Segmentation Networks

The backbone architectures for landslide recognition currently face a dual challenge of balancing efficiency and precision. Traditional CNN-based networks and its variants remain foundational due to their local receptive fields and inductive biases, which provide high stability in capturing fine-grained landslide boundaries. For example, Liu et al. [24] introduced the AST-UNet model, integrating the channel attention and spatial cross-attention mechanisms with UNet, significantly enhancing the identification capability of landslide regions. Additionally, Lin et al. [25] developed the CBAM-UNet model, incorporating the convolutional block attention module to better capture landslide boundary features. Dimitrovski et al. [26] adopted a lightweight EfficientNet that enhanced semantic segmentation in remote sensing imagery.

The recent advent of Vision Transformers, exemplified by architectures like SegFormer and Cross-modal Transformers, has redefined state-of-the-art benchmarks by leveraging self-attention mechanisms to capture long-range dependencies in complex disaster environments [27,28]. However, despite their significant breakthroughs in accuracy, the substantial computational overhead and high parameterization of these models pose critical challenges for rapid deployment in resource-constrained emergency response settings [29]. In addition, compared with general semantic segmentation tasks, landslide segmentation presents several distinctive challenges. First, landslide regions are frequently confused with surrounding classes that exhibit similar optical appearance, leading to spectral ambiguity. Second, landslide shapes are often irregular and fragmented, making boundary recovery difficult. Third, the foreground class usually occupies only a small portion of the image, which aggravates optimization imbalance. These properties suggest that auxiliary modalities should not simply be stacked with RGB inputs but should instead be used to guide the learning of landslide-discriminative features.

2.2. Innovations in Feature Alignment and Guidance Mechanisms

Multi-modal data fusion is the process of integrating information from multiple data sources into a unified representation to improve model robustness and expressiveness. Traditional fusion typically operates at three primary levels: early fusion, late fusion, and intermediate fusion [30]. Early fusion, which involves concatenating multi-modal data at the input layer, is the simplest approach. This method, however, forces a single encoder network to process physically disparate data types from its initial layers, confounding the feature learning process. Conversely, late fusion strategies combine the outputs of separate, modality-specific encoders at a high level. While this avoids input-level dissonance, it severely restricts the potential for the network to learn rich, cross-modal correlations at intermediate stages of feature extraction [31,32].

To overcome these bottlenecks, recent multi-modal remote sensing segmentation studies have increasingly moved toward modality-aware feature alignment and guided fusion [33]. The first category includes dual-encoder or multi-branch CNN fusion architectures, represented by methods such as DE-Unet [34], FFSNet [35], and LMFNet [36]. These methods improve over naive early fusion by preserving modality-specific representations before feature interaction. Their central idea is to encode optical and auxiliary modalities separately and then fuse them at one or multiple feature stages. This design better exploits cross-modal complementarity and generally improves segmentation quality. However, these architectures still exhibit two common limitations. First, maintaining multiple feature extraction branches often introduces non-negligible architectural redundancy, increasing parameter count and computational cost. Second, although they perform feature fusion, the auxiliary modalities are usually merged through direct concatenation, summation, or symmetric interaction, without explicitly acting as selective guidance priors for the main visual stream. Consequently, the model may still be insufficiently targeted in suppressing spectrally confusing background responses. The second category includes attention-based and transformer-based cross-modal fusion methods, such as CM-TF [28], which strengthen interaction between heterogeneous modalities by explicitly modeling long-range dependencies and feature correlations. These methods are more expressive in theory and can capture richer cross-source relationships. Nevertheless, their gains often come at the price of heavier computation and more complex optimization. For landslide segmentation, where the target is sparse, boundaries are ambiguous, and rapid inference may be desirable, such heavy cross-modal interaction may not offer the best accuracy–efficiency trade-off.

2.3. Challenges and Design Motivation

Landslide segmentation from multi-modal remote sensing imagery remains challenging for three main reasons [37]. First, landslide scars may exhibit optical characteristics similar to those of bare soil, dry riverbeds, exposed rock, and anthropogenic disturbances, which causes severe spectral confusion. Second, landslide regions are often spatially sparse, fragmented, and boundary-irregular, making precise delineation difficult. Third, RGB imagery, multispectral bands, and DEM-derived terrain cues contain complementary but heterogeneous information, and directly concatenating them at the input level may introduce shallow layer interference rather than effective fusion [38,39]. These limitations motivate the design of SGH-Net, which uses auxiliary modalities as selective guidance signals for hierarchical RGB feature recalibration.

3. Methods

3.1. Proposed Method: Spectral-Guided Hierarchical Fusion Network (SGH-Net)

Multi-modal data provide complementary information for landslide segmentation. However, RGB imagery, multispectral bands, and DEM-derived terrain features differ substantially in physical meaning, information density, and statistical distribution. Traditional early fusion, which directly concatenates heterogeneous modalities at the input, forces a single encoder to process physically disparate signals in shallow layers and may therefore amplify spectral ambiguity rather than resolve it. To address this issue, we propose SGH-Net, an asymmetric encoder–decoder framework composed of one primary RGB backbone, two lightweight guidance encoders, and a multi-level guided fusion module. The guidance encoders first extract compact modality-specific representations from multispectral and terrain inputs. These features are then used to generate stage-wise guidance signals that selectively recalibrate RGB backbone features through Guided Attention Blocks. The framework is as shown in Figure 1.

Figure 1. The overall framework of the proposed SGH-Net for multi-modal landslide segmentation. The EfficientNet-B4 backbone extracts hierarchical RGB features at multiple resolutions. The multispectral guidance encoder and DEM guidance encoder generate complementary auxiliary representations from multispectral band and terrain inputs, respectively. These guidance features are injected into the RGB backbone through Guided Attention Blocks (GABs) at multiple encoder stages. SGM denotes spectral guidance modality. STEM denotes Shallow Feature Extraction Module. The decoder progressively aggregates the recalibrated multi-scale features to produce the final landslide probability map.

3.2. Primary Backbone Encoder

The core of our visual feature extraction is a pretrained EfficientNet-B4 encoder. Effective feature extraction is essential for achieving robust segmentation performance in multi-modal remote sensing.

Given an input RGB image, the Shallow Feature Extraction Module (STEM) first produces an initial feature map ${\mathbf{I}}_{\mathbf{R}\mathbf{G}\mathbf{B}}\in {\mathbb{R}}^{C\times H\times W}$. The MBConv block initiates with an expansion layer, where a pointwise 1 × 1 convolution increases the channel dimension by a ratio t:

$${\mathbf{F}}_{\mathrm{e}\mathrm{x}\mathrm{p}}=\phi \left(\mathrm{B}\mathrm{N}\left({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left({\mathbf{F}}_{\mathrm{i}\mathrm{n}}\right)\right)\right)$$

(1)

Subsequently, a depthwise convolution with kernel size k × k and stride s is applied to each channel independently, preserving spatial locality while reducing computational overhead:

$${\mathbf{F}}_{\mathrm{d}\mathrm{w}}=\phi \left(\mathrm{B}\mathrm{N}\left({\displaystyle {\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{\mathrm{k}·\mathrm{k}}^{\mathrm{d}\mathrm{w}}}\left({\mathbf{F}}_{\mathrm{e}\mathrm{x}\mathrm{p}}\right)\right)\right)$$

(2)

The intermediate feature maps are then passed through a Squeeze-and-Excitation recalibration stage. Global average pooling aggregates channel descriptors:

$$\mathbf{z}={\displaystyle \frac{1}{\mathbf{H}·\mathbf{W}}}{\displaystyle \sum _{i=1}^{\mathbf{H}}}{\displaystyle \sum _{j=1}^{\mathbf{W}}}{\mathbf{F}}_{\mathrm{d}\mathrm{w}}\left(c,i,j\right)$$

(3)

which are transformed by a bottleneck MLP with reduction ratio r:

$$s_i={\sigma \left({\mathbf{W}}_2·\delta \left({\mathbf{W}}_1·\mathbf{z}\right)\right)}_i$$

(4)

where $\mathbf{z}$ denotes the channel-wise statistic produced by a global average pooling operation on the input feature map U, $\delta$ is the Rectified Linear Unit (ReLU) function, σ is the sigmoid function, and ${\mathbf{W}}_1$ and ${\mathbf{W}}_2$ are the learned weight matrices of the two fully connected layers in the “Excitation” step. The final output ${\tilde{\mathbf{F}}}_{\mathrm{d}\mathrm{w}}$ is obtained by rescaling the input with the activations s:

$${\tilde{\mathbf{F}}}_{\mathrm{d}\mathrm{w}}=s_i\odot {\mathbf{F}}_{\mathrm{d}\mathrm{w}}$$

(5)

Finally, a projection layer consisting of a pointwise 1 × 1 convolution followed by batch normalization compresses the channels back to the desired output dimension:

$${\mathbf{F}}_{out}=\mathrm{B}\mathrm{N}\left({Conv}_{1\times 1}\left({\tilde{\mathbf{F}}}_{\mathrm{d}\mathrm{w}}\right)\right)$$

(6)

Through the stacking of MBConv blocks with SE modules, the backbone encoder captures both local spatial patterns and global contextual dependencies at multiple scales. The resulting hierarchical features not only preserve fine-grained texture details but also encode semantic information, providing a solid foundation for subsequent guided fusion and decoding stages.

3.3. Lightweight Guidance Encoders

A key innovation of our architecture is the use of two lightweight, custom-designed guidance encoders to process the auxiliary spectral and topographical information shown in Figure 2.

Figure 2. A flowchart of the two lightweight guidance encoders in SGH-Net. The spectral guidance encoder extracts discriminative spectral representations from multispectral inputs. The DEM guidance encoder processes elevation and slope channels in a similar manner to capture terrain-aware features.

• The spectral guidance encoder $E_{SGM}$ takes the spectral guidance modality (SGM) ${\mathbf{I}}_{SGM}\in {\mathbb{R}}^{9\times H\times W}$ as input and outputs ${\mathbf{F}}_{SGE}$.
• The DEM guidance encoder $E_{DEM}$ takes ${\mathbf{I}}_{\mathrm{D}\mathrm{E}\mathrm{M}}\in {\mathbb{R}}^{2\times H\times W}$ as input and outputs ${\mathbf{F}}_{\mathrm{D}\mathrm{E}\mathrm{M}}$.

Each guidance encoder is composed of three sequential blocks, each comprising a 3 × 3 convolutional layer, batch normalization, ReLU activation, and an SE layer.

The outputs from these two guidance encoders, ${\mathbf{F}}_{SGE}$ and ${\mathbf{F}}_{\mathrm{D}\mathrm{E}\mathrm{M}}$, are then concatenated along the channel dimension, yielding a unified combined guidance feature map ${\mathbf{F}}_{\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}\in {\mathbb{R}}^{{(C}_{D3}+C_{S3})\times H\times W}$:

$${\mathbf{F}}_{\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}=\mathrm{Concat}\left({\mathbf{F}}_{SGE},{\mathbf{F}}_{\mathrm{D}\mathrm{E}\mathrm{M}}\right)$$

(7)

The combined guidance feature map ${\mathbf{F}}_{\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}$ is computed once at the native input resolution by concatenating the outputs of the spectral and DEM guidance encoders. For each encoder stage k, a scale-specific guidance field is obtained by bilinear resampling and is then used to generate attention gates. The output channel widths of the multispectral and DEM guidance encoders are set to 64 and 32, respectively. This asymmetric allocation is motivated by the fact that multispectral inputs contain richer material cues and typically exhibit higher intra-class variability, whereas DEM-derived features mainly provide lower-dimensional geometric priors for boundary and terrain regularization. The adopted channel setting provides a practical balance between segmentation performance and computational cost.

3.4. Multi-Level Guided Attention Fusion Mechanisms

The core contribution of SGH-Net lies in injecting the combined guidance representation into the hierarchical features of the RGB backbone through a series of Guided Attention Blocks (GABs). One GAB is placed after each spatial downsampling stage so that auxiliary modality cues can influence feature learning at multiple semantic levels. This design is specifically motivated by the characteristics of landslide segmentation. In remote sensing scenes, landslide areas are frequently confused with background classes such as bare soil, riverbanks, exposed bedrock, and shadowed slopes. Meanwhile, landslide regions are often small, fragmented, and boundary-irregular. Simply concatenating auxiliary modalities does not ensure that physically meaningful cues will be used at the appropriate semantic stage. In contrast, the GAB converts multispectral band and terrain features into stage-wise guidance gates, enabling the backbone to suppress responses that are inconsistent with material or topographic evidence while preserving boundary-sensitive structures. Therefore, the GAB is not a generic attention add-on but a task-oriented mechanism for reducing spectral confusion and improving landslide delineation. A flowchart is illustrated in Figure 3.

Figure 3. A flowchart of the guidance attention fusion block. Guidance features are resampled and transformed into attention weights, which modulate backbone features through element-wise multiplication.

Formally, let ${\displaystyle {\mathbf{F}}_{\mathrm{R}\mathrm{G}\mathrm{B}}^{\left(\mathrm{k}\right)}}\in {\mathbb{R}}^{C_k\times H\times W}$ denote the feature map from the k stage of the RGB backbone. The full-resolution guidance feature is first resampled to the same spatial size:

$${\displaystyle {\mathbf{F}}_{\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}^{\left(\mathrm{k}\right)}}=\mathrm{Interpolate}\left({\mathbf{F}}_{\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}},\mathrm{size}=\left(H_k,W_k\right)\right)$$

(8)

where ${\displaystyle F_{Guidance}^{\left(k\right)}}$ ∈ R₉₆ × H_k × W_k.

Then, a 1 × 1 convolution followed by a sigmoid activation is used to generate the attention gate:

$${\mathbf{A}}^{\left(k\right)}=\sigma \left({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left({\displaystyle {\mathbf{F}}_{\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}^{\left(\mathrm{k}\right)}}\right)\right)$$

(9)

where A^(k) ∈ R_Ck × H_k × W_k denotes the channel-wise attention weights.

The backbone feature map is then modulated by these attention weights through element-wise multiplication and subsequently refined by a convolutional block consisting of a 3 × 3 convolution, batch normalization, and ReLU activation:

$${\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(\mathrm{k}\right)}}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}\left(\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{B}\mathrm{N}\left({\displaystyle {\mathbf{F}}_{\mathrm{R}\mathrm{G}\mathrm{B}}^{\left(\mathrm{k}\right)}}\odot {\mathbf{A}}^{\left(k\right)}\right)\right)\right)$$

(10)

where ⊙ denotes element-wise multiplication, Conv_3×3 is a 3 × 3 convolution, BN is batch normalization, and ReLU is the activation function. The output ${\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(\mathrm{k}\right)}}$ ∈ R^Ck×Hk×Wk represents the enhanced, multi-modally informed feature map for stage k.

This process is repeated for each of the five stages of the backbone encoder, producing a list of fused features $\left\{{\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(1\right)}},{\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(2\right)}},{\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(3\right)}},{\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(4\right)}},{\displaystyle {\mathbf{F}}_{\mathrm{F}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}^{\left(5\right)}}\right\}$. This list serves as the input to the UNet decoder via skip connections.

3.5. UNet-Style Decoder

The decoder of the proposed SGH-Net follows a classic UNet-style architecture designed to progressively recover spatial resolution and reconstruct fine-grained details of landslide regions. By integrating hierarchical encoder features with guided skip connections, the decoder ensures that both semantic and spatial information are preserved. This process is particularly important in landslide detection, where small-scale boundaries and subtle texture differences need to be accurately delineated.

The upsampling operation restores spatial resolution by doubling the size of the feature maps. In our framework, bilinear interpolation followed by a 1 × 1 convolution is adopted, which reduces computational cost while preserving feature smoothness. The upsampling can be expressed as follows:

$${\mathit{y}}_{up}={\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times 1}\left(\mathrm{I}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{p}\left({\mathit{y}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}},\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}=2\right)\right)$$

(11)

where $y_{up}$ denotes the upsampled feature, $y_{fuse}$ is the fused feature from the previous stage, and $Interp\left(·\right)$ represents bilinear interpolation. This step is crucial for recovering the resolution necessary for pixel-level segmentation.

The upsampled features are concatenated with the corresponding fused encoder features delivered by the Guided Attention Fusion Blocks. This skip connection mechanism allows the decoder to exploit both high-level semantics and low-level spatial cues, which are essential for delineating landslides with irregular boundaries. The concatenation process ${\mathit{y}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}$ is defined as follows:

$${\mathit{y}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({\mathit{y}}_{up},{\mathit{x}}_{skip}\right)$$

(12)

where $x_{skip}$ is the guidance-enhanced encoder feature at the corresponding resolution. This operation integrates complementary information across multiple scales.

Following concatenation, the feature maps are refined through two successive 3 × 3 convolutional layers, each followed by batch normalization and ReLU activation. This step enhances feature discriminability and enforces local consistency. The refinement process is expressed as follows:

$${\mathit{y}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{B}\mathrm{N}\left({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}({\mathit{y}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}},{\mathbf{W}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}})\right)\right)$$

(13)

where ${\mathbf{W}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}$ represents the convolutional weights. This operation ensures that the fused features adapt to the target task and suppress redundant or noisy information.

Finally, a segmentation head is applied at the highest resolution output. It consists of a 3 × 3 convolution that maps the feature representation to a single-channel probability map, followed by a sigmoid activation:

$${\mathit{y}}_{\mathrm{o}\mathrm{u}\mathrm{t}}=\mathrm{\sigma }\left({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{3\times 3}({\mathit{y}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}},{\mathbf{W}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}})\right)$$

(14)

where ${\mathbf{W}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}$ is the feature map of the target channel dimension, and $\sigma \left(·\right)$ denotes the sigmoid function. This generates the final landslide probability mask, ensuring the precise detection of landslide boundaries in remote sensing imagery.

In summary, the UNet-style decoder of SGH-Net progressively upsamples and refines features through interpolation, skip connection concatenation, and convolutional refinement, before producing the final segmentation map. This design effectively integrates multi-scale and multi-modal information, leading to accurate and robust landslide segmentation across diverse terrain conditions.

3.6. Hybrid Dice–Focal Loss

Landslide segmentation suffers from severe foreground–background imbalance because landslide pixels typically occupy only a small portion of the image. To improve optimization under this condition, we use a hybrid Dice–Focal loss:

$$\mathcal{L}={\lambda }_{dice}\ast {\mathcal{L}}_{\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}}+{\lambda }_{focal}{\mathcal{L}}_{\mathrm{f}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}$$

(15)

where ${\lambda }_{\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{e}}$ and ${\lambda }_{\mathrm{f}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}$ denote the weights of the two loss terms. Dice loss emphasizes region-level overlap and boundary integrity, while Focal loss down-weights easy background pixels and encourages the model to focus on hard landslide samples. In this work, equal weighting is adopted as a balanced default to jointly optimize overlap consistency and hard sample discrimination without introducing additional loss engineering complexity. So, we set ${\lambda }_{\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{e}}={\lambda }_{\mathrm{f}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}$ = 0.5.

The Dice loss is defined as follows:

$${\mathcal{L}}_{\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}}=1-{\displaystyle \frac{2{\sum }_i{p}_i{y}_i+ϵ}{{\sum }_i{p}_i+{\sum }_i{y}_i+ϵ}}$$

(16)

where $p_i$ and $y_i$ denote the predicted probability and ground truth label of pixel i, respectively, and ϵ is a small constant for numerical stability.

The Focal loss is defined as follows:

$${\mathcal{L}}_{\mathrm{F}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}=-{\displaystyle \frac{1}{N}}{\sum }_i\alpha {(1-p_i)}^{\gamma }\log (p_i)$$

(17)

where α is the class-balancing factor, and γ is the focusing parameter.

3.7. Inference with Test Time Augmentation

To further improve prediction robustness, we apply test time augmentation (TTA) during inference. Specifically, each test sample is transformed into augmented variants using a predefined set of geometric transformations, including horizontal flip, vertical flip, and 90° rotations. Each augmented sample is fed into the trained model independently. The resulting probability maps are then inversely transformed back to the original image coordinates and averaged pixel-wise to obtain the final prediction.

Unless otherwise stated, the same TTA protocol is applied to all compared methods for fairness. TTA improves robustness to orientation variation and local spatial perturbations without changing the model parameters or training procedure.

4. Experimental Setup

To rigorously evaluate the effectiveness of our proposed multi-modal fusion architecture for landslide detection, we designed a comprehensive set of experiments. This section details the dataset used, the data augmentation strategies employed to enhance model robustness, and the specific implementation parameters and evaluation metrics used for quantitative analysis.

4.1. Datasets

The dataset utilized in this study is a preprocessed version of the Landslide4Sense benchmark, a large-scale collection of multi-modal remote sensing images curated for landslide detection [40]. We evaluate the proposed method on the Landslide4Sense dataset, which contains 3799 multi-modal image patches annotated for binary landslide segmentation. Each sample is provided with 14 channels, including RGB imagery, nine multispectral bands, and DEM-derived terrain information. Following our experimental protocol, the dataset is partitioned into training, validation, and test subsets with a ratio of 60%/20%/20%. All comparative experiments are conducted under the same split, preprocessing pipeline, and optimization settings to ensure the internal consistency of the evaluation.

A critical characteristic inherited from the Landslide4Sense dataset is its severe class imbalance. Although landslide events are present in approximately 58% of the images in the source data, they constitute only 2.3% of the total pixels. Furthermore, the proportion of landslide pixels within a single image varies drastically from as low as 0.0061% to 47.53%. This inherent imbalance poses a significant challenge for training robust and accurate segmentation models. Landslide4Sense, as a competition benchmark for multi-modal landslide mapping, has encouraged the development of a range of fusion-oriented solutions that exploit the complementarity of spectral and topographic information. These efforts collectively suggest that benchmark performance depends not only on whether multiple modalities are used but also on how feature interaction is organized across the network hierarchy.

4.2. Data Augmentation

To enhance model generalization and robustness against overfitting, we applied a series of data augmentation strategies tailored for remote sensing imagery. Geometric transformations: Random rotations within ±15°, horizontal and vertical flipping, and 0.8 to 1.2 random scaling are applied to increase spatial diversity. Spectral augmentations: Random ±10% brightness adjustments, hue-saturation shifts, and Gaussian noise injection are employed to simulate atmospheric variability and sensor noise. Band-level dropout: Random spectral band dropout is used to improve robustness against missing or corrupted channels, forcing the model to adaptively exploit complementary information. Spatial consistency enforcement: For DEM-guided inputs, topographic maps are co-augmented with optical imagery to preserve cross-modal alignment during transformations.

These augmentations significantly expand the effective training set size and improve resilience to spectral ambiguity, illumination changes, and terrain-induced distortions.

4.3. Implementation Details

All experiments were conducted using the PyTorch 2.5.1 framework on a server equipped with an NVIDIA H100 GPU. To train all models, we employed the AdamW optimizer with an initial learning rate set to 2 × 10⁻³ and a weight decay of 1 × 10⁻², based on our extensive hyperparameter tuning. The batch size was set to 64. We used a Cosine Annealing learning rate scheduler over a maximum of 200 epochs. To ensure optimal convergence and prevent overfitting, we implemented an early stopping mechanism with a patience of 20 epochs, monitoring the validation loss. The model with the lowest validation loss was saved for final evaluation. For reproducibility, a fixed random seed of 42 was used across all experiments.

For the efficiency comparison, inference time is reported as the average forward-pass latency per image patch on a single NVIDIA H100 GPU under batch size 1 after warm-up. Unless otherwise stated, the reported latency excludes data loading and offline preprocessing. All compared models were benchmarked under the same implementation environment and with the same input resolution. TTA was disabled when measuring latency to ensure a fair comparison of model-side inference cost.

4.4. Evaluation Metrics

To provide a comprehensive assessment, we evaluated the models on the test set using five standard metrics for semantic segmentation.

Intersection-over-Union (IoU)—Measures overlap between predicted and ground truth segmentation masks:

$$\mathrm{IoU}={\displaystyle \frac{TP}{TP+FP+FN}}$$

(18)

Accuracy (ACC)—The ratio of correctly predicted observations to the total observations:

$$\mathrm{ACC}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(19)

Precision (Pre)—The ratio of correctly predicted positive observations to the total predicted positive observations:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(20)

Recall—The ratio of correctly predicted positive observations to all observations in the actual class:

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(21)

F1-Score (F1)—The harmonic mean of precision and recall, providing a single score that balances both metrics.

$$\mathrm{F}1=2\cdot {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\cdot \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(22)

In these formulas, TP, TN, FP, and FN represent the number of True Positives, True Negatives, false positives, and false negatives at the pixel level, respectively.

5. Results

5.1. Comparative Methods

To evaluate SGH-Net, we compare it with both conventional segmentation backbones and representative multi-modal fusion models. The single-stream baselines include UNet [41], DeepLabv3+ [42], EfficientNet-B4 [43], and SegFormer [27]. For these models, the full 14-channel multi-modal input is provided through direct channel concatenation. To better reflect the performance of architectures explicitly designed for multi-modal fusion, we further include DE-Unet [34], FFSNet [35], LMFNet [36], and CM-TF [28] as representative multi-modal baselines. All models are trained and evaluated under the same preprocessing pipeline and optimization settings. Unless otherwise stated, the same TTA protocol is applied to all models during final inference.

Table 1 reports a quantitative comparison between SGH-Net and the competing methods. The best performance is highlighted in bold in the table. Among the single-stream baselines, SegFormer achieves the strongest overall performance, whereas EfficientNet-B4 provides a comparatively favorable efficiency profile. The multi-modal baselines consistently outperform the single-stream models, confirming the value of incorporating auxiliary spectral and terrain cues for landslide segmentation. Among these multi-modal methods, FFSNet achieves the strongest baseline IoU, while DE-UNet and LMFNet also deliver competitive results.

Table 1. Performance comparison of different models in landslide segmentation.

Model	IoU	Acc	Recall	Pre	F1	FLOPs (G)	Para (M)	Time (ms)
UNet	72.49	98.09	67.24	59.97	63.57	10.71	32.52	4.26
DeepLabv3+	72.18	98.04	66.56	58.28	62.88	9.23	26.68	5.36
EfficientNet	73.08	98.23	67.62	63.08	66.95	2.70	20.23	13.47
Segformer	74.51	98.64	67.33	66.72	67.02	12.45	61.29	29.82
CM-TF	68.62	98.04	61.49	52.01	56.35	5.44	24.76	12.68
DE-UNet	78.83	98.91	75.64	72.49	74.03	9.60	38.24	27.37
FFSNet	79.04	98.93	75.4	73.32	74.35	10.18	38.70	27.51
LMFNet	78.87	98.9	76.28	72.04	74.1	10.19	38.71	28.2
SGH-Net	81.15	99.06	78.96	76.58	77.86	5.37	22.43	16.08

* IoU/Acc/Recall/Pre/F1 are reported in percentage (%).

SGH-Net achieves the best overall performance among all compared methods, reaching 81.15% IoU and a 77.86% F1-score. Compared with the strongest multi-modal baseline, SGH-Net improves IoU by 2.11 percentage points and the F1-score by 3.51 percentage points. These gains suggest that stage-wise guidance injection and selective feature recalibration are more effective than either naive early fusion or heavier symmetric multi-modal fusion strategies for this task.

In addition to accuracy, SGH-Net maintains a favorable efficiency–performance trade-off. In terms of efficiency, the lowest FLOPs and parameter count are achieved by EfficientNet, while UNet has the shortest inference time. Although SGH-Net is not the absolute lightest model, it requires substantially fewer FLOPs and parameters than several stronger multi-modal baselines such as DE-UNet, FFSNet, and LMFNet while delivering clearly superior segmentation accuracy. This result supports the fact that SGH-Net offers a favorable accuracy–efficiency trade-off. For intuitive analysis, Figure 4 presents visualizations of the segmentation results across methods together with error overlays.

Figure 4. Prediction results on Landslide4sense using representative methods (error overlays are shown on prediction masks, where red indicates false positives, and blue indicates false negatives).

Figure 4 presents qualitative comparisons on several challenging test scenes. To improve readability, we retain only the most representative methods in the main paper and move the full comparison of all evaluated models to the Appendix A. Compared with the baseline method, SGH-Net produces predictions that are visually closer to the ground truth masks, with fewer false alarms in confusing background regions and an improved recovery of fragmented landslide areas. The most evident improvements are observed along irregular boundaries and in small, scattered regions, where direct or weakly aligned fusion strategies tend to either over-segment background textures or miss subtle landslide pixels. These visual results are consistent with the quantitative findings and support the effectiveness of modality-guided feature recalibration.

Despite the overall improvement, SGH-Net still exhibits several failure modes in difficult scenes. First, false positives may occur in exposed bare soil or riverbank regions whose optical texture and multispectral response partially resemble those of landslide scars. Second, false negatives are more likely for very small or partially vegetation-covered landslide patches, where the target occupies only a limited number of pixels or is visually occluded. Third, slight boundary deviations may still appear in transition regions with strong shadows or abrupt terrain changes, indicating that local topographic cues are sometimes insufficient to fully resolve fine-scale ambiguity. These cases suggest that, while the proposed guided fusion substantially reduces spectral confusion, performance remains sensitive to extreme background similarity, occlusion, and very small target extent.

5.2. Quantitative Comparison

To analyze the contribution of each component in SGH-Net, we conduct a step-by-step ablation study, as summarized in Table 2. M0 is an RGB-only UNet with an EfficientNet-B4 encoder. M1 directly concatenates all 14 input channels at the image level. M2 introduces separate guidance encoders for multispectral and DEM-derived inputs. M3 further injects the guidance features into multiple encoder stages through simple fusion blocks. M4 replaces these simple fusion blocks with the proposed Guided Attention Blocks. M5 uses the same architecture as M4 but adopts the hybrid Dice–Focal loss. M6 applies TTA during inference on top of M5.

Table 2. Ablation study of SGH-Net on Landslide4Sense dataset.

Model	RGB	Spectral+DEM	Guidance Encoder	Fusion Block	Guidance Attention	Hybrid Loss	TTA	mIoU	Acc	Recall	Pre	F1
M0	√	×	×	×	×	×	×	72.64	98.13	67.30	62.35	66.88
M1	√	√	×	×	×	×	×	72.98	98.19	67.36	62.64	66.89
M2	√	√	√	×	×	×	×	77.40	98.79	74.84	69.10	71.81
M3	√	√	√	√	×	×	×	78.22	98.86	75.61	70.76	73.10
M4	√	√	√	√	√	×	×	80.41	99.01	78.15	74.74	76.41
M5	√	√	√	√	√	√	×	80.68	99.03	77.81	75.82	77.43
M6	√	√	√	√	√	√	√	81.15	99.06	78.96	76.58	77.86

* All values are reported in percentage (%).

The ablation results reveal a clear progression. First, naive early fusion in M1 brings only marginal improvement over the RGB-only baseline, indicating that direct channel concatenation is insufficient for exploiting heterogeneous modalities. Second, introducing dedicated guidance encoders in M2 yields a substantial gain, confirming that modality-specific feature extraction is important for reducing cross-modal interference. Third, replacing single-stage fusion with multi-stage guidance injection in M3 provides additional improvement, showing the value of hierarchical interaction across semantic scales.

The largest architectural gain is observed when M3 is upgraded to M4 by replacing simple fusion with the proposed Guided Attention Block. This modification improves IoU from 78.22% to 80.41%, indicating that selective recalibration is more effective than direct feature mixing for landslide segmentation. Adding the hybrid Dice–Focal loss in M5 further improves the balance between precision and recall, and TTA in M6 provides a final incremental gain. Overall, the ablation study demonstrates that the effectiveness of SGH-Net does not stem from a single component but from the coordinated combination of modality-specific encoding, hierarchical guidance injection, and task-aware feature recalibration.

Figure 5 provides a compelling visual narrative that complements our quantitative findings. The visual reduction in red and blue areas from left to right indicates a progressive improvement in model performance. The baseline model M0 produces a coarse segmentation with significant missed areas and numerous scattered false positives, indicating poor localization and high noise. M1 and M2, by incorporating multi-modal data, show improved completeness of the main landslide body but still suffer from imprecise boundaries and FP predictions. The introduction of multi-stage fusion in M3 results in a more cohesive prediction, yet the boundaries remain somewhat jagged. A clear visual breakthrough is seen with M4, which utilizes the Guided Attention Block. The segmentation mask becomes significantly cleaner, with much sharper and more accurate boundaries. The error map for M4 shows a dramatic reduction in both FP and FN compared to M3, visually confirming the quantitative leap in performance. This demonstrates the attention module’s ability to filter noise and focus on true landslide features. The results from M5 and M6 are visually the most accurate, closely aligning with the ground truth mask. The boundaries are smooth and precise, and spurious predictions are almost entirely eliminated. M6 captures the intricate shape of the landslide with high fidelity, which corroborates its superior quantitative scores.

Figure 5. A progressive qualitative comparison of the ablation models (M0–M6) on representative test scenes. (White indicates the prediction results, red indicates false positives, blue indicates false negatives, and the yellow dashed boxes mark representative local regions where the progressive improvements are the most visible).

In summary, both the quantitative metrics and the qualitative visualizations consistently demonstrate the value of each component of our proposed SGH-Net. The progressive reduction in prediction errors from M0 to M6 validates our architectural choices, with the Guided Attention Block being the most impactful contributor to the model’s final high-fidelity segmentation performance.

6. Discussion

The present results support our working hypothesis that, in multi-modal landslide segmentation, the effectiveness of auxiliary information depends not merely on its inclusion but on how it is integrated into the feature learning process. SGH-Net achieves the best overall performance among all compared methods, reaching 81.15% IoU and a 77.86% F1-score while maintaining a favorable accuracy–efficiency trade-off relative to stronger multi-modal baselines. The ablation study further clarifies this finding: direct early fusion yields only marginal improvement over the RGB-only baseline, whereas modality-specific guidance encoding, hierarchical feature injection, and especially the proposed Guided Attention Block lead to consistent and substantial gains. These results are in line with previous studies suggesting that the direct fusion of RGB, multispectral, and DEM data is often suboptimal because these modalities differ in physical meaning and statistical characteristics. By contrast, the proposed framework uses multispectral and terrain cues as selective guidance priors for hierarchical RGB feature recalibration, which appears more effective for suppressing background confusion and improving the delineation of fragmented and boundary irregular landslide regions.

From an application perspective, SGH-Net is particularly suitable for patch-based post-event landslide mapping from multi-modal satellite imagery, where the rapid delineation of affected regions is required with limited computational overhead. Potential use cases include regional landslide inventory generation after extreme rainfall or seismic events, prescreening candidate areas for manual interpretation, and serving as a segmentation module in broader geohazard assessment pipelines. Despite these promising results, several limitations should be acknowledged. The current evidence is limited to the Landslide4Sense benchmark. Therefore, the present results should be interpreted as demonstrating effectiveness under this benchmark setting rather than as proof of universal applicability across all geographic regions or sensor configurations. Future work should therefore focus on cross-region validation and broader evaluation under more diverse multi-modal remote sensing conditions.

7. Conclusions

In this study, we proposed SGH-Net, an efficient multi-modal landslide segmentation framework based on spectrally guided hierarchical fusion. The network adopts an asymmetric encoder–decoder architecture, in which a pretrained EfficientNet-B4 serves as the primary RGB backbone, while lightweight multispectral and DEM guidance encoders provide complementary auxiliary cues. Through multi-stage Guided Attention Blocks, these auxiliary modalities are injected into the RGB feature hierarchy to improve feature representation for landslide segmentation. A hybrid Dice–Focal loss and test time augmentation are further introduced to enhance optimization under class imbalance and improve inference robustness.

The experimental results on the Landslide4Sense dataset demonstrate that SGH-Net achieves the best overall performance under the adopted evaluation protocol, reaching 81.15% IoU and a 77.86% F1-score while maintaining competitive model complexity. The comparative and ablation results consistently verify the effectiveness of the proposed design, particularly the combination of modality-specific guidance encoding and hierarchical attention-based feature fusion. Overall, this work shows that lightweight guided fusion provides an effective solution for multi-modal landslide segmentation and offers a practical framework for accurate landslide mapping from remote sensing imagery.