YOLOv11-SAFM: Enhancing Landslide Detection in Complex Mountainous Terrain Through Spatial Feature Adaptation

Highlights

What are the main findings?

• YOLOv11-SAFM integrates a Spatially Adaptive Feature Modulation (SAFM) module, optimized MPDIoU bounding box regression loss, and a multi-scale training strategy, significantly improving small-scale landslide detection under complex mountainous conditions.
• Compared with Mask R-CNN and YOLOv8, the model shows notable improvements in precision, recall, F1-score, and mAP@0.5 for small landslide detection.

What are the implications of the main findings?

• The SAFM module and MPDIoU loss enhance feature representation and localization accuracy, enabling robust and efficient automatic landslide detection.
• YOLOv11-SAFM has strong potential for application in geohazard monitoring and early warning systems in complex plateau environments.

Abstract

Landslide detection in mountainous regions remains highly challenging due to complex terrain conditions, heterogeneous surface textures, and the fragmented distribution of landslide features. To address these limitations, this study proposes an enhanced object detection framework named YOLOv11-SAFM, which integrates a Spatially Adaptive Feature Modulation (SAFM) module, an optimized MPDIoU-based bounding box regression loss, and a multi-scale training strategy. These improvements strengthen the model’s ability to detect small-scale landslides with blurred edges under complex geomorphic conditions. A high-resolution remote sensing dataset was constructed using imagery from Bijie and Zhaotong in southwest China including GF-2 optical imagery at 1 m resolution and Sentinel-2 data at 10 m resolution for model training and validation, while independent data from Zhenxiong County were used to assess generalization capability. Experimental results demonstrate that YOLOv11-SAFM achieves a precision of 95.05%, recall of 90.10%, F1-score of 92.51%, and mAP@0.5 of 95.30% on the independent test set of the Zhaotong–Bijie dataset for detecting small-scale landslides in rugged plateau environments. Compared with the widely used Mask R-CNN, the proposed model improves precision by 13.87% and mAP@0.5 by 15.7%; against the traditional YOLOv8, it increases recall by 27.0% and F1-score by 22.47%. YOLOv11-SAFM enables efficient and robust automatic landslide detection in complex mountainous terrains and shows strong potential for integration into operational geohazard monitoring and early warning systems.

1. Introduction

Landslides are a prevalent geohazard in Southwest China, with hotspots concentrated in the mountainous provinces of Yunnan, Sichuan, and Guizhou, where steep relief, active tectonics, and recurrent heavy rainfall prevail [1]. Landslides represent a major hazard to human safety and property, while also compromising essential infrastructure, including water management, transportation, and communication systems. They lead to ecological degradation, thereby profoundly impacting regional sustainable development [2,3]. According to the report released by the Yunnan Provincial Department of Emergency Management, geological disasters in Yunnan Province in 2024 affected approximately 79,300 people and caused direct economic losses of about 1.517 billion yuan (source: https://yjglt.yn.gov.cn/html/2025/tjfx_0110/4029810.html, accessed on 10 January 2025). For instance, the landslide event in Liangshui Village, Zhenxiong County, which occurred suddenly on 22 January 2024, behind a low-lying village in winter, resulted in 44 fatalities [4]. This tragedy underscores the suddenness and high risk of landslides under unusual spatial-temporal conditions [5]. With the intensification of climate change and increasing human disturbances, landslide events have become more frequent, widespread, and abrupt, highlighting the urgent need for more effective identification methods and intelligent monitoring techniques [6]. This trend poses greater challenges and demands for the efficient identification and intelligent monitoring of geological hazards.

Visual interpretation is one of the earliest traditional methods used for landslide identification. Manual expert mapping exploits contrasts in texture, tone, and morphology observed in remote-sensing data and terrain information to separate landslides from surrounding areas [7,8]. Xue et al. [9] compiled an inventory of 3979 landslides in Zhenxiong County using high-resolution Google Earth imagery, which significantly supported the development of a regional landslide database and disaster risk assessment. However, this method is highly dependent on expert judgment, is time-consuming, and involves strong subjectivity, thereby limiting its applicability to large-scale, time-critical landslide detection.

The pixel-based feature thresholding method is one of the commonly used automated approaches for landslide identification. The pixel-based feature threshold method is one of the commonly used automated approaches for landslide identification. It determines landslide areas by analyzing differences in spectral, textural, and topographic features, and applying one or more predefined thresholds [10,11]. However, the threshold values are often region-specific and experience-driven, resulting in poor generalization performance. This limits the method’s applicability across diverse geomorphic settings and constrains its use in large-scale landslide detection. Machine learning approaches commonly draw on multi-dimensional inputs, including spectral, textural, and topographic information. They use classifiers such as Support Vector Machines (SVM) [12,13], Random Forests (RF) [14,15], and Artificial Neural Networks (ANN) [16,17] to distinguish landslide from non-landslide areas, offering high accuracy and moderate generalization capabilities. However, these methods rely heavily on manual feature engineering and selection, involve complex parameter tuning, and require substantial effort, which limits their applicability in large-scale and topographically complex regions.

In recent years, deep learning—particularly Convolutional Neural Networks (CNNs)—has demonstrated strong feature learning capability and generalization performance in remote-sensing-based landslide identification [18,19,20,21,22]. With the evolution of network architectures, advanced U-Net variants, lightweight and efficient CNN models, and attention-enhanced networks have gradually become important research directions in geospatial object recognition. For example, Ghorbanzadeh et al. [23] constructed the Landslide4Sense dataset and systematically evaluated 11 mainstream segmentation models, highlighting the stable multi-scale representation and boundary delineation abilities of the U-Net family. Building on this work, Ghorbanzadeh et al. [24] proposed an integrated approach combining ResU-Net with rule-based OBIA for multi-temporal Sentinel-2 landslide detection, achieving more than a 22% improvement in mean intersection-over-union compared with ResU-Net alone. Meanwhile, efficient CNN architectures have also shown considerable promise. Li et al. [25] introduced FCADenseNet, which integrates surface deformation features and incorporates an attention mechanism, markedly improving landslide detection accuracy and demonstrating the potential of combining InSAR data with deep learning. Ji et al. [26] constructed a landslide dataset for Bijie City using high-resolution optical imagery and DEMs, and proposed a CNN model enhanced with spatial–channel attention. Their model achieved an F1-score of 96.62%, significantly improving detection accuracy under complex background conditions. Overall, although U-Net variants, efficient CNN architectures such as DenseNet, and attention-enhanced networks have substantially advanced landslide detection, notable challenges remain, including limited inference efficiency, susceptibility to missing small-scale landslides, and vulnerability to interference from complex mountainous backgrounds.

The YOLO (You Only Look Once) series, as a representative one-stage object detection algorithm, offers advantages such as end-to-end training, fast inference speed, and a streamlined architecture [27,28,29,30]. By striking a strong speed–accuracy trade-off, YOLO is well suited to rapid landslide localization in remote-sensing imagery. It leverages multi-scale convolutional networks to extract morphological features of landslides (e.g., arcuate main scarps and accumulation bodies), spectral characteristics, and topographic context (e.g., steep-slope associations). During detection, predefined anchor boxes are used to match the aspect ratios of landslides, while classification loss (Focal Loss) and regression loss (CIoU) are jointly optimized to achieve accurate localization [31,32,33]. Ju et al. [34] employed YOLOv3 and RetinaNet to identify relict landslides, demonstrating the feasibility of using YOLO in complex background environments. This model readily accommodates the integration of attention mechanisms, thereby enhancing object detection performance [35,36,37]. Hou et al. [38] introduced YOLOX-Pro, an upgraded YOLOX variant that integrates Varifocal Loss and attention modules, substantially enhancing the identification of small and intricate landslides. Chen et al. [39] developed LSI-YOLOv8, a YOLOv8-based detector featuring a lightweight architecture and refined loss functions. The model outperforms existing mainstream approaches in both accuracy and detection speed. YOLOv11 [40] further incorporates novel architectural components such as C3k2 and C2PSA, achieving notable improvements in detection accuracy, boundary localization, and multi-task adaptability. It supports a wide range of functions including detection of objects, instance segmentation, pose inference, and oriented bounding box localization, facilitating use on diverse platforms, from lightweight edge devices to large-scale computing infrastructures. However, the original YOLO series still faces challenges in landslide detection, including missed detection of small targets, blurred boundaries, and significant interference from complex terrain backgrounds. These issues limit its practical applicability for automatic landslide identification in highland and mountainous regions. In high-resolution optical imagery of complex mountainous regions, small-scale landslides exhibit strong spectral heterogeneity, fragmented textures, and extremely small spatial scales. These characteristics can lead to feature loss during the downsampling process in convolutional neural networks, resulting in missed detections or unstable localization in traditional detectors. Moreover, landslide boundaries often present gradual or transitional characteristics, which conventional IoU-based loss functions struggle to capture accurately, causing localization deviations. To address these challenges, the proposed SAFM module, MPDIoU bounding box regression loss, and multi-scale training strategy are designed to enhance salient landslide features, improve localization accuracy, and effectively suppress complex background interference.

This study targets key obstacles in landslide identification across complex plateau mountainous terrains—namely small-target omission and boundary ambiguity. An improved YOLOv11-based landslide detection method is proposed, integrating attention mechanisms, multi-scale training strategies, and loss function optimization to boost detection accuracy, with pronounced gains for small-scale landslides and under complex background interference. The specific objectives are as follows:

(1)

To construct a high-quality landslide sample dataset by incorporating remote sensing imagery from the Zhaotong and Bijie regions, covering typical mountainous geomorphological features and providing abundant samples for model training and evaluation;

(2)

To embed a SAFM attention module that strengthens the model’s responsiveness to landslide spatial distributions, thereby improving detection in regions with subtle texture differences and high background complexity;

(3)

To employ a multi-scale training strategy and an optimized loss function, enhancing detection accuracy across varying landslide scales and improving boundary regression for finer detail capture.

2. Study Area and Datasets

2.1. Study Area

In this study, Zhenxiong County in Yunnan Province (26.34°N–28.41°N, 102.52°E–105.19°E) and Bijie City in Guizhou Province (26.35°N–27.77°N, 103.60°E–106.72°E) were chosen as the study areas (Figure 1). Both regions are located within the Wumengshan Tectonic Belt and represent typical highland mountainous terrain, characterized by pronounced topographic relief and complex geological conditions, making them high-risk areas for landslides and other geological hazards.

Zhenxiong County experiences a highland monsoon climate, with 900–1200 mm of annual rainfall concentrated in June–September, typically as short, intense storms. The steep slopes, fractured lithology, and loose accumulated layers contribute to the frequent occurrence of landslides, debris flows, and collapses. By comparison, Bijie City experiences a subtropical humid monsoon climate, receiving 850–1100 mm of rainfall annually, most of it during May–October and often as prolonged heavy rain. The area is dominated by karst landforms with extensive underground cavities, and anthropogenic disturbances such as mining and reclamation further increase the risk of ground subsidence and landslides. Overall, the distinct topography, climate, and geological conditions of these two regions provide typical cases for landslide and related hazard studies, offering a reliable foundation for constructing high-quality landslide detection models. Moreover, disparities in regional hazard attributes bolster the model’s generalization performance and stability under complex mountainous conditions.

2.2. Datasets

The optical imagery used for landslide detection includes the Bijie and Zhaotong landslide datasets, as well as imagery from a portion of the Zhenxiong region, which was excluded from the training phase, as shown in Figure 2. The Zhaotong landslide samples were derived from the 2020–2025 geological hazard data report of the Yunnan Geological Survey, and 272 landslide samples were manually delineated based on high-resolution GF-2 and Sentinel-2 optical imagery. The Bijie samples were sourced from publicly available remote-sensing data; in total, 770 images were extracted from TripleSat imagery [26]. To ensure consistency, all images were processed with atmospheric correction, geometric registration, cropping, and radiometric normalization.

To improve robustness and generalization, we expanded the original landslide set to 2352 images via data augmentation—rotation, flipping, affine transformations, and color perturbations. In addition, 1000 non-landslide background images (e.g., mountainous areas, forests, and urban regions) were included as negative samples, resulting in a final training dataset comprising 3352 images. Images in the Bijie and Zhaotong datasets were uniformly cropped to 256 × 256 pixels and labeled in the YOLO format.

Table 1. Dataset composition employed in this study.

Dataset	Data Sources	Spatial Resolution	Images	Total Images	Number of Augmented Data
Landslide dataset	Sentinel-2	10 m	272 samples of Zhaotong landslide	1042	2352
	GF-2	1 m
	TripleSat	1 m	770 samples of Bijie landslide
Negative sample datasets	TripleSat	1 m	1000	1000	1000

summarizes the dataset composition, and Figure 3 presents representative samples used in this study.

The dataset was further divided into training, validation, and test sets in a 14:3:3 ratio, corresponding to 2346, 504, and 502 images, respectively, as detailed in

Table 2. Detailed Information on Dataset Partitioning.

	Train	Val	Test	Total
Landslide images	1646	354	352	2352
Negative images	700	150	150	1000
Total	2346	504	502	3352

. The training split served model fitting, the validation split informed model selection, and the test split was held out without further processing for performance assessment. The training set covers diverse geomorphological features and environmental interferences, serving as a robust basis for model training and evaluation.

Imagery from the Zhenxiong region was excluded from training and employed exclusively to assess the model’s generalization performance. These images were also cropped to 256 × 256 pixels with a 50% overlap to ensure comprehensive coverage.

3. Methods

Based on the baseline YOLOv11 architecture, this study develops YOLOv11-SAFM, an efficient detection method tailored for automatic landslide identification in mountainous regions. First, the basic architecture and advantages of the YOLOv11 model are briefly introduced. Subsequently, the model is optimized in three key aspects: (1) The loss function is redesigned to better handle the irregular shapes and uneven spatial distribution of landslides. This improves boundary regression accuracy, reduces missed detections, and enhances the recall rate in dense and small-scale landslide areas. (2) To improve detection under challenging backgrounds, a Spatially Adaptive Feature Modulation (SAFM) mechanism is employed to better capture essential features in landslide imagery with indistinct boundaries. (3) A multi-scale training approach is adopted to enhance the model’s capability in identifying landslides across different scales, particularly improving detection of small-scale targets.

3.1. Basic YOLOv11 Model

YOLOv11, introduced by Ultralytics in 2024, as a further evolution of the YOLO series, it builds upon the anchor-free architecture and decoupled detection strategy of YOLOv8, introducing modular-level optimizations across multiple layers of the network to improve detection accuracy while maintaining inference efficiency. Its core improvements focus on the feature extraction backbone, integration of attention mechanisms, and the lightweight design of the detection head. An overview of the architecture is provided in Figure 4.

In terms of network design, YOLOv11 incorporates the C3k2 module as an alternative to the C2f block in YOLOv8, aiming to strengthen feature extraction capacity. Derived from the Cross Stage Partial (CSP) structure [41], the C3k2 module controls cross-stage connections of feature channels through a tunable parameter. This configuration limits low-level interference with high-level semantics in the shallow stages and strengthens feature fusion in deeper stages, thereby improving the network’s overall representational capacity. Additionally, for attention mechanisms, YOLOv11 adds the C2PSA module after the original SPPF module [42], integrating a lightweight convolutional structure with multi-head parallel spatial attention. This enhancement improves the model’s capture of local spatial detail and contextual cues, yielding more accurate boundary and shape discrimination in mid- and high-level feature maps and better recognition of small landslide features. At the detection head, the model retains a decoupled classification–regression design and incorporates DWConv modules [43], sharply reducing parameters and computation. This enables real-time inference on resource-constrained devices and improves bounding-box localization accuracy.

In landslide remote sensing applications, the YOLOv11 model is trained primarily to distinguish landslide regions from non-landslide backgrounds, such as cropland and other land cover types. The model exploits its CNN architecture to learn deep spatial representations from remote-sensing imagery, which are then used for landslide classification and bounding-box localization. Morphologically, landslide areas often exhibit irregular boundaries (e.g., tear-shaped or fan-shaped), whereas croplands typically display regular geometric patterns. In terms of texture, landslides tend to exhibit pronounced heterogeneity and surface roughness, while cropland areas feature relatively uniform textures, often characterized by ridge lines formed by farming activities:

3.2. Improved YOLOv11 Model: YOLOv11-SAFM

3.2.1. Loss Function

Bounding Box Regression (BBR) plays a pivotal role in object detection and instance segmentation [44], as it enables the accurate localization of target objects within an image:

$$\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{area(B_p\cap B_{gt})}{area(B_p\cup B_{gt})}}$$

(1)

where B_p is the model’s estimated bounding box and B_gt is the reference ground-truth box.

However, in landslide instance segmentation tasks, the highly irregular geometries of landslide bodies, combined with the abundance of small-scale targets, present considerable challenges. Traditional IoU-based loss functions (e.g., GIoU [45], DIoU [46], CIoU [47]) often suffer from gradient vanishing and unstable optimization when dealing with large mismatches or high overlaps between the predicted bounding boxes and the true object locations. These issues result in slow convergence during the bounding box regression stage and lead to suboptimal performance, particularly in detecting small targets.

To address this issue, this study introduces MPDIoU (Multi-Perspective Decoupled IoU) as an improved regression loss function [48]. Based on the fundamental framework of IoU, MPDIoU further decouples the spatial geometric relationship between bounding boxes into multiple components—such as the distance between centers, angular direction, and overlapping or non-overlapping areas. Each component is evaluated to represent the geometric discrepancies between predicted and ground-truth boxes, as shown in Figure 5. Furthermore, MPDIoU simplifies the computational process and enhances the spatial localization accuracy of landslide targets. Its formulation is defined as follows:

$$MPDIoU=IoU-{\displaystyle \frac{d_1^2}{h^2+w^2}}-{\displaystyle \frac{d_2^2}{h^2+w^2}}$$

(2)

Specifically, $d_1^2=(x_{B1}-x_{A1}{)}^2+(y_{B1}-y_{A1}{)}^2$ and $d_2^2=(x_{B2}-x_{A2}{)}^2+(y_{B2}-y_{A2}{)}^2$ represent the squared Euclidean distances between the top−left and bottom−right corners of the predicted and ground truth bounding boxes, respectively. The width (w) and height (h) of the predicted bounding box are employed to normalize the distance terms, ensuring that the resulting metric shares the same dimensionality as IoU. By simultaneously evaluating the spatial deviations of both diagonal corners, MPDIoU provides a more sensitive measure of misalignment and scale discrepancy between bounding boxes, thereby enhancing the stability of the regression process. To facilitate model optimization, the regression loss function based on MPDIoU is defined as follows:

$${\mathcal{L}}_{MPDIoU}=1-MPDIoU$$

(3)

Here, ${\mathcal{L}}_{MPDIoU}$ denotes the bounding box regression loss function based on MPDIoU. When the predicted and ground-truth boxes coincide exactly, MPDIoU reaches its maximum value of 1, and the loss function yields zero, indicating no localization error. Conversely, as the spatial overlap decreases and the positional deviation increases, the MPDIoU value drops and the corresponding loss increases, driving the model to iteratively improve bounding box localization during training.

In this study, MPDIoU is integrated into the YOLOv11 framework by replacing the original CIoU term in the bounding box regression branch, while the remaining loss components—including the BCE-based classification loss and the Distribution Focal Loss (DFL)—remain unchanged in both configuration and weighting. Compared to DIoU and CIoU, MPDIoU decouples spatial distances across multiple geometric channels, significantly enhancing sensitivity to boundary misalignment. This improves the stability and discriminative quality of gradient feedback, making MPDIoU particularly well-suited for detecting landslide targets in remote sensing imagery, where object shapes are complex and spatial distributions are uneven. As a result, it contributes to more accurate bounding box regression and improved overall detection performance.

3.2.2. Introduction of the Spatial Adaptive Feature Modulation (SAFM) Module

To improve the network’s ability to extract discriminative features and accurately represent small-scale landslides in complex backgrounds, this study incorporates the Spatially Adaptive Feature Modulation (SAFM) mechanism [49], originally proposed in the lightweight image super-resolution model SAFMN.

The network architecture comprises stacked Feature Mixing Modules (FMMs) together with a lightweight upsampling module. As illustrated in Figure 6, a 3 × 3 convolution is first applied to the low-resolution input image I_LR to project it into the feature space, generating the initial shallow feature map F₀. Subsequently, a stack of FMM blocks extracts deep features from F₀, producing high-level feature representations for reconstructing high-resolution images. Each FMM comprises two key submodules: Spatially Adaptive Feature Modulation (SAFM) and the Convolutional Channel Mixer (CCM). The SAFM submodule performs long-range feature modeling and adaptively adjusts the weights across spatial positions based on image content, significantly enhancing the representation of critical regions—such as the edges and textures of small-scale landslides—in remote sensing imagery. The CCM is then employed to extract local contextual information and efficiently fuse features across the channel dimension, thus strengthening the capability to model complex multi-scale target semantics. This architectural design is well-suited for landslide detection tasks in remote sensing imagery, where target sizes vary significantly, boundaries are often ambiguous, and background interference is substantial. The feature mixing process can be formally described as:

$$Y=\mathrm{S}\mathrm{A}\mathrm{F}\mathrm{M}\left(\mathrm{L}\mathrm{N}\right(X\left)\right)+X,$$

(4)

$$Z=\mathrm{C}\mathrm{C}\mathrm{M}\left(\mathrm{L}\mathrm{N}\right(Y\left)\right)+Y,$$

(5)

Then, LN(·) denotes the layer normalization operation, while X, Y and Z represent intermediate features at various stages of the network.

To better recover high-resolution details, we introduce a global residual path that facilitates learning of high-frequency content. Additionally, a lightweight upsampling module—consisting of a 3 × 3 convolution followed by a sub-pixel convolution—is used in the output stage to facilitate efficient and rapid image reconstruction. The overall network can be succinctly expressed as:

$${\mathrm{F}}_0={\mathcal{C}}_{\omega }\left(I_{LR}\right),$$

(6)

$$I_{SR}={\mathcal{U}}_{\gamma }\left({\mathcal{M}}_{\theta }\right({\mathrm{F}}_0)+{\mathrm{F}}_0),$$

(7)

Here, I_LR denotes the input image, ${\mathrm{F}}_0={\mathcal{C}}_{\omega }\left(·\right)$ represents the initial features extracted by the convolutional layer, ${\mathcal{M}}_{\theta }\left(·\right)$ denotes the Feature Mixing Module (FMM) composed of SAFM and CCM, and ${\mathcal{U}}_{\gamma }\left(·\right)$ is the upsampling operator responsible for restoring the spatial resolution of the output image I_SR. During training, we optimize the model with a composite loss that couples spatial- and frequency-domain terms, improving the perceptual quality of the reconstructed images. The loss takes the following form:

$$\mathcal{L}=\parallel I_{SR}-I_{HR}{\parallel }_1+\lambda \parallel \mathcal{F}\left(I_{SR}\right)-\mathcal{F}\left(I_{HR}\right){\parallel }_1,$$

(8)

Here, I_HR denotes the corresponding high-resolution ground-truth image, $\parallel ·{\parallel }_1$ represents the L1 norm, and $\mathcal{F}(·)$ denotes the Fast Fourier Transform (FFT), used to capture the image’s frequency-domain characteristics. The parameter $\lambda$ is an empirically set weighting factor, fixed at 0.05 in our experiments. This loss function is designed to ensure accurate image reconstruction while effectively preserving structural details and texture information, making it well-suited for high-quality image restoration tasks.

3.2.3. Multi-Scale Training Strategy

To improve YOLOv11’s detection across varied target scales in complex terrain, we adopt a multi-scale training regimen during model training [50,51,52]. This strategy dynamically adjusts the input image resolution, exposing the model to samples of different scales during training. As a result, the network exhibits improved multi-scale feature representation and greater scale-invariant performance. Given that landslides in remote sensing imagery typically exhibit substantial scale variations, blurred boundaries, and diverse morphologies, multi-scale training effectively addresses the limitations of fixed-resolution training—namely, the perceptual blind spots for small targets and redundant feature extraction for large ones.

From a methodological perspective, multi-scale training introduces spatial-scale variation by randomly resizing input images within a range of 320 × 320 to 640 × 640 at each training iteration, allowing the network to learn feature representations under varying resolution conditions. This strategy encourages the model to continuously adapt its convolutional kernels to features of varying scales, thereby improving its capacity to resolve fine-edge details of small landslides while effectively representing the overall structure of larger features. YOLOv11 inherently employs a multi-level feature fusion architecture. With the integration of multi-scale input, feature maps from different receptive fields can collaboratively learn spatial representations of landslide targets at various scales, effectively improving both detection precision and recall.

Moreover, multi-scale training introduces a form of regularization by varying the input distribution during training. This breaks the model’s dependence on fixed feature scales, reduces the risk of overfitting, and enhances robustness and generalization in real-world complex scenarios.

4. Results

This section presents a comprehensive assessment and comparative analysis of the proposed enhanced YOLOv11 framework for landslide detection. The experiments cover overall detection performance, visualization-based analysis of detection results, and generalization capability assessment, aiming to systematically evaluate the performance and reliability of the enhanced approach.

4.1. Experimental Setup

All experiments were run on a workstation with an Intel Core i7-12700 CPU, 32 GB RAM, and an NVIDIA RTX 3090 GPU (24 GB VRAM), using Python 3.10 with the PyTorch 12.8 framework.

During training, we adopted the cosine–annealing learning-rate scheduler with an initial warmup phase, following the official YOLO optimization strategy. This scheduler gradually reduces the learning rate throughout the training process, enabling stable convergence across 400 epochs. In addition, the data enhancement strategy adopts the Mosaic method. This approach randomly selects four training images, applies independent scaling and cropping operations, and then merges them into a single composite image. This process effectively increases sample diversity and enhances the model’s robustness to scale variations and complex background conditions.

To ensure consistency across all experiments, a unified set of hyperparameters was adopted throughout the training process. Before finalizing these settings, the official YOLOv11 configuration was used as the initialization baseline, and key optimizer parameters were further examined using the Ray Tune framework within a constrained search space to verify their stability and convergence on the landslide dataset. Subsequent validation-based adjustments were applied to obtain more reliable training performance. The finalized hyperparameter settings are summarized in

Table 3. Configuration of Experimental Training Process.

Hyper Parameters	Value
Epoch	400
Momentum	0.937
Initial learning rate	0.01
GPU mem	5.3
GFLOPs	8.20
Input image size	256 × 256
Weight decay	0.0005
Optimizer	SGD
Data enhancement strategy	Mosaic

.

4.2. Evaluation Metrics

To evaluate the recognition performance of the proposed YOLOv11-SAFM model, four standard metrics were employed: Precision, Recall, F1-Score, and mean Average Precision at an IoU threshold of 0.5 (mAP@0.5). The evaluation metrics are computed as follows:

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(9)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(10)

$$\mathrm{F}1=2\ast {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(11)

In these definitions, TP, FP, and FN represent the numbers of true positives, false positives, and false negatives, respectively. The F1 score, combining precision and recall harmonically, serves as an integrated measure of a model’s performance in capturing positive cases accurately and completely. A low F1-score typically indicates deficiencies in either precision or recall, which may result from poor recognition of true positives or a high number of false detections. Conversely, a high F1 score indicates strong performance in both precision and recall, reflecting well-balanced and overall optimized classification.

$$\mathrm{A}\mathrm{P}=\int \begin{array}{c}1\\ 0\end{array}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}\right)d(\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})$$

(12)

$$\mathrm{m}\mathrm{A}\mathrm{P}={\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{i}=1}^{\mathrm{N}} ={\mathrm{A}\mathrm{P}}_{\mathrm{i}}$$

(13)

Here, AP_i denotes the Average Precision for the i class, and N represents the total number of classes in the training dataset (in this study, N = 1). mAP@0.5 is the mean Average Precision over all classes evaluated at an IoU threshold of 0.5. In this study, Mask AP is adopted as the primary metric for evaluating the segmentation accuracy of the YOLO-based instance segmentation model. This metric integrates segmentation precision under varying levels of localization strictness, offering a more thorough evaluation of the model’s detection performance over a range of IoU thresholds.

4.3. Experimental Results

4.3.1. Landslide Detection Results

Representative detection results of the YOLOv11-SAFM model on the test set are illustrated in Figure 7 to assess its performance in real-world scenarios. In the figure, blue masks indicate the landslide boundaries automatically segmented by the model.

The results show that YOLOv11-SAFM can accurately extract landslide boundary contours. In scenarios with clearly defined single targets, the model produces compact and well-overlapping boundary predictions. Figure 7(b1–b4) depicts scenarios involving multiple landslide regions. In these complex cases—where targets appear simultaneously with considerable scale variation—YOLOv11-SAFM demonstrates strong spatial separation capability, clearly distinguishing the position and extent of each landslide body. Figure 7(c1–c4) showcases robust detection of small landslides, which frequently occur on mountain slopes or along roads. High localization accuracy in such cases is crucial for early landslide detection and effective risk mitigation. In addition, Figure 7(d1–d4) shows landslides in complex backgrounds, including areas covered by surface vegetation, where the surrounding environment appears visually similar. In such scenarios, the model must identify landslide features that are obscured by confounding visual elements. The detection results indicate that YOLOv11-SAFM effectively suppresses vegetation interference and correctly segments potential landslide regions, demonstrating strong background adaptability and robustness against noise.

In summary, YOLOv11-SAFM exhibits high accuracy and strong robustness in landslide detection tasks. It performs well not only in identifying large, clearly defined landslides but also in detecting small-scale targets and operating under complex background conditions.

To evaluate the stability and convergence of the model, the training and validation loss curves (Figure 8) and the Precision–Recall curve (Figure 9) were analyzed. All loss components decrease smoothly throughout training, and the validation curves closely follow the training trends, indicating stable optimization without overfitting. The Precision–Recall curve exhibits a near-ideal shape, maintaining high precision across a wide range of recall levels, with an mAP@0.5 of approximately 0.95. These results confirm that the model achieves stable training dynamics and strong discriminative performance under different confidence thresholds.

4.3.2. Comparing Different Detection Models

To validate the performance advantage of the proposed YOLOv11-SAFM model in landslide detection tasks, four representative object detection models were selected for comparison: Mask R-CNN [53], YOLOv8, YOLOv8-SAFM, YOLOv11, YOLOv12 and YOLOv11-SAFM. The evaluation metrics include Precision, Recall, F1-Score, and mAP@0.5. The experimental results are summarized in

Table 4. Comparison of Detection Performance Across Different Models.

Model	Precision (%)	Recall (%)	mAP0.5 (%)	F1 (%)
Mask R-CNN	81.18	78.47	79.60	79.82
YOLOv8	78.74	63.10	70.70	70.04
YOLOv8-SAFM	92.48	79.40	87.40	85.45
YOLOv11	92.21	77.90	88.08	84.45
YOLOv11-SAFM	95.05	90.10	95.30	92.51
YOLOv12	88.60	76.50	89.80	82.10

.

As shown in the table, refining the loss, integrating attention, and adopting multi-scale training substantially improve overall detection performance. For example, in the YOLOv8 series, the integration of SAFM increased the F1-score from 70.04% to 85.45%, mAP@0.5 from 70.70% to 87.4%, and recall from 63.10% to 79.4%, respectively. These results demonstrate that the SAFM module strengthens the model’s perceptual modeling, thereby improving both detection accuracy and recall.

Similarly, YOLOv11 exhibited further improvements after incorporating the SAFM module and optimizing the loss function. The F1-score increased from 84.45% to 92.51%, recall improved from 77.90% to 90.10%, and mAP@0.5 reached 95.30%. These findings further validate the adaptability and generalizability of the SAFM module and MPDIoU loss function across different YOLO architectures.

To further enhance the completeness of the comparison, we evaluated YOLOv11-SAFM against YOLOv12 under identical training configurations. YOLOv12 achieved a precision of 88.60%, recall of 76.50%, and mAP@0.5 of 89.80%. While YOLOv12 provides strong baseline performance, YOLOv11-SAFM outperforms it with a higher recall of 90.10% and mAP@0.5 of 95.30%. Despite YOLOv12 introducing innovative architecture and attention mechanisms (such as the R-ELAN module and Flash Attention), landslide detection was not the primary focus of YOLOv12’s optimization. While these innovations are effective in other tasks, they did not yield the expected results when handling the blurry boundaries and small-scale targets typical of landslides. Additionally, YOLOv12’s increased computational load resulted in slower inference speeds, which negatively impacted its performance in landslide detection. Our results suggest that the performance improvement of YOLOv11-SAFM is primarily attributed to the integration of the SAFM module and MPDIoU loss, rather than differences between YOLO architecture versions.

Notably, the improvement in recall attributed to the SAFM module and MPDIoU loss function is particularly significant, with increases of 16.30% and 12.20% observed in YOLOv8 and YOLOv11, respectively. This demonstrates the module’s strong effectiveness in enhancing the model’s capability to identify landslide regions, especially under complex backgrounds or in the presence of small-scale targets.

As indicated by Table 4, Mask R-CNN underperforms YOLOv11-SAFM across all evaluation metrics. Moreover, the F1-score, calculated as the harmonic mean of precision and recall, provides a balanced measure of the model’s overall detection performance. The marked gains in F1 score after integrating SAFM indicate that the module boosts precision while sustaining high recall, yielding a more balanced and stable detection performance.

Under the same testing scenario (Figure 10), we conducted a side-by-side comparison of Mask R-CNN, YOLOv8, YOLOv8-SAFM, YOLOv11, YOLOv12 and YOLOv11-SAFM. The results show that Mask R-CNN, affected by its two-stage framework and feature sampling, tends to produce coarse masks, jagged boundaries, and partial misses in the presence of complex textures and shadow interference. YOLOv8 exhibits insufficient sensitivity to small-scale landslides, frequently resulting in partial or missed detections. With the introduction of SAFM, YOLOv8-SAFM improves the representation of fine-grained edges and weak-texture regions, resulting in better boundary continuity and small-object recognition. YOLOv11 achieves higher localization accuracy than YOLOv8, yet still presents edge omissions and over-segmentation in some cases. Although YOLOv12 provides a strong baseline, it still tends to incorrectly include nearby buildings as part of the landslide targets when dealing with small and densely distributed landslides, and its delineation of landslide boundaries is not sufficiently complete. By contrast, YOLOv11-SAFM delivers the best performance under similar conditions, more fully capturing fine-grained edge cues and effectively suppressing background interference, thereby demonstrating stronger adaptability and robustness while balancing precision and recall.

4.3.3. Heatmap Visualization Analysis

To further evaluate the contribution of the SAFM module to spatial feature adaptation and its influence on the model’s internal feature perception, Grad-CAM attribution maps were generated for both YOLOv11 and YOLOv11-SAFM. As shown in Figure 11, the baseline YOLOv11 presents relatively dispersed activation regions and is easily affected by background textures such as shadows and vegetation, making it difficult to consistently focus on landslide structures in complex environments.

In contrast, YOLOv11-SAFM produces more concentrated and semantically coherent activation patterns, with attention directed primarily toward the key structural components of the landslide while effectively suppressing background interference. These visual comparisons demonstrate that the SAFM module substantially enhances the model’s spatial feature representation capability, enabling more accurate perception of landslide features and thereby improving overall detection robustness.

4.3.4. Landslide Detection Results in Zhaotong

To evaluate the generalization capability of the YOLOv11-SAFM model in previously unseen geographic regions, this study conducted automatic landslide detection in Zhenxiong County, Yunnan Province, which was entirely excluded from the model’s training process. As shown in Figure 12, the detection results clearly illustrate the spatial distribution pattern of the identified landslides, with a total of 55 landslide instances detected. Overall, the model exhibited strong detection performance in this unfamiliar region, with only a few instances of missed or error detections. The yellow and blue markers indicate missed detections and error detections, respectively. As shown in Figure 13, some landslides marked by yellow points were missed due to the occlusion of key features by dense vegetation shadows. Meanwhile, the areas marked by blue points represent bare surfaces heavily eroded by rainfall, which were incorrectly detected as landslides due to their high visual similarity to actual landslide features. To address these challenges caused by specific scene complexities—such as vegetation-induced shadowing and erosion-related interference—additional training samples featuring similar occlusion conditions and easily confused erosional surfaces were incorporated into subsequent model training for targeted enhancement. In summary, this out-of-region generalization test demonstrates the model’s effective transferability and its promising potential for real-world landslide detection across diverse geographic environments.

5. Discussion

This study proposes an improved landslide detection framework built upon the YOLOv11 architecture, integrating a SAFM module, a refined loss function, and a multi-scale training scheme. Generalization experiments were conducted using high-resolution remote sensing data from Zhenxiong County, Zhaotong, Yunnan Province. Experimental evaluation indicates that the proposed model markedly enhances detection performance, improves the identification of small-scale landslides, and achieves more accurate boundary localization. To offer a thorough evaluation of the model’s performance and practical utility, the subsequent section analyzes the key findings from multiple perspectives.

5.1. Module Effectiveness Analysis

To further assess the individual contributions of each proposed component to landslide detection performance, we conducted component-wise ablation experiments using the YOLOv11 framework. Specifically, three improved models were constructed by sequentially integrating the MPDIoU bounding box regression loss, the SAFM spatial attention module, and the multi-scale feature fusion strategy. These variants were evaluated on a standardized landslide remote sensing dataset. The detection performance of each model, in terms of Precision, Recall, AP@0.5, and F1-score, is summarized in

Table 5. Evaluation of Detection Performance Across Model Variants.

Model	Precision (%)	Recall (%)	mAP0.5 (%)	F1 (%)
YOLOv11	92.21	77.90	88.08	84.45
YOLOv11+MPDIoU	92.33	83.21	90.71	87.51
YOLOv11+MPDIoU+SAFM	92.60	89.61	94.90	91.08
YOLOv11+MPDIoU+SAFM+ Multi-scale feature fusion	95.05	90.10	95.30	92.51

.

According to Table 5, the baseline YOLOv11 exhibited reliable detection performance, with Precision, Recall, AP@0.5, and F1-score of 92.21%, 77.90%, 88.08%, and 84.45%, respectively. Replacing the original loss with the proposed MPDIoU-based bounding-box regression loss markedly improves localization, with pronounced gains on small landslide instances. Specifically, Recall increased to 83.21%, AP@0.5 rose to 90.71%, and F1-score reached 87.51%. These results confirm that the MPDIoU loss effectively enhances localization accuracy while reducing missed detections.

Building on this improvement, incorporating the SAFM attention mechanism further strengthens the model’s discrimination of landslide regions under complex background conditions. As a result, Recall and F1-score increase to 89.61% and 92.60%, respectively, while mAP@0.5 improves to 94.90%. These findings indicate that the attention mechanism enables the model to focus on prominent landslide features while mitigating interference from background noise.

Finally, the adoption of a multi-scale feature fusion strategy leads to further performance gains. The final model, YOLOv11 enhanced with MPDIoU, SAFM, and multi-scale fusion, achieves the best results across all evaluation metrics, with Recall reaching 90.10%, F1-score attaining 92.51%, and mAP@0.5 peaking at 95.30%.

In conclusion, the ablation analysis demonstrates that the introduced enhancements significantly contribute to improved detection accuracy. The synergy among the introduced modules is evident, and the final YOLOv11-SAFM model exhibits strong generalization capability and adaptability to complex background conditions, rendering it highly applicable for real-world landslide detection in mountainous regions.

To further investigate the influence of each module on the training dynamics, we compared the training and validation loss curves of the four model configurations, as illustrated in Figure 14. The results show that the MPDIoU loss accelerates the convergence of the regression branch and effectively reduces loss fluctuations during the early stages of training. With the incorporation of the SAFM module, the segmentation loss exhibits a more stable downward trend, indicating that SAFM enhances spatial feature representation and contributes to a more stable optimization process. After integrating the multi-scale feature fusion structure, the overall loss curves become smoother, and the convergence trends of both the training and validation phases become more consistent. Overall, these components not only improve the final detection performance but also positively influence optimization efficiency, convergence stability, and the characteristics of the loss landscape throughout the training process.

5.2. Limitations and Future Challenges

While the improved model shows strong overall detection capability, its effectiveness is still constrained under real-world conditions. On one hand, optical remote sensing imagery is inherently susceptible to interference from terrain shadows, cloud cover, and vegetation, which often leads to boundary ambiguity or missed detections in complex terrains such as mountainous gorges. In addition, distinguishing natural landslides from human-induced excavated slopes remains challenging. Although YOLOv11-SAFM can leverage morphological and textural cues—such as arcuate scarps, heterogeneous roughness patterns, and irregular deposition zones—to differentiate the two to some extent, engineered cuts and quarry slopes often exhibit spectral and geometric characteristics highly similar to true landslides in high-resolution optical imagery. As a result, misclassifications may still occur under visually ambiguous or background-complex conditions. This reflects an inherent limitation of optical remote sensing in mountainous environments, where even expert interpreters may encounter difficulty achieving stable and reliable discrimination. Future work will therefore expand the diversity of engineered-slope samples and explore the integration of LiDAR-derived topographic metrics or SAR-based deformation features to enhance the model’s ability to separate natural landslides from human-modified surfaces.

To improve the model’s applicability and intelligence, future work may explore the following avenues. First, integrating multi-source remote sensing data—such as radar, thermal infrared, and LiDAR—with optical imagery could improve multi-perspective perception and all-weather detection capabilities. Second, expanding the model’s ability to classify different landslide types would allow for more refined disaster identification and severity assessment. Furthermore, incorporating landslide detection with InSAR-based deformation monitoring and meteorological early warning systems could facilitate the development of an integrated “detection–early warning–response” framework for intelligent geohazard monitoring, advancing landslide remote sensing applications toward greater operational viability and intelligence.

6. Conclusions

This study introduces YOLOv11-SAFM, an enhanced deep learning framework for automated landslide detection in complex mountainous environments. The model integrates a Spatially Adaptive Feature Modulation (SAFM) attention mechanism, an optimized MPDIoU bounding box regression loss, and a multi-scale training strategy, improving its ability to identify small-scale, blurred-edge, and heterogeneous landslide features. High-resolution remote sensing datasets from Bijie and Zhaotong in southwest China were used for training and validation, while independent imagery from Zhenxiong County evaluated generalization performance.

YOLOv11-SAFM demonstrates robust detection performance in rugged plateau terrains, achieving 95.05% precision, 90.10% recall, 92.51% F1-score, and 95.30% mAP@0.5 for small-scale landslides. Compared with Mask R-CNN, precision and mAP@0.5 increased by 13.87% and 15.7%, respectively; relative to traditional YOLOv8, recall and F1-score improved by 27.0% and 22.47%. The results indicate that the framework achieves superior precision, recall, and resilience under challenging background conditions.

In summary, YOLOv11-SAFM offers an efficient, robust, and generalizable framework for automated landslide detection. By combining fine-resolution satellite imagery with advanced deep learning approaches, the model substantially enhances the speed and accuracy of geohazard monitoring, providing a reliable technical foundation for landslide risk assessment, early warning, and emergency management. Moreover, the framework exhibits strong scalability and can be integrated with multi-source remote sensing data, InSAR-based deformation monitoring, and intelligent early warning systems, facilitating the development of comprehensive and smart landslide monitoring strategies in mountainous regions.