New Sampling Method for Landslide Susceptibility Evaluation with Consideration of Minimizing Potential Societal Losses

Abstract

In landslide susceptibility evaluation, scientific sampling minimizes potential societal losses and enhances the efficiency of disaster prevention and mitigation. However, traditional sampling methods, such as selecting landslide and non-landslide samples based on equal proportions or area proportions, overlook the different societal losses resulting from landslide omission and misreporting, and the potential societal losses faced by their evaluation results are often not minimized. Therefore, this study proposes a sampling method that takes potential societal losses into account and uses the Landslide Misjudgment Potential Societal Loss Evaluation Index (LMPSLEI) to quantify the total potential social losses in the area due to landslide omission and misreporting. The LMPSLEI is minimized by optimizing the sample ratio, thus minimizing the potential societal losses faced by the evaluation results and enhancing the scientific basis of disaster prevention and mitigation efforts. This study takes the Wenchuan earthquake area as the research region, selects 13 conditional factors and employs two models—Random Forest (RF) and Convolutional Neural Network (CNN)—to conduct case studies. We derive the recommended sample ratio based on the formula, hypothesizing that the LMPSLEI will be minimized under this ratio. The results show that the sample ratio for LMPSLEI minimization in the RF model is similar to the recommended sample ratio, while the sample ratio for LMPSLEI minimization in the CNN model is slightly higher than the recommended sample ratio. The recommended sample ratio can achieve the minimum of LMPSLEI or reach a lower value under different societal losses weights of landslide omission/misreporting, and thus it can be used as a preliminary choice of sampling for landslide susceptibility evaluation considering the potential societal losses.

1. Introduction

Landslides are a common global natural disaster, causing thousands of casualties and billions of dollars in economic losses each year [1]. According to statistics from the Centre for Research on the Epidemiology of Disasters (CRED), landslide-related deaths account for approximately 17% of total natural disaster fatalities. With the increasing frequency of extreme weather events and the accelerating process of urbanization, the casualties caused by landslides are likely to become even more severe. China is among the countries most affected by landslides, with 92,000 incidents over the past 11 years, representing 71.3% of all disasters, leading to nearly 50 billion yuan in direct economic losses and a significant number of casualties [2]. To effectively manage landslide risks, it is essential to identify potential landslide areas and implement preventive measures proactively. This is crucial for safeguarding lives and property, advancing disaster reduction efforts, and promoting sustainable development.

Landslide susceptibility evaluation (LSE) is an effective method for identifying landslide-prone areas and managing risks. It achieves the prediction of regional landslide occurrence probabilities by analyzing the relationship between historical landslides and evaluation factors [3]. Current methods fall into two categories: knowledge-driven and data-driven. Knowledge-driven methods [4,5] rely on expert knowledge, and their results are influenced by human subjective factors, resulting in considerable uncertainty [6]. Data-driven methods, such as machine learning [7,8,9,10,11], learn features directly from the input data, demonstrating higher evaluation accuracy [12]. Among these, random forest [13,14,15] and convolutional neural network [16,17,18] are widely applied in landslide susceptibility evaluation due to their outstanding performance. Importantly, the performance of data-driven methods largely depends on the quality and quantity of input data, making the sampling of landslide and non-landslide samples a key research topic.

Currently, most studies adopt a 1:1 sampling strategy for landslides and non-landslides. They focus on optimizing non-landslides selection through repeated random sampling [19,20], sampling outside the landslide buffer zone [21], and sampling in low susceptibility areas [22,23] to enhance prediction accuracy. Although these equal proportional sampling methods have made some progress in improving model performance, they overlook the significant area differences between landslide and non-landslide regions [24]. This may lead the model to over-identify landslides, resulting in a large number of landslide misreporting (FP) [25,26]. In contrast, Shao [27] and Xu [28] select samples based on the actual area proportion of landslides and non-landslides, and the results show that the predicted landslide area is relatively close to the actual landslide area. However, due to the imbalance in the number of positive and negative samples, the model may be biased toward the larger number of non-landslides, resulting in a significant number of landslide omission (FN). In addition, some studies have compared the performance of multiple sample ratios to select the optimal sample ratio [29,30,31,32]. The statistics for the specific sampling methods are shown in

Table 1. Statistics on previous research.

Researchers	Landslide/Non-Landslide Sample Ratio	Non-Landslide Sampling Strategy
Tang et al. [19]	1:1	Repeatedly random sampling
Oh et al. [20]		Iterative random sampling
Wang et al. [21]		Random sampling outside the buffer zone
Zhu et al. [22]		Random sampling in areas of low susceptibility in rapid evaluation results
Zhou et al. [23]		Random sampling in areas of low susceptibility in rapid evaluation results
Shao et al. [27]	based on the ratio of landslide to non-landslide areas	Random sampling Random sampling
Xu et al. [28]
Pourghasemi et al. [29]	1:2	Random sampling
Sun et al. [30]	1:10	Random sampling
Lv et al. [31]	1:1.2	Random sampling outside the buff-er zone
Yang et al. [32]	1:1.55 (SVM) 1:5 (RF) 1:6.16 (GBDT)	Random sampling outside the buff-er zone

. Most importantly, the evaluation standards for the sampling methods mentioned above are based on mathematical statistical metrics, such as accuracy and F1 score. In this context, the impacts of landslide omission and misreporting are considered equivalent. However, in reality, the societal losses (including direct economic losses from infrastructure damage, as well as indirect losses such as casualties, environmental degradation, and disruptions to social services) caused by landslide omission and misreporting often exhibit significant differences. Landslide omission may lead to failed disaster responses, resulting in casualties and substantial economic losses, whereas landslide misreporting typically results in the waste of human and material resources. Clearly, the consequences of landslide omission are often more severe [33]. Sampling methods that pursue accuracy and F1 score often fail to effectively reduce potential societal losses in a region because they ignore the asymmetry of risk associated with landslide omission and misreporting. Therefore, incorporating the different societal losses associated with omission and misreporting of landslides and selecting an appropriate sampling strategy to reduce potential regional losses has become a problem that needs to be solved.

To solve this problem, this study proposes a new evaluation metric—Landslide Misjudgment Potential Societal Loss Evaluation Index (LMPSLEI), which represents the potential societal losses incurred by a region due to landslide omission and misreporting. We select two commonly used models: Random Forest and Convolutional Neural Network. By adjusting the societal loss weights for omission/misreporting, we compare the LMPSLEI values under different sample ratios. Based on this analysis, the aim is to determine sampling methods that can minimize LMPSLEI under different societal loss weights. We propose a recommended sample ratio and compare it with actual results to verify its feasibility. The focus of this study is to propose a sampling method for landslide susceptibility evaluation that takes into account regional societal losses.

2. Materials and Methods

2.1. Study Area

The research area is selected as a narrow strip oriented from northeast to southwest in the northern part of Sichuan Province, China (Figure 1), which passes through the main disaster areas of the Wenchuan earthquake, including Wenchuan County, Shifang City, Beichuan Qiang Autonomous County, and Pingwu County (103.27° E–104.93° E, 32.28° N–30.96° N). The Wenchuan earthquake occurred on 12 May 2008, with a magnitude of 8. This earthquake caused tremendous economic losses and casualties, claiming the lives of 69,229 people, injuring 374,643, leaving 17,923 missing, and destroying 21,426,600 houses [34]. The earthquake formed two long surface ruptures along the central fault and the foreland fault, triggering numerous landslide disasters.

The research area is located in mountainous terrain, exhibiting a topographical feature of low elevation in the east and high elevation in the west, as well as low elevation in the north and high elevation in the south, with a maximum elevation reaching 5944 m. The undulating terrain reduces the stability of the geomorphological structure, providing a rich material basis for geological disasters such as landslides. The region has a well-developed water system, the dense river network exerts strong cutting effects on the slopes, increasing the likelihood of landslide occurrences. In addition, the annual rainfall in the research area ranges from 486 to 1419 mm, creating sufficient moisture conditions for landslides to occur.

2.2. Landslide-Related Conditional Factors

2.2.1. Landslide Inventory

The landslide data in this study mainly consist of earthquake-induced landslides, which were obtained through visual interpretation of high-resolution aerial imagery (0.3–0.5 m) after the earthquake, and were validated through field verification [35]. A total of 1148 landslides were interpreted, with an average area of 58,500 square meters, a maximum area of 153,000 square meters, and a minimum area of 715 square meters.

The vector data of the landslide areas obtained from visual interpretation was converted into 30 m × 30 m raster data in ArcGIS 10.8, where the landslide areas were set to 1 and the non-landslide areas were set to 0, resulting in the final binary landslide label data. The number of rasters for landslides is 71,287, while the number of rasters for non-landslides is 2,525,042, with an area ratio of approximately 1:36 between the two.

2.2.2. Landslide-Related Conditional Factors

After obtaining the landslide label data, identifying and collecting the condition factors related to landslide occurrence is another key step in conducting the susceptibility evaluation. This study selects 13 factors as evaluation factors for landslides (Figure 2), including elevation, slope, aspect, multi-scale topographic position index (mTPI), topographic wetness index (TWI), lithology, distance to fault, distance to river, annual rainfall, normalized difference vegetation index (NDVI), distance to road, land use type, and peak ground acceleration (PGA).

Table 2. Data sources for evaluation factors.

Conditional Factors	Data Sources
Elevation	SRTM DEM (30 m) obtained from Google Earth Engine
Slope, Aspect, TWI	Calculated from DEM
mTPI	Global ALOS mTPI (270 m) obtained from Google Earth Engine
Lithology, Distance to faults	Extracted from the geological map with the scale 1:200,000
NDVI	Landsat 5 TM image (30 m)
Annual rainfall	Global Resources Data Cloud (1 km)
Land use	China Land Cover Dataset (CLCD, 30 m)
Distance to roads, Distance to rivers	Obtained from the National Geomatics Center of China (NGCC) with the scale 1:200,000
PGA	Obtained from USGS

summarizes the sources of these data. The calculation formulas for NDVI and TWI are as follows:

$$\mathrm{N}\mathrm{DVI} = {\displaystyle \frac{\mathrm{NI}\mathrm{R} - \mathrm{Red}}{\mathrm{N}\mathrm{IR} + \mathrm{Red}}}$$

(1)

$$\mathrm{TWI}=\ln \left({\displaystyle \frac{\mathrm{SCA}}{\tan (\mathrm{Slope})}}\right)$$

(2)

In the formulas, NIR represents the near-infrared band, Red represents the visible red band, and SCA stands for the unit area runoff.

Elevation reflects the undulations of the terrain, significantly influencing surface hydrology, vegetation growth, and soil characteristics [36]. Steeper slopes experience stronger gravitational forces, making them more susceptible to instability [37]. Aspect plays a crucial role in determining sunlight exposure and moisture distribution, which in turn affects soil moisture levels and vegetation cover [38]. The Topographic Wetness Index (TWI) indicates the wetness of regional soils, while soil moisture content impacts the physical properties and shear strength of geological formations [39]. The multi-scale topographic position index (mTPI) is employed to differentiate ridges from valleys, affecting the convergence of water flow. This calculation involves subtracting the average elevation within a specified area from the elevation at each specific location [40]. Collectively, these geomorphological features influence the occurrence and distribution of landslides.

The type of rock determines its weathering and deformation resistance, which directly affects the stability of slopes [41]. According to the engineering rock mass classification standards, rocks are divided into five categories: hard rock, relatively hard rock, relatively soft rock, soft rock, and extremely soft rock, in descending order of hardness. The distance to the fault also affects slope stability; the closer the distance to the active fault, the greater the impact and the higher the risk of landslides [42].

The erosion and scouring by rivers can destabilize soil structures [43]. Meanwhile, rainfall allows water to infiltrate the soil, increasing the gravitational load on slopes and weakening their stability, thereby promoting the occurrence of landslides. Additionally, the normalized difference vegetation index (NDVI) reflects the health of vegetation. Soils with sparse vegetation are more prone to erosion, which further increases the likelihood of landslides.

The distance to roads and the type of land use are indicators of human impacts on slope stability [44]. Inappropriate excavation practices can create steep, high faces on slopes, thereby increasing the risk of instability. Under different land use types, the physical properties and erosion resistance of soil and rock masses vary, leading to differences in slope stability. The Peak Ground Acceleration (PGA) represents the intensity of ground shaking at the surface during an earthquake [45]. Higher PGA values correspond to stronger shear forces and vibrations acting on soil and rock masses, which in turn increase the likelihood of landslides.

To more accurately evaluate the impact of the NDVI and annual rainfall on landslide occurrence, pre-earthquake data are essential. The NDVI is derived from Landsat 5 Surface Reflectance data spanning 2003 to 2007, using the QA_PIXEL band to mask clouds and cloud shadows, and employing the maximum value composite method. Annual rainfall is calculated by averaging the total rainfall amounts from 2003 to 2007.

Due to the different data types and spatial resolutions of these condition factors, in ArcGIS 10.8, the spatial resolution is standardized to 30 m through projection, polygon to raster, and resampling tools, with a coordinate system of WGS_1984_UTM_Zone_48 N. Due to the differences in dimensions, maximum–minimum normalization is applied to the evaluation factors when inputting into the model.

2.3. Methods and Techniques Process

The sampling ratio of landslides and non-landslides directly affects the sensitivity of the model to landslide identification, thereby affecting the number of rasters with landslide omission and misreporting in landslide susceptibility evaluation in the region. With the selection of the study area, the relative social losses due to omission and misreporting of landslides are also determined. Multiply the number of rasters for landslide omission and misreporting by their corresponding societal losses, and then add these two values together to obtain the potential societal losses faced by the evaluation result under a single sampling ratio. This sum is also the value of the new indicator we propose, LMPSLEI. Under the condition that the societal losses from landslide omission and misreporting remain constant, there will always be an optimal sample ratio that minimizes the LMPSLEI. However, the societal losses from landslide omission and misreporting vary across different regions. Therefore, this study explores the trends in LMPSLEI and the sample ratio that minimizes it by establishing multiple groups of societal loss values for landslide omission and misreporting. At the same time, we derive a recommended sample ratio based on the LMPSLEI formula and believe LMPSLEI will achieve its minimum value at this sample ratio. The validity of the recommended sample ratio is tested in the study results. The specific workflow is shown in Figure 3:

(1)

Collecting evaluation factors and landslide inventory.

(2)

Correlation test of evaluation factors.

(3)

Setting 9 groups for the sample ratios of landslides to non-landslides and randomly selecting landslides and non-landslides to form sample sets.

(4)

Dividing the sample set into training set and testing set in a 7:3 ratio and utilizing random forest and convolutional neural network for training and testing.

(5)

Evaluating model results using Accuracy, Recall, Specificity, F1 score, and LMPSLEI.

(6)

Drawing and evaluating the landslide susceptibility map (LSM).

2.3.1. Pearson Correlation Coefficient

The formation of landslide disasters is quite complex and is controlled by numerous factors. Highly correlated conditional factors can increase the dimensionality of the data, reduce the efficiency of training and evaluation, and potentially impact the model’s final predictive results [46]. Therefore, it is necessary to conduct a correlation analysis to filter the conditional factors. The Pearson correlation coefficient is widely used to measure the linear relationship between two factors, ranging from −1 to 1. It is generally considered that when the absolute value of the Pearson correlation coefficient is ≥0.7, there is a strong correlation between the two factors, suggesting that one of the factors should be removed [47].

2.3.2. Random Forest (RF)

Random Forest is a classifier composed of multiple decision trees, originally proposed by Leo Breiman [48]. It constructs several decision trees using different subsets of data and then votes on their outcomes to obtain the final result. When performing binary classification, a random selection of certain features is made to serve as candidate features, from which the optimal features are then selected. This ensures that the decision trees in the random forest are diverse, thereby enhancing classification performance [49]. The flowchart is presented in Figure 4.

The hyperparameter tuning of the Random Forest model primarily focused on the number of trees. Under a sample ratio of 1:1, we tested four settings for the number of trees: 100, 200, 300, and 400. The results showed that the model achieved the highest overall accuracy with 100 trees. Therefore, we set the number of trees to 100 and applied this setting to all sample ratios in subsequent experiments.

2.3.3. Convolutional Neural Network (CNN)

Convolutional neural network (CNN) is one of the most commonly used methods in deep learning. It primarily extracts information and deep features from input data through convolutional layers and pooling layers and then passes these features to the output layer for classification or regression via of fully connected layers. Compared to point-to-point machine learning methods, the convolutional layers of CNN not only consider the features of the current position but also take into account the influence of surrounding features [50]. This is why CNNs usually perform better.

In this study, the original image input size is 15 × 15, with the convolutional kernel size set to 3 × 3. And the pooling layer uses a 2 × 2 max pooling. The number of neurons in the fully connected layers is 384 and 2, respectively. To prevent overfitting, a dropout layer is added after the convolutional layer and the fully connected layer, with a dropout rate of 0.2. The loss function used is the binary cross-entropy loss function, and the optimizer is set to Adam. Additionally, the learning rate was tuned between 0.0001 and 0.00001 with an interval of 0.00001. Results showed that the highest accuracy was achieved at a learning rate of 0.00003. Therefore, we set the learning rate to 0.00003 and applied this value to all sample ratios in subsequent experiments. The structure diagram is shown in Figure 5.

2.3.4. Evaluation Indicators

Accuracy, Recall, and Specificity are used to evaluate the classification performance of the model for all samples, landslide samples, and non-landslide samples. F1 score is a comprehensive metric that aims to simultaneously balance Precision and Recall. Their calculation formulas are shown as follows:

$$\mathrm{A}\mathrm{ccuracy} = {\displaystyle \frac{\mathrm{T}\mathrm{P} + \mathrm{TN}}{\mathrm{T}\mathrm{P} + \mathrm{TN} + \mathrm{FP} + \mathrm{FN}}}$$

(3)

$$\mathrm{R}\mathrm{ecall} ={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{FN}}}$$

(4)

$$\mathrm{S}\mathrm{pecificity} ={\displaystyle \frac{\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{N}+\mathrm{FP}}}$$

(5)

$$\mathrm{Precision} ={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(6)

$$\mathrm{F}1 \mathrm{score} = 2 \times {\displaystyle \frac{\mathrm{P}\mathrm{recision} \mathrm{R}\mathrm{ecall}}{\mathrm{P}\mathrm{recision}+\mathrm{R}\mathrm{ecall} }}$$

(7)

In this context, TP represents true positives (landslides predicted as landslides), TN represents true negatives (non-landslides predicted as non-landslides), FP represents false positives (non-landslides predicted as landslides), and FN represents false negatives (landslides predicted as non-landslides).

Opposite to Recall and Specificity are the FNR and FPR, which represent the probabilities of misclassifying landslides and non-landslides, respectively. The FNR (False Negative Rate) is related to landslide omission, while the FPR (False Positive Rate) is associated with landslide misreporting. They can be calculated using the following formula:

$$\mathrm{FNR} = {\displaystyle \frac{\mathrm{F}\mathrm{N}}{\mathrm{FN} + \mathrm{TP}}}= 1 - \mathrm{Recall}$$

(8)

$$\mathrm{F}\mathrm{PR}={\displaystyle \frac{\mathrm{F}\mathrm{P}}{\mathrm{FP}+\mathrm{TN}}}=1 - \mathrm{Spec}\mathrm{i}\mathrm{ficity}$$

(9)

Before calculating LMPSLEI, we need to introduce two parameters. The first parameter is the area of landslides and non-landslides, where the area of non-landslides is generally much larger than that of landslides. For convenience of calculation, the area of landslides in the selected region is set to 1, while the area of non-landslides is set to the ratio of the two, defined as Weight_area. The second parameter is the societal losses caused by landslide omission and misreporting, where the consequences of omission are generally more severe than those of misreporting. For computational convenience, the social losses from landslide misreporting are set to 1, while the social losses from omission are set to the ratio of the two, defined as Weight_loss. Weight_area and Weight_loss can be expressed by the following formulas:

$${\mathrm{Weight}}_{\mathrm{area}} = \mathrm{non}-\mathrm{landslide}\mathrm{s} \mathrm{area} ÷ \mathrm{landslides} \mathrm{area}$$

(10)

$${\mathrm{Weight}}_{\mathrm{loss}} =\mathrm{omission} \mathrm{societal} \mathrm{loss} ÷ \mathrm{misreporting} \mathrm{societal} \mathrm{loss}$$

(11)

LMPSLEI represents the total potential societal losses resulting from landslide omission and misreporting in the selected area. After obtaining the above indicator, we multiply the probability of landslide omission (FNR) by the area of the landslide and then by the societal losses from landslide omission to obtain the potential societal losses resulting from landslide omission. Similarly, by multiplying the probability of landslide misreporting by the area of non-landslides (FPR) and then by the societal losses from landslide misreporting, we can determine the potential social losses resulting from landslide misreporting. The sum of the two yields the LMPSLEI. The formula can be expressed as (the multiplier 1 is omitted in the LMPSLEI formula):

$$\mathrm{Omission} \mathrm{potential} \mathrm{societal} \mathrm{loss} = \mathrm{FNR} \times 1 \times {\mathrm{Weight}}_{\mathrm{loss}}$$

(12)

$$\mathrm{Misreporting} \mathrm{potential} \mathrm{societal} \mathrm{loss} =\mathrm{FPR} \times {\mathrm{Weight}}_{\mathrm{area}} \times 1$$

(13)

$$\mathrm{LMPSLEI} =\mathrm{FNR} \times {\mathrm{Weight}}_{\mathrm{area}}+\mathrm{FPR} \times {\mathrm{Weight}}_{\mathrm{loss}}$$

(14)

In the case of a single weight_loss, there is always an optimal sample ratio that can minimize LMPSLEI. Is there a regularity in the sample ratio that minimizes LMPSLEI across different weight_loss scenarios? We define the ratio of the number of non-landslide rasters to landslide rasters used for model training as Ratio. In Equation (14), both the FNR and FPR are functions of Ratio, and so is the LMPSLEI. Therefore, we take the derivative of Equation (14) with respect to Ratio and set the derivative to zero to find the critical points. The calculation process is as follows:

$${\displaystyle \frac{\mathrm{d}\left(\mathrm{LMPSLEI}\right)}{\mathrm{d}\left(\mathrm{Ratio}\right)}}= {\displaystyle \frac{\mathrm{d}\left(\mathrm{FNR}\right)}{\mathrm{d}\left(\mathrm{Ratio}\right)}} \times {\mathrm{Weight}}_{\mathrm{loss}} + {\displaystyle \frac{\mathrm{d}(\mathrm{F}\mathrm{P}\mathrm{R})}{\mathrm{d}\left(\mathrm{Ratio}\right)}} \times {\mathrm{Weight}}_{\mathrm{area}} = 0$$

(15)

$${\displaystyle \frac{\mathrm{d}\left(\mathrm{FNR}\right)/\mathrm{d}\left(\mathrm{Ratio}\right)}{\mathrm{d}\left(\mathrm{FPR}\right)/\mathrm{d}\left(\mathrm{Ratio}\right)}}=- {\displaystyle \frac{{\mathrm{Weight}}_{\mathrm{area}}}{{\mathrm{Weight}}_{\mathrm{loss}}}}$$

(16)

In the case of fixed Weight_loss and Weight_area, the trends of FNR and FPR vary inversely with Ratio. Correspondingly, during the process of minimizing LMPSLEI, the societal loss due to landslide omission and misreporting change in opposite directions—one increases while the other decreases. When LMPSLEI reaches its minimum, the losses from omission and misreporting achieve a corresponding balance. We assume that the losses from both are equal, then we can derive the following equation:

$$\mathrm{FNR} \times {\mathrm{Weight}}_{\mathrm{loss}} = \mathrm{FPR} \times {\mathrm{Weight}}_{\mathrm{area}}$$

(17)

$${\displaystyle \frac{\mathrm{FNR}}{\mathrm{FPR}}}={\displaystyle \frac{{\mathrm{Weight}}_{\mathrm{area}}}{{\mathrm{Weight}}_{\mathrm{loss}}}}$$

(18)

By combining Equations (16) and (18), we deduce that the LMPSLEI reaches its minimum value when the Ratio equals Weight_area/Weight_loss. This sample ratio is defined as the recommended sample ratio.

2.3.5. Setting the Sample Ratio and Selection of Samples

The derived recommended sample ratio is expressed in the form of a division. Therefore, in the process of verifying its applicability, the sample ratios are selected as the divisors of Weight_area. Correspondingly, Weight_loss is chosen as divisors of Weight_area. In this study, the Weight_area for the Wenchuan earthquake research area is 36, so the selected divisors are 1, 2, 3, 4, 6, 9, 12, 18, and 36. At the same time, since the Weight_loss for the entire area uses a unified value, this approach assumes that the societal losses from landslides omission are the same at different raster locations, and the same applies to landslides misreporting.

In the process of selecting landslide and non-landslide samples, we use the Python 3.8.19 language to read the label file and randomly select the indices of 10,000 landslide pixels and 10,000 non-landslide rasters. The values of 0 and 1 at these positions are used as label inputs. Subsequently, the dataset is split into training and testing sets at a ratio of 7:3. The landslide labels are derived from manually interpreted landslide vectors, which accurately reflect the boundaries of landslides. Therefore, by randomly selecting non-landslide samples, we can ensure the randomness and diversity of the data, thereby better representing the characteristics of the entire study area.

3. Results

3.1. Correlation Test

Figure 6 shows the correlation among 13 evaluation factors. Among them, the correlation coefficient between elevation and rainfall is the highest at 0.68, which does not exceed the set threshold of 0.7. Therefore, it is considered that the factor selection is reasonable, with no high correlation, and all factors are used for modeling the susceptibility to landslides.

3.2. Comparison of Evaluation Indicators

3.2.1. Comparison of Accuracy, Recall, Specificity, F1 Score

To reduce the randomness of a single training, the average of five trials is taken as the result of the evaluation metrics for each N/P Ratio.

Table 3. Statistical results of Accuracy, Recall, Specificity, and F1 score for different N/P ratios.

N/P Ratio	RF Recall	CNN Recall	RF Specificity	CNN Specificity	RF Accuracy	CNN Accuracy	RF F1 Score	CNN F1 Score
1	0.9372	0.9150	0.8779	0.8856	0.9075	0.9003	0.9102	0.9017
2	0.8661	0.8638	0.9296	0.9221	0.9084	0.9027	0.8631	0.8555
3	0.8169	0.8296	0.9498	0.9425	0.9166	0.9143	0.8304	0.8287
4	0.7687	0.8062	0.9627	0.9492	0.9239	0.9206	0.8016	0.8024
6	0.6905	0.7733	0.9760	0.9635	0.9352	0.9363	0.7528	0.7763
9	0.5921	0.7639	0.9858	0.9686	0.9465	0.9481	0.6885	0.7465
12	0.5278	0.7072	0.9913	0.9773	0.9557	0.9565	0.6467	0.7146
18	0.4138	0.6583	0.9958	0.9847	0.9651	0.9675	0.5552	0.6805
36	0.2877	0.6168	0.9989	0.9918	0.9797	0.9816	0.4332	0.6453

presents the statistics for Accuracy, Recall, Specificity, and F1 score across nine different N/P Ratios

As shown in Table 3, with the increase in the N/P Ratio, Specificity of both RF and CNN continues to increase, suggesting that as the proportion of non-landslide samples increases, the models’ ability to identify non-landslides is enhanced. In terms of Recall, both models exhibit a significant downward trend, indicating that as the proportion of landslide samples decreases, the models’ ability to capture landslide samples is weakening. These phenomena are common when machine learning algorithms handle imbalanced samples [51]. As the number of negative samples increases, the model’s ability to recognize tends to favor the majority samples, resulting in insufficient attention to the minority samples. Compared to RF, CNN demonstrates significantly better stability in Recall, showing higher reliability in identifying minority samples within imbalanced datasets.

Accuracy represents the classification correctness of all samples. In this study, Accuracy of both RF and CNN shows an increasing trend as the N/P Ratio continues to rise. In contrast, F1 score of both RF and CNN show a downward trend, with CNN F1 score declining more slowly, mainly because it can maintain a higher Recall even in the presence of imbalanced sample ratios.

3.2.2. Comparison of LMPSLEI

We obtain Recall and Specificity for 9 groups of N/P Ratios. By setting multiple Weight_loss, the LMPSLEI under different N/P Ratios can be calculated. The results from 5 predictions at each N/P Ratio are imported into Origin 2021 to complete the box plot for potential societal losses. We analyze the trend of LMPSLEI as it varies with N/P Ratios.

Figure 7 presents the box plots of LMPSLEI for RF at different N/P Ratios under nine Weight_loss values. The results indicate that when Weight_loss is 1 (Figure 7a), LMPSLEI exhibits a trend of initially decreasing and then increasing, reaching its minimum at an N/P ratio of 18. As Weight_loss continues to increase (Figure 7b–h), the N/P Ratio that minimizes LMPSLEI decreases correspondingly. When Weight_loss is 36 (Figure 7i), LMPSLEI shows a gradually increasing trend, reaching its minimum at an N/P ratio of 1.

Figure 8 shows the box plots of LMPSLEI for CNN at different N/P Ratios under nine Weight_loss values. Overall, the results of CNN are consistent with those of RF. As Weight_loss increases, the N/P Ratio at which the LMPSLEI is minimized continuously decreases. When Weight_loss is 1 and 36 (Figure 8a,i), the LMPSLEI reaches its minimum at N/P Ratios of 36 and 1, respectively. Unlike RF, for Weight_loss of 2, 3, 4, and 6 (Figure 8b–e), the N/P Ratio that minimizes the LMPSLEI for CNN remains at 36, only beginning to decrease when Weight_loss reaches 9.

The results from RF and CNN indicate that as Weight_loss increases, the N/P Ratio at which LMPSLEI is minimized shows a decreasing trend until the N/P Ratio reaches 1:1. In other words, when the societal loss of landslides omission is significant, reducing the number of landslides omission is essential to effectively decrease LMPSLEI. At this point, we need to reduce the proportion of non-landslide samples in the dataset to ensure that the model focuses more on landslides. This suggests that traditional studies using a 1:1 sampling ratio for positive and negative samples have already considered that the consequences of omitting a landslide are more severe than misreporting one.

3.2.3. Comparison with Recommended Sample Ratio

We infer that when the N/P Ratio is Weight_area/Weight_loss (recommended sample ratio), the LMPSLEI reaches its minimum value. To test the applicability of the recommended sample ratio under different Weight_loss values, the N/P ratios that minimize LMPSLEI for RF and CNN under different Weight_loss values are calculated in Figure 7 and Figure 8 and plotted in Figure 9. In this figure, the x-axis represents Weight_loss, while the y-axis represents the N/P ratio that minimizes LMPSLEI corresponding to each Weight_loss. The Weight_area of the study area is 36, so the recommended sample ratio is calculated based on the value of Weight_loss.

As shown in Figure 9, the N/P Ratio that minimizes LMPSLEI in RF aligns closely with the recommended sample ratio; although there are some discrepancies when Weight_loss is 1, 3, and 4, the difference in LMPSLEI between the two is minimal in Figure 7a,c,d, and can be approximated as compliant. The N/P Ratio that minimizes LMPSLEI in CNN aligns with the recommended sample ratio when Weight_loss is 1 and 36. When Weight_loss is relatively small (Weight_loss is 2–4), the N/P Ratio that minimizes LMPSLEI in CNN significantly differs from the recommended sample ratio, and the N/P Ratio that minimizes LMPSLEI consistently remains at 36 in Figure 8b–d. When Weight_loss is larger (Weight_loss is 6–18), the N/P Ratio that minimizes LMPSLEI is closer to the recommended sample ratio, and the difference in LMPSLEI between the two is also not significant in Figure 8e–h.

Overall, when Weight_loss is determined, the N/P Ratio that minimizes LMPSLEI in CNN is greater than the recommended sample ratio, while the N/P Ratio that minimizes LMPSLEI in RF is less than or equal to the recommended sample ratio (Figure 9). The reason for this overall deviation lies in that during the derivation of the recommended sample ratio, we assume that when LMPSLEI is minimized, the potential societal losses caused by landslide omission and misreporting in the region are equal. However, in reality, the potential societal losses from landslide omission and misestimation are difficult to be completely equal; alternatively, the increases and decreases in both reach a certain balance. The reason for the significant deviation of CNN when Weight_loss is small lies in its relatively stable performance when handling imbalanced sample training, with smaller fluctuations in Recall and Specificity. At the same time, the small Weight_loss value leads to a smaller variation in LMPSLEI, which is why the minimum N/P Ratio for LMPSLEI remains at 36. Additionally, the actual sample ratio that minimizes LMPSLEI does not follow a simple linear relationship, so we can only propose a sample ratio as a reference. The recommended sample ratio presented in the study has certain value as a reference and is capable of minimizing or reducing LMPSLEI in most cases, making it suitable as a reference sample ratio for considering societal loss modeling. However, in the case of modeling with a small Weight_loss in CNN, due to the significant discrepancy between the recommended sample ratio and the N/P Ratio that minimizes LMPSLEI, it is advisable to select a sample ratio larger than the recommended one.

3.3. Landslide Susceptibility Mapping

To highlight the significant impact of different sample ratios on landslide susceptibility zoning, Figure 10 and Figure 11 present landslide susceptibility maps predicted by RF and CNN under four N/P Ratios. In the landslide susceptibility map, the susceptibility index is classified into five levels: very low, low, moderate, high, and very high using the natural breaks method. It can be observed that as the N/P Ratio increases, the area of high and very high susceptibility zones gradually decreases in Figure 10 and Figure 11. This is because the proportion of landslide samples in the dataset gradually decreases, leading the model to be more inclined to predict the samples as non-landslides.

To further evaluate the effectiveness of the landslide susceptibility classification, statistics are calculated for the proportion of landslides area in each susceptibility zone to the total area of all landslides (Figure 12), and the frequency ratio of the proportion of landslides area to the proportion of area in susceptibility zones (Figure 13).

As shown in Figure 12, with the increase in the N/P Ratio, RF and CNN exhibit similar trends; the proportion of landslide area in the very high susceptibility zone gradually decreases, the proportion of landslide area in the high susceptibility zone shows a trend of first increasing and then decreasing, while the proportions in the very low, low, and moderate susceptibility zones gradually increase. In contrast, the proportion of landslide area in very low and low susceptibility zones for RF is lower than that for CNN, while CNN consistently has a higher proportion of landslide area in very high susceptibility zone compared to RF.

As shown in Figure 13, as the N/P Ratio gradually increases, the frequency ratio of the landslide area proportion to the area proportion in each susceptibility level also continuously rises. In the high and very high susceptibility zones, the frequency ratio of RF is higher than that of CNN, while there is little difference in the moderate, low, and very low zones.

In the landslide susceptibility evaluation, when calculating accuracy and other indicators, the landslide susceptibility index is generally divided into landslide and non-landslide categories using a threshold of 0.5. In the susceptibility zones defined by natural breaks, the susceptibility indices of very low and low susceptibility zones are typically less than 0.5. Therefore, landslides that occur in these two zones predicted as non-landslide can be regarded as landslide omission. Correspondingly, the susceptibility indices of very high and high susceptibility zones are usually more than 0.5; the non-landslide in these two zones predicted as landslide can be regarded as landslides misreporting. Moreover, 0.5 typically falls within the range of landslide susceptibility indices for moderate susceptibility zone; therefore, both landslides omission and misreporting exist in this area. When the N/P Ratio is relatively low, the proportion of landslide area distributed in the low and very low susceptibility zones is below 10%, indicating that landslide omission is relatively minor (Figure 12). In the high and very high susceptibility zones, the frequency ratio of landslide area to the area of susceptibility zones is small, suggesting that there are more non-landslide areas within high and very high susceptibility zones, leading to more landslide misreporting (Figure 13). Conversely, when the N/P Ratio is relatively large, the situation is the opposite. The societal losses incurred from landslide omission and misreporting are generally not the same. Therefore, the significance of this study is to minimize the potential societal losses shouldered by the region through a reasonable selection of the N/P Ratio, as discussed in Section 3.2

4. Discussion

4.1. The Best Sample Ratio Can Be Adjusted Around the Recommended Sample Ratio

Weight_loss varies across different study areas, and this study simulates the differences in Weight_loss values across different study areas by setting different societal loss weights for landslide omission/misreporting in the current study area, thereby proving the effectiveness of the recommended sample ratio. Once Weight_loss has been determined for other study areas, the recommended sample ratio can be calculated based on the Weight_area of that area and used as a preliminary selection for sampling

Due to differences in model performance, differences in model performance across different study areas, and the fact that the sample ratios setting in this study only use the Weight_area divisors, there is a certain deviation between the recommended sample ratio and the best sample ratio that truly minimizes LMPSLEI in practical applications. The recommended sample ratio can minimize or sub-optimize LMPSLEI in most cases. If further reduction in LMPSLEI is needed, adjustments can be made to the left or right of the recommended sample ratio based on the trend of LMPSLEI as the sample ratio changes, in order to find the best sample ratio.

As shown in Figure 7e, when Weight_loss is 6, LMPSLEI first decreases and then increases with the increase in the N/P Ratio, reaching its minimum value at a sample ratio of 6 (the recommended sample ratio). Therefore, the best sample ratio should fall within the ranges of 4 to 6 or 6 to 9, where the trend of LMPSLEI changes. Additionally, the recommended sample ratio itself may also be the best sample ratio. Based on this, we add the results for N/P ratios of 5 and 7 for comparison (Figure 14). The results indicate that when the N/P ratio is 5, its LMPSLEI is lower than that when the N/P ratio is 6. Therefore, the best sample ratio is 5. However, the difference in LMPSLEI between the best sample ratio and the recommended sample ratio is relatively small.

4.2. Setting of Weight_loss

In the Weight_loss settings of this study, the societal losses due to landslides omission and misreporting at different raster locations throughout the study area are set to be consistent. Although this method of using the same value is simple to operate, it ignores the differences in societal value between different land use types. For example, the societal value of densely populated, economically developed areas is higher than that of sparsely populated, economically underdeveloped areas. In future studies, population density and GDP will be used to create a Weight_loss weight layer through graded assignment to show the differences in societal value between different raster locations [52,53]. This weight layer will be used in the calculation of LMPSLEI.

5. Conclusions

Addressing the issue that traditional sampling methods in landslide susceptibility evaluation overlook the differences in societal losses caused by landslides omission and misreporting, this study proposes a new sampling method that considers regional societal losses and provides a recommended sample ratio to minimize these losses. This study takes the Wenchuan earthquake research area as an example and uses RF and CNN to conduct research. By setting multiple societal loss weights for landslide omission/misreporting, the effectiveness of the recommended sample ratio in reducing regional social losses is verified. The following conclusions are ultimately reached:

• The “Landslide Misjudgment Potential Societal Loss Evaluation Index” (LMPSLEI) indicator is proposed to measure the regional potential societal losses caused by the landslide omission and misreporting. Unlike simple mathematical indicators (such as accuracy), the LMPSLEI not only considers the societal losses associated with landslide omission and misreporting, but also fully accounts for the significant area discrepancy between landslide and non-landslide within the region. Minimizing LMPSLEI by adjusting the sample ratio can help make landslide susceptibility evaluation results more practical.
• The recommended sample ratio (N/P Ratio = Weight_area/Weight_loss) is proposed as a reference sampling strategy to minimize LMPSLEI. Under different Weight_loss scenarios, this sample ratio can minimize or sub-optimize LMPSLEI. Therefore, it is suitable as an initial choice for sampling when considering societal losses.