Performance Evaluation and Comparison of Bivariate Statistical-Based Artiﬁcial Intelligence Algorithms for Spatial Prediction of Landslides

: The purpose of this study is to compare nine models, composed of certainty factors (CFs), weights of evidence (WoE), evidential belief function (EBF) and two machine learning models, namely random forest (RF) and support vector machine (SVM). In the ﬁrst step, ﬁfteen landslide conditioning factors were selected to prepare thematic maps, including slope aspect, slope angle, elevation, stream power index (SPI), sediment transport index (STI), topographic wetness index (TWI), plan curvature, proﬁle curvature, land use, normalized di ﬀ erence vegetation index (NDVI), soil, lithology, rainfall, distance to rivers and distance to roads. In the second step, 152 landslides were randomly divided into two groups at a ratio of 70 / 30 as the training and validation datasets. In the third step, the weights of the CF, WoE and EBF models for conditioning factor were calculated separately, and the weights were used to generate the landslide susceptibility maps. The weights of each bivariate model were substituted into the RF and SVM models, respectively, and six integrated models and landslide susceptibility maps were obtained. In the fourth step, the receiver operating characteristic (ROC) curve and related parameters were used for veriﬁcation and comparison, and then the success rate curve and the prediction rate curves were used for re-analysis. The comprehensive results showed that the hybrid model is superior to the bivariate model, and all nine models have excellent performance. The WoE–RF model has the highest predictive ability (AUC_T: 0.9993, AUC_P: 0.8968). The landslide susceptibility maps produced in this study can be used to manage landslide hazard and risk in Linyou County and other similar areas. the conditioning factor and the occurrence of landslide were calculated by three di ﬀ erent bivariate models, and the weights were obtained. Then, the weights of the bivariate model were brought into the machine learning model to form six integrated models. Next, these nine di ﬀ erent models were applied to the spatial prediction of landslides in Linyou County. Finally, the accuracy of the generated model was veriﬁed by the ROC curve, the success rate curve and the prediction rate curve using two ways of solving the AUC value, and the best model was obtained by comparison. The AUC_T values of the CF, EBF, WoE, CF–RF, EBF–RF, WoE–RF, CF–SVM, EBF–SVM and WoE–SVM models were 0.8324, 0.8244, 0.8251, 0.9996, 0.9991, 0.9993, 0.8558, 0.8461 and 0.8484, respectively. The results show that all models have good prediction accuracy, but the prediction ability is di ﬀ erent. After comparing the performance of the single model and the hybrid model, it was found that the performance of the single model is signiﬁcantly improved after the collection. As an integrated model, the WoE–RF model has the highest prediction rate (AUC_T: 0.9993, AUC_P: 0.8968). Therefore, the nine landslide susceptibility maps produced by Linyou County can be used as useful tools for government personnel and local authorities to carry out land management and planning.


Introduction
Landslides cause various types of damage and affect people's lives and property [1]. In order to reduce these losses and hazards, relevant assessments of slope conditions where landslides are likely to occur should be made, and a series of countermeasures should be developed based on of Wei River. Its topography is complex, and the density of rivers and gullies is large, with 0.79 per square kilometer. The study area belongs to a temperate semi-humid-humid monsoon climate zone. There are four distinct seasons, with short summers and long winters. In the past 30 years (1990-2019), the maximum annual rainfall was 925.4 mm, the minimum annual rainfall was 355.8 mm and the average annual precipitation was 603.3 mm (Figure 2).

Data Preparation
The thematic map of landslide inventory constitutes the basis for the prediction of landslide susceptibility [58]. In this study, 152 landslides were identified through early reports, remote sensing image interpretation and field investigations [45]. Each landslide polygon in the landslide dataset is represented by a centroid [59]. In order to identify whether landslides are likely to occur in the area, it is necessary to prepare the same number of non-landslide data. Specifically, 152 non-landslide points were randomly selected from areas where landslides did not occur. The landslide data were randomly divided into two parts according to a ratio of 70/30, respectively, forming two sets of data, namely training data and verification data [60].
Relevant factors were selected to predict the occurrence of landslide hazards according to the characteristics of the study area and previous similar studies [61,62]. In the study area, landslide susceptibility evaluation was carried out using fifteen landslide conditioning factors, namely slope aspect, slope angle, elevation, stream power index (SPI), sediment transport index (STI), topographic wetness index (TWI), plan curvature, profile curvature, land use, normalized difference vegetation index (NDVI), soil, lithology, rainfall, distance to rivers and distance to roads ( Figure 3).
The slope aspect affects the discontinuity, the sunshine time and the intensity of solar radiation [63]. The slope aspect map includes nine categories: flat, south, north, southwest, northeast, west, east, northwest and southeast. The slope angle indicates the steepness of the slope, which affects the size and shear strength of the potential slip surface. The slope angle ranges from 0 • to 64.67 • , and it was divided into 6 categories with 10 • intervals. Elevation has a great influence on the degree of rock weathering, and it is an indispensable factor for the prediction of landslide susceptibility [64]. The elevation range is 724-1661 m, which is divided into ten subcategories. SPI indicates the erosion capacity of the water flow [65]. STI describes the potential erosive force of slope flow [66]. SPI and STI were divided into five categories: <10, 10-20, 20-30, 30-40 and >40. TWI can indicate runoff trends and catchment locations. TWI was divided into five categories: <2, 2-3, 3-4, 4-5 and >5. Plan curvature and Profile curvature were divided into three categories: concave, plan and convex. Landuse was divided into six categories: farmland, forestland, grassland, water, residential areas and bareland. NDVI is used to measure vegetation trends that affect the mechanical properties of slopes and hydrological processes related to the instability of the land and slopes in the study area [67,68]. The NDVI values range from −0.02 to 0.58 and were divided into five subcategories. Soil affects the movement of groundwater and surface water, and its physical and mechanical properties vary depending on the type of soil [69,70]. The soil map includes six categories, namely fimic anthrosol, calcaric cambisol, eutric cambisol, gleyic cambisol, calcaric regosol and eutric regosol. Lithology is very important for the occurrence of landslides, and has a great influence on the scale, type, distribution and activity nature of landslides [71]. The lithology is divided into 13 groups. Rainfall can cause a decrease in shear strength and induce landslides [72,73]. Rainfall categories in the study area include: <400, 400-500, 500-600 and >600. Distance to rivers determines the water content of the rocks and soil that make up the main slope [74]. The distance to rivers map includes five buffers: <200, 200-400, 400-600, 600-800 and >800. The construction of a road will affect the stability of the slope angle [75,76]. Therefore, distance to roads affects slope stability. The distance to roads map includes five buffers: <500, 500-1000, 1000-1500, 1500-2000 and >2000. Finally, all thematic maps were resampled and converted to the same resolution (20 m × 20 m).

Certainty Factors (CFs)
The CF model was first proposed in 1990 [77] and modified subsequently [78]. CF can be expressed as follows: where PPa is the conditional probability in class a, PPs is the prior probability of the total study area [79].
The change range of CF to [−1, 1] is such that the larger the value is, the more likely the landslide will occur. When the CF value approaches 0, it is difficult to give the certainty of the landslide, because the prior probability is the same as the conditional probability.

Weights of Evidence (WoE)
WoE is based on the Bayesian probability model and uses logarithmic linear form [80]. The main objective of WoE is to determine the spatial relationship between landslides and conditioning factors [81].
where P represents the probability, B represents each category of each factor and B represents each category of non-members of each factor. The pixel point A represents the occurrence of landslide, while A represents non-landslide. W + indicates the positive correlation between each category and landslide. W − indicates the negative correlation between category and landslide. C indicates the correlation between landslide and category characteristics [82].

Evidential Belief Function (EBF)
EBF method is mainly based on the evidence theory algorithm of Dempster-Shafer [83]. EBF is the sum of Bel (degree of belief), Dis (degree of disbelief), Unc (degree of uncertainty) and Pls (degree of plausibilities), and its range is [0, 1] [84]. Bel is used to expressing the correlation between landslide and conditioning factors in the study, and it is expressed using the following formula [85]: where Bel n denotes the elements of each type or range.

Random Forest (RF)
RF is a powerful integrated learning model proposed by Breiman in 2001 [86]. It has outstanding performance in classification, regression and unsupervised learning [87,88]. In the RF model, the landslide conditioning factor iP (random variable) and a previous random variable are generated independently and distributed on the binary decision tree. The training and verification dataset and random vector i P are grown on two trees (landslides and non-landslides), and the set of the tree structure classification h x, i p , p = 1, 2, . . . n of input variable x is obtained. Generally, in RF method, generalization error is shown as follows [89]: generalization error = P x,y (mg(x, y) < 0) (6) mg(x, y) = av p I h p (x) = y − max j y av p I h p (x) = j where x and y represent the landslide conditioning factors, mg is the margin function and I is the indicator function [86].

Support Vector Machine (SVM)
SVM is a non-linear classification system based on structural risk minimization and Vapnik-Chervonenkis dimension, which can be used for assortment and recurrence [90]. It separates classes from decision-making areas and maximizes the gap between classes [91]. This method can find an optimal hyperplane for binary classification problems to separate the two classes [92].
In this study, we used the radial basis function with optimum ϕ(m, The decision function of SVM can be expressed by the following formula [92]: where γ is used as the kernel parameter and η i is used as the Lagrange multiplier.

Application of the CF Model
The CF model was applied to each class of the landslide conditioning factors, and the CF weights were computed as shown in Figure 4. It is noted that the positive CF weights indicate the higher impact of the class of the conditioning factors on landslide occurrence. Accordingly, in terms of slope aspect, the results show that southeast (CF = 0.475), south (CF = 0.264), southwest (CF = 0.180) and east (CF = 0.074) were the most important classes for landslide incidence. However, the northeast, northwest, north, west, and flat had the least effect due to having negative CF weights. In terms of slope angle, the results showed that the slope angle between 30 • and 40 • had the highest potential for landslide occurrence (CF = 0.196). Slope angles lower than 10 • were ranked in the next position with CF weight equal to 0.029, and other slope angle classes were not more effective and assigned by negative weights so that slope angles more than 40 • did not play any role in landslide occurrence (CF = −1). Elevations between 800 and 900 m (CF = 0.587), 1100 and 1200 m (CF = 0.341), 1200 and 1300 m (CF = 0.089) and 1400 and 1500 m (CF = 0.116) were positively correlated with the occurrence of landslides in the study area. Another conditioning factor, SPI, analyzed based on CF weight, showed that SPI between 30 and 40 (CF = 0.271) had the greatest potential for landslide occurrence. It is followed by SPI between 10 and 20 (CF = 0.193) and SPI lower than 10 (CF = 0.046). This indicates that these classes of SPI were more prone to landslide occurrence compared to other classes. Analysis of CF weights for STI illustrated that the values between 30 and 40 and lower than 10 assigned a CF equal to 0.342 and 0.067 and thus concluded that they were more susceptible compared to other classes of STI for landslide occurrence. In the case of TWI, the result of CF weights indicated that lower TWI values (<2) obtained the highest CF value (0.098). According to Figure 1, 66 landslide locations occurred only at this class of TWI. Additionally, results showed that the higher the TWI was, the lower the potential for landslide incidence would be. In terms of plan curvature, results showed that plan (CF = 0.216) and convex (CF = 0.144) slope forms and, in the case of plan curvature, concave slope form (CF = 0.059) were the most important on landslide occurrence in the study area.

Application of EBF Model
In this study, the EBF model was performed, and the result is reported based on the "Bel" index, which is shown in Figure 4. The higher the Bel weight of each class of the conditioning factor was, the higher the potential of the class would be. The southeast slope aspect was the most important class among other slope aspect classes (Bel = 0.235) to landslide occurrence. It is followed by south (Bel = 0.168), southwest (Bel = 0.151), east (Bel = 0.133), northeast (Bel = 0.094), northwest (Bel = 0.078), north (Bel = 0.072), west (Bel = 0.068) and flat (Bel = 0.000). In the case of slope angle, results showed that slope angles lower than 40 • were more susceptible and had more potential for landslide occurrence. Among all classes, slope angles between 30 • and 40 • had the greatest impact on landslides of the study area (Bel = 0.294). This was followed by 0 • -10 • (0.244), 10 • -20 • (0.232) and 20 • -30 • (0.230). Although only two landslides occurred in elevations between 800 and 900 m, the number of equipped pixels for this class is low and thus this class of elevation obtained the highest Bel weights (0.292), indicating the highest susceptibility for landslide occurrence. This was followed by 1100-1200 (0.183), 1400-1500 (0.136), 1200-1300 (0.132), 1000-1100 (0.105), 1300-1400 (0.086) and 1500-1600 (0.066). In the case of SPI and Bel weights, results showed that SPI between 30 and 40 (0.291), 10 and 20 (0.263), <10 (0.223), >40 (0.174) and between 20 and 30 (0.076) were the most important SPI values for landslide occurrence. Among all classes of STI, factor results showed that STI between 20 and 30 had the highest Bel weight (0.459), and thus it was shown to be the most susceptible class. It was followed by STI < 10 (0.324) and between 10 and 20 (0.218). However, according to TWI results, with increasing TWI values, the probability of landslide occurrence decreased, such that the first class (TWI < 2) was shown to be the critical class for landslide occurrence (Bel = 0.332).

Application of WoE Model
The WoE model was performed on a training dataset, and the weights (C) were calculated for each class of each conditioning factor. A positive C weight of the class indicated greater importance of landslide occurrence in the class. Results showed that in the case of slope aspect, the highest C weight was obtained for southeast (0.775) because most of the landslide locations (24 cases) occurred in this class. This was followed by south (C = 0.349), southwest (C = 0.230), east (C = 0.091), northeast (C = −0.304), northwest (C = −0.506), north (C = −0.594), west (C = −0.658) and flat (C = 0.000). In terms of slope angle, although only nine landslide locations occurred at a slope angle between 30 • and 40 • (C = 0.236), this class was shown to be the most susceptible class to landslide occurrence compared to other classes. This was due to this fact that these few landslides occurred in a low number of pixels with a slope angle between 30 • and 40 • . Slope angles between 0 • and 10 • (C = 0.038), 10 • and 20 • (C = −0.036) and 20 • and 30 • (C = −0.038) were shown to be ranked as next in importance in terms of susceptibility to landslide occurrence. However, slope angles higher than 40 • had no susceptibility to landslide occurrence because they did not have any landslides. The most susceptible range of elevation above sea level for landslide incidence in this study was obtained for 800-900 m (C = 0.896). Elevations between 1100 and 1200 m (C = 0.507), 1400 and 1500 m (C = 0.152) and 1200 and 1300 m (C = 0.126) were assigned positive weights and hence had more landslide incidence compared to those with negative C weights such as elevation ranges of 1000-1100 m (C = −0.149), 1300-1400 m (C = −0.448) and 1500-1600 m (C = −0.616). However, there were no landslides at elevations lower than 800 m (C = 0.000) or higher than 1600 m (C = 0.000) and therefore no possibility of landslides at these elevations. For SPI factor, results showed that the highest C weight was computed for SPI between 30 and 40 (0.333), followed by 10 and 20 (C = 0.272), <10 (C = 0.133), >40 (C = −0.407) and 20 and 30 (C = −1.085). Results of C weight for STI showed that the most susceptible class for landslide incidence was STI between 20-30 (C = 0.439), followed by STI < 10 (C = 0.337) and 10-20 (C = −0.383). Results also indicated that STI > 30 had no contribution to landslide occurrence in the study area. There was a reverse relationship between TWI and probability of landslide occurrence, so the higher the TWI values, the less the C weights would be and thus the lower the probability of landslide occurrence. In other words, the lowest TWI values had the highest potential for landslide incidence. For example, TWI < 2 with C weight equal to 0.254 was more susceptible compared to other classes such as TWI between 2 and 3 (C = −0.177), 3 and 4 (C = −0.196), 4 and 5 (C = −0.680) and >5 (C = 0.000). Similar to CF and Bel indexes, the C weight for plan and profile curvatures had the same result; however, in terms of profile curvature, concave slope forms were more important and had the highest contribution to landsides in the study area.

Hybrid Integration of CF, EBF and WoE with RF Model
The bivariate models, namely CF, EBF and WoE, used in this study to landslide modeling have the weakness that they only consider the sub-factor weights for landslide susceptibility assessment whereas not all of the factors have the same effect on landslide occurrences. Therefore, it was assumed that if the RF decision tree classifier were to be integrated with the mentioned bivariate models, the goodness-of-fit and prediction accuracy would be enhanced by decreasing the noise and over-fitting problems of the RF classifier. At first, the landslide susceptibility index (LSI) for each class of each landslide conditioning factor were computed and assigned by CF, EBF and WoE methods. Then, the landslide training dataset was overlaid with the obtained results, and it was considered as input to the RF classifier. Finally, the LSIs were computed in Weka software [94] and then transformed in ArcGIS software.

Hybrid Integration of CF, EBF and WoE with the Benchmark SVM Model
In this study, to check the prediction power of the bivariate models integrated with the SVM model, we integrated the bivariate models of CF, EBF and WoE with SVM as a benchmark model. We first computed the landslide susceptibility index (LSI) for each class of each landslide conditioning factor by CF, EBF and WoE bivariate methods. In the next step, we overlaid landslide training locations on each bivariate model, and for each landslide location, a feature was extracted. All of these features were the considered as input to the SVM classifier. Consequently, the LSIs were computed in Weka software and then were transformed in ArcGIS software.

Model Validation and Comparison
In this study, the performance and prediction accuracy of the individual bivariate models and its ensembles with RF and SVM models were checked and analyzed using training and validation datasets by plotting the ROC curve [96][97][98] (Table 1; Table 2, Figures 6 and 7). The SVM was used as a benchmark model for assessing the ensemble of the bivariate models with RF classifiers. The SVM is known as a powerful state-of-the-art soft computing machine learning model that outperformed other models in many studies of susceptibility mapping [47,99]. The AUC, SE and CI at 95% confidence level statistical metrics were used for this aim. According to Table 1, results showed that the AUC values were 0.827, 0.816, 0.819, 0.982, 0.980, 0.978, 0.856, 0.856 and 0.857 for CF, EBF, WoE, CF-RF, EBF-RF, WoE-RF, CF-SVM, EBF-SVM and WoE-SVM models, respectively. According to Table 1, results showed that among individual models, the CF had the highest performance (AUC = 0.827), followed by WoE and EBF models. Moreover, it shows that among ensemble models, the highest performance was obtained for the CF-RF model, followed by EBF-RF and WoE-RF models. In other words, the bivariate models, when integrated with the RF model, had the highest performance/goodness-of-fit compared to its ensembles with the SVM model.   The results of the prediction accuracy of the individual bivariate models and its ensembles with RF and SVM using the validation dataset are shown in Table 2 and Figure 7. Although according to the training dataset, the CF bivariate model was more powerful than the EBF and WoE models, the WoE model based on the validation dataset was selected as having the highest prediction accuracy. Additionally, it indicated that the AUC values were 0.727, 0.725, 0.733, 0.861, 0.848, 0.860, 0.802, 0.811 and 0.795 for CF, EBF, WoE, CF-RF, EBF-RF, WoE-RF, CF-SVM, EBF-SVM and WoE-SVM models, respectively. It is also similar to the training dataset that of all of these models: the CF bivariate model integrated with RF model outperformed and outclassed other models with the highest prediction accuracy (AUC = 0.861). This was followed by WoE-SVM (AUC = 0.860) and EBF-SVM (AUC = 0.848). However, the individual bivariate models, when integrated with the SVM model, showed improved prediction accuracy.

Validation of Landslide Susceptibility Maps
In this study, in addition to checking the goodness-of-fit and prediction accuracy by the ROC curve, we evaluated the usability of the proposed models using success and prediction rate curves. The ROC curve was plotted based on training and validation datasets, that are used to evaluate the performance and prediction accuracy of a given model [100,101]. However, the success and prediction rate curves were designed based on only landslide locations divided into training and validation datasets, respectively [95]. Table 3 and Figures 8 and 9 show the validation of the proposed models by success (AUC_T) and prediction (AUC_P) rate curves. It can be observed that individual models had a higher performance in comparison to the ROC curve (AUC > 0.82). Among individual bivariate models, results illustrated that the CF model had the highest performance (AUC_T = 0.8324) and prediction accuracy (AUC_P = 0.7679) for landslide susceptibility mapping in the study area, followed by WoE and EBF models. Results according to training dataset showed that the AUC-Ts were 0.8324, 0.

Discussion
Landslides are one of the most significant natural disasters for all countries in the world, and landslide prediction is getting more and more attention [102]. To obtain higher accuracy of landslide susceptibility maps, there are higher requirements for the quality of the collected data, the selection of suitable models and the selection of effective parameters of the models [7]. The main advantage of the bivariate model is that it is easy to understand, does not require too much training and does not require parameter adjustment. However, bivariate algorithm modeling must be based on strict compliance with assumptions and does not analyze the relationships between each factor. Therefore, the bivariate model always ignores the importance of parameters. For machine learning models, these problems are avoided. Machine learning algorithms can determine the best parameters, but they cannot determine the weight of each factor category. It is therefore particularly necessary to introduce some integrated models to eliminate these limitations. In recent years, some integrated machine learning methods have been applied to landslide susceptibility research. For example, Nguyen et al. [103] proposed a new hybrid machine learning model for spatial prediction of landslides, namely particle swarm optimization adaptive neural fuzzy inference system (PSOANFIS), particle swarm optimization artificial neural network (PSOANN) and rotation forest based on optimal first decision tree (RFBFDT). The results showed that RFBFDT (AUC = 0.826) is the best method compared with other hybrid models and a promising hybrid machine learning method. Tien Bui et al. [104] proposed an integrated model (ABSGD) combining a function algorithm, stochastic gradient descent (SGD) and AdaBoost (AB) Meta classifier to predict the spatial distribution of landslides in Iran's Sarkhoon watershed. The results showed that the performance of the ABSGD model (AUC = 0.860) is better than other models. The combined use of the function algorithm and the Meta classifier can reduce noise, prevent overfitting, and improve the prediction ability of a single SGD algorithm in landslide spatial prediction. Pham et al. [105] proposed a new hybrid intelligent model MBSVM for landslides in the Uttarakhand State, Northern India, which is an integration of MultiBoost integration and support vector machine (SVM). Comparison results show that MBSVM (AUC = 0.966) is superior to LR, single SVM and mixed ABSVM models.
In this article, the landslide susceptibility was simulated for 15 landslide conditioning factors such as the slope aspect, slope angle, elevation, SPI, STI, TWI, plan curvature, profile curvature, landuse, NDVI, soil, lithology, rainfall, distance to rivers and distance to roads. The CF, EBF and WoE models as bivariate models and the RF and SVM models as machine learning models were integrated and compared to choose the best model. The CF, WoE and EBF models calculated the correlation between landslide conditioning factors and landslide occurrence. The weights of the CF, EBF and WoE models are used to judge the importance of each type of factor for landslides. In order to get an integrated model with the RF model, firstly, the CF, EBF and WoE models were calculated and the LSI of various landslide conditioning factors was recorded. Then, LSI was superimposed with the landslide dataset, and the corresponding features were extracted for each landslide point in the dataset to form new model training input data. Finally, the LSI calculated by the Weka software was converted into ArcGIS software, and the landslide susceptibility map is obtained and partitioned. The same is true for the comparative SVM integration model. In order to obtain the advantages and disadvantages between the models, the ROC curve and the necessary parameters are used for comparison between the models. The comparison of the nine models shows that all the integrated models are better than all the single models, and the CF-RF model has the best performance among the nine models. The integration model mixed with RF is superior to the integration model mixed with the comparison model SVM selected after many tests. The CF-RF model in the validation data set had the largest AUC value (0.848) and the smallest SE (0.0375). In order to obtain the most suitable model for this study area more accurately, the success rate (AUC_T) curve and the prediction rate (AUC_P) curve were introduced to verify the AUC value obtained by the nine models. As stated in the results, the success rate (AUC_T) curve and the prediction rate (AUC_P) curve only consider the landslides that have occurred, and will have better performance than the AUC value obtained by the ROC curve. For the landslide susceptibility map, this classification method will be better matching. The results show that the WoE-RF model as an integrated model is superior to the other eight models (AUC_T: 0.9993, AUC_P: 0.8968). The other eight models also produced reasonable and good performance.

Conclusions
Landslide susceptibility prediction is critical to spatial planning and civil safety. Landslide susceptibility mapping is an indispensable step for spatial prediction of landslide susceptibility. This study introduced RF as a machine learning algorithm combined with three bivariate models (the CF, WoE and EBF models) to solve this problem. For this reason, 15 conditioning factors were selected in the study area. These factors are slope aspect, slope angle, elevation, SPI, STI, TWI, plan curvature, profile curvature, landuse, NDVI, soil, lithology, rainfall, distance to rivers and distance to roads. In this paper, the correlation between the conditioning factor and the occurrence of landslide were calculated by three different bivariate models, and the weights were obtained. Then, the weights of the bivariate model were brought into the machine learning model to form six integrated models. Next, these nine different models were applied to the spatial prediction of landslides in Linyou County. Finally, the accuracy of the generated model was verified by the ROC curve, the success rate curve and the prediction rate curve using two ways of solving the AUC value, and the best model was obtained by comparison. The AUC_T values of the CF, EBF, WoE, CF-RF, EBF-RF, WoE-RF, CF-SVM, EBF-SVM and WoE-SVM models were 0.8324, 0.8244, 0.8251, 0.9996, 0.9991, 0.9993, 0.8558, 0.8461 and 0.8484, respectively. The results show that all models have good prediction accuracy, but the prediction ability is different. After comparing the performance of the single model and the hybrid model, it was found that the performance of the single model is significantly improved after the collection. As an integrated model, the WoE-RF model has the highest prediction rate (AUC_T: 0.9993, AUC_P: 0.8968). Therefore, the nine landslide susceptibility maps produced by Linyou County can be used as useful tools for government personnel and local authorities to carry out land management and planning.