Performance Evaluation of Five GIS-Based Models for Landslide Susceptibility Prediction and Mapping: A Case Study of Kaiyang County, China

: This study evaluated causative factors in landslide susceptibility assessments and compared the performance of ﬁve landslide susceptibility models based on the certainty factor (CF), logistic regression (LR), analytic hierarchy process (AHP), coupled CF–analytic hierarchy process (CF-AHP), and CF–logistic regression (CF-LR). Kaiyang County, China, has complex geological conditions and frequent landslide disasters. Based on ﬁeld observations, nine inﬂuencing factors, namely, altitude, slope, topographic relief, aspect, engineering geological rock group, slope structure, distance to faults, distance to rivers, and normalized difference vegetation index, were extracted using the raster data model. The precision of the ﬁve models was tested using the distribution of disaster points for each grade and receiver operating characteristic curve. The results showed that the landslide frequency ratios accounted for more than 75% within the high and very high susceptibility zones according to the model prediction, and the AUC evaluating precision was 0.853, 0.712, 0.871, 0.873, and 0.895, respectively. The accuracy sequencing of the ﬁve models was CF-LR > CF-AHP > LR > CF > AHP, indicating that the CF-AHP and CF-LR models are better than the others. This study provides a reliable method for landslide susceptibility mapping at the county-level resolution.


Introduction
Landslides occur when gravity and other geologic forces move surface material downward along a slope; often, landslides cause loss of human life and property or environmental damage [1][2][3][4]. In China, many areas have been critically affected by landslides in recent years. This severely hinders the sustainable development of society and the economy and threatens the safety of people's lives and property [5][6][7][8]. Therefore, sensitivity analysis of regional landslides has important research value and practical significance for land-use planning and disaster management [9,10].
Susceptibility assessment of landslides is the first step in the hazard and risk assessment of landslides [11]. It includes summarizing the characteristics, distribution rules, spatial density, and sensitivity that influences factors of regional landslides to complete a sensitivity assessment and develop a sensitivity zoning map [12]. In the past few decades, Kaiyang County is located in Guizhou Province (Figure 1a), China (26 • Figure 1b) and serves as the study area for this research. The region ranges in altitude from 515 to 1664 m (Figure 1c), with the highest elevations recorded in the southwest and the lowest in the northeast. Presinian, Cambrian, Carboniferous, Diassic, Triassic, Ordovician, Tertiary, and Quaternary strata are distributed across the county. The Baimadong Fault is the main fault in the region and is located in central Guizhou. The fold structure has the characteristics of a short and wide dome-shaped anticline and a narrow and compact linear syncline. Kaiyang County has a subtropical monsoon climate. Greater precipitation occurs in the west and south, while less precipitation occurs in the north. The Wujiang, Qingshui, and Gusa rivers are the main rivers in the study area and belong to the Yellow River network. The average annual temperature is between 10.6 and 15.30 • C. July is the hottest month (average temperature = 22.3 • C, maximum temperature = 35.4 • C), and January is the coolest (average temperature = 2 • C, minimum temperature = −10.1 • C) [50]. , and January is the coolest (average temperature = 2 °C, minimum temperature = −10.1 °C) [50].

Methodology
A digital elevation model (DEM) with 30 m × 30 m resolution was used to extract a set of topographic factors for the study area. The DEM was sourced from the International Scientific and Technical Data Mirror Site, Computer Network Information Center, and Chinese Academy of Sciences (http://www.gscloud.cn, accessed on 23 July 2010). LAND-SAT-8 satellite images with a spatial resolution of 30 m × 30 m were also provided by the same institutions. Lithology maps for the study area at a scale of 1:50,000 were collected from the local Land and Resources Bureau.

Landslide Inventory Map
A total of 141 landslides were delineated through field investigations and landslide mapping. The landslide classification system developed by Varnes (1984) [1] was used in this study; 102 earth slides, 35 rock falls, and 4 debris flows were identified, accounting for 72%, 25%, and 3% of the total number of landslides in the region, respectively. Among these, earth slides are the most serious type of landslide in Kaiyang County (Figures 1c  and 2).

Methodology
A digital elevation model (DEM) with 30 m × 30 m resolution was used to extract a set of topographic factors for the study area. The DEM was sourced from the International Scientific and Technical Data Mirror Site, Computer Network Information Center, and Chinese Academy of Sciences (http://www.gscloud.cn, accessed on 23 July 2010). LANDSAT-8 satellite images with a spatial resolution of 30 m × 30 m were also provided by the same institutions. Lithology maps for the study area at a scale of 1:50,000 were collected from the local Land and Resources Bureau.

Landslide Inventory Map
A total of 141 landslides were delineated through field investigations and landslide mapping. The landslide classification system developed by Varnes (1984) [1] was used in this study; 102 earth slides, 35 rock falls, and 4 debris flows were identified, accounting for 72%, 25%, and 3% of the total number of landslides in the region, respectively. Among these, earth slides are the most serious type of landslide in Kaiyang County (Figures 1c and 2).

Landslide Conditioning Factors
The landslide susceptibility evaluation is based on the selection of evaluation factors [51,52]. The selection of factors that contribute to the development of landslides should be performed in combination with field investigation data. Based on the detailed investigation and study of typical disaster sites, this paper concludes that landslides in Kaiyang County are mainly controlled by topography, landform, stratigraphic lithology, geological structure, and water system dynamics. Landslides are mainly distributed alongside rivers and in areas with dense faults. Low-altitude areas recorded a significantly higher number of landslides than high-altitude areas, and 58% of landslides occurred in Cambrian strata.

Landslide Conditioning Factors
The landslide susceptibility evaluation is based on the selection of evaluation factors [51,52]. The selection of factors that contribute to the development of landslides should be performed in combination with field investigation data. Based on the detailed investigation and study of typical disaster sites, this paper concludes that landslides in Kaiyang County are mainly controlled by topography, landform, stratigraphic lithology, geological structure, and water system dynamics. Landslides are mainly distributed alongside rivers and in areas with dense faults. Low-altitude areas recorded a significantly higher number of landslides than high-altitude areas, and 58% of landslides occurred in Cambrian strata.
To evaluate landslide susceptibility for this study, previous research conclusions on the influencing factors of landslides were considered, as well as the nine additional factors: altitude, slope, topographic relief, slope direction, engineering rock group, slope structure, fault, drainage system, and normalized difference vegetation index (NDVI) [53,54].

Correlations between Landslides and Their Conditioning Factors
Using the GIS software, the element layers were converted into raster data. Each evaluation factor layer and the landslide distribution layer were superimposed for analysis. The distribution rules of different factors and landslides were counted, and the correlation analysis between landslide pregnancy factors and landslide occurrence was carried out for the study area. Percentage of area is the ratio of a graded area to the total area of the study area, and percentage of landslide number is the ratio of the number of landslides to the total number of landslides within a certain classification. Further, Pn refers to the landslide density in the grading interval, that is, the number of landslides occurring in the unit area of the grading interval (units: km 2 ). To evaluate landslide susceptibility for this study, previous research conclusions on the influencing factors of landslides were considered, as well as the nine additional factors: altitude, slope, topographic relief, slope direction, engineering rock group, slope structure, fault, drainage system, and normalized difference vegetation index (NDVI) [53,54].

Correlations between Landslides and Their Conditioning Factors
Using the GIS software, the element layers were converted into raster data. Each evaluation factor layer and the landslide distribution layer were superimposed for analysis. The distribution rules of different factors and landslides were counted, and the correlation analysis between landslide pregnancy factors and landslide occurrence was carried out for the study area. Percentage of area is the ratio of a graded area to the total area of the study area, and percentage of landslide number is the ratio of the number of landslides to the total number of landslides within a certain classification. Further, Pn refers to the landslide density in the grading interval, that is, the number of landslides occurring in the unit area of the grading interval (units: km 2 ).

Altitude
Kaiyang County has an elevation of 515-1664 m, with a relative elevation difference of 1149 m. The elevation is divided into five class ranges: 515-827, 828-993, 994-1133, 1134-1287, and 1288-1664 m (Figure 3a). The statistical results of the elevation classification and spatial distribution of landslides are shown in Figure 4a. It was observed that elevation is mainly concentrated between 828 and 1287 m, accounting for 78.5% of the total area of Kaiyang County; notably, 93.8% of the disaster sites developed in areas below 1287 m. The height range of 515-827 m developed the highest density of landslides, up to 0.14 km 2 .

Slope
Slope is an important factor that affects the occurrence of landslides; it affects the direction of water flow and soil development [55]. The Kaiyang landscape is predominated by a staggered distribution of mountain basins. The range in slopes found within the region was divided into six classes: 0-10 • , 10-20 • ,20-30 • , 30-40 • , 40-50 • , and >50 • ( Figure 3b). The statistical results of the slope classification and spatial distribution of landslides are shown in Figure 4b. The slopes are mainly concentrated within 0-30 • , accounting for 94.3% of the total area. Moreover, 49.6% of the slopes conducive to geological disasters appeared in the range of 20-50 • , accounting for 20.8% of the total area. The 30-40 • slopes contained the largest density of landslides, up to 0.23 km 2 , indicating that this slope range is the most prone to landslides.

Slope Structure
The slope structure plays an important role in landslide distribution [54]. The types of slope structure were divided into four classes (i.e., transverse slope, reverse slope, oblique slope, and dip slope) according to the relationship between rock strata inclination and the included angle of the topographic aspect ( Figure 3f). The study area was dominated by oblique slopes, accounting for 43.8% of the total area, and 85.5% of the disaster sites were developed in the transverse, normal, and oblique slopes. The density was the highest in the Shunxiang slope area, accounting for 14.4% of the total area. This area is highly prone to landslides, accounting for 9.6% of the total area ( Figure 4f).

Distance to Faults
Faults are an important influencing factor on landslide occurrence. Under the influence of faults, rock mass is broken, and joints and fissures develop, which have an adverse influence on slope deformation [58]. The distance to faults in the study area was estab-

Topographic Relief
Topographic relief can reflect the macroscopic features of the terrain within a certain range [47]. Based on 30 m resolution DEM data, this study calculated the topographic relief of the study area and divided it into five classes (<18, 18-32, 32-50, 50-79, and >79 m) from Jenks natural breaks method ( Figure 3c). The statistical results of the topographic relief classification and spatial distribution of landslides are shown in Figure 4c. The topographic relief is mainly concentrated at 0-79 m, accounting for 90.9% of the total area of Kaiyang County. The density of topographic relief is the highest in the >79 m topographic relief region, up to 0.25 km 2 , indicating that this is a highly prone area for landslides.

Aspect
The influence of aspect on the occurrence of landslides is mainly manifested in the microclimate and the regular change of the hydrothermal ratio on the hillside [56]. The aspect factor was divided into 9 classes: (−1) flat, north (0-25.  Figure 4d. The results show that southeast, west, and northwest aspects are most conducive to landslides in the study region, with the number of disaster points accounting for 48.9% of the total area and the area accounting for 40.0% of the total area of the study area. Among them, the aspect associated with the highest density of landslides is the west aspect, accounting for up to 0.11 km 2 , indicating that this aspect is the most conducive to landslides.

Engineering Geological Rock Group
The engineering geological rock group is the material basis for landslides. The rock type and degree of hardness determine the weathering resistance of the rock mass [57]. Different lithological units were grouped into three different categories (hard rocks, soft and hard interbedded rocks, and soft rocks) based on landslide susceptibilities (Figure 3e). Hard rocks dominate in Kaiyang County (54.6% of the total area), while the density of soft rocks was the highest. The number of developmental disasters accounted for 45.5% of the total disasters, and the area accounted for 22.3% of the total area ( Figure 4e).

Slope Structure
The slope structure plays an important role in landslide distribution [54]. The types of slope structure were divided into four classes (i.e., transverse slope, reverse slope, oblique slope, and dip slope) according to the relationship between rock strata inclination and the included angle of the topographic aspect ( Figure 3f). The study area was dominated by oblique slopes, accounting for 43.8% of the total area, and 85.5% of the disaster sites were developed in the transverse, normal, and oblique slopes. The density was the highest in the Shunxiang slope area, accounting for 14.4% of the total area. This area is highly prone to landslides, accounting for 9.6% of the total area ( Figure 4f).

Distance to Faults
Faults are an important influencing factor on landslide occurrence. Under the influence of faults, rock mass is broken, and joints and fissures develop, which have an adverse influence on slope deformation [58]. The distance to faults in the study area was established at a distance of 500 m (Figure 3g), and the sensitivity of each classification to landslides was statistically analyzed. The results are presented in Figure 4g. The results show that there are 65 landslides from 0 to 500 m on both sides of the fault, accounting for 46% of the total, and 36 disaster points from 500 to 1000 m away from the fault, accounting for 25.5% of the total. Generally, the landslide is negatively correlated with the distance from the fault within 1000 m. The farther the fault from the area, the lower the landslide probability.

Distance to Rivers
Proximity to rivers can also control the distribution of landslides. Rivers have an erosion effect on the bank slope, which can change the internal stress state of the slope body and increase the possibility of bank slope instability and slippage [59]. For the rivers in the study area, buffer zones were established at intervals of 200 m (Figure 3h), and the sensitivity of the buffer zones to landslides was analyzed. The results are presented in Figure 4h. As shown in the figure, in the region within 800 m of the water system, the disaster points accounted for 71.4%, which is the area prone to landslides, accounting for 41.6% of the total area. Within the range of more than 600 m, the density showed a trend of gradual decrease with the increase in distance, indicating that the influence of the water system on landslides decreases with an increase in distance within this range.

Normalized Difference Vegetation Index (NDVI)
The NDVI is often used to reflect human engineering activities [60]. As an index representing vegetation characteristics, vegetation coverage can represent external factors that contribute to landslides. The NDVI is divided into five class ranges: −0.098-0.070, 0.070-0.137, 0.138-0.181, 0.182-0.221, 0.222-0.274, and 0.275-0.514 (Figure 3i). Overall, vegetation coverage was relatively high in Kaiyang County. The NDVI is mainly concentrated in the range of 0.070-0.274, accounting for 93.7% of the total area; 62.8% of the disaster sites are developed in the range of NDVI= −0.098-0.181. This indicates that this area is conducive to the occurrence of landslides, accounting for 48.1% of the total area. In particular, the density of landslides is the highest in the range of 0.070-0.138, up to 0.11 km 2 , which is the area most prone to landslides (Figure 4i).

Correlations Analysis of Landslide Conditioning Factors
In the evaluation of landslide susceptibility, each conditioning factor is not independent but has a correlation with each other. If it is not dealt with, the weight of each conditioning factor may overlap with each other, leading to errors in the evaluation results. Therefore, this study used the variance inflation factor (V IF) and tolerance (TOL) to check for multicollinearity of the conditioning factors and used the Pearson correlation coefficient (PCC) to analyze the correlation between two conditioning factors. The V IF refers to the ratio of the variance between explanatory variables in the case of multicollinearity and variance in the case of no multicollinearity, so it can reflect the increased degree of variance caused by multicollinearity. Its formula is [61] where R 2 i represents the coefficient of determination between the ith factor X i and other factors. The tolerance and variance inflation factor are reciprocal of each other. A V IF > 5 or a TOL < 0.2 indicates a potential multicollinearity problem in the dataset [61].
Suppose the sample dataset (X i , Y j ) = (x 1 , y 1 ), (x 2 , y 2 ), . . . ,(x n , y n ). Then, the calculation formula of the correlation coefficient among evaluation factors is [62,63] where x i and y i are the variable values of X i and Y j , respectively, and x and y are the mean values of X i and Y j , respectively. The greater the absolute value of the PCC, the stronger the correlation of factors. A Pearson's correlation coefficient >0.7 indicates high collinearity [64]. The multicollinearity among the remaining 9 landslide conditioning factors was identified using TOL, V IF (Table 1), and Pearson's correlation coefficient ( Table 2). The results show that the highest V IF value is 3.943 for topographic relief, which is less than the critical value of 5. The highest correlation value (−0.267) appeared between slope structure and topographic relief, which is also less than the critical value of 0.7, indicating no multicollinearity. Therefore, all remaining 9 factors were included in the present analysis. The CF model was first proposed by Shortliffe and Buchanan in 1975 [65] and improved by Heckerman (1986) [66]. The CF is a probability function used to analyze the sensitivity of various factors affecting the occurrence of events. CF is widely used in landslide susceptibility assessments. The basic assumption is that the susceptibility of a landslide can be determined based on the statistical relationship between previous landslides and datasets determined as environmental factors [29,67,68]. The equation is expressed as follows: where PP a is the conditional probability of events occurring in class a; that is, the ratio of the number of disaster points to the area in the unit of class a. Further, PP s is the prior probability of an event occurring; that is, the ratio of the number of potential landslide points to the area in the study area.
The CF values ranged from −1 to 1. Positive numbers show an increase in certainty, while negative numbers correspond to a decrease in certainty. Numbers close to zero imply that the prior probability is similar to the conditional probability and do not provide any indication of certainty.
The weighted sum is carried out to obtain the landslide susceptibility index of each unit, which can be calculated by where I j is the susceptibility index of the jth evaluation unit and CF i is the CF value of each grade of the ith impact factor.

Analytic Hierarchy Process (AHP)
The AHP is a multi-index analysis and evaluation method with high precision and ease of use [69,70]. The principle is as follows: first, the evaluation factors are selected, a hierarchical structure model is established, and the relative importance of the selected evaluation factors is scored by experts. Then, the 1-9 scale method (Table 3) given by Saaty is used for scoring to construct a pair judgment matrix and test the consistency [19,71]. The consistency test index is expressed as follows: where CI is the causative index, RI-Mean is the random consistency index of the same order, and CR-random is the consistency ratio. Table 3. Judgment matrix scale and its meaning [69].

1
Indicates that two factors are of equal importance compared to each other 3 The former is slightly more important than the latter 5 Indicates that the former is significantly more important than the latter 7 The former is more important than the latter 9 The former is more important than the latter 2, 4, 6, 8 Represents the median value of the above adjacent judgments The bottom Contrary to the above effects The judgment criterion for this study was <0.1, indicating that the test passed and the judgment matrix constructed was reasonable. The weight of each factor was calculated using the judgment matrix. The different grading state values for each factor were normalized. Finally, the susceptibility index was calculated using Equation (7) [72].
where S i is the AHP susceptibility index, W i is the different grading weights of the evaluation factors, and L i is the normalized standard value of different grades of evaluation factors.

Logistic Regression (LR) Model
The LR model is a regression analysis model of binomial categorical variables. It describes the relationship between binary dependent variables and a series of independent variables [73][74][75][76]. When conducting susceptibility assessment, P is the probability of landslide occurrence, and Q = (1 − P) is the probability of non-occurrence of a landslide. If the logarithm of P/Q is ln(P/Q), then the probability of landslide occurrence is regarded as the dependent variable, and the influencing factor set [x 1 , x 2 , . . . , x n ] is considered an independent variable. The regression Equation (8) is established as follows [77]: where α is the regression constant, β i (i = 1, 2, . . . , n) is the regression coefficient, and x i (i = 1, 2, . . . , n) is the index value of the impact factor. Based on this, the LR Equation (9) can be obtained:

CF-AHP Integrated Model
As the deterministic coefficient model fails to consider the difference in the impact of each evaluation factor on landslide susceptibility, Fan (2017) [46] proposed a new method combining the CF and the AHP. In this method, first the CF method is used to calculate the relative weight of each evaluation factor index grading, namely, the CF value. Then, the hierarchy division of each evaluation index is carried out by combining the AHP to allocate weight scientifically and verify the reliability of weight allocation. Finally, the weight value is introduced to carry out the weighted sum of all the CF values of the impact factors, and the geological hazard susceptibility index (I j ) of each unit is calculated using the formula [46]: where I j is the susceptibility index of the jth evaluation unit and ω i is the weight of the ith evaluation factor. CF i is the CF value of each grade of the ith impact factor.

CF-LR Integrated Model
To consider the weight of each category of indicators and the weight of indicators at the same time, Cao (2020) [47] coupled the CF with LR and proposed the CF-LR integrated model. First, the CF value of each grade of evaluation factors calculated by the CF model was used as the index value of the LR analysis. Binomial LR analysis was performed to obtain the regression coefficient of each impact factor. On this basis, a LR equation was established to evaluate the landslide susceptibility [47].

Certainty Factor (CF) Model
The element layers were converted into raster data using GIS software, and the size of each layer cell was set to 30 m × 30 m. The layer of each evaluation factor ( Figure 3) and the distribution layer of the landslides were superimposed for analysis, and a total of 2,217,519 independent attribute units were obtained.
The area of each classification and the number of landslides contained in each classification were counted, and the CF value of each classification was calculated using the CF model (Table 4). On this basis, the weighted sum is performed to obtain the geological disaster susceptibility index of each unit, I j .

AHP
We built a judgment matrix (Table 4). After calculation, CI = 0.144 and CR = 0.0986 < 0.1, indicating that the construction of the judgment matrix is reasonable. The judgment matrix maximum characteristic root was determined to be Max = 10.152. Finally, the weights of each factor were determined (Table 5).

LR Model
A total of 260 independent units (130 landslide-present pixels and 130 randomly selected landslide-absent pixels.) were randomly selected as statistical samples for susceptibility assessment. Among them, 1 indicates occurrence, and 0 indicates non-occurrence. The grading index values of nine impact factors were considered independent variables, and whether landslides occurred was regarded as the dependent variable. Binary LR analysis was conducted on the samples, and the regression results are shown in Table 6. The results of the LR analysis show that the significance of all nine impact factors was <0.05, indicating that the nine factors were effective. The regression coefficients of the nine factors were substituted into the model to obtain the LR formula for susceptibility evaluation: where P is the probability of landslide and x 1−9 respectively represent the index values of each grade of nine impact factors, including altitude, slope, topographic relief, aspect, engineering geological rock group, slope structure, fault, river, and NDVI. Note: B represents the regression coefficient of each factor in the model, Se is the standard deviation, Wals is the chi-square, df is the degree of freedom, and Sig represents significance.

CF-AHP Integrated Model
AHP was used to calculate the importance of the selected index factors, and the CF method was used to calculate the CF values of different factors. Based on this, the CF weighted sum of the index factors was used to obtain the calculation formula of the sensitivity index of the evaluated region.

CF-LR Integrated Model
Based on the CF model, CF values of each grade of the nine impact factors were considered independent variables, and whether landslides occurred was regarded as the dependent variable. Binary LR analysis was conducted (Table 7) to obtain the LR formula: 1+exp(z) z = 0.169 + 0.795x 1 + 0.610x 2 + 0.693x 3 +0.207x 4 + 1.727x 5 + 0.753x 6 + 1.584x 7 +1.784x 8 + 1.336x 9 (13) where P is the probability of landslide and x (1-9) represents the CF values of each grade of the aforementioned nine impact factors. Note: B represents the regression coefficient of each factor in the model, Se is the standard deviation, Wals is the chi-square, df is the degree of freedom, and Sig represents significance.

Landslide Susceptibility Mapping
The grid calculator tool of GIS software was used to calculate the susceptibility index of each unit superposition. Using the Jenks natural breaks method, the susceptibility of Kaiyang County was divided into four classes: very low, low, moderate, and high ( Figures 5 and 6). Town and Jinzhong Town that were densely populated, as well as in areas with faultconcentrated belts. Areas of extremely low susceptibility to landslides are mainly distributed in Longgang Town in the south, and Nanmudu Town in the north, where the terrain is relatively flat and the geological conditions are not conducive for landslides. Figure 5 shows that the area ratios of each classification of the CF and LR model are similar. In the CF-LR model, the area of high susceptibility was 20.68%, which was higher than the 16.60% in the CF model or the 16.94% in the LR model. The area of the very low susceptibility area was 42.68%, which was higher than that of the CF model (24.23%) or the LR model (27.27%).
In the CF-AHP model, the area of high susceptibility was 19.72%, which was higher than that of the CF model (16.60%) or the LR model (118.95%). The area ratio of the extremely low susceptibility area was 18.51%, which was higher than that of the CF model (24.23%) and similar to that of the AHP model (18.51%).

Distribution of Landslides
The accuracy of the evaluation results can be tested by counting the percentage of the number of landslides in each susceptibility classification in the total number of landslides. The statistical results (Figure 7) show that 89.11% and 86.90% of landslides in the CF-AHP and CF-LR models fall in the medium to high-susceptibility areas, respectively; these values are higher than those recorded by the CF-AHP (82.89%), LR (80.65%), and AHP (76.19%) models, respectively. The results show that the evaluation results using the CF-AHP and CF-LR models are more reasonable than the CF, AHP, and LR models.

Receiver Operating Characteristic Curves
The receiver operating characteristic curve (ROC) is a common method for accuracy verification of landslide susceptibility evaluation, and it is widely used in the accuracy verification of landslide susceptibility evaluation. To show the evaluation effect more clearly, the area under the receiver operating curve (AUC) is typically used as an objective and quantitative evaluation index to measure the accuracy of the model's prediction [32]. The AUC value ranges between 0 and 1; the closer it is to 1, the higher is the prediction accuracy. The ROC test results for the five models in this study are shown in Figure 8. The test results show that the AUC values of the CF, AHP, LR, CF-AHP, and CF-LR models were 0.853, 0.712, 0.871, 0.873, and 0.895, respectively. With the exception of the AHP model, the AUC value of the other models is greater than 0.75, indicating that these models accurately evaluate the landslide susceptibility. The accuracy of the five models ranged from the highest to the lowest as follows: CF-LR, CF-AHP, LR, CF, and AHP. The success rates of the CF-AHP and CF-LR models were higher than those of the models using CF-AHP, AHP, and LR, respectively. The resultant landslide susceptibility maps ( Figure 5) suggest that the areas of high susceptibility to landslides are mainly distributed alongside rivers, in areas such as Huali Town and Jinzhong Town that were densely populated, as well as in areas with fault-concentrated belts. Areas of extremely low susceptibility to landslides are mainly distributed in Longgang Town in the south, and Nanmudu Town in the north, where the terrain is relatively flat and the geological conditions are not conducive for landslides. Figure 5 shows that the area ratios of each classification of the CF and LR model are similar. In the CF-LR model, the area of high susceptibility was 20.68%, which was higher than the 16.60% in the CF model or the 16.94% in the LR model. The area of the very low susceptibility area was 42.68%, which was higher than that of the CF model (24.23%) or the LR model (27.27%). In the CF-AHP model, the area of high susceptibility was 19.72%, which was higher than that of the CF model (16.60%) or the LR model (118.95%). The area ratio of the extremely low susceptibility area was 18.51%, which was higher than that of the CF model (24.23%) and similar to that of the AHP model (18.51%).

Distribution of Landslides
The accuracy of the evaluation results can be tested by counting the percentage of the number of landslides in each susceptibility classification in the total number of landslides. The statistical results (Figure 7) show that 89.11% and 86.90% of landslides in the CF-AHP and CF-LR models fall in the medium to high-susceptibility areas, respectively; these values are higher than those recorded by the CF-AHP (82.89%), LR (80.65%), and AHP (76.19%) models, respectively. The results show that the evaluation results using the CF-AHP and CF-LR models are more reasonable than the CF, AHP, and LR models.

Distribution of Landslides
The accuracy of the evaluation results can be tested by counting the percentage of the number of landslides in each susceptibility classification in the total number of landslides. The statistical results (Figure 7) show that 89.11% and 86.90% of landslides in the CF-AHP and CF-LR models fall in the medium to high-susceptibility areas, respectively; these values are higher than those recorded by the CF-AHP (82.89%), LR (80.65%), and AHP (76.19%) models, respectively. The results show that the evaluation results using the CF-AHP and CF-LR models are more reasonable than the CF, AHP, and LR models.

Receiver Operating Characteristic Curves
The receiver operating characteristic curve (ROC) is a common method for accuracy verification of landslide susceptibility evaluation, and it is widely used in the accuracy verification of landslide susceptibility evaluation. To show the evaluation effect more clearly, the area under the receiver operating curve (AUC) is typically used as an objective and quantitative evaluation index to measure the accuracy of the model's prediction [32]. The AUC value ranges between 0 and 1; the closer it is to 1, the higher is the prediction accuracy. The ROC test results for the five models in this study are shown in Figure 8. The

Receiver Operating Characteristic Curves
The receiver operating characteristic curve (ROC) is a common method for accuracy verification of landslide susceptibility evaluation, and it is widely used in the accuracy verification of landslide susceptibility evaluation. To show the evaluation effect more clearly, the area under the receiver operating curve (AUC) is typically used as an objective and quantitative evaluation index to measure the accuracy of the model's prediction [32]. The AUC value ranges between 0 and 1; the closer it is to 1, the higher is the prediction accuracy. The ROC test results for the five models in this study are shown in Figure 8. The test results show that the AUC values of the CF, AHP, LR, CF-AHP, and CF-LR models were 0.853, 0.712, 0.871, 0.873, and 0.895, respectively. With the exception of the AHP model, the AUC value of the other models is greater than 0.75, indicating that these models accurately evaluate the landslide susceptibility. The accuracy of the five models ranged from the highest to the lowest as follows: CF-LR, CF-AHP, LR, CF, and AHP. The success rates of the CF-AHP and CF-LR models were higher than those of the models using CF-AHP, AHP, and LR, respectively.

Uncertainty Analysis
As for landslides, there are some uncertainties in landslide susceptibility evaluation because of the complexity of the forming mechanism, the uncertainty of obtaining relevant data such as landslide cataloging, and the difference of prediction principles of different analysis models. For the AHP method, the rating system is mainly based on expert opinions, which is a subjective method and may have great defects. When making grading decisions on slopes, experienced experts generally assign a relatively high vulnerability index to a high slope and a relatively low vulnerability index to a low slope. In fact, the Pn values of landslide points in the six classes are 0.0229 (0-10°) and 0.0665 (10-20°), 0.1493 (20-30°), 0.2398 (30-40°), 0.2189 (40-50°), and 0 (>50°), respectively. The number of landslides in >50° region is not higher than those in 30-40° and 40-50° regions. Therefore, it is difficult to estimate the actual weight of factors when only starting from professional knowledge and general experience, and one cannot comprehensively analyze the actual situation. For the CF and LR models, the accuracy is mainly affected by the unbalanced sample sets. Landslide and non-landslide points are strongly unbalanced data sets. When using the sampling method to select non-landslide points, it is difficult to directly determine the specific location of non-landslide areas; therefore, there are differences in the selection methods of non-landslide points. In this study, through random sampling in the whole area, it is difficult to obtain good non-landslide points, which affects the accuracy of the model to some extent. These problems need to be solved in future research.

Conclusions
Although past studies have individually applied the AHP, CF, LR, CF-AHP, and CF-LR methods to examine landslide distribution and susceptibility characteristics, these methods have not previously been compared for accuracy. This study applied all five of these modeling approaches to the same area (Kaiyang County, China) to map landslide

Uncertainty Analysis
As for landslides, there are some uncertainties in landslide susceptibility evaluation because of the complexity of the forming mechanism, the uncertainty of obtaining relevant data such as landslide cataloging, and the difference of prediction principles of different analysis models. For the AHP method, the rating system is mainly based on expert opinions, which is a subjective method and may have great defects. When making grading decisions on slopes, experienced experts generally assign a relatively high vulnerability index to a high slope and a relatively low vulnerability index to a low slope. In fact, the Pn values of landslide points in the six classes are 0.0229 (0-10 • ) and 0.0665 (10-20 • ), 0.1493 (20-30 • ), 0.2398 (30-40 • ), 0.2189 (40-50 • ), and 0 (>50 • ), respectively. The number of landslides in >50 • region is not higher than those in 30-40 • and 40-50 • regions. Therefore, it is difficult to estimate the actual weight of factors when only starting from professional knowledge and general experience, and one cannot comprehensively analyze the actual situation. For the CF and LR models, the accuracy is mainly affected by the unbalanced sample sets. Landslide and non-landslide points are strongly unbalanced data sets. When using the sampling method to select non-landslide points, it is difficult to directly determine the specific location of non-landslide areas; therefore, there are differences in the selection methods of non-landslide points. In this study, through random sampling in the whole area, it is difficult to obtain good non-landslide points, which affects the accuracy of the model to some extent. These problems need to be solved in future research.

Conclusions
Although past studies have individually applied the AHP, CF, LR, CF-AHP, and CF-LR methods to examine landslide distribution and susceptibility characteristics, these methods have not previously been compared for accuracy. This study applied all five of these modeling approaches to the same area (Kaiyang County, China) to map landslide susceptibility and then compared the results obtained from the analyses. Five models were used to produce landslide maps, and the suitability of each model was evaluated using the area under the curve method and by comparing the maps with known landslide locations. The accuracies of the landslide maps produced by the CF, AHP, LR, CF-AHP, and CF-LR methods were 0.853, 0.712, 0.871, 0.873, and 0.895, respectively. The success rates of the CF-AHP and CF-LR models were higher than those of the CF, AHP, and LR models. By combining CF with the LR method, or CF with the AHP method, the shortcomings of the CF model were resolved, and the quadratic LR calculation of the landslide index factor was realized. The ROC precisions of the CF-AHP and CF-LR models were 0.838 and 0.836, respectively. Compared with the single AHP and CF models, the accuracy of landslide susceptibility assessment can be improved by approximately 8%, indicating that the CF-AHP and CF-LR models have a higher success rate than the single AHP and CF models in the assessment of landslide hazard susceptibility; these findings provide a reliable way to improve the assessment process of landslide hazard susceptibility in the Kaiyang region and elsewhere.
The comparison results of the present study indicate that except for the AHP model, the AUC value of all the other models was greater than 0.75, and the frequency ratio of extremely vulnerable areas and highly vulnerable areas accounted for more than 80% of the total frequency ratio. This indicates that the landslide susceptibility of Kaiyang County could be accurately evaluated using these methods.
By comparing and analyzing the value of CF and regression coefficient B of each characteristic value of the nine factors, it can be observed that the three influencing factors, namely formation lithology, distance from fault, and distance from the water system, contribute significantly to landslide susceptibility. In particular, landslides are more likely to occur within 1000 m of a fault, 800 m of a water system, and in lithologies of soft rock formation.
Landslide susceptibility maps provide decision support to planners and engineers who must choose suitable locations to implement development action plans. The landslide susceptibility maps produced in this study can aid planners and engineers in future development and land-use planning for Kaiyang County.
Author Contributions: All authors conceptualized the main research purpose of the paper. Y.Q. and G.Y. performed the analysis, prepared the data, tables, and figures, and wrote the majority of the first draft of the paper. K.L., Q.S., J.X. and Y.W. contributed with paper review and editing. All authors have read and agreed to the published version of the manuscript.