An Ensemble Approach of Feature Selection and Machine Learning Models for Regional Landslide Susceptibility Mapping in the Arid Mountainous Terrain of Southern Peru

Abstract

This study evaluates the utility of the ensemble framework of feature selection and machine learning (ML) models for regional landslide susceptibility mapping (LSM) in the arid climatic condition of southern Peru. A historical landslide inventory and 24 different landslide influencing factors (LIFs) were prepared using remotely sensed and auxiliary datasets. The LIFs were evaluated using multi-collinearity statistics and their relative importance was measured to select the most discriminative LIFs using the ensemble feature selection method, which was developed using Chi-square, gain ratio, and relief-F methods. We evaluated the performance of ten different ML algorithms (linear discriminant analysis, mixture discriminant analysis, bagged cart, boosted logistic regression, k-nearest neighbors, artificial neural network, support vector machine, random forest, rotation forest, and C5.0) using different accuracy statistics (sensitivity, specificity, area under curve (AUC), and overall accuracy (OA)). We used suitable combinations of individual ML models to develop different ensemble ML models and evaluated their performance in LSM. We assessed the impact of LIFs on ML performance. Among all individual ML models, the k-nearest neighbors (sensitivity = 0.72, specificity = 0.82, AUC = 0.86, OA = 78%) and artificial neural network (sensitivity = 0.71, specificity = 0.85, AUC = 0.87, OA = 79%) algorithms showed the best performance using the top five LIFs, while random forest, rotation forest, and C5.0 (sensitivity = 0.76–0.81, specificity = 0.87, AUC = 0.90–0.93, OA = 82–84%) outperformed other models when developed using all twenty-four LIFs. Among ensemble models, the ensemble of k-nearest neighbors and rotation forest, k-nearest neighbors and artificial neural network, and artificial neural network and rotation forest outperformed other models (sensitivity = 0.72–0.73, specificity = 0.83–0.84, AUC = 0.86, OA = 79%) using the top five LIFs. The landslide susceptibility maps derived using these models indicate that ~2–3% and ~10–12% of the total study area fall within the “very high” and “high” susceptibility. The obtained susceptibility maps can be efficiently used to prioritize landslide mitigation activities.

1. Introduction

Landslides are downslope movements of soil or rock materials along sliding planes that occur under the influence of gravitational forces [1]. This form of slope instability initiates when the force of the material’s weight exceeds the internal shear resistance of the slide materials [2]. Landslides are among the most deadly and common natural geohazards in mountainous regions across the globe [3]. According to the International Disaster Database, landslides comprised more than 4.9% of all natural disaster events and caused 1.3% of natural disaster fatalities between 1990 and 2015 [4].

The topographical, hydrological, and geo-environmental settings of southern Peru result in several types of landslides (e.g., rockfall, rock and soil slides, debris flows, shallow, and deep-seated landslides) that cause risk to human lives, infrastructure damage, economic instability, and landscape degradation. In general, landslides are triggered due to several factors, such as heavy precipitation, earthquakes, and volcanic and anthropogenic activities [5]. The frequency and impact of landslides are expected to increase in the future due to urbanization, highway construction, and deforestation [4,5,6]. Although landslides cannot be prevented, their impacts can be mitigated by developing spatial susceptibility models, which can be used in risk zonation and mitigation management [5,7,8].

Landslide susceptibility mapping/modeling (LSM) deals with the prediction of the probability of the occurrence of landslides in an area based on past landslides at different geo-locations with similar topographical, hydrological, and geo-environmental factors [6,9,10]. LSM approaches can be broadly categorized as being either qualitative or quantitative. Qualitative susceptibility maps are prepared based on geomorphological and field mapping by domain experts, whereas quantitative susceptibility maps rely on statistical models that explicitly identify the relationship between past landslides and geo-environmental factors. The advantages of quantitative approaches are that they are not subjective and produce repeatable results with higher accuracy [11]. Recently, GIS-based probabilistic models have also been developed for the kinematic susceptibility of landslides [12,13].

In the past few decades, LSM has been widely used as one of the most effective tools in landslide hazard management worldwide [7,14,15,16]. However, the accurate prediction of landslides is challenging due to their complex nature [10,16,17,18]. Successful LSM campaigns typically consist of three phases: preparation of a landslide and non-landslide inventory (i.e., training dataset), identification of relevant landslide influencing factors (LIFs) (e.g., topographical, hydrological, and geo-environmental), and implementation of appropriate prediction methods [16,19,20,21,22,23,24].

In recent years, several machine learning (ML) algorithms have been successfully used in LSM, such as linear discriminant analysis (LDA), mixture discriminant analysis (MDA), k-nearest neighbors (KNN), support vector machine (SVM), artificial neural networks (ANN), boosted logistic regression (BLR), bagged cart (BC), random forest (RF), rotation forest (RTF), and C5.0 [20,25,26,27,28,29,30,31,32,33]. ML algorithms offer several advantages over conventional statistical methods, such as the ability to learn the complex relationship between dependent and independent variables, proficiency in big data handling, geostatistical analysis, and the ability to update the developed model in the future [34,35,36,37,38]. Numerous studies evaluated the performance of ML methods to identify suitable methods for their study areas [39]. It should be noted that the performance of ML methods shows considerable variability between study sites due to differences in the complexity of each area, training datasets, availability of data summarizing LIFs, and ML implementation approaches. Therefore, the performance evaluation of different ML methods is recommended for different sites for accurate LSM [31].

The success of ML models highly depends on appropriate training data, optimal variable selection, and hyper-parameter optimization [40]. The selection of less correlated and important variables is typically obtained using feature selection (FS) methods. FS methods are broadly categorized as filter-based, wrapper, and embedded methods [41]. Filter-based FS methods use statistical measures (e.g., correlation, entropy, mutual information, etc.) to obtain the importance of given variables [41] and have been successfully used in LSM. Linear correlation, rank correlation, information gain (IG), gain ratio (GR), and relief-F (R_F) are common filter-based FS methods. The advantages of filter-based FS over the wrapper and embedded methods are that they are computationally efficient, reliable, and non-biased towards specific models [41]. Some studies have also highlighted the utility of ensemble FS (EFS), where the selection of variables results from multiple FS methods using majority voting [42,43].

Recently, a few studies have explored the utility of ensemble ML models for LSM and reported improvement in accuracy and generalization over individual ML models [42,44,45,46]. The application of ensemble learning is well-exploited in several fields, including data mining [47,48] and biological sciences [49], but comparatively less explored in geohazard applications, particularly in LSM [42,44]. To the best of our knowledge, the application of EFS and ensemble ML models together for landslide susceptibility prediction is rarely discussed in the literature. A few recent studies, such as Kadavi et al. [50], Arabameri et al. [51], and Fang et al. [42], used ensemble ML models for LSM. However, these studies do not investigate the impact of the number of LIFs on the performance of ML models, which is vital in obtaining important LIFs for regional- or global-scale mapping. Additionally, these prior studies evaluated the performance of ML models in relatively small geographical areas (i.e., ~100–400 km²) of tropical–subtropical climatic conditions and will not necessarily produce a similar performance in an arid climatic region at a regional scale.

To the best of our knowledge, the performance evaluation of a wide range of ML models and their ensemble for regional LSM in an arid climatic condition has not been presented in the literature. Therefore, this study attempts to highlight the utility of the ensemble approach of feature selection and ML models for regional LSM in the arid mountainous terrain of southern Peru using remotely sensed data and GIS. The objectives of this paper are three-fold: (a) We evaluate the performance of diverse sets of ML models (LDA, MDA, BC, BLR, KNN, ANN, SVM, RF, RTF, and C5.0) for LSM. (b) We evaluate the performance of different ensemble ML models developed in this study. (c) We investigate the impact of the number of LIFs derived using EFS on ML performance and their utility in developing robust ML models for regional LSM. From a practical perspective, the identified suitable LIFs coupled with robust ML models developed in this study should be useful in developing mitigation strategies to reduce the landslide impact in the area.

2. Materials and Methods

2.1. Study Area

The study area covers the Colca-Camana watershed in the southern part of Peru with a spatial extent of 16,955 km², which covers approximately 27% of the total area of the Arequipa region. The longitude and latitude of the study area are 72°45′29.45″W to 70°54′6.5″W and 14°55′48″S to 16°39′28.89″S, respectively. Figure 1 displays the geographical location of the study area. Arequipa is the second most important commercial hub of Peru and therefore plays a crucial role in the socio-economic development of the country. The study area was chosen for the LSM due to high landslide vulnerability, socio-economic importance, and the availability of suitable remote sensing and auxiliary datasets. The region is tectonically active, contains several major faults and volcanoes, and experiences frequent earthquakes. Faults are mainly oriented in the NW–SE direction.

The area shows a diversified topography, where the elevation ranges from 0 to 6389 m. The major landcover/land use of the area includes scrubland, barren land, snow cover, grassland, waterbody, cropland, and built-up land. A major portion of the area is covered by scrubland and barren land. The major soil types found in the area include cambisols, leptosols, and regosols. The area receives an average rainfall of 96 mm/year, where January to March are the wettest months and the rest of the months are predominantly dry. The surface temperature usually ranges from 5 to 25 °C. Camana, Aplao, Chuquibamba, Cabanaconde, and Chivay are major communities within the study area and are vulnerable to geologic hazards, including landslides. Figure 2 shows field photographs captured during field visits. These photographs are intended to provide a sense of the overall topography, vegetation cover, and climatic condition of the study area. Figure 2a shows a barren mountainous terrain susceptible to landslides that produce regular debris flows that have been cut by the river. The communities have built their houses on the old debris fans, which are highly vulnerable. Figure 2b presents a highly active rockfall area that is situated in the region of active geological faults. Figure 2c shows vegetation patterns in high-altitude areas of Chivay dominated by dryland grasses and bushes.

2.2. Datasets

Several datasets from different sources were used in this study, including remote sensing, auxiliary data, field observations, and published reports. A cloud-free Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) digital elevation model (DEM), and Landsat 8 (Level 2: surface reflectance) data of 30 m spatial resolution, were downloaded from the United States Geological Survey (USGS) Earth Explorer (https://earthexplorer.usgs.gov/, accessed on 30 March 2021) to derive the topographical, hydrological, and environmental LIFs. Multi-temporal high-resolution (i.e., spatial resolution 0.4–1 m) Google Earth imageries were accessed using open-source Google Earth Pro software (https://earth.google.com/, accessed on 15 March 2021), primarily for manual development of the landslide inventory. Geology (scale: 1:50,000), structural (1:100,000), geomorphology (1:50,000), and hydrogeology (1:50,000) maps were obtained from Universidad Nacional de San Agustín (UNSA), Arequipa, as Geographic Information System (GIS) layers. An Environmental Systems Research Institute (ESRI) global land use/landcover (LULC) map from 2020 with a spatial resolution of 10 m and 10 classes derived using Sentinel-2 data and a deep learning model with an overall accuracy of ~86% was used in this study (https://www.arcgis.com/, accessed on 15 April 2021) [52]. A soil-type map of the study area at a spatial resolution of 250 m was obtained from Soil Grids (https://soilgrids.org/, accessed on 30 April 2021) [53]. Ten years of averaged annual rainfall at a 0.1 by 0.1 degrees (i.e., 10 by 10 km) grid size were retrieved using global precipitation measurement (GPM) data and a continuous raster of rainfall was generated using the inverse distance weighted (IDW) interpolation method in GIS. Historical earthquake data over the area were accessed from the USGS earthquake catalog (https://earthquake.usgs.gov/, accessed on 30 June 2021).

2.3. Methods

A workflow diagram illustrating the overall methodology adopted in this study is presented in Figure 3. The methods are described in terms of landslide inventory development, preparation of different LIFs, multi-collinearity and variable importance analysis, performance evaluation of individual and ensemble ML models, and LSM followed by ground-truthing.

2.3.1. Landslide Inventory and Training Data Preparation

The preparation of an accurate landslide inventory is essential for successful LSM [54]. It contains the geo-location and other characteristics of past landslides as a point or polygon feature class. In this study, we define landslides to include rockfall, debris and mud flows, and translational and rotational landslides composed of soil and weathered rock. We used the geo-location of the landslide initiation area as the landslide location as is commonly practiced in LSM [33,50,55]. The landslide inventory is crucial in analyzing the spatial relationship between past landslides and LIFs because it is generally assumed that future landslides are most likely to occur when similar conditions exist to those that caused landslides in the past [3]. Therefore, multi-temporal high-resolution Google Earth imageries, field data, and input from different local communities were used together to prepare a historical landslide inventory.

Along with a landslide inventory, information about areas where landslides have not previously occurred is also crucial in LSM. We generated well-distributed random non-landslide points in a GIS environment using a buffer distance of at least 250 m from identified landslide points as a constraint to avoid non-landslide points overlapping on mapped landslides.

Figure 4 presents the spatial distribution of landslide and non-landslide locations. A total of 1460 landslide and 2400 non-landslide locations were further randomly divided into partitions of 70% (2702) for training and 30% (1158) for testing data to be used in the development and evaluation of ML models. The use of a greater number of non-landslide samples in this study relative to the number of landslide samples was intended to reflect the diverse set of non-landslide conditions that might not be captured by a smaller number of non-landslide points considering the large spatial area covered in the study and the variability in the large number of LIFs considered.

2.3.2. Preparation of Landslide Influencing Factors (LIFs)

LIFs are vital in understanding the complex nature of landslides, and LIFs are considered to have a significant impact on the accuracy of LSM. However, there is no universal rule for selecting LIFs, as the relative importance of LIFs varies between study areas and depends on the scale of mapping, availability of suitable remote sensing, and auxiliary datasets. We derived twenty-four different LIFs to characterize topographical, hydrological, and geo-environmental characteristics to be used in the LSM (

Table 1. Remotely sensed and auxiliary datasets were used to prepare different LIFs.

S. No.	LIFs	Data and Scale/Resolution
	Topographical and hydrological LIFs
1	Elevation	ASTER DEM (30 × 30 m)
2	Aspect
3	Slope
4	Profile curvature
5	Topographical position index
6	Topographical roughness index
7	Topographical wetness index
8	Stream transportation index
9	Stream power index
10	Surface relief ratio
11	Stream density
12	Direct radiation
13	Direct duration radiation
	Geo-environmental LIFs
14	NDVI	Landsat 8 OLI/TIRS (2020) (30 × 30 m)
15	Geology	Reference maps (Scale: 1:50,000)
16	Hydrogeology
17	Geomorphology
18	Land use/landcover	ESRI LULC map of 2020 (10 × 10 m)
19	Rainfall	10 years of averaged GPM data (2010–2020) (10 × 10 km)
20	Soil type	A global soil type map (250 × 250 m)
21	Distance to faults	Reference map, scale: 1:50,000
22	Earthquake magnitude	USGS historical earthquake data (1973–2021)
23	Distance to roads	Road networks (2021)
24	Distance to epicenter	USGS historical earthquake data (1973–2021)

). The ASTER DEM was pre-processed to address data imperfections (i.e., sink and anomalous pixel value) using the fill and cubic convolution resampling method in ArcMap 10.8. The pre-processed ASTER data were then used to derive topographical and hydrological factors, including altitude, slope, aspect, curvature, surface area ratio, relief ratio, flow accumulation, topographical wetness index (TWI), topographical position index (TPI), topographical ruggedness index (TRI), stream power index (SPI), sediment transportation index (STI), stream/drainage density, direct radiation, and direct duration radiation. These are briefly discussed below.

• Altitude is one of the crucial factors in determining the stress distribution along a slope, and it influences environmental conditions such as surface temperature, solar radiation, and rainfall [56,57].
• The slope is a measure of the steepness of topography, where the driving force of material increases with the slope angle. The slope angle also controls the subsurface flow, which impacts the moisture content and is therefore directly related to the occurrence of landslides [58]. Aspect can impact slope stability as it influences the wind direction, solar radiation, evapotranspiration, surface moisture, and vegetation cover [34,59].
• Curvature quantifies the terrain’s complexity and morphology. Plan curvature influences runoff acceleration and the erosion rate, whereas profile curvature affects the runoff velocity direction [60].
• The surface area ratio is a measure of the landscape’s topographic roughness: it is the ratio of surface area to its planimetric area [61]. A value close to 1 indicates a smoother surface, whereas values greater than 1 correspond to a rough surface [62]. The relief ratio is the ratio of basin relief and basin length, which represents the overall steepness of a basin [63]. The relief ratio plays a significant role in several geomorphic processes, such as drainage development, erosion on the slope, surface and subsurface water flow, moisture content, and landform development [64].
• Flow accumulation is derived from flow direction. It is computed by a cumulative count of other pixels that flow through that pixel. Regions of higher accumulation values are most likely to experience landslides, as they tend to concentrate a high volume of rainfall water [16,65]. Stream density refers to the total stream length per unit area. It indicates the closeness of the spacing of streams, which controls the landscape dissection and runoff [63]. High stream density usually occurs in impermeable areas, high relief, and barren surfaces, while low stream density is mostly associated with highly permeable surfaces, low relief, and densely vegetated surfaces [66]. Low drainage density develops a coarser drainage texture and implies low runoff and high infiltration, whereas high drainage density leads to the formation of fine drainage texture, higher runoff, and low infiltration [67].
• SPI is a measure of the erosive capacity of streams that predominantly modify the terrain through gully erosion and sediment transportation [68], whereas STI describes the process of erosion and deposition [69]. Higher values of STI indicate a high potential for erosion and vice versa.
• TWI represents the flow accumulation and slope of the area and typically corresponds to the water saturation zone [70]. Lower and higher values of TWI are typically associated with steep and flat or valley regions, respectively [71]. TRI describes surface heterogeneity as concave upward and convex slopes [72], whereas TPI computes the difference between the elevation of each pixel and its neighbors within a specified radius [73]. TPI can also be used to define geomorphic landforms as ridges (positive TPI), valleys (negative TPI), and flat areas (~0).
• Incoming solar radiation has been rarely used in LSM but it plays a significant role in a variety of physical processes that occur on the Earth’s surface, and therefore could be relevant to slope stability [74,75], particularly when considering a large spatial extent. Direct radiation represents the direct incoming solar radiation and direct duration radiation represents the duration of direct incoming solar radiation for each location. These were computed using the area solar radiation tool of the spatial analyst with default settings in ArcMap 10.8.
• The normalized difference vegetation index (NDVI) indicates vegetation coverage, which plays a significant role in decreasing the surface runoff and increasing the shear resistance of soil and rock types [76]. The roots of vegetation improve the stability of slope regions [77]. The NDVI was derived using near-infrared and red spectral bands of Landsat 8 reflectance data.
• Geology, hydrogeology, and geomorphology are commonly considered in most LSM as different rock types and landforms vary in their physical and mechanical properties, such as overlying soil strength, the intensity of weathering, porosity, and permeability, and therefore have a significant impact on slope stability [78,79].
• Geo-environmental LIFs such as geology, hydrogeology, geomorphology, LULC, 10 years annual average rainfall, soil type, distance from roads, distance from faults, distance from streams, distance from epicenters, and earthquake magnitude density were prepared in a GIS environment. The LIFs were resampled to 30 m using the nearest neighbor resampling method in a GIS environment to match the pixel size of remotely sensed data. Table 1 presents different data sources used in deriving the LIFs. Figure 5 displays six important LIFs derived in this study. The remaining LIFs are presented in the Supplementary Data (Figure S1).

2.3.3. Multicollinearity and Feature Selection (FS)

Optimal selection of variables or features is one of the most important steps in achieving a reliable result from ML methods. A variable set consisting of highly correlated variables has the potential to reduce performance, increase complexity, and reduce the generalizability of ML models [80]. In this study, the optimal selection of LIFs is carried out in two phases. The first phase performs the multicollinearity test using the VIF and tolerance statistics (TOL), and the second phase employs the ensemble FS for identifying important LIFs.

VIFs, Tolerance, and Pearson Correlation

The VIFs and tolerance (TOL) detect the multicollinearity between two or more variables. The VIF and TOL have been widely used in geohazard studies for discarding highly correlated variables [11,42,81,82]. To calculate VIF, let $\mathrm{X}={ \mathrm{X}}_1,{\mathrm{X}}_2,{ \mathrm{X}}_3, \dots ,{ \mathrm{X}}_{\mathrm{N}}$ represent the given LIFs (i.e., independent variable set) and ${\mathrm{R}}_{\mathrm{j}}^2$ represents the multicollinear coefficient between ${\mathrm{X}}_{\mathrm{j}}$ and other LIFs. The VIF is the reciprocal of TOL and can be computed using Equation (1):

$$VIF=\frac{1}{\left(1-R_j^2\right)}$$

(1)

In general, VIF > 10 and TOL < 0.1 indicate higher multicollinearity among variables, and these variables should be discarded in predictive modeling [83]. The Pearson correlation coefficient measures the linear correlation between continuous independent variables, where highly correlated variables have a similar impact on the dependent variable [21]. It can be computed using Equation (2):

$$r_{xy}={\displaystyle {\sum }_{i=1}^n}\frac{X_i-\overline{X}}{{{\displaystyle \sum }}_{k=1}^n\left(X_i-\overline{X}\right)}\times \frac{Y_i-\overline{Y}}{{{\displaystyle \sum }}_{k=1}^n\left(Y_i-\overline{Y}\right)}$$

(2)

$X_i$ and $Y_i$ denote the corresponding value of $X$ and $Y$ for the i-th independent variable. $\overline{X}$ and $\overline{Y}$ represent the mean of $X$ and $Y$. In general, a variable yielding an $r$ value of 0.7 or higher indicates a higher linear correlation and should be excluded in predictive modeling [81]. We also used the correlation plot to visually interpret the correlation between different continuous variables.

Feature Selection Methods

FS is a process of optimal selection of variables to improve the prediction or classification accuracy and reduce a model’s complexity. Chi-square [32,84], IG [11,32,85], GR, and relief-F (R_F) [32,42,84] have been successfully used as FS methods in LSMs. We used these FS methods to develop an EFS method for the optimal selection of LIFs. The FS methods are summarized below in

Table 2. Summary and mathematical framework of implemented FS methods.

Summary	Mathematical Framework
a. Chi-square: It measures the degree of independence between the dependent (i.e., landslide inventory) and independent variables (i.e., LIFs) [86]. Higher values of Chi-square indicate a greater probability of independence among variables.	$Chi-square={\displaystyle \sum _{i=1}^l}{\displaystyle \sum _{j=1}^s}\frac{{(o_{ij}^{\left(k\right)}-e_{ij}^{\left(k\right)})}^2}{e_{ij}^{\left(k\right)}}$	(3)
	$o_{ij}^{\left(k\right)}$ denotes the observed number of samples with the dependent variable $i, i\in \left\{1, \dots , l\right\}$, and a value of $X_k$ of the $j$th category, $j\in \left\{1, \dots , s\right\}$. $e_{ij}^{\left(k\right)}=\frac{1}{n} {\displaystyle \sum }_{i=1}^l{o}_{ij}^{\left(k\right)}{\displaystyle \sum }_{j=1}^l{o}_{ij}^{\left(k\right)}$ is the expected number of samples under the hypothesis of independence.
b. Gain ratio (GR): GR is a modified version of information gain (IG). It is the ratio of IG and entropy of an independent variable [87]. It uses entropy to rank variables of similar information content to reduce the bias of IG [41]. IG is also known as mutual information, which uses entropy to estimate the amount of information contained in the variable to predict the dependent variable [88]. IG and GR can be computed using Equations (4) and (5), respectively.	$IG=I\left(Y;X_k\right)$	(4)
	$Y$ and $X_k$ represent the dependent and independent variables, respectively. The higher the value of IG, the greater the importance of the corresponding variable.
	$GR=\frac{I\left(Y;X_k\right)}{H\left(X_k\right)}$	(5)
	$H\left(X_k\right)$ is the entropy of an independent variable $X_k$. In general, a variable that yields IG or GR values ≤ 0 should be excluded.
c. Relief-F (RF): It computes the importance of independent variables to the dependent variable. It looks for two closest neighbors, one from the same class (i.e., the nearest hit) and one from a different class (i.e., the nearest miss), by computing the Manhattan distance [89]. A variable that discriminates the sample from its neighbors belonging to different classes achieves a higher weightage [90]. The average score of independent variables is used to rank them, where a higher-ranked variable has a better ability to predict the dependent variable. $R_F$ can be computed using Equation (6).	$\begin{array}{c}{R}_F=\frac{1}{N}{\displaystyle \sum _{t=1}^N}\{-\frac{1}{k}{\displaystyle \sum _{x_i\in NH\left(y\right)}}diff\left(x_{t,i},x_{j,i}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\displaystyle \sum }\frac{1}{k}\frac{N\left(y\right)}{1-N\left(y_i\right)}{\displaystyle \sum _{x_j\in NM\left(x_i,y\right)}}diff\left(x_{t,i},x_{j,i}\right)\}\end{array}$	(6)
	$N$ is the number of samples of the training dataset and $N\left(y\right)$ is the probability of a sample being from class $y$ of the dependent variable. $x_{t,i}$ denotes the value of $x_t$ on variable $x_i$ and the $diff\left(.\right)$ function calculates the difference between $x_{t,i}$ and $x_{j,i}$. $k$ is a user-defined parameter that is used to define the number of nearest neighbors in computing the nearest hit (i.e., $NH$) and the nearest miss (i.e., $NM$).
d. Ensemble feature selection (EFS): The EFS subsets the variables based on the outputs of multiple FS methods and usually yields better results than the individual FS methods [43]. Chi-square, GR, and RF were used to develop the EFS using the robust rank aggregation (RRA) technique and it was implemented in R using the ‘RobustRankAggreg’ package [91].	$\rho \left(r\right)=mi{n}_{k=1,\dots ,n}{\beta }_{k,n}\left(r\right)$	(7)
	$r$ is the normalized rank variables vector, where $r_1,\dots ,r_n$ is reordering of $r$ such that $r_1\le \dots \le r_n$. ${\beta }_{k,n}\left(r\right)$ is the binomial probability and $\rho \left(r\right)$ denotes the order rank of variables to their $\rho$ scores. We further scaled the score of $\rho \left(r\right)$ as $1-\rho \left(r\right)$ for simplicity and obtained EFS scores.

.

2.3.4. Frequency Ratio (FR)

The FR is one of the most extensively used bivariate statistical methods in LSM to illustrate the correlation between the spatial distribution of past landslides (i.e., landslide inventory) and LIFs [13,92]. The class of LIF that yields values of FR ≥ 1 exhibits a higher correlation between the landslides and the corresponding LIF, and vice versa [79]. The FR of LIF i for the j-th class can be calculated using Equation (8):

$$F{R}_{ij}=\frac{\raisebox{1ex}{$N_{ij}$}\!\left/ \!\raisebox{-1ex}{$N_T$}\right.}{\raisebox{1ex}{$A_{ij}$}\!\left/ \!\raisebox{-1ex}{$A_T$}\right.}$$

(8)

$N_{ij}$ denotes the number of landslides distributed within the j-th class of LIF i, $N_T$ denotes the total number of landslides within the study area, $A_{ij}$ is the spatial extent of the j-th class of LIF I, and $A_T$ represents the total spatial extent of the study area.

2.3.5. ML Methods

Ten different ML algorithms were evaluated, as were certain combinations of algorithms within an ensemble framework. ‘Caret’ [93] and ‘CaretEnsemble’ [94] packages of R were used in implementing ML methods. Brief descriptions of each of the implemented ML methods are presented below.

LDA

LDA is a simple and computationally efficient statistical technique that projects the original dimension of data to a lower dimension in a linear combination of input variables to maximize the separation between two or more classes [95]. LDA has been widely used for dimensionality reduction and image classification. The performance of LDA highly depends on the collinearity among variables [96]. In LSM, the LDA finds a decision boundary that separates landslide and non-landslide regions. The implemented LDA algorithm in this study has no tuning hyperparameters.

MDA

Unlike LDA, the MDA assumes that each class is a Gaussian mixture of subclasses. The MDA uses the expectation-maximization technique to estimate the maximum likelihood of different classes [34]. The MDA can produce non-linear classification and an easy interpretation due to its simple structure, similar to LDA [97]. The MDA is most likely to perform better than LDA and has been widely used in natural resource modeling [97]. The implemented MDA consists of a tuning hyperparameter called subclasses, i.e., the number of subclasses per class.

BC

The CART algorithm builds generalized two-stage decision trees (DTs) using Gini’s impurity index to perform regression or classification tasks [97]. BC is an ensemble ML algorithm that applies bootstrap aggregation, also called bagging, to reduce the model’s variance and improve the accuracy and stability of the ensembled model compared to the individual model. BC provides a basis for developing ensemble ML models and has been successfully used in LSM [97]. The implemented BC does not have any hyperparameter to tune.

BLR

BLR is a boosting classification algorithm that performs an additive logistic regression which applies a regression scheme as a meta-learner to solve binary and multi-class classification problems [50]. BLR is a modified version of the AdaBoost algorithm, which replaces the exponential loss of the AdaBoost algorithm with conditional Bernoulli-likelihood loss. The common advantage of BLR is that it does not require a normality distribution assumption and can deal with noisy training data [98]. The implemented BLR has a hyperparameter denoted as niter that describes the number of boosting iterations to be run.

KNN

KNN is a non-parametric and simple ML algorithm that stores the available training data and performs classification on new data based on similarity metrics such as Euclidean distance. The samples of similar properties will have a lower Euclidean distance between them, and vice versa [18]. The implemented KNN has a tuning hyperparameter denoted as k, i.e., the number of neighbors should be considered in class voting. An odd number of k is suggested for binary classification to avoid issues with tied votes [24]. The larger value of k can reduce the effects of noise available in the training data but would produce a less distinct boundary between classes, whereas smaller k values are prone to overfitting and can produce a complex boundary [99].

ANN

ANN is developed based on the principle of biological neural networks. The architecture of a classical ANN consists of three layers, such as input, hidden, and output. The hidden layer applies some transformation to find the pattern and structure of input data to obtain the desired output. The ANN consists of many interconnected nodes, called artificial neurons. ANN uses the backpropagation technique, which allows the hidden layer to adjust the weights of neurons to meet the expectation of output [100]. ANNs have been successfully used in landslide detection and susceptibility mapping [17,101]. The implemented ANN in this study was obtained by averaging several neural network models, called model-averaged neural networks. It contains three hyperparameters, such as size, decay, and bag. Size represents the number of nodes in the hidden layer and decay is the regulation parameter preventing the overfitting. The bag is a logical parameter allowing bagging for every repeat.

SVM

SVM is a supervised ML classification algorithm, that provides a variety of kernel functions (i.e., radial basis function, linear, polynomial, and sigmoid) to find a hyperplane for separating two or multiple classes [30]. The radial basis function has been widely used in various applications due to its higher accuracy [9]. SVM is widely used in LSM to find an optimal hyperplane to distinguish between landslides and non-landslides [9,102,103]. The implemented SVM algorithm consists of two hyperparameters, i.e., cost and sigma. Cost deals with the penalty for the misclassified data samples, whereas the sigma controls the complexity of the hyperplane.

RF

RF is an ensemble learning approach which consists of many decision tress (DTs), where the decision in assigning the final class label is made by the majority vote of all DTs [104]. The RF overcomes the limitation of DTs of overfitting to training data [105]. The success of RF is due to its method of developing different DTs, where each tree is grown on the new training data using random variables. It uses a bagging technique that generates a random training set with the replacement of the original training data for each variable or a variable combination [104]. The implemented RF method consists of one hyperparameter, i.e., mtry, which is the number of input variables randomly sampled at each split when creating DTs.

RTF

RTF uses the principal component analysis (PCA) technique to transform the training data to train the base classifiers. The implemented RTF consists of two hyperparameters, such as K (i.e., the number of variable subsets) and L (i.e., the number of base classifiers (i.e., trees)). In the ensemble process, the training data are split into K subsets (i.e., user-defined) and PCA is applied to each training data subset acquired through the bootstrap sampling to construct a rotation-sparse matrix [27]. Subsequently, several classifiers are developed to improve accuracy and diversity. The confidence for each class is measured using the average combination method across all the classifiers and the final class label is assigned with the maximum confidence value [29].

C5.0

C5.0 is an improved version of the C4.5 ML algorithm of the DTs family. The C5.0 provides improved computational and memory usage that generates a smaller number of DTs and incorporates boosting and weighing techniques to improve the accuracy of the model [106]. The implemented C5.0 consists of three tuning hyperparameters, including trials, model, and winnow. The trials are boosting iterations, the model provides two options such as rules or tree-based, and winnow is a logical parameter: true or false.

Ensemble ML

An ensemble ML model makes use of two or more individual ML models, combining their predictions to improve prediction accuracy and generalization. Ensemble approaches can be broadly classified as homogeneous and heterogeneous. Homogenous ensembling combines the same model multiple times to obtain an aggregated ML model, whereas heterogenous ensembling integrates different ML models, showing considerable non-linearity among them to make a final prediction [42]. Heterogenous ensemble techniques tend to result in a more accurate and generalized model as compared to homogeneous ensembling [42]. There are several techniques of ensemble learning, such as stacking, blending, and averaging. Averaging is a common and simple ensemble technique and has been widely used in hazard susceptibility prediction [42]. Averaging can be performed as a simple averaging (SA) of the predictions of all base models using Equation (9). N and $p_i$ are the number of base models and prediction probability of the i-th base model, respectively.

$$SA=\frac{1}{N}{\displaystyle {\sum }_{i=1}^N}{p}_i$$

(9)

2.3.6. Performance Measures

Overall accuracy (OA) (Equation (10)), sensitivity or recall (Equation (11)), specificity (Equation (12)), and AUC of the ROC curve (Equation (13)) are among the most used performance measures in LSM and have been used in this study. OA indicates the total proportion of accurately classified landslide and non-landslide pixels. The sensitivity and specificity indicate the proportion of landslide and non-landslide pixels classified correctly, respectively [107]. The AUC values of the ROC curve can be used to quantitively assess the model accuracy. The values of AUC range from 0.5 to 1 and can be categorized as follows: AUC ≥ 0.7 is acceptable, AUC ≥ 0.8 is excellent, and AUC ≥ 0.9 is outstanding [108,109,110].

$$OA=\frac{TP+TN}{TP+FP+TN+FN}$$

(10)

$$Sensitivity=\frac{TP}{TP+FN}$$

(11)

$$Specificity=\frac{TN}{TN+FP}$$

(12)

$$AUC=\frac{{\displaystyle \sum }TP+{\displaystyle \sum }TN}{TP+FP+TN+FN}$$

(13)

where TP (i.e., true positive) and TN (i.e., true negative) indicate correctly classified landslide and non-landslide pixels, respectively. FP (i.e., false positive) and FN (i.e., false negative) indicate landslide pixels incorrectly classified as non-landslides and non-landslide pixels wrongly classified as landslides, respectively.

3. Results

3.1. Optimal Selection of LIFs

The selection of optimal LIFs is crucial in developing robust ML models to accurately discriminate between landslide and non-landslide areas. The selection of optimal LIFs was performed using multi-collinearity statistics and EFS, as described in Section 2.3.3. The multi-collinearity test enables the detection of collinearity among different LIFs, whereas EFS helps to assess their relative importance to discriminate between landslide and non-landslide areas. Figure 6 displays the Pearson correlation among continuous LIFs. The major advantage of using EFS over an individual FS approach is that it reduces the biases and uncertainty in evaluating the LIFs’ importance and yields stable results. The 24 LIFs were within the acceptable VIF (i.e., 2.03–6.19) and tolerance range (i.e., 0.49–0.16) (

Table 3. VIF and tolerance statistics of LIFs.

Code	Symbol	LIFs	Tolerance	VIF
1	Elv	Elevation	0.194	5.144
2	Asp	Aspect	0.953	1.050
3	Slp	Slope	0.203	4.928
4	Prc	Profile curvature	0.184	5.437
5	Tpi	Topographical position index	0.161	6.193
6	Tri	Topographical roughness index	0.666	1.501
7	Twi	Topographical wetness index	0.492	2.031
8	Sti	Stream transportation index	0.982	1.018
9	Spi	Stream power index	0.965	1.037
10	Srr	Surface relief ratio	0.724	1.381
11	Rnf	Rainfall	0.883	1.133
12	Std	Stream density	0.626	1.599
13	Drr	Direct radiation	0.187	5.358
14	Ddr	Direct duration radiation	0.327	3.057
15	Ndv	Normalized difference vegetation index	0.954	1.048
16	Lit	Lithology	0.827	1.209
17	Hdg	Hydrogeology	0.930	1.075
18	Gmr	Geomorphology	0.676	1.480
19	Luc	Land use/landcover	0.902	1.109
20	Som	Soil type	0.508	1.970
21	Flb	Distance from faults	0.940	1.064
22	Eqd	Epicenter density	0.895	1.118
23	Rdb	Distance from roads	0.767	1.304
24	Ebf	Distance from epicenter	0.969	1.032

) and were used in EFS.

Table 4. Relative importance (RI) of different LIFs derived from FS and EFS methods. LIFs are sorted in descending order based on their RI. The abbreviation of LIFs is provided in Table 3.

Chi-Square		Gain Ratio		Relief-F		EFS
LIFs	RI	LIFs	RI	LIFs	RI	LIFs	RI
Slp	0.542	Slp	0.109	Asp	0.078	Slp	0.994
Drr	0.418	Drr	0.079	Gmr	0.034	Drr	0.986
Twi	0.411	Twi	0.064	Slp	0.028	Twi	0.871
Prc	0.378	Prc	0.052	Drr	0.026	Prc	0.842
Tpi	0.346	Tpi	0.044	Som	0.025	Ddr	0.783
Elv	0.331	Ddr	0.043	Ddr	0.024	Tpi	0.664
Srr	0.322	Srr	0.043	Eqd	0.015	Asp	0.640
Rnf	0.320	Tri	0.036	Rdb	0.013	Gmr	0.625
Gmr	0.319	Elv	0.034	Prc	0.010	Tri	0.523
Ddr	0.295	Rnf	0.034	Std	0.004	Srr	0.383
Tri	0.261	Spi	0.033	Ebf	0.002	Std	0.268
Spi	0.259	Gmr	0.026	Hdg	0.001	Spi	0.111
Asp	0.219	Asp	0.023	Srr	0.000	Elv	0.051
Ndv	0.166	Std	0.020	Tri	0.000	Sti	0.000
Std	0.165	Ndv	0.019	Sti	0.000	Rnf	0.000
Luc	0.120	Som	0.012	Spi	0.000	Ndv	0.000
Som	0.116	Luc	0.012	Ndv	0.000	Lit	0.000
Rdb	0.089	Hdg	0.010	Rnf	0.000	Hdg	0.000
Flb	0.089	Rdb	0.007	Luc	−0.001	Luc	0.000
Hdg	0.083	Flb	0.006	Tpi	−0.003	Som	0.000
Sti	0.000	Sti	0.000	Lit	−0.014	Flb	0.000
Lit	0.000	Lit	0.000	Twi	−0.015	Eqd	0.000
Eqd	0.000	Eqd	0.000	Flb	−0.016	Rdb	0.000
Ebf	0.000	Ebf	0.000	Elv	−0.020	Ebf	0.000

and Figure 7 present the relative importance of different LIFs derived using Chi-square, GR, R_F, and EFS. It should be noted that we do not intend to compare the individual FS methods but are rather interested in using them to develop EFS for an optimal selection of LIFs. The score magnitudes of different FS methods are not comparable. The higher score of LIFs derived from different individual FS methods indicates a higher potential contribution towards discriminating between landslides and non-landslides. Similarly, a higher EFS score for a given LIF indicates that the variable was chosen by most of the individual FS methods and was therefore more discriminative in classifying landslide and non-landslide areas. There is a considerable discrepancy among individual FS methods in ranking the importance of LIFs. For example, slope and direct radiation are marked as the top two most important LIFs by the Chi-square and GR methods, whereas R_F marked aspect and geomorphology as the top two most important LIFs. In general, the results of Chi-square and GR are similar to each other and differ from those of R_F. Based on the score/rank derived from EFS, slope, direct radiation, TWI, profile curvature, and direct duration radiation were among the most important LIFs, whereas soil type, distance to faults, earthquake magnitude density, distance to roads, and distance to epicenters were among relatively the least important LIFs.

3.2. Spatial Relationship between Landslides and LIFs

Figure 8 shows the FR plots of six important LIFs. The remaining FR plots and FR statistics are presented in Figure S2 and Table S1. The landslide events show a linear relationship with slope steepness, and as expected, higher slope angles had higher landslide susceptibility. Furthermore, the southern and western slopes aspects were highly susceptible to landslides. Along these lines, moderate slope curvature values had lower susceptibility than low or high (negative or positive) curvature. The regions of low direct radiation and low direct duration radiation showed higher landslide susceptibility. The regions of low TWI Indicated higher landslide susceptibility. Similarly, the areas that receive low rainfall showed higher landslide susceptibility than areas with higher rainfall. This may seem counterintuitive, but the areas with higher rainfall also have more vegetation, which reduces the runoff and improves the soil strength. This relation was confirmed with NDVI correlations, where areas with low NDVI values had a higher susceptibility to landslides. This relation also manifests in terms of land use/landcover classes, where the barren land and snow cover were more susceptible than other categories. The regions of negative and positive TPI values indicated higher landslide susceptibility as these regions represent valleys and ridges. The regions with high TRI values were highly susceptible to landslides. The regions with a high surface relief ratio indicated higher susceptibility to landslides. The regions of lower elevation (<~2075 m) showed slightly higher landslide susceptibility than regions of higher elevation.

Among geomorphic classes, alluvial high terraces showed higher landslide susceptibility. Among geological classes, the highest correlation with landslide occurrence was seen for colluvial deposits, Ambo group, Huaylillas, and Camana formation. For hydrogeology classes, Holocene-continental quaternary had a uniquely high correlation with landslides, followed by Paleogene tonalite and granodiorite, continental lower carboniferous, and upper cretaceous granodiorite. Among the soil types, fluvisols, regosols, acrisols, and arenosols had higher landslide susceptibility, and chernozems, gypsisols, kastanozems, phaeozems, and vertisols had no landslide occurrence. Areas closer to faults had higher susceptibility, as found with areas closer to earthquake epicenters (except areas within 1 km). However, there was no clear correlation found between earthquake magnitude and historical landslides. The distance to roads did not exhibit a clear influence on landslide occurrence, making it among the least important LIFs in categorizing landslide susceptibility in the area.

3.3. Performance Evaluation of ML Models

The selection of optimal LIFs is crucial in successful LSM using ML models. For each ML model considered, we evaluated performance when the model utilized different sets of LIFs. We computed different accuracy statistics (e.g., sensitivity, specificity, AUC, and OA) of ML models with the top 5 LIFs (per EFS ranking—see Table 4 and Figure 7), subsequently added the next 5 top variables, and so on (i.e., 5, 10, 15, 20, and all 24 LIFs). Before evaluating the performance of ML models, their hyperparameters were optimally selected using the grid search method. We were exclusively interested in ML models that produced reasonable accuracy statistics with a relatively lower number of LIFs. This is because minimizing the number of LIFs mitigates overfitting and model complexity issues. Specifically, the removal of less significant variables improves the model generalization and computational cost.

Table 5. Accuracy statistics of ML models using different sets of LIFs.

	LDA				MDA
Number of LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.65	0.87	0.86	79	0.68	0.84	0.86	78
10	0.67	0.89	0.86	80	0.67	0.87	0.87	79
15	0.64	0.90	0.88	80	0.66	0.88	0.88	80
20	0.64	0.90	0.88	80	0.65	0.89	0.88	80
24	0.65	0.90	0.88	81	0.65	0.89	0.88	79
Mean statistics	0.65	0.89	0.87	80	0.66	0.87	0.87	79
	BC				BLR
Number of LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.68	0.82	0.84	77	0.47	0.90	0.81	74
10	0.67	0.84	0.86	78	0.66	0.82	0.81	79
15	0.70	0.85	0.88	79	0.76	0.81	0.85	79
20	0.71	0.86	0.89	80	0.70	0.83	0.85	78
24	0.73	0.87	0.89	82	0.68	0.82	0.82	76
Mean statistics	0.70	0.85	0.87	79	0.64	0.83	0.83	77
	KNN				ANN
Number of LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.72	0.82	0.86	78	0.71	0.85	0.87	79
10	0.71	0.85	0.87	80	0.69	0.86	0.88	80
15	0.68	0.86	0.88	79	0.70	0.85	0.88	79
20	0.67	0.86	0.88	79	0.72	0.86	0.87	81
24	0.69	0.87	0.88	80	0.72	0.86	0.88	81
Mean statistics	0.70	0.85	0.87	79	0.71	0.86	0.88	80
	SVM				RF
Number of LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.68	0.85	0.86	79	0.69	0.82	0.85	77
10	0.68	0.87	0.87	80	0.72	0.83	0.87	79
15	0.69	0.87	0.88	80	0.75	0.86	0.90	82
20	0.71	0.89	0.90	82	0.74	0.86	0.90	81
24	0.72	0.89	0.90	82	0.76	0.87	0.91	82
Mean statistics	0.69	0.87	0.88	81	0.73	0.85	0.89	80
	RTF				C5.0
Number of LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.68	0.84	0.86	78	0.64	0.84	0.85	76
10	0.71	0.86	0.88	80	0.76	0.82	0.88	80
15	0.73	0.87	0.90	81	0.81	0.86	0.91	84
20	0.76	0.85	0.90	81	0.81	0.86	0.92	84
24	0.76	0.87	0.90	83	0.81	0.87	0.93	84
Mean statistics	0.73	0.86	0.89	81	0.76	0.85	0.90	82

and Figure 9 present the accuracy statistics of the ML models. The fact that we used a relatively greater number of non-landslide samples (non-1:1 ratio of landslide and non-landslide samples) has the potential to influence the overall performance of models. Therefore, we have assigned more weight to sensitivity statistics in evaluating the performance of different models over other metrics as it illustrates the rate of success in accurately predicting landslide regions [107].

Different ML models showed different responses to the number of LIFs used in model development. Among all developed ML models, the ANN and KNN achieved the maximum sensitivity value (i.e., 0.71–0.72) with the top 5 LIFs. The sensitivity values of KNN and MDA decreased by 0.03 when the number of LIFs increased. The LDA produced similar sensitivity values for all numbers of LIFs, except 10 LIFs (where sensitivity was slightly maximized). The BLR showed the worst performance (relative to other ML models) when developed using the top 5 LIFs. The BC, SVM, RF, and RTF showed a slight improvement (3–8%) in their sensitivity values when developed using all LIFs. The C5.0 yielded the lowest sensitivity value (i.e., 0.64) (except BLR) using the top 5 LIFs but outperformed other models when developed using ≥15 LIFs. The top seven performing models (C5.0, KNN, ANN, RTF, RF, SVM, and BC) were considered in developing the ensemble ML models.

3.4. Performance Evaluation of Ensemble ML Models

Pearson’s linear correlation matrix of the top seven best-performing ML models is presented in

Table 6. Pearson’s correlation coefficient of the selected best-performing ML models.

	KNN	ANN	SVM	RF	RTF	C5.0
BC	0.68	0.75	0.77	0.86	0.92	0.83
KNN	1	0.74	0.76	0.76	0.72	0.78
ANN		1	0.87	0.82	0.77	0.88
SVM			1	0.83	0.84	0.89
RF				1	0.94	0.94
RTF					1	0.91

. A higher correlation between the prediction of the two models indicates a higher similarity between the models. An ideal ensemble ML model would be the one developed using the least correlated models. In this study, most of the ML models were highly correlated with each other, which induced a challenge in selecting candidate models for the ensemble model. Therefore, we combined pairs of individual ML models showing a correlation < 0.8. Given a preference, the KNN + BC (0.68) or KNN + ANN (0.74) would be a better choice for the ensemble model over the RF + C5.0 (0.94) or RTF + C5.0 (0.91).

An ensemble of less correlated models may offer complementary information in the final prediction and a better generalization. We restricted the ensembles to two individual ML models, as adding more models would further increase the correlation between them and might offset the advantages of the ensemble approach. Based on this criterion, we developed nine different ensemble models: KNN + BC, KNN + ANN, KNN + SVM, KNN + RF, KNN + RTF, KNN + C5.0, ANN + BC, ANN + RTF, and SVM + BC. Among all ensemble models, the correlation between the individual models of KNN + BC, KNN + ANN, and KNN + RTF was less compared to other models (Table 6).

Table 7. Accuracy statistics of ensemble ML models using different sets of LIFs. MS: mean statistics.

	KNN + BC				KNN + ANN				KNN + SVM
LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.70	0.83	0.86	78	0.72	0.84	0.86	79	0.70	0.84	0.86	79
10	0.70	0.85	0.87	80	0.69	0.86	0.87	80	0.69	0.86	0.87	80
15	0.73	0.87	0.90	82	0.72	0.87	0.88	81	0.70	0.87	0.89	80
20	0.72	0.87	0.90	82	0.72	0.86	0.88	81	0.71	0.87	0.89	81
24	0.75	0.87	0.90	82	0.73	0.86	0.89	81	0.71	0.89	0.90	82
MS	0.72	0.86	0.89	81	0.72	0.86	0.88	80	0.70	0.87	0.88	80
	KNN + RF				KNN + RTF				KNN + C5.0
LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.68	0.83	0.86	78	0.73	0.83	0.86	79	0.70	0.83	0.86	78
10	0.72	0.84	0.87	80	0.71	0.87	0.88	81	0.68	0.85	0.88	79
15	0.74	0.86	0.90	82	0.72	0.87	0.89	81	0.74	0.87	0.91	82
20	0.75	0.87	0.90	82	0.75	0.87	0.91	82	0.76	0.88	0.92	84
24	0.77	0.86	0.91	83	0.75	0.87	0.91	82	0.78	0.89	0.93	85
MS	0.73	0.85	0.89	81	0.73	0.86	0.89	81	0.73	0.86	0.90	82
	SVM + BC				ANN + BC				ANN + RTF
LIFs	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA	Sen	Spec	AUC	OA
5	0.67	0.85	0.86	78	0.70	0.84	0.87	78	0.72	0.83	0.86	79
10	0.68	0.87	0.88	80	0.67	0.88	0.87	80	0.70	0.86	0.88	80
15	0.72	0.88	0.90	82	0.70	0.88	0.89	81	0.72	0.87	0.90	81
20	0.72	0.89	0.91	82	0.71	0.88	0.90	81	0.73	0.86	0.90	81
24	0.73	0.89	0.91	83	0.74	0.87	0.90	82	0.73	0.86	0.90	81
MS	0.70	0.88	0.89	81	0.70	0.87	0.89	81	0.72	0.86	0.89	80

and Figure 10 display the accuracy statistics of different ensemble ML models. As observed with the individual ML models, the ensemble ML models also showed different degrees of sensitivity to the number of LIFs used in model development. We intended to achieve the best-performing ensemble models with the least possible number of suitable LIFs. Based on this criterion, the KNN + ANN, ANN + RTF, and KNN + RTF produced the highest sensitivity values (i.e., 0.72–0.73) using the top 5 LIFs. These models only showed a marginal improvement in their sensitivity values (i.e., 1–2%) when developed using all 24 LIFs. It can be noticed that the KNN + C5.0 yielded the highest sensitivity value (i.e., 0.78) when developed using all 24 LIFs. The KNN + SVM, ANN + BC, and SVM + BC slightly underperformed, with an average sensitivity value of 0.70, as compared to other ensemble models (i.e., 0.72–0.73). The ANN + BC, KNN + BC, SVM + BC, KNN + RF, and KNN + C5.0 improved their sensitivity values by 4–8% when developed using all 24 LIFs (Table 7).

3.5. Landslide Susceptibility Mapping

The best-performing ensemble ML models (KNN + RTF, KNN + ANN, and ANN + RTF) developed using the top 5 LIFs were used to map the landslide susceptibility of the study area (Figure 11, Figure 12 and Figure 13). The probability values of ML models were categorized into five different classes: very low (<0.2), low (0.2–0.4), moderate (0.4–0.6), high (0.6–0.8), and very high (>0.8), in a GIS environment [81]. The majority of high to very high susceptibility categories fell within the central part of the study area along the steep slopes of mountainous terrain. The majority of high to very-high susceptibility regions are categorized by low to moderate direct radiation and direct duration radiation. The regions of the north, northeast, and southeast parts of the area are mainly categorized as low to very low susceptibility. These regions are categorized as flat and low–moderate steep slopes and have comparatively higher direct radiation and higher direct duration radiation. The spatial statistics of landslide susceptibility derived from these models are presented in Figure 14. The susceptibility maps derived from the KNN + RTF and ANN + RTF models showed higher similarity between them than the susceptibility map derived from the KNN + ANN model. Relative to the KNN + RTF and ANN + RTF models, the KNN + ANN model showed a lower spatial extent of very low and high susceptibility categories, but more spatial extent coverage of low, moderate, and very high susceptibility categories (Figure 14).

The spatial statistics of the susceptibility maps Indicated that ~2–3% (i.e., ~308–465 km² of ~16,955 km²) and 10–12% (i.e., ~1768–2030 km²) of the total area fell within very high and high landslide susceptibility categories, respectively. The major communities Chivay, Cabanaconde, Chuquibamba, Aplao, and their vicinity fell within high to very high landslide susceptibility categories, which is consistent with observations made during field mapping around these communities. Figure 15 displays the robustness of the developed ensemble models in accurately predicting landslide susceptibility within the vicinity of major communities.

4. Discussion

Feature selection is an important step in machine learning that aims to remove redundant and less useful variables to reduce the potential for overfitting and improve generalization. We used an EFS method derived using the Chi-square, gain ratio, and relief-F methods to select the most important LIFs in LSM for our study area. The EFS reduces the uncertainty in selecting the best possible variables as different feature selection methods may rank the variables in different orders of importance, as seen in this study (Figure 7). Among twenty-four derived LIFs (elevation, aspect, slope, profile curvature, TPI, TRI, TWI, STI, SPI, SRR, rainfall, stream density, direct radiation, direct duration radiation, NDVI, lithology, hydrogeology, geomorphology, LULC, soil type, distance from faults, earthquake magnitude, distance from roads, and distance from epicenter), the slope, direct radiation, TWI, profile curvature, and direct duration radiation were the top five LIFs ranked by the EFS method in this study (Figure 7).

Direct radiation and direct duration radiation (solar radiation) are rarely used in LSM [99,100] but were found to be important for landslide susceptibility prediction in this study. The relevance of solar radiation is interpreted to be linked with the cold-arid climatic condition of the area, where the amount of solar radiation plays a significant role in evapotranspiration, growth of vegetation, and minimizes the frost action, which improves slope stability [99]. Direct radiation and direct duration radiation are negatively correlated with slope steepness, indicating a strong association between low solar radiation areas and high slope angles (Figure 6). The relevance of solar radiation can also be illustrated using frequency ratio plots, where the areas of relatively low direct radiation and low to moderate direct duration indicate a very high frequency of landslides in the area (Figure 8).

The slope had a negative correlation with TWI, indicating less moisture content (and reduced vegetation growth) along steep slopes and correspondingly more frequent landslides. The frequency of landslides was noted to be higher within the low- to moderate-elevation areas, as these regions mostly consist of relatively soft rock [111]. This was also confirmed by the geomorphology and lithology layers, indicating the highest landslide frequency within the areas of alluvial terraces and colluvial deposits. The frequency plots of these LIFs also confirmed that these regions have a higher frequency of landslides (Figure 8 and Figure S2). The distance to faults, earthquake magnitude, and distance to earthquake epicenter were among the least important variables, as earthquake-triggered landslides usually occur in the vicinity of active geologic faults and their impacts are likely to be limited within a certain distance [111]. Similarly, distance to roads was among the least important variables in this study, which could again be because the impact of engineering practices is impactful within a certain distance and may have a negligible impact on regional landslide susceptibility, as found in this study.

We evaluated the performance of ten different ML models in regional LSM using different sets of LIFs derived from the EFS method. Table 5 and Figure 9 show that the LDA, MDA, KNN, and ANN did not improve their sensitivity values when the number of LIFs increased. Relative to other models, the SVM, RF, and RTF have shown a slight improvement in their performance when developed using a greater number of LIFs. The RF and RTF improved their performance by ~7–8% when the number of LIFs increased from 5 to 24. The RF also achieved the best accuracy statistics in previous LSM studies [55,111]. The C5.0 showed a better improvement in performance when developed using a greater number of LIFs, as compared to other models. It is interesting to note that the C5.0 underperformed the other models when developed using the top 5 LIFs but outperformed other models when developed using ≥15 LIFs. Specifically, the sensitivity value of C5.0 increased from 0.64 to 0.81 when the number of LIFs increased from 5 to 15 or more. However, we suggest that the performance of ML models when using a small number of LIFs is more important as it reduces the risk of overfitting and may result in improved generalization performance.

A few ML models may Improve their performance marginally at the cost of higher model complexity. For example, the RF and RTF can provide ~3–4% better accuracy statistics (sensitivity, specificity, AUC, and OA) than the KNN and ANN, but only when approximately five times as many LIFs are considered (24 vs. 5). Interestingly, the C5.0 yielded ~9% better sensitivity than the KNN and ANN when the number of LIFs increased to 15 or higher. There is often a tradeoff between the model’s performance and complexity (i.e., models developed using a higher number of variables (LIFs)) [112]. The common drawbacks of complex models include issues related to overfitting, interpretability, generalization, and computation cost [112]. It is challenging to comment on the performance of ML models solely based on statistical inferences if they exhibit slight differences in their performance, as most of them have a similar ability to represent complex non-linear relationships [55]. Furthermore, the minor differences in accuracy statistics on LSM at a regional scale may not be practically significant in the spatial domain.

Different ensemble ML models using a suitable combination of best-performing individual ML models were developed in this study. The ensemble ML models should reduce the problem of overfitting and improve the generalization over the individual models [42,50]. The KNN + RTF, KNN + ANN, and ANN + RTF were among the best-performing ensemble models when developed using the top five LIFs. However, the experimental results did not exhibit notable improvement in the accuracy statistics of ensemble models over their individual models (Table 5 and Table 7 and Figure 9 and Figure 10), likely due to the high correlations among individual models (Table 6). In general, the performance of the ensemble models falls somewhere between the performance of the individual models used in ensemble development. For example, the sensitivity value of KNN + C5.0 (0.70) was lower than the sensitivity value of KNN (0.72) but higher than that of C5.0 (0.64) when developed using the top five LIFs. Conversely, the accuracy statistics of KNN + C5.0 were slightly lower and higher than C5.0 and KNN, respectively, when developed using a greater number of LIFs (≥15). This could be due to the performance of KNN decreasing as the number of LIFs increased, whereas C5.0 improved its performance as the number of LIFs increased. In other words, the KNN and C5.0 showed an opposite response to the number of LIFs.

Most of the ensemble ML models yielded very good accuracy statistics (sensitivity ≥ 0.70, specificity ≥ 0.80, AUC ≥ 0.86, and OA ≥ 78%) using the top five LIFs in this study due to the well-distributed training dataset and suitable LIFs. Moreover, the consistent performance of different models ensures the reliability of the derived results and their subsequent utilization in susceptibility categorization (Figure 11, Figure 12 and Figure 13). This can be further supported by Figure 15, which displays the robustness of developed ensemble models in accurately predicting landslide susceptibility within the vicinity of major communities. The spatial statistics of susceptibility categories indicate that 2–3% and 10–12% of the total study area fell within the “very high” and “high” susceptibility categories, respectively, (Figure 14), which are predominantly characterized by barren steep slopes, low solar radiation, low to moderate elevation, and sedimentary deposits.

Around 80% of the historical landslide points (i.e., 1168 out of 1460) fell within the moderate to very high landslide susceptibility derived from the ensemble models. The remaining ~20% of the historical landslide points fell within the regions of very low to low susceptibility. This could be due to the significant difference between the spatial resolution of the remotely sensed data used in preparing the landslide inventory (~≤1 m) and most of the LIFs (~30 m). There is a possibility that some of the landslides present in the inventory do not cover the sufficient spatial extent to be predicted by LIFs of coarser resolution using ML models. This could be further attributed to uncertainty in training data and LIFs, ML’s ability to learn the complex non-linear relationship between the historical landslides and LIFs, and spatial prediction at a regional scale.

There are some limitations of this study that are recommended to be considered in future work. In this study, we considered a relatively greater number of non-landslide samples than landslide samples, which may influence the performance of different ML models. Different proportions of landslide and non-landslide samples and their influence on ML performance in LSM at a regional scale could be explored in future studies.

Suitable LIFs are crucial in obtaining an accurate susceptibility map of landslides using any ML models. A few LIFs may provide an added value when their multi-temporal series are considered in context to pre- and post-landslides. In this study, we used a single-year NDVI and land use/landcover map in characterizing the landslide susceptibility of the area, which may produce a landslide detection model rather than a susceptibility model. It would be interesting to see the utility of multi-temporal NDVI and land use/landcover in assessing pre-landslide susceptibility and post-landslide occurrences. Additionally, we used an average of ten years of annual precipitation map in assessing their impact on landslide susceptibility. Future studies could consider using multi-temporal and extreme precipitation events in characterizing landslide susceptibility.

The correlation among ML models is crucial in finding the ideal combination of ML models to obtain robust ensemble ML models. Most of the ML models implemented in this study indicated higher correlation among them, which induced limitations in developing the ensemble models. Ideally, the candidate models for the ensemble should have less correlation. We recommend exploring a wide range of ML models in selecting candidate models for optimal ensemble and their assessment in LSM.

Regarding the validation of the models, we assessed the performance of ML models based on a training and testing data split. However, an intensive iteration-based cross-validation approach can be considered in future studies to assess the robustness of ML models. This could provide further information about the robustness and generalization potential of the models.

5. Conclusions

The identification of landslide-prone areas can be valuable for land use planners or disaster management agencies to aid in the process of appropriately allocating resources to forecast and mitigate landslide impacts. We derived a regional landslide susceptibility of the Colca-Camana watershed in the south of Peru using an ensemble approach of feature selection and machine learning (ML) models. The ensemble feature selection successfully identified the most important landslide influencing factors (LIFs) (e.g., slope, direct radiation, topographical wetness index, profile curvature, and direct duration radiation) to predict the landslide susceptibility in the area.

We evaluated the performance of ten individual ML models using different sets of LIFs ranked by ensemble feature selection. The k-nearest neighbors (KNN) (sensitivity = 0.72, specificity = 0.82, area under curve (AUC) = 0.86, overall accuracy (OA) = 78%) and artificial neural network (ANN) (sensitivity = 0.71, specificity = 0.85, AUC = 0.87, OA = 79%) outperformed other models when developed using the top five LIFs. The RF, RTF, and C5.0 outperformed other models when developed using all 24 LIFs (sensitivity = 0.76–0.81, specificity = 0.87, AUC = 0.90–0.93, OA = 82–84%). Among ensemble ML models, the ensembles of KNN and rotation forest (KNN + RTF), KNN + ANN, and ANN + RTF models outperformed other models using the top five LIFs (sensitivity = 0.72–0.73, specificity = 0.83–0.84, AUC = 0.86, OA = 79%). The ensemble models did not show significant improvement in their statistical performance but should reduce the uncertainty in the spatial prediction of susceptibility over the individual models. The accuracy statistics of different ML models using all LIFs showed small improvements, but arguably not enough to justify the additional complexity introduced by including more LIFs. This justifies the robustness of the proposed ensemble approach to obtain a reliable landslide susceptibility at a regional scale.

The susceptibility maps derived using ensemble models suggested that approximately 2–3% and 10–12% of the total study area fell within the “very high” and “high” landslide susceptibility categories, respectively. These regions are mainly categized by barren steep slopes of low to moderate elevation, southerly slope aspects, low solar radiation, low topographical wetness, and loose sedimentary deposits. The landslide susceptibility maps of the area derived in this study have the potential to be used by policymakers to develop an effective mitigation strategy to reduce the landslide risk for the sustainable development of the area.