Automated Hyperparameter Optimization of Gradient Boosting Decision Tree Approach for Gold Mineral Prospectivity Mapping in the Xiong’ershan Area

Abstract

The weak classifier ensemble algorithms based on the decision tree model, mainly include bagging (e.g., fandom forest-RF) and boosting (e.g., gradient boosting decision tree, eXtreme gradient boosting), the former reduces the variance for the overall generalization error reduction while the latter focuses on reducing the overall bias to that end. Because of its straightforward idea, it is prevalent in MPM (mineral prospectivity mapping). However, an inevitable problem in the application of such methods is the hyperparameters tuning which is a laborious and time-consuming task. The selection of hyperparameters suitable for a specific task is worth investigating. In this paper, a tree Parzen estimator-based GBDT (gradient boosting decision tree) model (TPE-GBDT) was introduced for hyperparameters tuning (e.g., loss criterion, n_estimators, learning_rate, max_features, subsample, max_depth, min_impurity_decrease). Then, the geological data of the gold deposit in the Xiong ‘ershan area was used to create training data for MPM and to compare the TPE-GBDT and random search-GBDT training results. Results showed that the TPE-GBDT model can obtain higher accuracy than random search-GBDT in a shorter time for the same parameter space, which proves that this algorithm is superior to random search in principle and more suitable for complex hyperparametric tuning. Subsequently, the validation measures, five-fold cross-validation, confusion matrix and success rate curves were employed to evaluate the overall performance of the hyperparameter optimization models. The results showed good scores for the predictive models. Finally, according to the maximum Youden index as the threshold to divide metallogenic potential areas and non-prospective areas, the high metallogenic prospect area (accounts for 10.22% of the total study area) derived by the TPE-GBDT model contained > 90% of the known deposits and provided a preferred range for future exploration work.

1. Introduction

Mineral prospectivity mapping (MPM) can guide us to conduct deep and peripheral ore prospecting in chosen study area. From the spatial dimension, it can be divided into two-dimensional MPM and three-dimensional MPM [1,2]. Two-dimensional MPM can facilitate the regional-scale prospecting and delineation of the prospective area [3,4,5,6,7,8], while three-dimensional MPM can be used to guide the delineation of the deep metallogenic target area at the deposit scale [9,10,11,12,13,14,15,16]. In terms of methods, MPM can be divided into knowledge-driven MPM and data-driven MPM. Knowledge-driven MPM is mainly based on geological expert experience for statistical analysis methods [17,18,19,20,21,22,23,24], and data-driven MPM is mainly based on big data methods (i.e., machine learning or deep learning algorithms) [25,26,27,28,29,30,31,32,33,34,35].

No matter which MPM is performed, at either the regional scale or at the deposit scale, MPM is carried out by way of data-driven or knowledge-driven approaches, geological exploration data are essential. The quality of exploration data collected matters a great deal for the accuracy of MPM. However, the geoscientific data available are generally limited, and there are a few target/positive samples (ore spots) in the real world. Moreover, machine learning and deep learning algorithms are data-hungry [36,37,38,39,40,41]. Therefore, there are some inevitable problems for the application of big data algorithms based on geological data (such as with the data structure, extreme imbalance of positive (prospect) and negative samples (non-prospect) etc.) [42].

In resolving these problems, the current research mainly focused on data augmentation and algorithm optimization, among which the data augmentation mainly included: multi-sample data augmentation (e.g., under-sampling, over-sampling, synthetic sampling) [43,44,45,46,47,48,49,50,51], single-sample data augmentation (e.g., image up-and-down, left-right inversion and center inversion, etc.) [52,53,54,55,56,57], and unsupervised data enhancement (e.g., GAN, AutoML) [58,59,60,61]. The improved algorithms mainly included design cost sensitive functions and used ensemble learning, such as cost-sensitive neural networks [62,63], balanced fuzzy support vector machine algorithms [64,65,66,67,68], a CBP-SVM algorithm based on a hybrid model [69], the RSBoost algorithm [70,71,72], anomaly detection algorithms (e.g., isolation forest) [73,74,75,76], differential Siamese convolutional neural network [77,78,79,80,81], and the MLA (machine learning algorithms) based on the decision tree model (e.g., GBDT, weighted random forest) [82,83,84]. Among them, the tree model was suitable for few-shot data training [85,86,87].

The MLA based on the tree model has high computational efficiency and strong model discrimination ability with a simple principle. It is one of the few “white box models” in MLA [88,89,90,91]. Moreover, in terms of the function of the tree model itself, the decision tree can be used for classification and regression, which also produce additional criteria such as importance of features and continuous variable box indexes. In ensemble learning, the tree model is the most commonly used primary classifier [92]. These advantages make the tree model one of the most important in MLA. The GBDT model is an essential boosting algorithm, based on a decision tree, that utilizes the strategy of multi-model integration to fit the residuals for reducing the deviation and variance of the model [93,94]. Nevertheless, it is also faced with an important problem: hyperparametric optimization (HPO). However, there are few studies on HPO with GBDT. Many researchers tend to set default parameters values for the model, leading to weak end results [95,96]. The optimization of hyperparameters, for certain case studies, has a great impact on the effectiveness of the algorithm, and it plays a crucial role in the accuracy and generalization ability of the final generated model. There are a large number of hyperparameters (e.g., n_estimation, learning_rate, max_features, subsample, etc.) in the GBDT, with different parameters having different effects on the GBDT model. Therefore, the selection of the hyperparameter combination is important to achieve the optimal model.

With regard to its algorithms, we aspire to eventually realize the perfect automation of all processes. The discipline specializing in automation machine learning is called AutoML, and hyperparameter automatic optimization is the most mature, in-depth, and well-known direction [97,98,99]. Theoretically, when the computational power and data are sufficient, the performance of HPO should exceed that of human beings. HPO can reduce human workload, and the results obtained by HPO are more likely to be reproduced than searched, so HPO can greatly improve the reproducibility and fairness of scientific research [100,101,102,103]. Contemporary HPO algorithms can be mainly divided into grid-based search (grid), Bayesian optimization (Bayesian), gradient-based optimization (gradient-based), and population-based optimization (evolutionary algorithm, genetic algorithm, etc.), among which the grid search and Bayes-based optimization are the most popular [104,105,106,107,108,109,110]. These HPO methods have great effect and significance on optimizing complex ensemble algorithms.

In this paper, a knowledge-driven synthetic sampling method was proposed for dealing with imbalanced data, which made the data form more suitable for MLA. The GBDT algorithm based on the decision tree model for few-shot geological data was selected. The random grid search and the Bayesian optimization based on the TPE algorithm were used to study the hyperparametric of the GBDT model, taking the gold deposit in Xiong ‘ershan area as an example, in order to verify the difference of MPM based on two algorithms for HPO with few shot geological data. The study can be used as a reference for the research of MLA based on HPO in the MPM.

2. Methodology

2.1. GBDT Algorithm

The gradient boosting decision tree (GBDT) is a representative boosting algorithm, which is the cornerstone of XGBoost, LightGBM and other tree model algorithms. It is also one of the most widely used MLAs in the industry and the most stable one in practice. When it was first proposed, GBDT was written as gradient boosting machine (GBM), which integrated bagging and boosting ideas and can accept all types of weak estimators as input. After the weak estimator was defined as a decision tree, it was slowly renamed the gradient boosting tree.

Inspired by the boosting algorithm, GBDT naturally contains three boosting elements: loss function $\mathrm{L}\left(\mathrm{x},\mathrm{y}\right),$ weak estimators $\mathrm{f}\left(\mathrm{x}\right)$ and comprehensive integration results $\mathrm{H}\left(\mathrm{x}\right)$ [111]. Yet, some improvements have since been made: (1) The weak estimator output type of GBDT is no longer consistent with the ensemble algorithm. For AdaBoost or random forest (RF) algorithms, weak estimators are regressions when ensemble algorithms perform regression tasks. When ensemble algorithms perform classification tasks, weak estimators are classifiers; no matter whether the GBDT is performing regression or classification tasks as a whole, the weak estimator must be a regression. It outputs specific classification results by sigmoid or softmax functions, but the actual weak estimator must be a regression; (2) The loss function has been extended to any differentiable function in the mathematical principle, and is no longer limited to a fixed or single loss function; (3) Before each weak estimator is established, the sample weight is not modified, but the residual error is fitted to affect the structure of the subsequent weak estimator; (4) It joins the idea of random sampling from RF, allowing samples and features to be sampled before each tree is built to increase the independence between weak estimators (thus allowing for out-of-bag datasets to be used to verify the establishment results of each weak evaluator). It can further increase the stability of boosting algorithm.

As is well known, MPM is considered a classification problem in the MLA, and is categorized as either a prospect or non-prospect in the study region. So, we should know the GBDT binary classification algorithm flow [112]:

First, initialize weak classifier as:

$${\mathrm{H}}_0=\log \frac{\mathrm{P}\left(\mathrm{Y}=1|\mathrm{x}\right)}{1-\mathrm{P}\left(\mathrm{Y}=1|\mathrm{x}\right)}$$

(1)

where the $\mathrm{P}\left(\mathrm{Y}=1|\mathrm{x}\right)$ is the proportion of positive (as y = 1) in train datasets, using prior information to initialize the learner.

Second, the loss function is defined, and the GBDT binary classification algorithm uses a logarithmic loss function:

$$\mathrm{L}\left(\mathsf{\theta }\right)=-{\mathrm{y}}_{\mathrm{i}}\log \widehat{{\mathrm{y}}_{\mathrm{i}}}-\left(1-{\mathrm{y}}_{\mathrm{i}}\right)\log \left(1-\widehat{{\mathrm{y}}_{\mathrm{i}}}\right)$$

(2)

where the $\widehat{{\mathrm{y}}_{\mathrm{i}}}$ is the result of logistic regression prediction $\mathrm{H}\left(\mathrm{x}\right).$

$$\mathrm{H}\left(\mathrm{x}\right)=\frac{1}{1+{\mathrm{e}}^{-\mathrm{f}\left(\mathrm{x}\right)}}$$

(3)

where the $\mathrm{f}\left(\mathrm{x}\right)$ are the weak estimators. Flowing Formula (3) into Formula (2) as:

$$\mathrm{L}\left({\mathrm{y}}_{\mathrm{i}},\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)\right)={\mathrm{y}}_{\mathrm{i}}\log \left(1+{\mathrm{e}}^{-\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)}\right)+\left(1-{\mathrm{y}}_{\mathrm{i}}\right)\left[\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)+\log \left(1+{\mathrm{e}}^{-\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)}\right)\right]$$

(4)

Then, calculation of negative gradient of loss function (pseudo residual):

$${\mathrm{r}}_{\mathrm{m},\mathrm{i}}=-{\left|\frac{\partial \mathrm{L}\left({\mathrm{y}}_{\mathrm{i}},\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)\right)}{\partial \mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}\right)}\right|}_{\mathrm{f}\left(\mathrm{x}\right)={\mathrm{f}}_{\mathrm{m}-1}\left(\mathrm{x}\right)}={\mathrm{y}}_{\mathrm{i}}-\frac{1}{1+{\mathrm{e}}^{-\mathrm{f}\left(\mathrm{x}\right)}}={\mathrm{y}}_{\mathrm{i}}-\widehat{{\mathrm{y}}_{\mathrm{i}}}$$

(5)

Subsequently, calculate the best residual fitting value of each leaf node as:

$${\mathrm{C}}_{\mathrm{m},\mathrm{j}}=\mathrm{argmin}{\displaystyle \sum }_{{\mathrm{x}}_{\mathrm{i}}\in {\mathrm{R}}_{\mathrm{m},\mathrm{j}}}\mathrm{L}\left({\mathrm{y}}_{\mathrm{i}},{\mathrm{f}}_{\mathrm{m}-1}\left({\mathrm{x}}_{\mathrm{i}}\right)+\mathrm{C}\right)$$

(6)

where the m is the mth trees. In order to solve the ${\mathrm{C}}_{\mathrm{m},\mathrm{j}}$ value, we introduce the second-order Taylor expansion as:

$${\mathrm{C}}_{\mathrm{m},\mathrm{j}}=\frac{{{\displaystyle \sum }}_{{\mathrm{x}}_{\mathrm{i}}\in {\mathrm{R}}_{\mathrm{m},\mathrm{j}}}{\mathrm{r}}_{\mathrm{m},\mathrm{i}}}{{{\displaystyle \sum }}_{{\mathrm{x}}_{\mathrm{i}}\in {\mathrm{R}}_{\mathrm{m},\mathrm{j}}}\left({\mathrm{y}}_{\mathrm{i}}-{\mathrm{r}}_{\mathrm{m},\mathrm{i}}\right)\left(1-{\mathrm{y}}_{\mathrm{i}}+{\mathrm{r}}_{\mathrm{m},\mathrm{i}}\right)}$$

(7)

Meanwhile, update strong learners as:

$${\mathrm{H}}_{\mathrm{m}}={\mathrm{H}}_{\mathrm{m}-1}\left(\mathrm{x}\right)+{\displaystyle \sum }_{\mathrm{j}=1}^{{\mathrm{J}}_{\mathrm{m}}}{\mathrm{C}}_{\mathrm{m},\mathrm{j}}\mathrm{I}\left(\mathrm{x}\in {\mathrm{R}}_{\mathrm{m},\mathrm{j}}\right)$$

(8)

Finally, the ${\mathrm{H}}_{\mathrm{M}}\left(\mathrm{x}\right)$ as:

$${\mathrm{H}}_{\mathrm{M}}\left(\mathrm{x}\right)={\mathrm{H}}_{\mathrm{m}}+{\displaystyle \sum }_{\mathrm{m}=1}^{\mathrm{M}}{\displaystyle \sum }_{\mathrm{j}=1}^{{\mathrm{J}}_{\mathrm{m}}}{\mathrm{C}}_{\mathrm{m},\mathrm{j}}\mathrm{I}\left(\mathrm{x}\in {\mathrm{R}}_{\mathrm{m},\mathrm{j}}\right)$$

(9)

In summary, the GBDT binary classification algorithm uses multiple CART regression trees to fit the log probability with the positive label (as y = 1), and the loss function uses the logarithmic loss, but it is necessary to replace the predicted value $\widehat{{\mathrm{y}}_{\mathrm{i}}}$ (the probability predicted as y = 1) with the logarithmic probability $\mathrm{H}\left(\mathrm{x}\right)$ predicted by the regression tree, and then fit the residuals each round. After the final classifier output is obtained, the probability that the prediction label is positive (as y = 1) can be obtained by the sigmoid function.

2.2. Hyperparameter Optimization Method

Model optimization is one of the most difficult challenges in MLA implementation. Parameter adjustment is the core of model optimization, but the process is complex and cumbersome. Especially, there are a large number of hyperparameters in the MLA and deep learning algorithms (DLA), which not only makes the method extremely flexible but also affects the algorithm performance through the combination hyperparameters. Therefore, the application and selection of an HPO method that can automatically obtain hyperparameters is crucial. In this paper, the stochastic network search (RandomizedSearchCV) and a tree-structured Parzen estimator approach (TPF) based on the Bayesian optimization algorithm was used for HPO with a GBDT model.

2.2.1. Random Grid Search Optimization

The random grid search method is the advanced version of the grid optimization algorithm. Based on the original grid search optimization, it abandons the global hyperparameter space that was used in the original search, and instead randomly selects some parameter combinations to construct the hyperparameters subspace, and searches only in the subspace [113]. This greatly reduces the search space, the number of parameter groups that need to be enumerated and compared, and shortens the overall search time. Additionally, the minimum loss obtained by random grid search is very close to the minimum loss obtained by enumeration grid search. It improves the computing speed without reducing the search accuracy. In random search, random sampling is not put back, so there will be no problem for extracting the same set of parameters twice. We can assign a fixed amount of computation to the random grid search, and when all computation is consumed, the random grid search is completed.

2.2.2. Tree Parzen Estimator in Bayesian Optimization

Bayesian optimization is a parameter adjustment method with a prior process. It is the SOTA in the current hyperparameter optimization and is the most advanced optimization framework. It can be used in various fields of AutoML, and it can also be used in advanced fields such as neural network architecture search for NAS and meta-learning [114]. The mathematical process of Bayesian optimization mainly includes the following processes:

(1)

Define the objective function $\mathrm{f}\left(\mathrm{x}\right)$ to be estimated and the definition domain of $\mathrm{x}$;

(2)

Take out the values on finite $\mathrm{nx}$, and solve the $\mathrm{f}\left(\mathrm{x}\right)$ of these $\mathrm{x}$ (solve the observed values);

(3)

According to the limited observations, the function is estimated (this assumption is called the prior knowledge in Bayesian Optimization), and the target value (maximum or minimum) on the estimated $\mathrm{f}(\widehat{\mathrm{x}})$ is obtained;

(4)

Define a rule to determine the next observation point to be calculated.

Continue to cycle in steps (2)–(4) until the target value on the assumption distribution reaches, or all computing resources are used up (e.g., up to m observations or up to t minutes allowed to run).

With the tree Parzen estimator (TPE), a different idea is used to model the probability distribution. According to Bayesian theorem:

$$\mathrm{P}(\mathrm{Y}|\mathrm{X})=\frac{\mathrm{p}(\mathrm{X}|\mathrm{Y})\mathrm{P}\left(\mathrm{Y}\right)}{\mathrm{P}\left(\mathrm{X}\right)}$$

(10)

TPE divides $\mathrm{P}(\mathrm{X}|\mathrm{Y})$ as:

$$\mathrm{P}(\mathrm{X}|\mathrm{Y})=\{\begin{array}{c}\mathrm{l}\left(\mathrm{x}\right),\mathrm{Y}<{\mathrm{Y}}^{*}\\ \mathrm{g}\left(\mathrm{x}\right),\mathrm{Y}>{\mathrm{Y}}^{*}\end{array}$$

(11)

In other words, the different distribution of TPE for observation points on both sides of the threshold ${\mathrm{Y}}^{*}$ can be regarded as the hyperparametric probability distribution of a good and a bad score. The threshold ${\mathrm{Y}}^{*}$ is determined by the hyperparametric $\mathsf{\gamma }$, that is the quantile of Y. Through the above division, we can obtain:

$$\mathrm{P}\left(\mathrm{X}\right)={{\displaystyle \int }}_{\mathrm{R}}^{}\mathrm{P}((\mathrm{X}|\mathrm{Y})\mathrm{P}\left(\mathrm{Y}\right)\mathrm{dy}=\mathsf{\gamma }\mathrm{l}\left(\mathrm{x}\right)+\left(1-\mathsf{\gamma }\right)\mathrm{g}\left(\mathrm{x}\right)$$

(12)

Then, bring the Formula (12) into the EI formula as:

$${\mathrm{EI}}_{{\mathrm{Y}}^{*}}={{\displaystyle \int }}_{-\infty }^{+\infty }\max ({\mathrm{Y}}^{*}-\mathrm{Y},0){\mathrm{P}}_{\mathrm{M}}(\mathrm{Y}|\mathrm{X})\mathrm{dy}=\frac{{{\displaystyle \int }}_{-\infty }^{{\mathrm{Y}}^{*}}\left({\mathrm{Y}}^{*}-\mathrm{Y}\right)\mathrm{P}\left(\mathrm{Y}\right)\mathrm{d}\left(\mathrm{y}\right)}{\mathsf{\gamma }+\left(1-\mathsf{\gamma }\right)\frac{\mathrm{g}\left(\mathrm{x}\right)}{\mathrm{l}\left(\mathrm{x}\right)}}.$$

(13)

Thus, ${\mathrm{EI}}_{{\mathrm{Y}}^{*}}\left(\mathrm{x}\right)\propto {(\mathsf{\gamma }+\left(1-\mathsf{\gamma }\right)\frac{\mathrm{g}\left(\mathrm{x}\right)}{\mathrm{l}\left(\mathrm{x}\right)})}^{-1}$, where the ${\mathrm{EI}}_{{\mathrm{Y}}^{*}}$ is proportional to the reciprocal of the denominator, and when $\mathsf{\gamma }$ is determined, the denominator value only depends on the ratio $\frac{\mathrm{g}\left(\mathrm{x}\right)}{\mathrm{l}\left(\mathrm{x}\right)}$ of the two segments of $\mathrm{x}$, and the physical meaning of this ratio is the probability of $\mathrm{x}$ being a good score to the probability of $\mathrm{x}$ being a bad score. Therefore, the result is used to find the max $\mathrm{x}$ to obtain the ratio maximum.

2.3. GBDT Modeling

GBDT, as the representative boosting algorithm, has a large number of parameters which can be roughly divided into five categories, such as parameters based on an the iterative process; parameters of a weak evaluator structure; parameters for an early stop; training data parameters of weak evaluator; and others (

Table 1. Categories of GBDT hyperparameters.

Type	Parameter
boosting	n_estimators, learning_rate, loss, alpha, init
Weak evaluator structure	criterion, max_depth, min_samples_split, min_samples_leaf, min_weight_fraction_leaf, max_leaf_nodes, min_impurity_decrease
early stop	validation_fraction, n_iter_no_change, tol, n_estimators_
Weak evaluator training data	subsample, max_features, random_state
others	ccp_alpha, warm_start

). Among them, the number of iterations and the parameters of the weak evaluator have a great influence on the GBDT model. For example, then estimators parameters determine the number of iterations, the learning rate parameters that affect the overall learning efficiency of the algorithm, and the max depth, min impurity decrease parameters are used to prune the tree model to reduce the complexity of the model. Faced with so many parameters, quickly selecting the appropriate parameters to optimize the model is a challenge. This paper selected eight parameters that greatly impact the GBDT model (

Table 2. GBDT hyperparameters selected and default values in Sklearn.

Parameter	Function of Parameters	Default Value
loss	loss function	“deviance”
criterion	Impurity measurement index of weak estimate when branching	“friedman_mse”
n_estimators	The Actual number of iterations	100
learning_rate	Weighted summation process affecting weak estimator results	0.1
max_features	Maximum number of features considered in constructing optimal CART tree model	None
subsample	The Proportion of random samples released from the full dataset before each CART tree is built	1.0
max_depth	Maximum allowable depth of weak estimator	3
min_impurity_decrease	The minimum reduction in impurity is allowed when the weak evaluator branches	0.0

), and carried out parameter optimization based on the Sklean package in Python.

The basic process of hyperparametric optimization is as follows: First, the objective function is defined. We used the Formulation (2) as the objective function. Second, the search space is determined. For the GBDT model, most parameters have a fixed range, so we chose to explore the unbounded parameters. Generally, a large space will be set initially, and during the iterative optimization classification model, the parameter space range and dimension are gradually reduced many times. Finally, the function is optimized, and the iterative optimization model is trained.

3. Study area and Geological Data

3.1. Geological Setting

The Xiong’ershan area is located in the southern margin of the North China Craton (NCC) and the eastern segment of the Qinling orogenic belt, which is an important part of Huaxiong block (Figure 1A,B). The strata in this area are mainly composed of the Neoarchean-Paleoproterozoic Taihua Group, the Paleoproterozoic-Mesoproterozoic Xiong’er Group, the Mesoproterozoic Guankou Group, the Luanchuan Group, and finally Cenozoic formations. The Taihua Group is composed of Late Archean-Early Proterozoic metamorphic rocks (e.g., amphibolite gneiss, amphibolite, leptynite, granulite), exposed along the central and western in the southern margin of the NCC. It is one of the metamorphic crystalline basements in NCC [115,116,117]. Xiong’er Group is the product of important magmatic events after the stability of the NCC. It comprises intermediate-acid volcanic strata distributed in the southern margin of the NCC. The main lithologies are andesites, rhyolites, dacites, etc. [118,119]. The Guandaokou Group is mainly composed of a series of shallow marine terrigenous clastic-carbonate rock (e.g., dolomite and quartz sandstone) formations located in the southern part of the Xiong’ershan area, overlain by the Xiong’er group in an unconformable contact relationship. The Luanchuan Group is distributed in the south of Luanchuan, and is mainly a set of shallow metamorphic clastic and carbonate rocks (e.g., sandstone, mudstone, limestone, etc.) (Figure 1C) [120,121,122].

The regional structure is mainly composed of three EW trending deep regional faults (e.g., Luonan-Luanchuan fault, Machaoying fault and Sanmenxia-Baofeng fault). A series of NE and NW trending faults are distributed among the three main faults. The Machaoying fault zone experienced at least five deformation cycles and seven generations of tectonic events, which is the main ore-guiding-hosting structure in this area [129].

The magmatic activity is relatively frequent. According to the age of magmatic activity, the characteristics of magmatic rocks and the corresponding geodynamic background are roughly divided into three stages: (1) The first cycle mainly occurred in the Late Archean-Early Proterozoic, during which a large number of intermediate-acid volcanic rocks and TGG granites were produced, and the Taihua Group metamorphic basement was formed by metamorphism in the later period. (2) The second cycle mainly occurred in the Middle-Late Proterozoic, forming the Xiong’er Group of volcanic strata, which is discordant to the Taihua Group. (3) The third magmatic cycle took place mainly in the Mesozoic, being the most important magmatic activity in the Xiong’ershan area. This magmatic activity is also an important tectonic-magmatic thermal event that formed a large number of deposits in the central and eastern China (Figure 2a), hosted in the Wuzhangshan, Huashan, Heyu and other granite batholiths [130,131].

3.2. Geological Exploration Datasets

The used datasets were: (1) the geological map of Xiong’ershan district at 1:50,000 scale provided by the Geological Survey in the Henan province; (2) geochemical data derived from 1156 stream sediment samples at the scale of 1:200,000 with the density of 2 Km ∗ 2 Km; and (3) gravity anomaly data at scale of 1:200,000.

The metallogenic model needed to be translated into predictor layers for better support to the MPM of the Xiong’ershan area. This section describes the methods used to generate the predictor layers based on the exploration criteria.

3.2.1. Source

The source of ore-forming material provides the basis for the deposit formation [132,133]. A large number of studies on fluid inclusions and stable isotopes (O, H, C) in the Xiong ‘ershan area, show that the main metallogenic mechanism of gold deposits is fluid boiling, and the source of ore-forming fluids is mainly deep source materials [134,135,136,137,138]. The large-scale tectonic-magmatic-metallogenetic thermal event in the Early Cretaceous is characterized by the widespread exposure of faults and granitic batholiths in the study area (Figure 2). Granite batholith provides a heat source for the formation of gold and molybdenum deposits; therefore, the contemporaneous granite base in the region can be used as an important prediction factor. Reasonable estimation of the influence distance between granite rock mass and gold deposits indicates the positioning function for metallogenic prediction. Here, we analyzed the buffer distance layer by the distribution histogram (Figure 3) as one of the prediction factors. It showed the relationship between the granite rock mass (GRB) and ore-occurrence (Figure 4p). The granite intrusions usually exhibit low and gentle gravity anomalies owing to their low density. Figure 4l shows that the low gravity is roughly consistent with granitic intrusions, and the outline of several huge granitic intrusions can be recognized. The residual gravity anomaly (RGA) can highlight concealed granite geological bodies. We adopted the RGA as another prediction factor for MPM in the Xiong’ershan area.

Figure 2. Metallogenic model map of Xiong’er shan gold district: (a) Late Mesozoic tectonic-magmatic evolution scenario in Xiong’er shan Area; and (b) metallogenic model for the Au polymetallic deposits in Xiong’ershan Area (Modified after [139,140]).

Figure 2. Metallogenic model map of Xiong’er shan gold district: (a) Late Mesozoic tectonic-magmatic evolution scenario in Xiong’er shan Area; and (b) metallogenic model for the Au polymetallic deposits in Xiong’ershan Area (Modified after [139,140]).

3.2.2. Transport and Deposition

Most gold deposits are hosted in faults. The ore-bearing faults in the study area are mainly NE and NWW faults, and a few are NW faults (i.e., Mao Chaoying fault). The Mao Chaoying fault zone is a deep fault that acted as pathways for the upward flow of deep-seated fluids (Figure 2b). The thick ore bodies (such as Qianhe and Hongzhuang gold deposits) are common in the intersection of NWW and NE faults in the Xiong’ershan area. Therefore, the fault buffer zone (FB), the European direction of the fault (FT), and the fault intersection density (FID) were selected as the prediction factor layers (Figure 4m–o).

Stream sediment geochemical data can reveal regional to district-scale patterns associated with Au mineralization in the study area. We adopted the geochemical anomalies of Au, Ag, Pb, Zn, As, Sb, Hg, W and Mo, associated with fault-controlled mineralization on a regional scale (Figure 4a–k). In addition, considering the compositional nature of geochemical data (i.e., to address the constant sum problem existing in compositional data), we applied centered log-ratio transformation (clr) prior principal component analysis (PCA) [141,142,143]. Figure 5 shows the positive loading for Zn, Sb, Pb, Mo, Hg, Au and Ag elements, and the negative loading for As in the first principal component (PC1), where the Ag and Pb had a positive high factor load. This indicated that the high-value area of PC1 is more closely related to Ag, and Pb metallogenetic elements. Where Sb, Au, As, and Ag had a positive loading for the second principal component (PC2), the As and Sb as the front elements had a positive high factor load, and the high-value area was consistent with the low-value of PC2. This indicated that the low-value area of PC2 had great metallogenic potential.

Therefore, the distribution histogram was used for the statistical analysis of the quantitative prediction factor as shown in Figure 3, along with the sixteen evidence maps derived by the prediction factor layers, which have been processed using a grid with a pixel size 130 m × 130 m (Figure 4).

3.2.3. Training and Validation Data

In this paper, 45 gold deposits developed in the study area were used as the training set, and each gold deposit includes 16 prediction variables (such as Au, Ag, Pb, Zn, Mo, W, Hg, Sb, As, PC1, PC2, RGA, FB, FT, FID, RB). The classification for MLA has a common basic assumption: the number of positive and negative datasets (samples) should be balanced [144,145]. If the positive and negative datasets are extremely imbalanced, the predicted results may be biased towards the majority classes with a large number of datasets [146,147]. Mineralization is a rare event, resulting in an insufficient number of training samples by MLA, and the number of deposits (positive samples) and non-deposits (negative samples) is not equal. In this paper, we used the synthetic minority over-sampling technique (SMOTE) based on geological knowledge constraints to balance data. The specific process is as follows: (1) based on the data of 45 known Au deposits (including 16 features) in the Xiong’ershan area and constrained by the optimal fault buffer radius of 3 km (Figure 3), yielded 945 SMOTE-augmented positive samples in the optimal ore-controlling adjacent area; (2) A total of 900 SMOTE-augmented negative samples were randomly selected from outside the 3 km threshold range of the non-anomalous area; (3) The knowledge-driven SMOTE datasets (generated by the (1) and (2)) are called MS (with the 945 deposits/positive and the 940 non-deposits/negative datasets) as the first datasets. The second datasets (denoted hereafter as OS datasets) were composited with 90 datasets (with the original known 45 deposits, 45 non-deposits are randomly selected non-mineralized locations)l (4) Then, we used the sixteen derived evidence maps, and the optimal threshold was used to show the Kernel density estimation curve (Figure 3) needed to accomplish the metallogenic predictionl (5) Finally, the dataset (with OS and MS) was divided into a test set and a train set (ratio of 2:8), and the test set was used to evaluate the accuracy and robustness of the model.

4. Results and Discussion

4.1. Parameters Optimization

Based on the Section 2.3 hyperparametric optimization process, the initially selected parameter space and the final reduced parameter space are shown in

Table 3. GBDT parameter space for training datasets.

Parameter	Initial Parameter Space	Final Parameter Space
loss	[“deviance”,”exponential”]	[“deviance”,”exponential”]
criterion	[“friedman_mse”, “squared_error”]	[“friedman_mse”, “squared_error”]
n_estimators	(25,200,25)	(55,200,1)
learning_rate	(0.1,2.1,0.1)	(0.05,1,0.005)
max_features	(4,20,2)	(1,16,1)
subsample	(0.1,0.8,0.1)	(0.5,1.0,0.05)
max_depth	(2,30,2)	(10,35,1)
min_impurity_decrease	(0,5,1)	(0,5,0.1)

.

In order to compare and verify the performance of the two hyperparameter optimization algorithms on the final classification model, we plotted the kernel density estimates curve to understand the distribution of hyperparameters in both algorithms. Figure 6a–h shows that the distribution density curves of two hyperparameter optimization algorithms are consistent in the same domain space. It indicates that the hyperparameter distribution of the two algorithms is basically consistent. However, TPE tends to concentrate near the high-density area (placing more probabilities), resulting in the minimum loss in cross-validation. Figure 6i shows that the parameters criterion is mainly “friedman_mse”in random search and TPE-GBDT model. For pruning parameters “max_depth”, ”min_impurity_decrease” TPE has relatively lower values. Figure 7 shows that the value of the loss function is related to the spatial relationship based on the iterative process and the weak estimators. The random search reached the minimum loss function value at 77 iterations, while TPE reached the minimum loss function value at 54 iterations. Figure 6g also verified that in the same domain control interval, the training time of TPE was lower than that of the random search. The aforementioned results indirectly prove that the core idea of TPE is to spend more time evaluating promising hyperparametric values (i.e., the minimum loss function value). The loss value decreases with the increase of “learning rate” and “max_feature” (Figure 7b,d). The subsample and the “max_depth” shows a low loss value between 0.8–0.95, 15–25, respectively (Figure 7a,c). There is no obvious monotonic relation between the “min_impurity_decrease” parameter and loss value in the Xiong’ershan dataset. Compared with random search and TPE, the latter has a less objective function evaluation and better generalization performance on test sets. The optimal hyperparameters determined by the minimum loss function are shown in

Table 4. The Optimized parameters for the GBDT model.

Parameter	Parameter Value Based on the Random Search	Parameter Value Based on the TPE
loss	“deviance”	“deviance”
criterion	“friedman_mse”	“friedman_mse”
n_estimators	186	69
learning_rate	0.09	0.7
max_features	9	12
subsample	0.95	0.8
max_depth	12	10
min_impurity_decrease	0.1	0.1

.

4.2. Performance Evaluation

Model evaluation plays a vital role in machine learning models. It helps to find the best model to represent our data. Different types of models (regression model and classification model) have different evaluation indexes. The evaluation indexes of the regression model include mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and Coefficient of determination (R2) etc. The evaluation methods commonly used for classification models include k-fold cross validation to verify the model fitting effect, accuracy, recall rate and AUC index to evaluate the model generalization ability [148,149,150,151]. In our study, we selected the five-fold cross-validation, confusion matrix and the AUC as the performance evaluation for the GBDT model.

The five-fold cross-validation refers to roughly dividing the datasets into five parts, one of which is reserved for the validation model; the other samples are used for training. Cross-validation is repeated five times, with each sub-sample verified iterately. The results of an average of five times or other combination methods are used to finally obtain a single estimation. In this way, the model is more accurate. The GBDT model was trained according to the hyperparameters in Table 4, and the average accuracy of the five-fold cross-validation results obtained based on random search and TPE on the MS test set was 0.963 and 0.966, and on the OS test set was 0.764 and 0.786, respectively. This showed that the augmented datasets of the MS and the parameter adjustment improved the accuracy of the GBDT model (detail shows in

Table 5. The result of five-fold cross-validation for GBDT model.

Model	Train Accuracy	Test Accuracy	Five-Fold Cross-Validation Time	Datasets
GBDT	0.981	0.940	0.55 s	MS
GBDT-Random	1.000	0.963	3.16 s
GBDT-TPE	1.000	0.966	0.71 s
GBDT	1.000	0.754	0.07 s	OS
GBDT-Random	1.000	0.764	0.33 s
GBDT-TPE	1.000	0.786	0.09 s

). The results of the random search and TPE hyperparameter optimization were compared and analyzed after the five-fold cross-validation. Figure 8 shows that the results based on TPE had the best anti-overfitting effect for the GBDT model with the higher accuracy compared with the test sets.

The confusion matrix is a situation analysis table that summarizes the prediction results of the classification model in MLA. In the form of the matrix, records in datasets are summarized according to two criteria: the real category and the prediction category. The rows of the matrix represent the real value, and the columns of the matrix represent the predicted value.

This mainly includes the real value being negative that is composed of the TP (true positive) and FN (false negative), and the real value being positive that is composed of the TN (true negative) and FP (false positive). The model’s generalization ability was analyzed and verified by the test sets (20% of the total training data), and it selected 369 data from the MS and 18 from OS datasets. Figure 9a,b shows that the random search-GBDT model and TPE-GBDT model have the same performance when the real value is negative with the MS datasets, as TN = 165, FP = 12, and a slightly different performance when the real value is positive, as FN = 3, TP = 189 in the TPE-GBDT model, that is, three of the positive samples were incorrectly predicted in this model, while the four of positive samples were incorrectly predicted in the Random search-GBDT model. Figure 9c,d shows that with the OS datasets, the random search-GBDT model and TPE-GBDT model had the same TP and FP. However, the FN with the TPE-GBDT model was lower than the random search-GBDT model, and the TN with the TPE-GBDT model was the opposite. According to the confusion matrix value, the secondary indicators (i.e., accuracy, recall, specificity) can be calculated to evaluate the model.

Table 6. The secondary indicators of the GBDT model base on the confusion matrix.

Model	Accuracy	Recall	Precision	Specificity	Datasets
GBDT-Random	0.9566	0.9791	0.940	0.9322	MS
GBDT-TPE	0.9593	0.9844	0.941	0.9322
GBDT-Random	0.777	0.636	0.875	0.857	OS
GBDT-TPE	0.833	0.727	0.875	0.875

shows that Accuracy = 0.9593, recall = 0.984, and Precision = 0.941 with the TPE-GBDT model with the MS datasets. The values were slightly higher than the random search-GBDT model, indicating that the TPE-GBDT model performed better than the random search-GBDT model.

Another indicator to evaluate the performance of overall classification of the model is the ROC curve and the AUC value. The ROC (short for Receiver Operating Characteristic) is a curve in two-dimensional plane space. The AUC is the calculation result of the area under the ROC curve, which is a specific value. For any model, the closer the ROC curve is to the upper left, the larger the area is under the ROC curve, and the better the classification performance of the model. Figure 10 shows that ROC curve with the MS datasets all in the upper left corner; the AUC values of the TPE-GBDT model and random search-GBDT model were 0.985 and 0.981, respectively.

4.3. Mapping of Mineral Prospectivity

The MPM of the Xiong’ershan area using ArcGIS software was finally obtained through the above hyperparameter determination and model evaluation. The evaluation of prospectivity models with continuous probabilities (ranging from 0–1) of random search-GBDT (Figure 11a) and TPE-GBDT (Figure 11b) was conducted by measuring the correlation between the prospectivity values and known mineral occurrences, and for each model, the success-rate curves were plotted. The success-rate curve is a capture efficiency curve that indicates the relationship between the probability distribution and Au deposit locations. Figure 12 shows the proportion of the known gold deposits according to different percentages of prospective areas. It can be observed that the random search-GBDT and TPE-GBDT start from a similar success-rate curve, although the slope of the success-rate curve is steeper in the TPE-GBDT model, indicating that the TPE improved the performance of the GBDT predictive modeling.

Then combined with the fact that the high probability part of the Random search-GBDT and TPE-GBDT models occupy 25% of the total study area. However thisthis 25% study area contains more than 93% of the known gold deposits (Figure 12). Finally, in order to delineate high-favorable targets in the study area. The cut-off values of Youden index in the ROC curve were adopted to discretize the TPE-GBDT and random search-GBDT predictive models, with the thresholds of 0.9211 and 0.7371, divided the prospectivity map of TPE-GBDT and Random search models into high potential (favorable) and low potential (non-favorable) areas, respectively. The highly favorable areas in the TPE-GBDT model (Figure 13b) captured 91% of known Au deposits within only 10.22% of the Xiong’er shan area, while the highly favorable areas of the Random search-GBDT model (Figure 13a) contained the same percentage of known Au occurrences but within larger areas (16.84%). Results suggested that the TPE-GBDT model is more consistent with the actual geological conditions, and more suitable for the next step of the exploration work.

5. Conclusions

In this study, we proposed a new method of TPE-GBDT generating MPM, which solved the problem of hyperparametric optimization in MLA. To some extent, it improved the accuracy of the MPM. The summary includes the following points:

(1) In view of the data imbalance of geological datasets, we used the knowledge-driven SMOTE method for data augmentation (as the MS datasets), which reached the balance of positive and negative sets, and then compared and analyzed these with the OS datasets (original 45 ore occurrence and the randomly selected 45 non-ore occurrence datasets), showing that it improved the precision and increased the interpretability of the model;

(2) The comprehensive AUC value and the accuracy of the models indicated that the GBDT models suit small data (insufficient known mineral deposits) training. The proposed random search-GBDT model and the TPE-GBDT model were able to adjust the GBDT for HPO automatically, and the AUC value was higher than that in the conventional GBDT, indicating that the HPO increased GBDT model accuracy;

(3) The spatial distribution of the random search-GBDT model and the TPE-GBDT model predicted results was consistent. A comparison with known Au deposits indicated that the TPE-GBDT model required less training time and had a more reasonable probability distribution than the random search-GBDT model. Thus, TPE-GBDT modeling can further reduce the uncertainty of predictions and enhance the predictive accuracy of mineral exploration.