## 1. Introduction

There are several types of deep foundations, for instance, piles and caissons, which are required in situations where the soil is not able to support structural loads at a shallow depth. The main objective of the pile foundation is to transmit the structural load to deeper bearing strata in order to withstand the axial, lateral, and uplift load and to minimize the settlement. The load applied at the top of the pile head is transferred to the soil where the load is partially taken by normal stress at the pile base and the remaining load is taken by the lateral pile-soil interface via shear stress [

1]. Thus, the piles can be classified into two types, which are end bearing pile and friction pile. The end bearing pile is a pile that transmits the structural load to a hard and incompressible stratum where the required bearing capacity is derived from end bearing at the pile base [

2]. As for the friction pile, the pile-bearing capacity is derived from skin friction and cohesion between the pile surface and the shaft that is encompassed by soil or rock along a pile [

3]. Hence, the base and friction capacity of piles are crucial for carrying the axial loading. In the event of no stiff stratum at a reasonable depth, the loads are required to transfer by friction through the pile shafts [

4].

The pile-bearing capacity can be governed by soil and pile properties [

5,

6]. The contribution of soil typically consists of cohesion and friction between the pile and the shaft of a pile at a depth. The pile friction capacity is calculated by a combination of the interface shear strength (τ

_{m}) along the pile length and the pile surface area to compute the shaft resistance (Q

_{su}) [

7]. In addition, during the installation of driven piles, which are usually prefabricated, the loose deposit soil that encompasses the pile can be locally densified due to soil displacement and, thus, the pile capacity can be increased [

7]. As such, it can be stated that the installation method of piles can be one of the factors that contributes to the pile capacity [

8].

The pile-bearing capacity can be determined using several techniques, such as empirical, semi-empirical and finite element (FE). The extent of the FE model and computation time is limited with model boundaries in contexts where this can be done by redoing the model boundaries by taking the boundaries to be further away from the modeling object and comparing the results. This process can be more time consuming [

9]. In industry practice, the Standard Penetration Test (SPT)-N is widely-used to determine the pile capacity [

8,

10,

11,

12]. There are many empirical formulas of pile friction bearing correlated with SPT-N in a general form of equation, as shown below:

where,

q_{s} is the limit skin friction stress at a given depth, which is proportional to the

N value at the particular depth, and

n_{s} is the skin friction factor proposed by researchers as presented in

Table 1. However, according to previous studies, these empirical equations are not reliable in terms of accuracy [

13,

14]. This is because some of the pile’s empirical analysis relationships are made by simplification in contexts where a large factor of safety is applied. This factor reduces the accuracy of the predictions and the deprivation of resources [

15]. Other than this, there is a simple correlation between the pile bearing capacity and in-situ tests, for instance, the Cone Penetration Test (CPT) or SPT. However, this correlation method overestimates the pile bearing capacity [

16].

Pile tests are required during the construction process to reassure the design calculation, because the estimation of axial pile capacity at various soil types will never be more accurate than approximately 30% [

25]. There are a few methods of pile tests used to calculate the axial capacity of the piles. The typical methods are Static Load Test (SLT) and High Strain Dynamic Testing (HSDT) [

26]. SLT is considered as the most reliable predictor of long term pile performance. However, this testing is expensive and time consuming [

27,

28,

29]. Other than the SLT test, HSDT is one of the methods used to determine the pile bearing capacity. In comparison with SLT, HSDT is quick and economical [

26]. This test is carried out based on the theory of one-dimensional wave propagation and is given by a Pile Driving Analyzer (PDA). PDA has proven that the predicted bearing capacity values are closely related to SLT results [

30]. Nevertheless, all pile tests are, basically, expensive and time consuming to set up at the site [

31,

32]. Due to the aforementioned situation, it is important to predict pile bearing capacity using new and effective calculation approaches, such as Machine Learning (ML) and Artificial Intelligence (AI).

AI, ML and data mining techniques have been used widely to solve many civil engineering and more specifically geotechnical problems [

33,

34,

35,

36,

37,

38,

39,

40,

41,

42,

43,

44,

45]. In terms of piling related issues, such as pile bearing capacity, there are several studies that have applied and proposed AI and ML techniques [

12,

46,

47,

48,

49,

50]. One of the most-used models in this regard is the Artificial Neural Networks (ANN). These approaches demonstrated a number of successful predictions [

29,

50]. As discussed before, pile driving formulae were used to provide an approximate estimation of the driven pile capacity. This formula is derived from impulse-momentum principles. However, the accuracy of neural network predictions are significantly higher compared to the conventional pile driving formulae [

51]. In another study, Pal [

52] stated that the General Regression Neural Network (GRNN) model has shown higher accuracy of the pile load bearing capacity prediction in comparison to empirical approaches, but slightly lower accuracy than the ANN technique. In addition, the Gene Expression Programming (GEP) model was in good agreement with the results of the experiment, indicating that pile capacity has a good relationship with some inputs, such as pile geometry [

53]. Alavi et al. [

54] concluded that Linear Genetic Programming (LGP) model is the best behavior in modelling uplift capacity of suction caissons, followed by the GEP and tree-based genetic programming models in comparison with regression and FE models. A Gaussian Process Regression (GPR) approach was suggested by Momeni et al. [

55] in the area of pile capacity after comparison with other ANN-based models. Another group of scholars applied and proposed a combination of at least two AI techniques for prediction of pile bearing capacity [

26,

27,

28,

31,

47]. Actually, these combined techniques enjoyed the advantages of all the used AI models for prediction purposes and, due to that, they achieved higher performance compared to the single AI models.

Table 2 presents the most important AI and ML studies for predicting the pile capacity, together with their soil types, number of data, model performance, and input parameters.

According to

Table 2, many studies used ANN, ANN-based and genetic-based models for estimating pile capacity. In addition, several studies applied and proposed neuro-fuzzy and Support Vector Machine (SVM) models for the same purpose. However, there are very few approaches using tree-based techniques, like Random Forest (RF), to predict the pile capacity as far as the authors know. Due to this, this paper is aimed at applying and proposing the full applications of tree-based models only, i.e., Decision Tree (DT), RF and Gradient Boosted Tree (GBT) for the prediction of pile friction bearing capacity. To do this, a feature selection (i.e., input selection) will be conducted to select the most crucial input variables for pile friction bearing capacity. The mentioned models will then be constructed and the model with the highest accuracy will be selected and introduced for estimating the pile friction bearing capacity.

## 4. Modeling and Results

In order to have an accurate model proposal, both model development and model evaluation parts should be at an acceptable level. In the first stage of modeling all data samples were normalized using the following equation in the range of [0–1]:

where,

X presents each parameter that needs to be normalized (i.e., each input and output),

min(

X) and

max(

X) are the minimum and maximum values of whole data of that specific parameter, respectively.

For this purpose, there is a need to divide the whole database into two groups: train and test. In this study, among all of the available suggestions in the literature, the authors decided to use a combination of 80–20% for train and test phases, respectively. Therefore, before starting the modeling, all 125 data samples were divided to 25 and 100 data samples for model evaluation and model development, respectively. As discussed before, three tree-based ML techniques were employed to determine the most accurate model for predicting the pile friction bearing capacity. To do this, several parametric investigations were performed for different parameters of DT, RF and GBT techniques. In these analyses, three and five model inputs were utilized. Finally, the model performance results with three and five input variables are presented in

Table 6 and

Table 7, respectively. In these tables, train and test results of R

^{2}, RMSE and absolute error were presented. According to the results, GBT technique achieved the highest accuracy rate for both models with five and three variables, with R

^{2} equal to 0.911 and 0.901, respectively, for training datasets in predicting pile friction bearing capacity. In addition, the GBT model achieved the lowest RMSE and absolute error compared to the RF and DT models. The next best model after GBT is related to RF for three and five input variables, followed by the DT model. The R values of (0.813 and 0.761) and (0.773 and 0.712) were obtained for testing data samples of RF and DT techniques for three and five model inputs, respectively. It is obvious that the results obtained by the GBT technique are more accurate compared to the RF and DT models for both three and five input parameters.

As was expected, the accuracy of the models decreased by decreasing the number of variables after feature selection. However, considering the train results of the GBT technique, the accuracy reduction is not significant (only 1%). For testing data, the GBT results based on R

^{2} are 0.841 and 0.816 for three and five models, respectively, which show a close model accuracy when three input parameters are used. Therefore, as discussed before, proposing a new predictive model with a lower number of model predictors is of importance in the area of piling and geotechnical engineering. The other researchers and designers can easily use a simpler model because they need a lower number of parameters to be measured. Therefore, in this study, the authors decided to propose and introduce a predictive model with lower model inputs, even though it has lower performance prediction results. Hence, the results presented in

Table 7 will be considered in this study and, as such, the GBT model for three inputs will be discussed in more detail in the following paragraphs.

After reducing the number of variables through the feature selection process, the GBT model was conducted using three important selected variables.

Table 8 presents the importance of input variables using the GBT technique with the final three variables. The importance values of 0.81, 0.21 and 0.075 were obtained, respectively, for pile length, SPT-N average, and hammer drop height. According to the results, pile length, with an importance of 0.81, plays the most important role in predicating pile friction bearing capacity using the GBT technique. On the other hand, hammer drop height has the lowest impact on the model output, which is the pile capacity.

In modelling GBT, there were many models constructed in order to see the difference between different parameters of GBT on the system performance. As presented in

Table 9, 27 GBT models were built in this study with different properties in order to predict pile bearing capacity. In these 27 models, the authors considered different values for the number of trees, maximum depth and learning rate in the modeling. In addition, error results are presented in

Table 9 for each GBT model. As a result, the optimal/best model is achieved when the number of trees is 90 with a maximum depth of two and 0.1 learning rate (i.e., GBT model number 20). The lowest error rate of 0.1889 and the highest accuracy (R values of 0.901 and 0.816 according to

Table 7) were observed at the described point.

Figure 10 shows the schematic tree generated by the proposed GBT model. More discussions regarding this technique will be given in the next section.

## 5. Discussion

In this study, a series of experimental data were measured and recorded during SLT tests, and the capacity values of friction piles, together with some other important parameters on them, were collected. The idea was to propose a series of fully tree-based techniques, i.e., DT, RF and GBT, for estimation of the pile bearing capacity. Through feature selection, in order to propose a simpler model, the three most important parameters were identified as pile length, SPT-N average and hammer drop height. The mentioned tree-based models were then built to predict pile friction bearing capacity. In order to construct DT, RF and GBT models, many attempts have been made to achieve higher performance capacities based on the used statistical indices. These attempts were performed by setting different values for the most influential DT, RF and GBT parameters. As expected, the developed GBT model was able to provide a better performance capacity in estimating the actual results of pile friction bearing capacity. The training and testing results of the proposed GBT model are presented in

Figure 11 and

Figure 12, respectively. It is important to note that the pile capacity values presented in these figures are normalized between [0–1], as described previously. The R

^{2} and other statistical indices are presented in these figures, which confirms that the GBT is a powerful tree-base technique in both phases of model development and model evaluation. RMSE results and absolute error results of (0.094, and 0.077) and (1.27, and 0.098) for train and test data samples, respectively, reveal that the GBT tree model is applicable in the field of piling and deep foundation. It is able to predict pile bearing capacity values with a low level of system error, which is of importance and advantage in the geotechnical engineering field.

Compared to the previous ML related studies, this study focuses on only tree-based ML techniques. To date, only a few researchers have proposed similar techniques in this regard. The developed GBT model in this study was based on only three input parameters, and based on these three inputs, the GBT model provided R

^{2} values of 0.901 and 0.816 for the train and test phases, respectively. The results of the GBT model are not better than many of the relevant studies presented in

Table 2. However, as presented in

Table 2, most of the studies used five or more input parameters to predict the pile bearing capacity. This makes them complicated models for further use by other researchers. This is because they have to provide the related values for all inputs if they want to use the proposed models. Nevertheless, in this study, the presented results were constructed based on only three model inputs/predictors. In other words, the proposed GBT technique in this study is easier to implement by other researchers, designers, or engineers. Hence, the modelling process and the proposed GBT model in this study can be suggested as a reliable and applicable technique/process with a high level of accuracy in forecasting pile bearing capacity.

It is good to know that the GBT model can depict the promising accuracy of the prediction, provided that this study is carried out for different types of soils, piles, installation methods, and types of hammers. This study was carried out with limited data with one hammer weight, two types of pile dimension, an SPT-N value of generally about 10, and one installation method. Thus, it is highly recommended to further carry out this study with more variables in order to provide higher accuracy of the prediction.

## 7. Summary and Conclusions

The purpose of this research is to propose a more accurate and applicable model/approach for predicting pile friction bearing capacity, which is fully tree-based with a limited number of model inputs/predictors. To achieve this aim, first, among the initial five input variables, the three most effective ones, i.e., pile length, SPT-N average, and hammer drop height, were selected for the modelling part, based on a comprehensive feature selection process. Three tree-based techniques, i.e., DT, RF and GBT, were then built to estimate pile friction bearing capacity. In building these models, a series of parametric investigations based on their effective variables were planned and performed in order to obtain the best model in each category. In the next step, model assessment has been done using different performance prediction indices, and their results have been compared to each other. Overall, the findings demonstrated the successful application of tree-based techniques for the purpose of this paper. However, the best tree model was related to GBT with R^{2} values of 0.901 and 0.816 for model development and model assessment parts, respectively. It should be noted that the other tree-based models received acceptable and applicable results for prediction of pile friction bearing capacity. The parametric study results showed that the optimum values of pile length, hammer drop height and SPT-N average are 44, 1.1, and 6, respectively, in order to get the maximal pile capacity values. The proposed tree-based techniques and their processes are easy to implement and can be used by other researchers and designers to obtain very accurate pile capacity values for similar conditions. However, other researchers can prepare a larger database for the same problem and develop more comprehensive tree-based techniques, or even a combination of these techniques with new optimization techniques, such as the sparrow search algorithm, in order to achieve a higher accuracy level.