Machine learning is a key branch of artificial intelligence (AI) that possesses adaptive capabilities [
18]. In recent years, with the development of machine learning approaches, time-series forecasting methods have been widely used as an effective means of addressing data complexity, including hidden internal patterns, outliers and noise, and variations in characteristics across domains, periods, and fields of data collection. This is in contrast with traditional time-series forecasting approaches, which often struggle to identify complex patterns due to their reliance on predefined linear relationships, while machine learning does not require such assumptions because it learns nonlinear patterns directly from the data [
19].
One group of methods within machine learning is tree-based machine learning (TBML). These algorithms include several implementations, such as Random Forest, Gradient-Boosted Decision Trees (GBDT), and more recent methods like XGBoost, LightGBM, and CatBoost. Ensemble learning is a common strategy in TBML based on the “no free lunch” theorem, which states that no single algorithm is consistently superior to all others across all types of problems [
20]. Ensemble learning differs from traditional methods that generate a single hypothesis from the training data. Instead, it combines two or more models (or learners) to produce predictions that are more accurate than those generated by a single model and to help prevent overfitting. This method is widely used in supervised learning for predictive tasks. In addition, TBML is computationally efficient, relatively easy to optimize, and competitive with deep learning approaches [
21].
XGBoost is a boosting-based ensemble learning approach introduced by Chen and Guestrin in 2016 that has become a benchmark for evaluating the performance and scalability of machine learning algorithms [
22]. As a gradient-boosting algorithm, XGBoost builds models sequentially in a forward stage-wise manner, with each iteration involving the training and addition of a weak learner (a decision tree) that attempts to minimize the residual errors of previous predictions [
21,
22]. The learning process is expressed as the minimization of an objective function, which includes loss functions to improve predictive accuracy and regularization terms to limit model complexity [
22]. Through its boosting mechanism and the weighted aggregation of all trees, XGBoost can generate a strong learner with high accuracy and stable predictive performance in both classification and regression problems [
21].
2.4.2. XGBoost Modeling
XGBoost is a boosting-based ensemble learning algorithm in which the predictive model is constructed from a combination of multiple decision trees [
22]. The prediction of the decision tree ensemble model for the
-th input
is defined as follows [
25]:
with
where
denotes the predicted value for the
-th observation,
represents the number of decision trees constructed as weak learners, and
denotes the
-th decision tree. The vector
represents the input features for the
-th observation and has dimension
. Furthermore,
denotes the space of regression tree functions,
represents the weights assigned to each tree leaf,
denotes the tree structure that maps each data point to a leaf index, and
indicates the number of leaves in each tree.
Each regression tree partitions the feature space into several leaf nodes, where each leaf is associated with a prediction score. The function
assigns each observation to a specific leaf node based on its feature values, while the corresponding leaf weight
represents the prediction contribution generated by that leaf. The final prediction is obtained by summing the prediction contributions from all trees in the ensemble. Through this additive tree structure, XGBoost is able to capture complex nonlinear relationships and interactions among variables [
25].
The objective function used in XGBoost training is a combination of a loss function and a regularization term that controls model complexity to avoid overfitting and is defined as follows [
22]:
where
denotes the objective function to be minimized,
represents the number of observations, and
is the actual value of the
-th observation. The function
serves as the loss function, while
represents the regularization term.
In this study, the loss function employed is the Mean Squared Error (MSE), which measures the average of the squared differences between the actual values and the predicted values produced by the XGBoost model. The MSE is intended to impose a higher penalty on prediction errors of greater magnitude, thereby encouraging the model to generate more accurate predictions [
25].
Furthermore, the objective function in XGBoost includes a regularization term that helps prevent overfitting by controlling model complexity and penalizing overly complex decision trees, thereby contributing to better model generalization [
25].
where
is the penalty coefficient for adding leaf nodes,
is the penalty coefficient for leaf weights (L2 regularization factor), and
is the weight value (score) of each leaf.
XGBoost builds the model in a stage-wise manner through an iterative process, which can be substituted into the objective function as shown in the following equations [
25]:
where
denotes the prediction at the
-th iteration,
is the prediction at the
-th iteration,
is the learning rate parameter, and
represents the newly added tree.
To accelerate convergence and improve model predictive accuracy, the objective function at each iteration is approximated using a second-order Taylor expansion. This approach yields a new optimization formulation involving the first-order and the second-order gradient of the loss function [
25].
where
represents the first-order gradient, indicating the direction and magnitude of the prediction error, and
denotes the second-order Hessian, which measures the curvature or sensitivity of the loss function with respect to the predicted values. These quantities are incorporated into the optimization process through the second-order Taylor expansion approximation, enabling XGBoost to efficiently minimize the objective function and determine the optimal tree structure and leaf weights during the boosting process [
25].
After applying the second-order Taylor expansion, the objective function at the
t-th iteration can be expressed as follows:
The constant terms in the objective function can be omitted, allowing the objective function at the
t-th iteration to be reformulated as the optimization target for constructing a new tree, as expressed in the following equation [
25]:
Next, by defining the set of data in the
-th leaf as
, and by substituting
based on the tree structure while expanding the regularization components, the objective function can be expressed in terms of the gradient contributions and leaf weights, together with a penalty on tree complexity, as follows [
25]:
Based on this formulation, the optimal weight for each leaf can be determined by setting the derivative equal to zero [
25].
Using these optimal weights, the minimum value of the objective function for a given tree structure can be computed and subsequently used as a scoring function to evaluate the quality of the resulting decision tree structure [
25].
When all observations are present at a single node
, the value of the objective function for that node can be expressed as follows [
22]:
Based on the optimal leaf weights obtained from the optimization process, the objective function can be rewritten in a simplified form that depends only on the aggregated gradients and Hessians of the observations within a node. After determining the optimal weight for each leaf, the next step in constructing the tree is to evaluate possible splits at each node. This evaluation is performed by measuring the gain produced by a split, which represents the reduction in the loss function achieved by splitting the node compared to leaving it unsplit. Let
and
denote the sets of data in the left and right child nodes after the split, and let
denote the set of data in the parent node; then, the loss reduction resulting from the split can be expressed as follows [
22]:
A node will split only if the candidate split has a gain greater than zero, indicating that after the split, the loss is reduced. Further, the splitting criterion also takes the regularization parameters (regularized weight) into account, such that splitting is beneficial only when the gain exceeds a specified threshold value. This condition aims to avoid the creation of unnecessarily complex tree structures and to maintain the generalization capability of the model [
22].