Empirical Calibration of XGBoost Model Hyperparameters Using the Bayesian Optimisation Method: The Case of Bitcoin Volatility

Abstract

Ensemble learning techniques continue to show greater interest in forecasting the volatility of cryptocurrency assets. In particular, XGBoost, an ensemble learning technique, has been shown in recent studies to provide the most accurate forecast of Bitcoin volatility. However, the performance of XGBoost largely depends on the tuning of its hyperparameters. In this study, we examine the effectiveness of the Bayesian optimization method for tuning the XGBoost hyperparameters for Bitcoin volatility forecasting. We chose to explore this method rather than the most commonly used manual, grid, and random hyperparameter choices due to its ability to predict the most promising areas of hyperparameter spaces through exploitation and exploration using acquisition functions, as well as its ability to minimize error with a reduced amount of time and resources required to find an optimal configuration. The obtained XGBoost configuration improves the forecast accuracy of Bitcoin volatility. Our empirical results, based on letting the data speak for itself, could be used for a comparative study on Bitcoin volatility forecasting. This would also be important for volatility trading, option pricing, and managing portfolios related to Bitcoin.

1. Introduction

Volatility is a key indicator of uncertainty and risk in financial markets. For the case of Bitcoin, an innovative and highly speculative digital asset, this volatility exceeds that observed in traditional markets by far. While this instability creates opportunities for substantial returns, it also complicates forecasting and risk management. In this context, reliable predictive models that can estimate volatility are essential for investors and quantitative finance researchers alike.

Among modern predictive modeling approaches, the XGBoost (Extreme Gradient Boosting) algorithm is one of the most effective at processing complex, nonlinear financial data. However, the quality of the predictions obtained depends heavily on calibrating its hyperparameters correctly, such as tree depth, learning rate, and regularization coefficients. Inadequate calibration can lead to overfitting or a loss of performance, thereby compromising the robustness of the model.

To optimize this calibration, Bayesian optimization (BO) (Brochu et al., 2010; Garnett, 2023) offers an effective, mathematically rigorous solution. Unlike traditional methods such as grid or random searches, BO uses a probabilistic model of the objective function and an acquisition function to intelligently guide exploration of the hyperparameter space. This approach reduces computational cost while increasing the likelihood of identifying optimal configurations.

This paper proposes performing empirical calibration of the XGBoost model using BO for Bitcoin volatility forecasting. The objective is twofold: firstly, to evaluate the effectiveness of this tuning strategy compared to conventional calibration methods; and secondly, to demonstrate how tools from probability, statistics, and optimization can be integrated into a practical approach that links mathematical rigour with real-world financial market issues.

2. Literature Review

In recent years, modeling and forecasting the volatility of financial assets, particularly cryptocurrencies such as Bitcoin, has attracted increasing attention in the literature. Several traditional approaches, such as GARCH models and stochastic models (Nadarajah et al., 2025), have been studied extensively to capture the complex dynamics and high variability inherent in these markets. However, machine learning methods such as gradient boosting algorithms, notably XGBoost, have emerged as powerful tools capable of modeling complex nonlinear relationships in large, noisy datasets.

Nevertheless, the performance of these models hinges on the precise calibration of their hyperparameters. In this regard, Bayesian optimization has become the preferred method for efficiently exploring the hyperparameter space, often outperforming classical methods such as exhaustive or random search. While several recent studies have demonstrated the advantages of this approach in various fields, few have specifically examined the combined use of XGBoost (Friedman, 2001) and Bayesian optimization for predicting cryptocurrency volatility.

The authors in Cowen-Rivers et al. (2022) worked on a black-box optimization about hyperparameter settings with HEBO (Heteroscedastic Evolutionary Bayesian Optimization) enriching Bayesian Optimization (BO) algorithms (Brochu et al., 2010). The authors in Cowen-Rivers et al. (2022) created a wide range of hyperparameter tasks (108) in a variety of classification and regression problems, using nine models (such as multi-layer perceptrons, support vector machines, and others), six datasets (two regression and four classification) from the University of California Irvine (UCI) repository, and two measures per dataset (such as negative log-likelihood or mean-squared error). Each model had adjustable hyperparameters, such as the number of hidden units in a neural network. The objective was to adjust these hyperparameters so as to maximize/minimize one of the specified measures. The black-box objective values were stochastic, with noise contributions from the training–test separations used to calculate losses. The paper found results with a wide range of solvers that rely on BO strategies or follow zero-order techniques such as differential evolution or particle swarms.

The authors in Joy et al. (2016) proposed a framework for fitting hyperparameters to big data using BO. The framework divides the big data set into chunks and generates the hyperparameter fit on these chunks in parallel. The framework then uses this information to tune the hyperparameters of the big data set using transfer learning, and evaluates its performance on the hyperparameter tuning tasks of two machine learning algorithms: a deep neural network and an SVM with a radial basis function kernel. These tasks were performed on two real-world data sets.

The authors in Bentéjac et al. (2021) observed that XGBoost and gradient boosting, when trained using the package defaults, were the least efficient methods. Consequently, the paper concluded that a meticulous search for parameters was necessary to create accurate gradient boosting-based models. This was not the case for random forest, whose performance was slightly better on average when the default parameter values were used. In fact, adjusting the randomization parameters, the subsampling rate and the number of features selected at each division in XGBoost proved useless as long as a certain randomization was used. In experiments, the paper set the subsampling rate to 0.75 without replacement and the number of features to the square root of the total number of features. This reduced the size of the parameter search grid by a factor of 16 and improved the average performance of XGBoost.

The authors in Putatunda and Rama (2019) proposed a new method for optimizing the hyperparameters of an extreme gradient boosting algorithm: Randomized-Hyperopt. Its performance was compared with other existing techniques, such as grid search, random search, and hyperopt, taking into account both prediction accuracy and time required. The paper found that Randomized-Hyperopt outperforms Grid Search and Hyperopt, and is close to or better than the average Gini of Hyperopt for all datasets. Furthermore, the Randomized-Hyperopt method has the shortest execution time for all datasets.

The authors in van Hoof and Vanschoren (2021) discussed Hyperboost, an AutoML (Automated Machine Learning) tool based on SMAC (Sequential Model-Based Algorithm Configuration). They described the challenges of adapting the Gradient Boosting model to BO and then explained how they adapted this model using quantile regression. However, they acknowledged that quantile regression alone is insufficient as it does not account for epistemic uncertainty. When estimating the 90th quantile, they provided optimistic estimates for the candidate configurations and combined these with the quantile regression estimates. They then combined these with a bonus for Manhattan distance from the nearest configuration. This bonus encourages users to avoid configurations that have already been observed. They demonstrated that Hyperboost outperformed SMAC on a reasonable set of classification problems when used with a small set of configuration spaces.

The authors in Lévesque (2018) discussed hyperparameter tuning using BO. This paper evaluated the impact of overlearning on optimization and proposed strategies to reduce it, as well as applying it to generate heterogeneous sets of classifiers and methods for optimizing hyperparameters with conditional structures. The hyperparameters of the following models were optimized: LDA, QDA, SVM, KNN, and Adaboost. The paper also used the Laplace kernel and Modrian with several types of Gaussian processes (GPs) (Rasmusse & Williams, 2005).

The authors in Eggensperger et al. (2013) worked on a reference library for hyperparameter optimization and provided the first in-depth comparison of three substitution models: Spearmint, SMAC, and TPE (van Hoof & Vanschoren, 2021), which are substitute models in the GP family. The paper provided a common interface for three optimization software packages and made it easy to integrate new ones. The authors in (Anghel et al., 2018) does a job similar to (van Hoof & Vanschoren, 2021), working with the hyperparameters of XGBoost, LightGBM, and Catboost.

The authors in Joy et al. (2020) developed a rapid hyperparameter tuning framework called Hypertune that draws on insights from the learning theory of CAPs. The paper also identified a new way of exploiting trends in generalization performance from a subset of the data to the whole dataset. The paper used directional derivatives to report monotonic trends in model generalization performance for hyperparameters that directly dictate model complexity. The paper then evaluated the effectiveness of the algorithm at fitting hyperparameters from several machine learning algorithms to real-world benchmark datasets. The results show that Hypertune outperforms Fabolas (Klein et al., 2017) and generic BO. The paper also showed that HyperTune is compatible with the state-of-the-art hyperparameter tuning algorithm, Hyperband (Li et al., 2018), and that it is widely applicable to optimization of hyperparameters.

Our objective is to optimize the hyperparameters of the XGBoost model using BO. Our study will begin with an introduction to the theory of Bayesian optimization and its application to Bitcoin volatility. This will be followed by a discussion of the results and a conclusion. All computations for the application were performed using R, version 4.1.3. The R codes used are given in Appendix A.

3. Bayesian Optimization

3.1. Overview

BO is a powerful strategy for locating the extrema of objective functions when evaluations are costly and an explicit analytical form of the function is unavailable. Instead of requiring derivatives or convexity assumptions, BO relies on observations obtained from sampled evaluations, making it particularly well suited for problems where the following apply:

• The objective function has no closed-form expression.
• Evaluations are expensive or time-consuming.
• Derivatives are unavailable.
• The optimization problem is non-convex.

A key strength of BO lies in its efficiency—it typically requires fewer evaluations than alternative optimization methods. This efficiency comes from incorporating prior knowledge about the problem, which guides the sampling process and reduces unnecessary exploration of the search space.

The term “Bayesian” reflects the use of Bayes’ theorem to update beliefs about the objective function as new data is observed. The prior distribution encodes assumptions about plausible functions, such as smoothness. These assumptions make certain function shapes more likely than others.

Formally, let $x_i$ denote a sample point and $f\left(x_i\right)$ denote the value of the objective function at that point. After t observations,

$$\begin{array}{c}\hfill {\displaystyle D_{1:t}=\left\{x_{1:t},f\left(x_{1:t}\right)\right\},}\end{array}$$

(1)

and the prior distribution over functions $P\left(f\right)$ is updated using the likelihood $P\left(D_{1:t}\mid f\right)$ to yield the posterior

$$\begin{array}{c}\hfill {\displaystyle P(f\mid D_{1:t})=\frac{P\left(D_{1:t}\mid f\right)P\left(f\right)}{P\left(D_{1:t}\right)}.}\end{array}$$

(2)

This posterior distribution then guides the selection of the next evaluation point, balancing the exploration of uncertain regions with the exploitation of promising areas.

3.2. Bayesian Optimization Approach

We begin by formulating the problem as $x^{*}=\underset{x}{\mathrm{argmin}}f\left(x\right)$, where the objective function f is assumed to be continuous and Lipschitz continuous. That is, for all $x_1,x_2$, there is a constant C such that

$${\displaystyle \parallel f\left(x_2\right)-f\left(x_1\right)\parallel \le C\parallel x_2-x_1\parallel .}$$

(3)

Our focus is on global rather than local optimization. In a minimization setting, we seek $x^{*}$ satisfying

$${\displaystyle f\left(x^{*}\right)\le f\left(x\right),\forall x,\parallel x^{*}-x\parallel <\epsilon .}$$

(4)

If f is concave, every local minimum is also a global minimum. In practice, however, concavity cannot be confirmed in advance, as the explicit form of f is unknown and its gradient is unavailable. We do, however, assume that the boundaries of the search space are known.

3.3. A Priori Function

Every Bayesian method begins with an initial distribution. A BO method can be considered optimal if the following conditions hold:

• The acquisition function is continuous and approximately minimizes the risk with respect to the global minimum at a fixed point.
• The variance converges to zero (or to a positive minimum in the presence of noise) if and only if the distance to the nearest observation is zero.
• The objective function itself is continuous.
• The a priori function is homogeneous.
• The method is independent of specific variants and applicable to a wide range of optimization tasks.

A GP on a space $\chi$ is a random process mapping $\chi \to \mathbb{R}$. In non-parametric Bayesian methods, GPs are often used as priors over functions. A GP is fully specified by its mean function $\mu \left(x\right)$ and its covariance function $k:{\chi }^2\to \mathbb{R}$. For all $x\in \chi$, the distribution of $f\left(x\right)$ is given by

$$\begin{array}{c}\hfill {\displaystyle f\left(x\right)\sim \mathcal{N}\left(\mu \left(x\right),k\left(x,x^{\prime }\right)\right).}\end{array}$$

(5)

Formally,

$$\begin{array}{c}\hfill {\displaystyle \mu :\chi \mapsto \mathbb{R},f\sim GP(\mu ,k).}\end{array}$$

(6)

Intuitively, a GP can be thought of as a distribution over functions. Instead of returning a single value $f\left(x\right)$ at a point x, it returns a mean and variance describing the uncertainty over possible values of f at that point. This is why stochastic processes are sometimes referred to as “random functions”, by analogy to random variables.

3.3.1. Covariance Function

We choose the squared exponential (radial basis function) kernel

$$\begin{array}{c}\hfill {\displaystyle k\left(x,x^{\prime }\right)=exp\left(-\gamma \parallel x-x^{\prime }{\parallel }^2\right),}\end{array}$$

(7)

which takes values close to 1 for nearby points and approaches 0 for distant points. This reflects the intuitive idea that close points are strongly correlated, while far-apart points influence each other minimally.

Given a set of inputs $x_{1:t}$ and their function values $f_{1:t}=f\left(x_{1:t}\right)$, the joint distribution is

$$\begin{array}{c}\hfill {\displaystyle f_{1:t}\sim \mathcal{N}(0,K),}\end{array}$$

(8)

where the kernel matrix K is

$$\begin{array}{c}\hfill {\displaystyle K=\left(\begin{array}{ccc}{\displaystyle k\left(x_1,x_1\right)}& \dots & k\left(x_1,x_t\right)\\ ⋮& \ddots & ⋮\\ {\displaystyle k\left(x_t,x_1\right)}& \dots & k\left(x_t,x_t\right)\end{array}\right).}\end{array}$$

(9)

The diagonal entries are 1, reflecting perfect correlation of each point with itself—valid in the noise-free case. For simplicity, we assume a zero mean function $\mu \left(x\right)=0$.

3.3.2. Posterior Prediction

Suppose we have previous observations $\left\{x_{1:t},f_{1:t}\right\}$. We wish to use BO to select the next evaluation point $x_{t+1}$. Let $f_{t+1}=f\left(x_{t+1}\right)$ be the unknown function value at this point. By the properties of GPs, $\left(f_{1:t},f_{t+1}\right)$ is jointly Gaussian:

$$\begin{array}{c}\hfill {\displaystyle \left(\begin{array}{c}{f}_{1:t}\\ f_{t+1}\end{array}\right)\sim \mathcal{N}\left(0,\left(\begin{array}{cc}K& k\\ k^T& K\left(x_{t+1},x_t\right)\end{array}\right)\right),}\end{array}$$

(10)

where

$$\begin{array}{c}\hfill {\displaystyle k={\left[k\left(x_{t+1},x_1\right),\dots ,k\left(x_{t+1},x_t\right)\right]}^{\top }.}\end{array}$$

(11)

Using the Sherman–Morrison–Woodbury formula (Shermen & Morrison, 1949), the predictive distribution is

$$\begin{array}{c}\hfill {\displaystyle P\left(f_{t+1}\mid {\mathcal{D}}_{1:t},x_{t+1}\right)=\mathcal{N}\left({\mu }_t\left(x_{t+1}\right),{\sigma }_t^2\left(x_{t+1}\right)\right)}\end{array}$$

(12)

with ${\mu }_t\left(x_{t+1}\right)=k^T{K}^{-1}{f}_{1:t}$ and ${\sigma }_t^2\left(x_{t+1}\right)=k\left(x_{t+1},x_{t+1}\right)-k^T{K}^{-1}k$.

3.4. Choice of Covariance Function

Selecting an appropriate GP covariance function is critical, as it governs the smoothness and regularity of functions drawn from the process. The commonly used squared exponential kernel makes a simplifying assumption: it treats all input features as contributing equally to the covariance, ignoring potential differences in their influence.

To address this limitation, isotropic kernels can be extended by introducing hyperparameters. In the isotropic case, a single length-scale parameter $\theta$ controls the kernel width

$${\displaystyle k\left(x_i,x_j\right)=exp\left(-{\displaystyle \frac{1}{2{\theta }^2}}{\parallel x_i-x_j\parallel }^2\right).}$$

(13)

Reducing $\theta$ narrows the kernel, allowing the GP to model functions that vary more rapidly.

For anisotropic (direction-dependent) models, the most common choice is the squared exponential kernel with a vector of length-scale parameters $\theta$ for automatic relevance determination (ARD)

$${\displaystyle k\left(x_i,x_j\right)=exp\left[-{\displaystyle \frac{1}{2}}{\left(x_i-x_j\right)}^T\mathrm{diag}{\left(\theta \right)}^{-1}\left(x_i-x_j\right)\right],}$$

(14)

where $\mathrm{diag}\left(\theta \right)$ is a diagonal matrix with d entries, one per dimension. Intuitively, a small ${\theta }_l$ implies that the lth input dimension has little influence on the covariance, effectively rendering it irrelevant and removing it from the model.

Hyperparameters are typically learned by seeding the GP with a small set of random samples and maximizing the log marginal likelihood (Jones et al., 1998; Santner, 2003; Sasena, 2002; Williams & Rasmussen, 2006). This process can be aided by informative priors over the hyperparameters (Frean & Boyle, 2008; Lizotte, 2008). Efficient and robust methods for hyperparameter learning remain an active research area (Brochu et al., 2010; Osborne & Osborne, 2010).

3.5. Acquisition Function in BO

After covering the implementation of the prior on smooth functions and its updates with new observations, we now turn to the acquisition function in BO.

The acquisition function directs the search toward the optimum by assigning high values to locations that are promising—due to high predicted objective values, high uncertainty, or both. By maximizing this function, we determine the next point to evaluate. Formally, the next sample location is chosen as

$$\begin{array}{c}\hfill {\displaystyle x_{\mathrm{next}}=\underset{x}{\mathrm{argmax}}u\left(x\mid \mathcal{D}\right),}\end{array}$$

(15)

where $u(·)$ denotes the acquisition function, and $\mathcal{D}$ is the set of current observations.

3.5.1. Expected Improvement (EI) and Probability of Improvement (PI)

The earliest acquisition function, proposed is aimed to maximize the Probability of Improvement (PI) over the current best observation. Let

$$\begin{array}{c}\hfill {\displaystyle x^{+}=\underset{x_i\in x_{1:t}}{\mathrm{argmax}}f\left(x_i\right)}\end{array}$$

(16)

denote the best location found so far. The PI function is defined as

$${\displaystyle PI\left(x\right)=P\left(f\left(x\right)\ge f\left(x^{+}\right)\right)=\Phi \left(\frac{\mu \left(x\right)-f\left(x^{+}\right)}{\sigma \left(x\right)}\right),}$$

(17)

where $\Phi (·)$ is the standard normal cumulative distribution function. While intuitive, PI is purely exploitative, favoring points with a high chance of beating $f\left(x^{+}\right)$, even if the potential improvement is small. To encourage exploration, an offset parameter $\xi \ge 0$ is often introduced

$${\displaystyle PI\left(x\right)=P\left(f\left(x\right)\ge f\left(x^{+}\right)+\xi \right)=\Phi \left(\frac{\mu \left(x\right)-f\left(x^{+}\right)-\xi }{\sigma \left(x\right)}\right).}$$

(18)

The choice of $\xi$ is left to the user. The authors in Kusner and Yin (1997) recommend starting with a large $\xi$ (to promote exploration) and decreasing it over time. Empirical studies (Jones, 2001; Lizotte, 2008; Törn & Žilinskas, 1989) show that the choice of $\xi$ can have a substantial effect depending on the problem domain.

PI can be useful in perceptual or preference-based models, where thresholds of noticeable change exist, since it naturally targets points likely to achieve a specified level of improvement.

A more balanced approach is the Expected Improvement (EI), which accounts for both the probability and magnitude of improvement. Let the improvement function be

$${\displaystyle I\left(x\right)=max\left\{0,f_{t+1}\left(x\right)-f\left(x^{+}\right)\right\}.}$$

(19)

Then EI is defined as

$${\displaystyle EI\left(x\right)={\int }_0^{\infty }I·L\left(I\right)dI,}$$

(20)

where $L\left(I\right)$ is the likelihood of achieving improvement I given the posterior normal distribution:

$${\displaystyle L\left(I\right)=\frac{1}{\sqrt{2\pi }\sigma \left(x\right)}exp\left[-\frac{{\left(\mu \left(x\right)-f\left(x^{+}\right)-I\right)}^2}{2{\sigma }^2\left(x\right)}\right].}$$

(21)

Evaluating the integral yields the closed-form (Jones, 2001; Mockus et al., 1978)

$${\displaystyle EI\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \left[\mu \left(x\right)-f\left(x^{+}\right)\right]\Phi \left(Z\right)+\sigma \left(x\right)\varphi \left(Z\right),}\hfill & \sigma \left(x\right)>0,\hfill \\ \\ {\displaystyle 0,}\hfill & \sigma \left(x\right)=0,\hfill \end{array}\right.}$$

(22)

with $Z={\displaystyle \frac{\mu \left(x\right)-f\left(x^{+}\right)}{\sigma \left(x\right)}}$, and $\varphi (·)$ is the standard normal probability density function.

While EI is typically applied myopically (one step ahead), multi-step extensions exist, including closed-form two-step EI (Ginsbourger et al., 2008) and multi-step Bayesian optimization strategies (Garnett et al., 2010).

3.5.2. Upper Confidence Bound (UCB) Acquisition Functions

Unlike PI and EI, which focus on local improvement near the current maximum, UCB-based strategies (Srinivas et al., 2009) aim to reduce uncertainty across the entire search space. A naïve rule is to select

$$\begin{array}{c}\hfill {\displaystyle x_t=\underset{x\in \mathcal{D}}{\mathrm{argmax}}{\mu }_{t-1}\left(x\right),}\end{array}$$

(23)

which is purely exploitative and risks premature convergence.

The Gaussian Process Upper Confidence Bound (GP-UCB) balances exploration and exploitation via

$${\displaystyle x_t=\underset{x\in \mathcal{D}}{\mathrm{argmax}}\left[{\mu }_{t-1}\left(x\right)+\sqrt{{\beta }_t}{\sigma }_{t-1}\left(x\right)\right],}$$

(24)

where ${\beta }_t$ controls the exploration–exploitation trade-off. The first term favors high predicted values, while the second favors uncertain regions. The quantity inside the maximization can be interpreted as an upper quantile of the posterior marginal $P\left(f\left(x\right)\mid y_{t-1}\right)$.

From a theoretical perspective, GP-UCB is related to functions in a Reproducing Kernel Hilbert Space (RKHS) $H_k\left(D\right)$ (Wahba, 1990) associated with kernel $k(·,·)$. In RKHS, the norm

$$\begin{array}{c}\hfill {\displaystyle {\parallel f\parallel }_k=\sqrt{{\langle f,f\rangle }_k}}\end{array}$$

(25)

quantifies smoothness: kernels producing smoother functions induce larger penalties. This RKHS framework underpins theoretical regret bounds for GP-UCB under both Bayesian and non-Bayesian settings.

3.5.3. Extreme Gradient Boosting (XGBoost)

Consider the objective function deduced by the XGBoost model, as follows:

$${\displaystyle L\left(y\right)=\sum _{i=1}^{n}l\left({\widehat{y}}_i,y_i\right)+\sum _{m=1}^M\Omega \left({\delta }_m\right)}$$

(26)

with

$${\displaystyle \Omega \left(\delta \right)=\alpha \left|\delta \right|+\frac{1}{2}\beta {\parallel \omega \parallel }^2,}$$

(27)

where l is the loss function, $\left|\delta \right|$ is the number of leaves in the regression tree $\delta$, $\omega$ is the vector of values assigned to each of its leaves, $\alpha$ is the Lasso penalty coefficient, and $\beta$ is the ridge regularization coefficient. Our objective is to find ${\theta }^{*}$ such that ${\theta }^{*}=\underset{\theta }{min}L\left(y\right)$ with $\theta$ denoting the set of hyperparameters.

The hyperparameters include eta, max-depth, subsample, colsample-by-tree, and min-child-weight. eta is a hyperparameter that controls the learning rate, which determines how much each tree’s predictions contribute to the overall model. eta scales the contribution of each new tree by multiplying its leaf weights before adding them to the model’s running prediction. It prevents the model from overfitting by making the boosting process more conservative.

The max-depth is a parameter that controls the maximum number of levels (depth) a decision tree can grow during training. A higher max-depth allows the model to capture more complex patterns and interactions between features, but it also increases the risk of overfitting-especially when combined with a small dataset or insufficient regularization. A shallower tree (low max-depth) tends to be more generalizable but may underfit by missing important feature relationships. In practice, max-depth works in tandem with other parameters like min-child-weight and gamma to balance model complexity and performance, and it is typically tuned through cross-validation.

The subsample parameter controls the fraction of the training data randomly sampled (without replacement) for growing each individual tree. It is a form of stochastic gradient boosting, similar to bagging, that helps reduce overfitting and improve model generalization. For example, setting subsample = 0.8 means each tree is built using only 80% of the training samples, chosen at random, which introduces diversity among trees and makes the ensemble less prone to fitting noise Lower values can improve robustness but may also increase bias if set too low, while subsample = 1.0 means all data is used for every tree.

colsample-by-tree is a hyperparameter that controls the fraction of features (columns) randomly selected to train each individual tree in the boosting process. Its value ranges from 0 to 1, where 1 means all features are used, and smaller values make the model use only a subset of features per tree, introducing randomness that can help reduce overfitting and improve generalization—similar to the feature bagging technique in Random Forests. For example, setting colsample-by-tree = 0.8 means each tree will be built using 80% of the available features, chosen randomly before the tree grows. This is especially useful when working with high-dimensional datasets or when some features are highly correlated.

min-child-weight is a regularization parameter that controls the minimum sum of instance weights (or Hessians) needed in a child node before further splitting is allowed. It essentially sets a threshold on how much “training data mass” must exist in a branch, helping to prevent overfitting by stopping overly specific splits on small, noisy subsets of data. Higher values make the model more conservative in that it requires larger, more statistically significant splits, while lower values allow the model to create deeper, more fine-grained splits. In regression, this value relates to the sum of Hessians; in classification, it is influenced by class probabilities.

4. Application with the Volatility of Bitcoin

The volatility of Bitcoin has been the subject of extensive academic research, indicating that it exhibits significantly higher fluctuations compared to traditional asset classes like stocks and commodities. Empirical studies have consistently shown that Bitcoin’s price volatility is notably distinct due to factors including speculative trading, market sentiment, and external economic events. Some studies emphasize the asymmetric nature of Bitcoin’s volatility, whereby positive shocks exert a more substantial influence on price fluctuations than negative shocks of equal magnitude (Bouri et al., 2023; Tunahan et al., 2020). Additionally, Bitcoin options display similar implied volatility patterns to traditional financial markets, highlighting its speculative nature and susceptibility to shifts in investor sentiment (Zulfiqar & Gulzar, 2021). Various econometric models, particularly GARCH-type models, have been utilized to capture the inherent volatility dynamics of Bitcoin, revealing that trading volume and market sentiment are crucial determinants of Bitcoin’s pricing behavior (Jalal et al., 2020; Sapuric et al., 2022). The increase in Bitcoin’s popularity and its integration into financial markets has led to heightened volatility spillovers to other assets, reflecting its evolving role as both an investment vehicle and a speculative bubble (Bouri et al., 2018; Melawat & Gunarsih, 2023). Furthermore, the onset of the COVID-19 pandemic intensified volatility across cryptocurrency markets, demonstrating how macroeconomic uncertainties can impact Bitcoin’s trading patterns (Karim et al., 2014; Maghyereh & Abdoh, 2022). Overall, the literature indicates that Bitcoin’s volatility arises from a complex interplay of market factors, including its relatively young market status, regulatory developments, and global economic contexts, making it a focal point for research within financial economics.

4.1. Data

The daily Bitcoin data used is available at https://fr.investing.com/crypto/bitcoin/historical-data (accessed on 28 August 2025). The price series is between 1 January 2017 and 31 January 2024. The Bitcoin data series $p_t$ are transformed into returns using $r_t=log{p}_t-log{p}_{t-1}$.

The series of Bitcoin returns $r_t$ is shown in Figure 1. Between 2017 and 2018, the line shows high-frequency oscillations with occasional sharp spikes, suggesting periods of heightened volatility. In 2019, similar dense oscillations continue, though volatility appears slightly reduced compared to 2017. In early 2020, a sharp, pronounced downward spike occurs, likely corresponding to a significant market event (possibly the COVID-19 crash if this is financial data like Bitcoin or stock returns). Between 2021 and 2023, the returns continue fluctuating with relatively consistent volatility, though the frequency of extreme spikes appears somewhat diminished.

Some basic statistics are summarized in

Table 1. Basic summary statistics of the returns.

Statistics	Bitcoin Returns
Minimum	$-0.2160$
Average	$0.0006$
Maximum	$0.0988$
Variance	$0.0003$
Standard deviation	$0.0170$
Skewness	$-0.8287$
Kurtosis	$12.8550$

.

The largest single-day loss in the dataset is about −21.60%, showing that Bitcoin can have very sharp drops. On average, Bitcoin’s daily return is a small positive value of roughly 0.063%, indicating slight upward drift over the period. The largest single-day gain is about 9.88%, which is much smaller in magnitude than the worst loss. The variability of returns is 0.0003, a measure of dispersion in squared units. The typical daily fluctuation is about 1.7%, reflecting high volatility. The skewness is −0.8287, meaning the distribution of returns is moderately left-skewed. Large negative returns are more frequent or severe than large positive ones. At 12.855, the kurtosis is far above the normal distribution’s value of 3, indicating very heavy tails. Extreme returns (both gains and losses) occur much more often than in a normal distribution. Overall, the numbers suggest Bitcoin returns are volatile, prone to large swings, and have a distribution with a bias toward severe losses and frequent extreme movements.

4.2. Tuning Hyperparameters with Bayesian Optimization

We used the GP with exponential kernel during the optimization. The ranges of hyperparameters used respectively with their initializations are in

Table 2. Initializations and hyperparameter ranges.

Hyperparameters	Initialization	Ranges During Optimization
eta	$runif(5,0.001,0.3)$	$c(0.001,0.3)$
max-depth	$sample(3$:$15,5,replace=TRUE)$	$c(3,15)$
subsample	$runif(5,0.5,1)$	$c(0.5,1)$
colsample-bytree	$runif(5,0.1,1)$	$c(0.1,1)$
min-child-weight	$sample(1$:$20,5,replace=TRUE)$	$c(1,20)$

. The choices of the space for the hyperparameters were taken from (Verma, 2024).

In Table 2, $runif(b,c,a)$ denotes the uniform distribution between a and b in steps of c. $c(a,b)$ denotes minimum and maximum margins of a hyperparameter. $sample(a$:$b,c,replace=TRUE)$ creates a random sequence of numbers between a and b in steps of c. Replace being equal to TRUE means that the values are replaced if several sequences are drawn.

The range defines the search space that the Bayesian optimization will explore, while the initializations are the random evaluation of a number of initial points before the Bayesian optimization begins using the GP to guide the search. At each iteration, cross-validation based on time series splitting was performed.

After the optimization,

Table 3. The values of hyperparameters are related to their respective acquisition functions and the best RMSE.

Hyperparameter	EI	POI	UCB
eta	$0.2020$	$0.0697$	$0.1026$
max-depth	3	4	4
subsample	$0.7003$	$0.7278$	$0.8393$
colsample-bytree	1	$0.7046$	$0.6788$
min-child-weight	10	6	4
RMSE	$0.0170$	$0.0170$	$0.0170$

gives the optimal hyperparameter values with the acquisition functions used and the best root mean squared error (RMSE). RMSE is a common metric for evaluating models that measures the average magnitude of prediction errors, giving higher weight to larger errors due to squaring. It is calculated by taking the square root of the mean of the squared differences between predicted values and actual values. Because the errors are squared before averaging, RMSE penalizes large deviations more severely, making it useful when large mistakes are especially undesirable. The result is expressed in the same units as the target variable, which makes it more interpretable than some other metrics, though it can be sensitive to outliers.

4.3. Graphical Representation of Various Results

The three figures (Figure 2—EI; Figure 3—UCB; Figure 4—POI) all visualize the behavior of different Bayesian Optimization acquisition functions when tuning hyperparameters for XGBoost. Each figure includes the hyperparameter search space, optimization convergence, and the resulting learning curve. Although the structure of the plots is consistent, the outcomes and optimization behaviors differ noticeably.

Starting with the explored hyperparameter spaces, the EI and UCB acquisition functions (Figure 2 and Figure 3) explore a much wider and more uniform grid across the eta and max-depth dimensions. Their plots show a broader scatter of evaluated points, reflecting a tendency to explore more. In contrast, POI (Figure 4) focuses its search more tightly in a specific region, particularly at lower eta values and mid-to-lower max-depth levels. This tighter clustering implies that POI (Figure 4) tends to exploit early and refine in a focused region, while the UCB and EI (Figure 3 and Figure 2) prioritize broader exploration.

Looking at the color scale of RMSE scores, the POI (Figure 4) plot uses a much narrower and higher-performing score range (with RMSE values from around −0.02 to −0.05). On the other hand, both the EI and UCB (Figure 2 and Figure 3) show RMSE values reaching as low as −0.4, but this is due to using a different scaling, likely to represent greater error magnitudes. The key difference is that POI’s plot (Figure 4) shows consistently better scores (smaller errors), suggesting a more effective optimization in that specific range.

In the 3D hyperparameter space plots, the same trends persist. POI (Figure 4) shows its best results clustered in a narrow zone, especially at lower eta and higher subsample values. The UCB and EI (Figure 3 and Figure 2) have a more scattered spread, with the UCB (Figure 3) displaying several outlier points—evidence of aggressive exploration, which sometimes leads to poor-performing configurations.

The convergence plots offer some of the clearest distinctions. POI (Figure 4) converges the fastest and most smoothly, reaching a low RMSE within just a few iterations and staying stable afterward. The EI (Figure 2) shows one major dip (possibly an early good point) and then flattens, indicating slightly slower convergence. The UCB (Figure 3), however, is the most erratic. Its convergence curve fluctuates sharply several times, which reflects instability and inconsistent improvement. This instability, while potentially leading to better global optima, makes the UCB (Figure 3) less reliable for fast or resource-constrained tuning.

Finally, examining the XGBoost learning curves, all three acquisition functions eventually achieve low RMSE values for both training and test sets, suggesting successful model training. However, POI (Figure 4) achieves this with the smoothest and steepest descent, stabilizing around iteration 25 and maintaining close performance between training and test curves. The EI (Figure 2) also converges well, though over a slightly longer horizon. The UCB curve (Figure 3) takes the longest to settle, needing almost 70 iterations, but still manages to reach comparable final performance.

In summary, POI (Figure 4) demonstrates the most efficient and stable convergence, exploiting early good results and focusing its search effectively. The EI (Figure 2) represents a middle ground, balancing exploration and exploitation. The UCB (Figure 3) is the most exploratory and erratic, showing more variation in optimization quality, which could be beneficial in avoiding local minima but comes at the cost of stability and speed.

5. Conclusions

This study focused on empirically calibrating the XGBoost model for predicting Bitcoin volatility using Bayesian optimization to adjust the hyperparameters. The explanatory variables included technical indicators, historical volatility measures, and exogenous factors related to the cryptocurrency market. This allowed us to capture the complexity and specific dynamics of this volatile financial asset.

The main innovation lies in applying a Bayesian optimization method based on a Gaussian process, which intelligently and efficiently explores the hyperparameter space. This reduces computational cost while improving the model’s predictive accuracy. This approach optimized performance in terms of root mean square error (RMSE) and preserved the robustness of the model by avoiding overfitting thanks to the dynamic management of the exploration–exploitation balance during the optimization process.

The results highlight the importance of adopting advanced optimization techniques in financial time series modeling, especially for highly volatile assets like Bitcoin. By strengthening the model’s predictive capacity and generalization, this approach opens up promising prospects for risk management and decision-making in digital financial markets.

The lack of interpretability of machine learning techniques regarding features and stylized facts, as seen in Bitcoin volatility, is not a barrier to its use when forecasting accuracy is the focus. Work on the interpretability of machine learning techniques continues in various fields.