An Efficient and Accurate Random Forest Node-Splitting Algorithm Based on Dynamic Bayesian Methods

Abstract

Random Forests are powerful machine learning models widely applied in classification and regression tasks due to their robust predictive performance. Nevertheless, traditional Random Forests face computational challenges during tree construction, particularly in high-dimensional data or on resource-constrained devices. In this paper, a novel node-splitting algorithm, BayesSplit, is proposed to accelerate decision tree construction via a Bayesian-based impurity estimation framework. BayesSplit treats impurity reduction as a Bernoulli event with Beta-conjugate priors for each split point and incorporates two main strategies. First, Dynamic Posterior Parameter Refinement updates the Beta parameters based on observed impurity reductions in batch iterations. Second, Posterior-Derived Confidence Bounding establishes statistical confidence intervals, efficiently filtering out suboptimal splits. Theoretical analysis demonstrates that BayesSplit converges to optimal splits with high probability, while experimental results show up to a 95% reduction in training time compared to baselines and maintains or exceeds generalization performance. Compared to the state-of-the-art MABSplit, BayesSplit achieves similar accuracy on classification tasks and reduces regression training time by 20–70% with lower MSEs. Furthermore, BayesSplit enhances feature importance stability by up to 40%, making it particularly suitable for deployment in computationally constrained environments.

1. Introduction

It is widely known that the performance of multiple combined classifiers far exceeds that of using any single classifier independently [1]. Tree ensembles such as Random Forest (RF) [2] have achieved impressive success across various classification and regression tasks. RF builds a number of decision trees in the training process, which contributes to the final outcome through an aggregation method known as bagging [3]. Each decision tree is built using a bootstrap sample of the data and random subsets of features at each split. The predictions from decision trees are combined to produce the final output of RF, delivering high accuracy and robustness across different datasets and outperforming single classifiers, especially on complex predictive tasks [4].

RF has established itself as an indispensable tool in data mining, with applications covering bioinformatics [5], image processing [6], financial analytics [7], and natural language processing [8]. However, traditional RF algorithms face substantial challenges when handling high-dimensional data or running on hardware-constrained devices such as smartphones and Internet-of-Things (IoT) systems [9]. In these scenarios, the traditional algorithms often exhibit lower computational efficiency and reduced predictive accuracy. These limitations become even more pronounced due to the rapidly evolving computational landscape, where edge computing and IoT devices demand faster and more reliable algorithms [10].

To address these computational demands, researchers have explored various optimization techniques, particularly those focusing on decision-tree construction [11]. Decision trees represent the relationship between features and target variables through a hierarchical structure of conjunctive conditions, where each internal node corresponds to a feature and a threshold. In RF, finding the optimal split by identifying the best feature–threshold pair is crucial for both computational efficiency and predictive accuracy [12]. Accordingly, optimizing the node splitting process can significantly improve the overall performance of RF [13].

The traditional RF algorithms determine optimal splits by exhaustively searching through all possible splitting points, resulting in a computational complexity of $O\left(N\right)$ for each split, where$N$is the number of instances. They are computationally heavy and inefficient. To address this problem, algorithms like XGBoost [14] and LightGBM [15] have employed histogram-based techniques to reduce the complexity to $O\left(B\right)$, where $B\ll N$, by grouping feature values into discrete bins. While histogram-based methods can substantially accelerate training, they remain wasteful because they allocate the same computational effort to all features, including those that are not particularly informative.

1.1. Related Works

In 2001, Breiman [2] introduced the RF algorithm by combining the bagging technique, the random subspace method, and the Classification and Regression Trees (CART) algorithm. With the rapid growth of information technology, datasets have expanded dramatically in both size and complexity. Traditional RF algorithms face significant challenges meeting time and computational demands [16]. Thus, enhancing RFs to handle large datasets has become a major focus in both research and industry [17].

Recent advancements have emerged in algorithmic optimizations and parallel computing techniques. Yates and Islam [9] developed FastForest, which incorporates Subbagging to reduce the size of bootstrap samples, Logarithmic Split-Point Sampling (LSPS) to decrease the computational cost of node splits, and Dynamic Restricted Subspacing (DRS) to adjust the feature subset size during feature selection. These optimizations collectively reduce the data processed by each tree, thereby accelerating node splitting. In the context of incremental learning, Domingos and Hulten [18] introduced Hoeffding Tree (HT) in Very Fast Decision Tree (VFDT), which utilizes the Hoeffding Bound to guide the choice of decision nodes, ensuring a rapid and effective approximation of the optimal decision with limited samples. The Extremely Fast Decision Tree (EFDT) algorithm [19] builds upon HT by introducing Hoeffding Anytime Tree (HATT), which continuously monitors and updates splits as more data becomes available. Both HT and HATT are incremental decision tree learning techniques, particularly suited for big data environments and data stream applications.

Parallel computing frameworks have also been leveraged to enhance decision tree algorithms. Mu et al. [20] proposed a parallel decision tree algorithm based on MapReduce, using Pearson’s correlation coefficient for optimal split selection. Xu [21] introduced an improved RF algorithm based on Spark, leveraging the Fayyad boundary point principle for efficient feature discretization. Yin et al. [22] presented a fast parallel RF algorithm on Spark, integrating a modified Gini coefficient to reduce feature redundancy and applying an approximate equal-frequency binning method for split optimization.

Although these algorithms have sped up the training process for decision tree ensembles, most of them still rely on exhaustive searches to find the optimal split at each node, which can be costly for large datasets. MABSplit presents a cutting-edge solution to overcome this bottleneck [23]. At its core, MABSplit treats each candidate feature–threshold pair as an arm in a multi-armed bandit (MAB) problem, aiming to efficiently identify the optimal split by estimating each candidate’s impurity reduction through sampling. Specifically, the algorithm iteratively samples batches of data points, updates confidence intervals of impurity reduction estimates for each candidate split, and eliminates splits whose lower confidence bound exceeds the upper confidence bound of the currently best-performing candidate. This adaptive procedure significantly reduces computational complexity from linear to logarithmic with respect to the number of samples, achieving substantial speed-ups without compromising predictive accuracy.

Despite its considerable strengths, MABSplit has certain aspects that could be further enhanced:

Lack of Memory Mechanism: MABSplit lacks some related mechanisms to leverage information gathered from previously explored splits. The valuable information from earlier computations is not utilized to improve subsequent split evaluations. Consequently, the exploration in the early phase would be less efficient.

Lower Efficiency for Similar Candidates: The computational efficiency of MABSplit relies on sufficient heterogeneity among the true impurity reductions of different feature–threshold pairs. When most splits have similar impurity reductions (as in highly symmetric datasets), MABSplit fails to achieve its promised logarithmic sample complexity and reduces to a batched version of the naïve approach, resulting in no significant speed advantage.

Reduced Accuracy with Limited Samples: MABSplit employs confidence intervals to estimate split quality, which are derived from some statistical properties of impurity measures such as Gini impurity and entropy. However, these estimates may become less reliable when sample sizes are small or probabilities approach extreme values.

1.2. Our Contributions

In this paper, we introduce BayesSplit, a novel node-splitting algorithm that extends MABSplit to improve the computational efficiency and predictive accuracy of RFs. BayesSplit treats the probability of impurity reduction as a Beta posterior distribution, which is iteratively refined based on batched observations. Based on this, posterior confidence intervals are used to adaptively select splits most likely to maximize impurity reduction, ensuring statistically robust and data-driven tree construction. Our key contributions include:

(1) A Novel Bayesian-Based Impurity Estimation Framework

A Bayesian framework is developed that treats each split’s impurity reduction as a random event with an unknown success probability. The framework initializes uninformative Beta priors that evolve into informative posteriors as observations accumulate. After each batch of samples is evaluated, the Beta parameters of candidate splits are updated according to the observed impurity reductions. These posterior distributions are used to derive confidence intervals that balance exploration of uncertain splits with exploitation of promising ones. By comparing the confidence bounds of each split across all candidates, BayesSplit eliminates suboptimal splits while allocating computational resources to the most promising candidates. This Bayesian-based Impurity Estimation Framework naturally accommodates prior knowledge and new data, enabling robust decision-making and faster convergence to optimal splits.

(2) Two Bayesian Optimization Strategies to Achieve High Computational Efficiency

Dynamic Posterior Parameter Refinement: Drawing inspiration from Thompson Sampling (TS), this strategy treats each split’s impurity reduction as a reward signal to update the parameters of Beta distributions. After a batch is sampled, whether each split reduced impurity is evaluated, $\alpha$is incremented when impurity decreases and $\beta$when it does not. This process creates a memory mechanism to accumulate evidence across iterations.

Posterior-Derived Confidence Bounding: This strategy uses each split’s posterior Beta distribution to establish confidence intervals. Rather than relying on frequentist approximations, the confidence bounds are set directly from the posterior parameters, enabling more accurate uncertainty quantification across diverse data conditions.

Our experiment results demonstrate that BayesSplit further minimizes quantization errors and more accurately captures the underlying data distribution than existing approaches.

2. Algorithmic Background

To facilitate understanding of the subsequent technical content, we summarize the key notation used in this paper.

Table 1. Notations in BayesSplit algorithm and analysis.

Symbol	Meaning
$D$	$\mathrm{dataset} \mathrm{with} N$  $\mathrm{data} \mathrm{points} {\left\{\left(x_i,y_i\right)\right\}}_{i=1}^N$
$N$	total number of data points
$x_i$	$i$-th feature vector
$y_i$	$\mathrm{target} \mathrm{value} \mathrm{corresponding} \mathrm{to}{ x}_i$
$R_{\nu }$	region in feature space corresponding to node$\nu$
$N_n\left(\nu \right)$	number of samples falling into region$R_{\nu }$
$\mathcal{M}$	set of features considered for splitting
$\mathcal{T}$	set of possible thresholds for each feature
$\left(f,t\right)$	feature–threshold pair (candidate split)
$I\left(·\right)$	impurity measure (Gini index, entropy, or MSE)
${\mu }_{ft}$	$\mathrm{optimization} \mathrm{objective} \mathrm{for} \mathrm{split} \left(f,t\right)$
${\widehat{\mu }}_{ft}$	$\mathrm{point} \mathrm{estimate} \mathrm{of} {\mu }_{ft}$from histogram
$n^{\prime }$	number of subsamples for confidence interval estimation
$C_{ft}\left(n\prime ,\delta \right)$	confidence interval width with confidence level$\left(1-\delta \right)$
$S_{solution}$	set of candidate splits maintained by algorithm
$\alpha ,\beta$	parameters of Beta distribution
$g_{ft}$	impurity gain
${\widehat{\mu }}_{post,ft}$	$\mathrm{posterior} \mathrm{mean}: {\widehat{\mu }}_{post,ft}=\alpha /\left(\alpha +\beta \right)$
$C_{post,ft}$	posterior confidence interval half-width
$B$	batch size for data sampling
${\theta }^{*}$	$\mathrm{true} \mathrm{probability} \mathrm{that} \mathrm{split} \left(f,t\right)$reduces impurity

provides a list of symbols and their meanings, serving as a reference for the algorithmic development and theoretical analysis of BayesSplit.

2.1. Node-Splitting Description in RFs and Decision Trees

An RF is composed of multiple classification or regression trees, where each tree$T$maps the feature space to the response. Consider a dataset$D$with$N$data points ${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^N$, where${ x}_{i }$is the$i$-th feature vector and $y_i$is the corresponding target. Each tree in an RF is built independently on a bootstrapped dataset $D^T$from the original data$D$, while a random subset of features$\mathcal{M}$is considered at each node.

In the decision tree node-splitting process, let $R_{\nu }$represent the region in the feature space corresponding to node$\nu$ (typically a hyper-rectangle) [24]. Using the pair$\left(f,t\right)$to split node$\nu$, $R_{\nu }$is partitioned into two subregions${ R}_{\nu ,left }$and $R_{\nu ,right }$, corresponding to the left and right child nodes of node$\nu$. For a node $\nu$ in decision tree $T$, $N_n\left(\nu \right)=\left|\left\{i\in D^T:x_i\in R_{\nu }\right\}\right|$denotes the number of samples falling into $R_{\nu }$. Finding the optimal split, i.e., determining the best pair $\left(f_b,t_b\right)$to maximize node-splitting effectiveness, is accomplished by maximizing the reduction in label impurity:

$$\underset{f\in \mathcal{M},t\in \mathcal{T}}{\mathrm{argmin}}\left\{{\displaystyle \frac{N_n\left({\nu }_{left}\right)}{N_n\left(\nu \right)}}I\left({\nu }_{left}\right)+{\displaystyle \frac{N_n\left({\nu }_{right}\right)}{N_n\left(\nu \right)}}I\left({\nu }_{right}\right)-I\left(\nu \right)\right\}$$

(1)

where$N_n\left({\nu }_{left}\right)$and$N_n\left({\nu }_{right}\right)$represent the number of samples in the left and right child nodes, respectively, $I\left(·\right)$represents the impurity measure, and$\mathcal{T}$denotes the permissible thresholds for feature $f$. Popular impurity measures include the Gini index and entropy for classification, as well as the mean-squared error (MSE) for regression [25]:

$$Gini=1-\sum _{k=1}^K{p}_k^2$$

(2)

$$E=-\sum _{k=1}^K{p}_k{\log }_2{p}_k$$

(3)

$$MSE={\displaystyle \frac{1}{N_n\left(v\right)}}\sum _{i:x_i\in R_v}{\left(y_i-\overline{y}\right)}^2$$

(4)

where $K$ represents the no. of classes in the target variable, $p_k$is the proportion of samples at node belonging to class$k$, and $\overline{y}$is calculated as $\overline{y}={\displaystyle \frac{1}{N_n\left(v\right)}}{\sum }_{i:x_i\in R_v}{y}_i$. In Equation (1), $I\left(\nu \right)$denotes the impurity of node$\nu$ before splitting and does not depend on the split feature $f$ or threshold $t$. Since $I\left(\nu \right)$ does not affect the minimization process, we simplify Equations (1)–(5) for a direct evaluation of the split effect, written as follows:

$$\underset{f\in \mathcal{M},t\in \mathcal{T}}{\mathrm{argmin}}\left\{{\displaystyle \frac{N_n\left({\nu }_{left}\right)}{N_n\left(\nu \right)}}I\left({\nu }_{left}\right)+{\displaystyle \frac{N_n\left({\nu }_{right}\right)}{N_n\left(\nu \right)}}I\left({\nu }_{right}\right)\right\}$$

(5)

We define ${\mu }_{ft}={\displaystyle \frac{N_n\left({\nu }_{left}\right)}{N_n\left(\nu \right)}}I\left({\nu }_{left}\right)+{\displaystyle \frac{N_n\left({\nu }_{right}\right)}{N_n\left(\nu \right)}}I\left({\nu }_{right}\right)$as the optimization objective. Note that lower values of ${\mu }_{ft}$correspond to higher impurity reductions, while higher values of ${\mu }_{ft}$correspond to lower impurity reductions.

2.2. Confidence Interval Estimation in MABSplit

Since computing ${\mu }_{ft}$exactly requires a full pass over the data, MABSplit draws $n^{\prime }$samples, ${\left\{\left(X_i,Y_i\right)\right\}}_{i=1}^{n^{\prime }}$, to construct point estimates ${\widehat{\mu }}_{ft}$and confidence intervals for impurity reduction. At the core of this approach is the delta method, which transforms the empirical estimates of class distributions into reliable estimates of impurity measures.

Let$p_{L,k}$and $p_{R,k}$represent the proportion of the full data points in class$k$and each of the two subsets created by the split $\left(f,t\right)$. For a given split $\left(f,t\right),$ MABSplit constructs empirical estimates ${\widehat{p}}_{L,k }$and ${\widehat{p}}_{R,k}$represents the proportion of class$k$samples in the left and right child nodes based on the $n^{\prime }$subsamples, respectively:

$${\widehat{p}}_{L,k }={\displaystyle \frac{1}{n^{\prime }}}\sum _{i=1}^{n^{\prime }}I\left(X_{if}<t,{ Y}_i=k\right)\hspace{1em}\hspace{1em}{\widehat{p}}_{R,k }={\displaystyle \frac{1}{n^{\prime }}}\sum _{i=1}^{n^{\prime }}I\left(X_{if}\ge t,{ Y}_i=k\right)$$

(6)

These estimates jointly follow a multinomial distribution with parameters $\left(n^{\prime },p\right)$, where $p={\left(p_{L,1},\dots ,p_{L,K},p_{R,1},\dots ,p_{R,K}\right)}^T$. By the Central Limit Theorem (CLT), we obtain the following:

$$\sqrt{n^{\prime }}\left(\widehat{\gamma }-\gamma \right)\stackrel{d}{\to }\mathcal{N}\left(0,Q_{\gamma }\right)$$

(7)

where $\gamma ={\left(p_{L,1},\dots ,p_{L,K},p_{R,1},\dots ,p_{R,K-1}\right)}^T$ and $\widehat{\gamma }={\left({\widehat{p}}_{L,1},\dots ,{\widehat{p}}_{L,K-1},{\widehat{p}}_{R,1},\dots ,{\widehat{p}}_{R,K-1}\right)}^T$. $Q_{\gamma }$ is the corresponding covariance matrix. Specifically, ${\mu }_{ft}$ could be written in terms of $\gamma$ for the impurity metrics (e.g., Gini or entropy). Let $\nabla {\mu }_{ft}\left(\gamma \right)$ be the derivative of ${\mu }_{ft}$ with respect to $\gamma$. Applying the delta method, we obtain the following:

$$\sqrt{n^{\prime }}\left({\widehat{\mu }}_{ft}\left(\gamma \right)-{\mu }_{ft}\left(\gamma \right)\right)\stackrel{d}{\to }\mathcal{N}\left(0,{\nabla {\mu }_{ft}\left(\gamma \right)}^T{Q}_{\gamma }\nabla {\mu }_{ft}\left(\gamma \right) \right)$$

(8)

This allows MABSplit to construct $\left(1-\delta \right)$ confidence intervals scale as $C_{ft}\left(n\prime ,\delta \right)=O\left(\sqrt{\log \left(1/\delta \right)/n^{\prime }}\right)$, these confidence intervals are asymptotically valid as $n^{\prime }\to \mathrm{\infty }$. In MABSplit, each batch of data points updates ${\widehat{p}}_{L,k}$and ${\widehat{p}}_{R,k}$, which in turn refine the point estimates ${\widehat{\mu }}_{ft}$and their corresponding confidence intervals.

While the delta method provides a theoretically sound framework for constructing confidence intervals, its practical reliability depends on the validity of asymptotic assumptions guaranteed by the CLT. Specifically, Equation (8) shows that the width of the confidence intervals depends on the sample size through a$\sqrt{1/n^{\prime }}$scaling. In the early stages of MABSplit, where only small batches of data are available, this approximation may break down. For example, when the class proportions approach 0, the derivative of impurity metrics such as entropy becomes unstable, leading to large variance and unreliable confidence intervals.

In summary, MABSplit exhibits reduced accuracy with limited samples, as statistical estimation under small-sample conditions can introduce substantial variance in impurity estimates.

3. BayesSplit: A Bayesian Node-Splitting Algorithm

3.1. Overview of the Framework

BayesSplit treats impurity reduction as a Bernoulli event with Beta-conjugate priors, as shown in Figure 1. The framework begins with uninformative priors, which are gradually updated into informative posteriors as observations accumulate. After each batch of samples is evaluated, the Beta parameters of candidate splits are updated based on the observed impurity reductions.

The framework consists of three key components: (1) InitializePosterior establishes posterior distributions for each candidate split based on initial impurity evaluations; (2) UpdatePosteriorAndBounds implements the Dynamic Posterior Parameter Refinement strategy, updating beliefs about split effectiveness based on empirical observations; and (3) Filter uses Posterior-Derived Confidence Bounds to eliminate splits that are demonstrably suboptimal, focusing resources on promising candidates.

The precise splitting approach BayesSplit is outlined in Algorithm 1. We preprocess the input data to identify candidate splits $\left(f,t\right)$, or arms, for each feature$f$. All potential solutions to Equation (5) are tracked by maintaining a set $S_{\mathit{solution}}$, which initially includes every candidate arm$\left\{\left(f,t\right)\right\}$.

Algorithm 1 BayesSplit $(\mathcal{X},\mathcal{M},\mathcal{T},I\left(\cdot \right),B)$

1: $S_{\mathit{solution}}\leftarrow \{\left(f,t\right),\forall f\in \mathcal{M},\forall t\in \mathcal{T}\}$ // Set of potential solutions to Equation (5)   2: $n_{used}\leftarrow 0$ // Number of data points sampled   3: for all $(f,t)\in S_{solution}$ do   4:  ${\widehat{\mu }}_{ft}\leftarrow \mathrm{\infty },C_{ft}\leftarrow \mathrm{\infty }$ // Initialize mean and CI for each arm   5: end for   6: for all $f\in \mathcal{M}$ do   7:  Create empty histogram $h_f$ with $\left|\mathcal{T}\right|=\mathrm{T}$ equally spaced bins   8: end for   9: InitializePosterior // Set up posterior distributions 10: while $n_{used}<n \mathrm{and} \left|S_{solution}\right|>1$ do 11:  Draw a batch sample ${\mathcal{X}}_{batch}$ of size B with replacement from $\mathcal{X}$ 12:  for all unique f in Ssolution do 13:   for all x in ${\mathcal{X}}_{\mathit{batch}}$ do 14:    Insert xf into histogram hf // Update histograms with sampled data 15:   end for 16:  end for 17:  for all $(f,t)\in S_{\mathit{solution}}$ do 18:   Update ${\widehat{\mu }}_{ft}\mathrm{and}{C}_{ft}$ based on histogram hf 19:  end for 20:  UpdatePosteriorAndBounds // Adjust posterior and refine bounds 21:  $S_{\mathit{solution}}\leftarrow \left\{\left(f,t\right):{\widehat{\mu }}_{post,ft}-C_{post,ft}\le {\min }_{f,t}\left({\widehat{\mu }}_{post,ft}+C_{post,ft}\right)\right\}$ // Retain promising splits 22:  $n_{used}\leftarrow n_{used}+B$ 23: end while 24: if $\left|S_{\mathit{solution}}\right|=1$ then 25:   return $\left(f_b,t_b\right)\in S_{\mathit{solution}}$ 26: else 27:  Compute ${\mu }_{ft}$ exactly for all $\left(f,t\right)\in S_{\mathit{solution}}$ 28:  return $\left(f_b,t_b\right)={\arg \min }_{\left(f,t\right)\in S_{\mathit{solution}}}{\mu }_{ft}$

Algorithm 1, as depicted in Figure 1, begins with candidate splits at the top and produces the optimal split $\left(f_b,t_b\right)$at the bottom. In each iteration, it samples a batch of data points and updates histograms to refine impurity estimates. When evaluating a candidate split, BayesSplit calculates the impurity gain $g_{ft}$and updates the posterior Beta parameters, which are then used to compute confidence intervals for each split. The algorithm renews $S_{\mathit{solution}}$until convergence by eliminating suboptimal splits.

This Bayesian framework fundamentally distinguishes BayesSplit from previous approaches by providing adaptive, accurate, and computationally efficient node-splitting decisions, thus enhancing both computational efficiency and predictive accuracy of RFs.

3.2. Dynamic Posterior Parameter Refinement

BayesSplit is inspired by TS, a seminal Bayesian heuristic for solving MAB problems [26]. TS has shown robust performance compared to alternatives like Upper Confidence Bound (UCB) algorithms [27]. It is widely applied in Bernoulli bandit problems, where rewards are binary, with a value of 1 for success and 0 for failure. In BayesSplit, “success” specifically corresponds to a split that successfully reduces node impurity. To clarify the application of TS within BayesSplit, we begin by briefly reviewing the Beta-Bernoulli Bandit.

Beta distributions serve as natural representations of uncertainty for Bernoulli trials due to their conjugacy properties, meaning that if the prior is Beta$\left(\alpha ,\beta \right)$, the posterior after observing outcomes from Bernoulli trials remains a Beta distribution with updated parameters. The probability density function (PDF) of the Beta distribution on the interval$\left[0, 1\right]$ is given by the following:

$$f\left(x\right)={\displaystyle \frac{1}{B\left(\alpha ,\beta \right)}}{x}^{\alpha -1}{\left(1-x\right)}^{\beta -1}$$

(9)

where $B\left(\alpha ,\beta \right)$ is the Beta function, defined as follows:

$$B\left(\alpha ,\beta \right)={\int }_0^1{x}^{\alpha -1}{\left(1-x\right)}^{\beta -1}dx={\displaystyle \frac{\mathsf{\Gamma }\left(\alpha \right)\mathsf{\Gamma }\left(\beta \right)}{\mathsf{\Gamma }\left(\alpha +\beta \right)}}$$

(10)

Here, $\mathsf{\Gamma }$ denotes the gamma function, generalizing factorials.

Suppose there are$A$actions, and when an action $a\in \left\{1,\dots ,A\right\}$is played, it yields a reward of 1 with probability${\theta }_a$and a reward of 0 with probability$1-{\theta }_a$[28]. Each${\theta }_a$represents the probability of success, or the mean reward. Once an action$a$is chosen, the resulting reward $r_1\in \left\{\mathrm{0,1}\right\}$is generated with success probability$P\left(r_1=1|a\right)={\theta }_a$. We assume an independent prior belief for each${ \theta }_a$, where these priors follow a Beta distribution with parameters $\alpha =\left({\alpha }_1,\dots ,{\alpha }_A\right)$ and $\beta =\left({\beta }_1,\dots ,{\beta }_A\right)$. For each action$a$, the prior PDF of ${\theta }_a$is calculated as follows:

$$p\left({\theta }_a\right)={\displaystyle \frac{\mathsf{\Gamma }\left({\alpha }_a+{\beta }_a\right)}{\mathsf{\Gamma }\left({\alpha }_a\right)\mathsf{\Gamma }\left({\beta }_a\right)}}{\theta }_a^{{\alpha }_a-1}{\left(1-{\theta }_a\right)}^{{\beta }_a-1}$$

(11)

If action$a$is chosen at step$t$, the parameters are updated based on the observed reward $r_t$. The posterior update for each action’s distribution follows the following simple rule:

$$\left({\alpha }_a,{\beta }_a\right)\leftarrow \left({\alpha }_a+r_t,{\beta }_a+\left(1-r_t\right)\right)$$

Initially, BayesSplit assigns a non-informative$\mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}\left(\mathrm{1,1}\right)$prior to each candidate split, representing uncertainty about split effectiveness. Before the main iterations begin, the algorithm performs a posterior initialization phase where each candidate split undergoes a preliminary impurity reduction evaluation. If this evaluation shows that a split reduces impurity (a success), its parameters update as $\alpha \leftarrow \alpha +1$; otherwise (a failure), as $\beta \leftarrow \beta +1$. This initialization step provides an informed starting point for the posterior distributions, leveraging global data characteristics to guide early exploration (see Algorithm 2). It ensures that all candidate splits begin without bias and are immediately updated based on observed impurity reductions. As more batches are processed, the influence of the prior becomes negligible, and split selection is determined by empirical evidence.

In each subsequent iteration, BayesSplit samples a new batch of data ${\mathcal{X}}_{batch }$to evaluate all candidate splits$\left(f,t\right)$in${ S}_{\mathit{solution}}$. This ongoing update process refines the value of ${\widehat{\mu }}_{ft}$, which represents the mean objective estimate computed from histograms. Using the current estimate, BayesSplit calculates the impurity gain $g_{ft}={\widehat{\mu }}_{ft}-{\widehat{\mu }}_{prev,ft}$between consecutive iterations. The posterior parameters are then updated based on this gain: specifically, when $g_{ft}<0$, confidence increases by incrementing$\alpha$; when $g_{ft}>0$, confidence decreases by incrementing $\beta$. This dynamic mechanism actively accumulates evidence across iterations, enhancing convergence efficiency (see Algorithm 3).

Algorithm 2 InitializePosterior $\left(\mathcal{X},\mathcal{H},I\left(\cdot \right),{\alpha }_1,{\beta }_1\right)$

Require: $\mathcal{X}$: input data; $\mathcal{H}$: list of histograms; $I\left(\cdot \right):$ impurity measure; ${\alpha }_1,{\beta }_1:$ non-informative prior parameters of Beta distribution Ensure: $\alpha ,\beta :$ initialized posterior parameters   1: $\mathrm{Initialize} \mathrm{success} \mathrm{counter} S_{ft}=0 \mathrm{and} \mathrm{failure} \mathrm{counter} F_{ft}=0 \mathrm{for} \mathrm{all} \left(f,t\right)\in S_{\mathit{solution}}$   2: for each feature f do // Iterate over all features   3:   for each bin b in hf do // Iterate over bins in histogram   4:     Compute impurity reduction Δ for splitting on bin b   5:     if $\Delta >0$ then // Check if split reduces impurity   6:       $S_{ft}=S_{ft}+1$   7:     else   8:       $F_{ft}=F_{ft}+1$   9:     end if 10:   end for 11: end for 12: $\alpha ={\alpha }_1+S_{ft },\hspace{1em}\beta ={\beta }_1+F_{ft}$ 13: return $\alpha ,\beta$

3.3. Posterior-Derived Confidence Bounding

Unlike the classical TS, which selects arms based on posterior sampling alone, BayesSplit uses posterior distributions to construct explicit confidence intervals for impurity reductions. Arms whose confidence intervals indicate potential optimality are retained for further exploration; others are eliminated early.

Confidence intervals in BayesSplit are derived directly from the statistical properties of the Beta distribution posterior, providing a theoretically sound basis for uncertainty quantification. The moment-generating function (MGF) of the Beta distribution can be expressed through a confluent hypergeometric function, written as follows:

$$\mathbb{E}\left[\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathsf{\lambda }\theta \right)\right]={}_{1}F_1\left(\mathsf{\alpha };\mathsf{\alpha }+\mathsf{\beta };\mathsf{\lambda }\right)=\sum _{j=0}^{\mathrm{\infty }}{\displaystyle \frac{\mathsf{\Gamma }\left(\alpha +j\right)\mathsf{\Gamma }\left(\alpha +\beta \right)}{\left(j!\right)\mathsf{\Gamma }\left(\alpha \right)\mathsf{\Gamma }\left(\alpha +\beta +j\right)}}{\mathsf{\lambda }}^j$$

(12)

From this, the $j^{th}$ raw moment of a Beta$\left(\alpha ,\beta \right)$random variable $\theta$is given by:

$$\mathbb{E}\left[{\theta }^j\right]={\displaystyle \frac{{\left(\alpha \right)}_j}{{\left(\alpha +\beta \right)}_j}}$$

(13)

where ${\left(x\right)}_j=x\left(x+1\right)\dots \left(x+j-1\right)={\displaystyle \frac{\mathsf{\Gamma }\left(x+j\right)}{\mathsf{\Gamma }\left(x\right)}}$, known as the Pochhammer symbol or rising factorial. The mean and variance are defined as follows:

$$\mathbb{E}\left[\theta \right]={\displaystyle \frac{\alpha }{\alpha +\beta }}\hspace{1em}\hspace{1em}\mathrm{V}\mathrm{a}\mathrm{r}\left[\theta \right]={\displaystyle \frac{\alpha \beta }{{\left(\alpha +\beta \right)}^2\left(\alpha +\beta +1\right)}}$$

(14)

For each candidate split $\left(f,t\right)$with posterior parameters $\alpha$and $\beta$, BayesSplit computes the posterior mean as ${\widehat{\mu }}_{post,ft}={\displaystyle \frac{\alpha }{\alpha +\beta }}$, which represents our current belief about the probability that the split reduces impurity based on all observed evidence. The half-width of the confidence interval $C_{post,ft}$is based on the standard deviation$\sqrt{\mathrm{V}\mathrm{a}\mathrm{r}\left[\theta \right]}$from the posterior Beta distribution, $C_{post,ft}=\mathrm{o}\left(\sqrt{{\displaystyle \frac{\alpha \beta }{{\left(\alpha +\beta \right)}^2\left(\alpha +\beta +1\right)}}}\right)$, which naturally decreases as more samples are accumulated.

Algorithm 3 UpdatePosteriorAndBounds $\left(\alpha , \beta ,{\widehat{\mu }}_{ft},S_{\mathit{solution}},B\right)$

$\begin{array}{cc}\mathbf{Require}:\hfill & \alpha , \beta :\mathrm{initialized} \mathrm{posterior} \mathrm{parameters} \mathrm{for} \mathrm{all} \left(f,t\right)\in S_{\mathit{solution}}\hfill \\ & {\widehat{\mu }}_{ft}:\mathrm{estimates} \mathrm{of} \mathrm{impurity} \mathrm{reductions} \mathrm{based} \mathrm{on} \mathrm{histogram} h_f\hfill \\ & B:\mathrm{batch} \mathrm{size}\hfill \\ \mathbf{Ensure}:\hfill & \mathrm{Updated} \alpha ,\beta ,{\widehat{\mu }}_{post,ft},C_{post,ft}\hfill \end{array}$   1: for all $\left(f,t\right)\in S_{\mathit{solution}}$ do   2:   Compute impurity gain $g_{ft}={\widehat{\mu }}_{ft}-{\widehat{\mu }}_{prev,ft}$ // Compute impurity gains from new batches   3:   if $g_{ft}<0$ then   4:     Set reduced flag $r_{ft}\leftarrow 1$   5:   else   6:     Set reduced flag $r_{ft}\leftarrow 0$   7:   end if   8:   Update posterior parameters:   9:     $\alpha =\alpha +r_{ft}$ // Increase confidence if split reduced impurity 10:     $\beta =\beta +\left(B-r_{ft}\right)$ // Decrease confidence if split increased impurity 11: end for 12: Update ${\widehat{\mu }}_{post,ft}\mathrm{and}{C}_{post,ft} \mathrm{using} \mathrm{updated} \alpha \mathrm{and} \beta$ 13: return Updated $\alpha ,\beta ,{\widehat{\mu }}_{post,ft},C_{post,ft}$

At each iteration, BayesSplit implements a filtering mechanism by retaining only those splits whose posterior lower confidence bound ${\widehat{\mu }}_{post,ft}-C_{post,ft}$falls below the upper confidence bound of the highest potential arm ${\min }_{f,t}\left({\widehat{\mu }}_{post,ft}+C_{post,ft}\right)$. This criterion ensures that only splits that are demonstrably suboptimal with high probability are removed from consideration. As sampling progresses, the confidence intervals narrow, enabling the algorithm to identify the optimal split while minimizing exploration of suboptimal splits. The iterative process continues until either a single optimal split remains or the algorithm reaches a predefined computational budget.

While both Dynamic Posterior Parameter Refinement and Posterior-Derived Confidence Bounding contribute to BayesSplit’s efficiency, their roles are complementary rather than independent. Dynamic Posterior Parameter Refinement is the primary driver of performance gains. It accumulates evidence across iterations to update split-quality estimates adaptively, addressing MABSplit’s lack of memory mechanism by leveraging historical data to inform future sampling. The accumulated evidence enables faster convergence to optimal splits, particularly when many feature–threshold pairs have similar impurity reductions, a scenario where MABSplit requires substantially more samples to distinguish among candidates. Posterior-Derived Confidence Bounding adds statistical robustness and accuracy under limited-sample conditions, since its confidence intervals are derived directly from the exact properties of the Beta distribution.

The synergy between the two strategies is crucial because the Dynamic Posterior Parameter Refinement provides increasingly informative posteriors that tighten confidence bounds, and the Posterior Derived Confidence Bounding ensures statistically sound elimination decisions, creating a feedback loop that focuses computational resources on the most competitive splits.

Implementation Details: In practice, sampling without replacement is utilized for efficiency, similar to MABSplit, achieving substantial computational savings without significantly impacting performance.

3.4. Convergence Analysis

In this section, we show that BayesSplit’s posterior estimation of impurity reduction for each feature–threshold pair converges to the true parameter as the number of observations increases. Based on these estimates, the algorithm eliminates suboptimal splits with high confidence, and the retained candidate converges to the optimal split with high probability. For any feature–threshold pair$\left(f,t\right),$ let ${\theta }^{*}$denote the true probability that split$\left(f,t\right)$successfully reduces node impurity. We treat the event of observing an impurity reduction as a Bernoulli trial with success probability ${\theta }^{*}$.

We make the following standard assumptions for Bayesian consistency:

1. The Bernoulli likelihood is different for different parameter values.

2. The prior Beta distribution assigns non-zero density to the true parameter ${\theta }^{*}$.

Lemma 1. 

For any feature–threshold pair $\left(f,t\right),$ given a sufficient number of samples$n^{\prime }$, the posterior estimate in BayesSplit converges to the true impurity reduction probability ${\theta }^{*}$. As$n^{\prime }\to \infty$, the posterior mean ${\widehat{\mu }}_{post,ft}$tends to ${\theta }^{*}$, and the posterior distribution concentrates its mass at ${\theta }^{*}$.

Proof. 

We use a conjugate Beta–Bernoulli model for inference. Suppose the prior for ${\theta }^{*}$is ${\theta }^{*}~Beta\left(\alpha ,\beta \right)$. After observing${ n}^{\prime }$independent impurity-reduction outcomes${ X}_1$,…,$X_{{\mathrm{n}}^{\prime }}$for split$\left(f,t\right)$, where $X\in \left\{\mathrm{0,1}\right\}$and it indicates whether impurity is reduced$\left(X=1\right)$or not $\left(X=0\right)$, the posterior distribution is as follows:

$${\theta }^{*}|{ X}_{1: n^{\prime }}~Beta \left({\alpha }^{\left(n^{\prime }\right)}, {\beta }^{\left(n^{\prime }\right)}\right)$$

with updated parameters, it is written as follows:

$${\alpha }^{\left(n^{\prime }\right)}=\alpha +\sum _{i=1}^{n^{\prime }}{X}_i,\hspace{1em}{\beta }^{\left(n^{\prime }\right)}=\beta +\sum _{i=1}^{n^{\prime }}\left(1-X_i\right)$$

The posterior mean at this stage is ${\widehat{\mu }}_{post,ft}={\displaystyle \frac{{\alpha }^{\left(n^{\prime }\right)}}{{\alpha }^{\left(n^{\prime }\right)}+{\beta }^{\left(n^{\prime }\right)}}}$. We can rewrite the posterior mean as follows:

$${\widehat{\mu }}_{post,ft}={\displaystyle \frac{\alpha +\beta }{\alpha +\beta +n^{\prime }}}·{\displaystyle \frac{\alpha }{\alpha +\beta }}+{\displaystyle \frac{n^{\prime }}{\alpha +\beta +n^{\prime }}}·{\displaystyle \frac{1}{n^{\prime }}}\sum _{i=1}^{n^{\prime }}{X}_i$$

(15)

As $n^{\prime }$grows large, the weight on the prior mean vanishes, while the weight on the sample mean approaches 1. By the Law of Large Numbers, the sample mean converges to $\mathbb{E}\left[X_i\right]={\theta }^{*}$. Therefore, ${\widehat{\mu }}_{post,ft}\to {\theta }^{* }$as $n^{\prime }\to \infty$.

In the Beta–Bernoulli model, the posterior variance of the random variable$\theta$given $X_{1:n^{\prime }}$is written as follows:

$$\mathrm{V}\mathrm{a}\mathrm{r}\left[\theta |X_{1:n^{\prime }}\right]={\displaystyle \frac{{\alpha }^{\left(n^{\prime }\right)}{\beta }^{\left(n^{\prime }\right)}}{{\left({\alpha }^{\left(n^{\prime }\right)}+{\beta }^{\left(n^{\prime }\right)}\right)}^2\left({\alpha }^{\left(n^{\prime }\right)}+{\beta }^{\left(n^{\prime }\right)}+1\right)}}=O\left({\displaystyle \frac{1}{n^{\prime }}}\right)$$

(16)

Thus, as$n^{\prime }$increases, the posterior variance shrinks, and the distribution becomes increasingly concentrated around ${\theta }^{*}$. Another perspective uses the Kullback–Leibler (KL) divergence [29]. For any candidate value ${\theta }^{\prime }\ne {\theta }^{* }$, the KL divergence between the Beta distributions with parameters ${\theta }^{* }$and${\theta }^{\prime }$is written as follows:

$$D_{KL}\left({\theta }^{\mathrm{*}}\left|\right|{\theta }^{\prime }\right)={\theta }^{\mathrm{*}}\ln {\displaystyle \frac{{\theta }^{\mathrm{*}}}{{\theta }^{\prime }}}+\left(1-{\theta }^{\mathrm{*}}\right)\ln {\displaystyle \frac{{1-\theta }^{\mathrm{*}}}{{1-\theta }^{\prime }}}$$

(17)

By Gibbs’ inequality, $D_{KL}\left({\theta }^{* }\left|\right| {\theta }^{\prime }\right)>0.$ With i.i.d. observations, the average log-likelihood ratio converges to $D_{KL}\left({\theta }^{* }\left|\right| {\theta }^{\prime }\right)$, implying the following:

$${\displaystyle \frac{1}{n^{\prime }}}\ln {\displaystyle \frac{P\left(X_{1:n^{\prime }}|{\theta }^{\mathrm{*}}\right)}{P\left(X_{1:n^{\prime }}|{\theta }^{\prime }\right)}}\stackrel{n^{\prime }\to \mathrm{\infty }}{\to }{D}_{KL}\left({\theta }^{\mathrm{*}}\left|\right|{\theta }^{\prime }\right)>0$$

(18)

The likelihood of the data under any ${\theta }^{\prime }\ne {\theta }^{* }$diminishes exponentially relative to the likelihood under${ \theta }^{* }$. Consequently, the posterior probability of such a ${\theta }^{\prime }$becomes negligible, and the posterior mass concentrates at ${\theta }^{* }$as $n^{\prime }\to \infty$. □

The above analysis shows that BayesSplit’s posterior estimate ${\widehat{\mu }}_{post,ft}$converges to the true impurity reduction probability ${\theta }^{*}$, and its uncertainty decreases as $O\left(\sqrt{1/n^{\prime }}\right)$. Given enough sampling, the algorithm will, with high probability, correctly identify the optimal split.

3.5. Optimal Solution Regret Bound

BayesSplit’s multi-armed bandit formulation allows us to derive a finite-time regret guarantee for the node-splitting process. Consider a node with $n$ data points $\mathcal{X}$, $m$ features $\mathcal{M}$, and $T$ possible thresholds for each feature ($\left|\mathcal{T}\right|=T$). Assume that $\left(f_b,t_b\right)$ is the optimal feature–threshold pair that maximizes node-splitting effectiveness, i.e., $\left(f_b,t_b\right)=\underset{f\in \mathcal{M},t\in \mathcal{T}}{\mathrm{arg\; min}}{\mu }_{ft}$. The following theorem provides an upper bound on the expected regret at total computation.

Theorem 1. 

Fix$ϵ\in \left(\mathrm{0,1}\right).$For the multi-armed bandit formulation of BayesSplit, the finite-time expected regret $\mathbb{E}\left[\mathcal{R}\left(M\right)\right]$ at total computation $M$ satisfies the following:

$$\mathbb{E}\left[\mathcal{R}\left(M\right)\right]\le \left(1+ϵ\right)\sum _{\left(f,t\right)\ne \left(f_b,t_b\right)}^{mT}{\displaystyle \frac{\mathit{ln}M}{d\left(p_{ft}, p_{f_b{t}_b}\right)}}{∆}_{ft}+O\left({\displaystyle \frac{mT}{{ϵ}^2}}\right)$$

(19)

where $d\left(p_{ft},p_{f_b{t}_b}\right)$ is the KL divergence between the probability of achieving impurity reduction of the optimal split$\left(f_b,t_b\right)$and that of any suboptimal split $\left(f,t\right)$, defined as $d\left(p_{ft},p_{f_b{t}_b}\right)=p_{ft}\log {\displaystyle \frac{p_{ft}}{p_{f_b{t}_b}}}+\left(1-p_{ft}\right)\log {\displaystyle \frac{\left(1-p_{ft}\right)}{\left(1-p_{f_b{t}_b}\right)}}$ with ${∆}_{ft}=p_{f_b{t}_b}-p_{ft}$.

A complete derivation and step-by-step proof of Theorem 1 can be found in Appendix A. The proof follows standard multi-armed bandit analyses by bounding the number of times a suboptimal split$\left(f,t\right)$can be selected. The regret bound implies that BayesSplit rapidly converges on the optimal split when one is clearly superior. In scenarios where several splits exhibit comparable performance, the cumulative regret increases slightly, yet it remains sublinear overall.

4. Performance Analysis

To evaluate the effectiveness of BayesSplit, we conducted a series of experiments. Initially, the wall-clock training time and corresponding generalization performance of decision tree ensembles utilizing BayesSplit were measured. Subsequently, we assessed the computational efficiency and resulting generalization performance under a fixed computational budget. All experiments were performed on a ThinkPad T14 Gen 3 laptop, equipped with a 12th Generation Intel Core i7-1260P processor and 32 GB of RAM.

In these comparative experiments, we evaluated the histogrammed versions of three decision tree ensembles with and without BayesSplit: Random Forest (RF), ExtraTrees [30], and Random Patches (RP) [31]. Random Forest creates an ensemble of decision trees built on bootstrapped samples, while a random subset of features is considered at every node split to balance variance reduction and computational complexity. ExtraTrees further randomizes this process by randomly selecting split thresholds, reducing variance and training time but potentially increasing bias. Random Patches samples subsets of instances and features simultaneously for each tree, enhancing ensemble diversity and improving generalization, particularly for high-dimensional datasets.

We performed experiments using all datasets originally employed by MABSplit, supplemented with additional publicly available classification and regression datasets from the UCI Machine Learning Repository. This dataset selection ensured coverage of varying complexities and data characteristics, while consistent preprocessing and benchmarking allowed for a fair comparison of BayesSplit, MABSplit, and the naïve approach. In addition to these core experiments, we conducted additional experiments to evaluate BayesSplit’s sensitivity to hyperparameter settings and its performance on imbalanced datasets, with detailed results provided in Appendix B.

4.1. Wall-Clock Time Comparisons

Classification: We assessed the performance of BayesSplit by comparing it with MABSplit and the baseline brute-force solver on several classification tasks, evaluating wall-clock training time, number of histogram insertions, and test accuracy. As shown in Figure 2, incorporating BayesSplit and MABSplit subroutines significantly reduces training time relative to the naïve approach, with BayesSplit providing up to a 95% reduction compared to baseline methods.

Table 2. Number of histogram insertions for classification models.

	Dataset
	MNIST	APS Failure	Forest Covertype	Online Retail	Dry Bean
RF	1.54 × 108$\pm$ 3.16 × 105	1.88 × 107$\pm$ 2.19 × 104	1.04 × 108$\pm$1.16 × 105	3.60 × 107$\pm$1.17 × 105	1.21 × 106$\pm$1.09 × 103
RF+MABSplit	3.93 × 106$\pm$ 5.64 × 103	7.85 × 105$\pm$ 1.43 × 104	1.04 × 106$\pm$ 1.26 × 104	3.68 × 105$\pm$ 7.62 × 103	4.00 × 105$\pm$1.01 × 104
RF+BayesSplit	3.93 × 106  $\pm$  5.64 × 103	7.85 × 105  $\pm$  1.43 × 104	1.04 × 106  $\pm$  1.26 × 104	3.68 × 105  $\pm$  7.62 × 103	4.00 × 105  $\pm$  1.01 × 104
RP	1.32 × 108$\pm$ 6.95 × 105	1.61 × 107$\pm$ 3.52 × 104	7.79 × 107$\pm$ 4.95 × 105	2.68 × 107$\pm$1.50 × 105	1.22 × 106$\pm$1.65 × 103
RP+MABSplit	3.17 × 106$\pm$ 1.40 × 104	6.44 × 105$\pm$ 2.43 × 104	3.59 × 105$\pm$1.71 × 104	2.74 × 105$\pm$ 1.48 × 104	4.07 × 105$\pm$2.31 × 103
RP+BayesSplit	3.17 × 106  $\pm$  1.40 × 104	6.44 × 105  $\pm$  2.43 × 104	3.59 × 105  $\pm$  1.71 × 104	2.74 × 105  $\pm$  1.48 × 104	4.07 × 105  $\pm$  2.31 × 103
ExtraTrees	1.68 × 108$\pm$ 0.00 × 100	1.89 × 107$\pm$ 3.85 × 102	1.04 × 108$\pm$1.29 × 105	3.55 × 107$\pm$2.12 × 105	1.22 × 106$\pm$1.91 × 103
ExtraTrees+MABSplit	4.32 × 106$\pm$ 7.69 × 103	8.03 × 105$\pm$ 2.45 × 104	2.24 × 106$\pm$8.11 × 105	6.12 × 106$\pm$9.15 × 105	4.15 × 105$\pm$1.30 × 104
ExtraTrees+BayesSplit	4.32 × 106  $\pm$  1.11 × 104	7.43 × 105  $\pm$  1.30 × 104	1.00 × 106  $\pm$  3.22 × 104	3.63 × 105  $\pm$  6.03 × 103	3.87 × 105  $\pm$  3.85 × 103

further confirms their efficiency by showing fewer histogram insertions, with BayesSplit’s refined histogram-based splits providing additional computational savings to the ExtraTrees model. The identical insertion counts between BayesSplit and MABSplit for RF and RP models occur because both algorithms eliminate candidates at similar rates when significant differences in split quality are evident. Given the strong correlation between histogram insertions and training time, improving node-splitting by reducing sample complexity is clearly justified.

Figure 3 illustrates that integrating BayesSplit and MABSplit produces test accuracy comparable to that of the baseline models. Notably, BayesSplit provides significant advantages to the ExtraTrees model, which typically relies on random split thresholds that can lead to suboptimal decisions. Through Bayesian optimization, BayesSplit refines those random splits, resulting in more precise decision boundaries and improved accuracy.

Regression: Across four regression datasets with diverse characteristics, BayesSplit consistently reduces training time by 20–70% compared to MABSplit, as shown in Figure 4. To ensure fairness given the varying histogram bin counts of the baseline regression models, we excluded the number of histogram insertions from our analysis.

Table 3. Regression test MSEs.

	Dataset
	Beijing Multi-Site Air Quality	SGEMM GPU Kernel Performance	Multivariate Gait Data	Clickstream Data
RF	$1138.284 \pm$4.066	$28,822.531 \pm$13.386	$66.477 \pm$0.066	$10.399 \pm$0.014
RF+MABSplit	1132.521$\pm$5.659	$27,646.146 \pm$391.684	$51.270 \pm$3.056	$10.083 \pm$0.136
RF+BayesSplit	1113.040  $\pm$  4.898	27,957.258  $\pm$  395.620	39.888  $\pm$  1.539	9.193  $\pm$  0.265
RP	$889.993 \pm$7.186	$41,998.543 \pm$5321.693	$61.942 \pm$6.699	$7.656 \pm$0.395
RP+MABSplit	$861.830 \pm$4.373	$41,998.543 \pm$5321.693	$60.756 \pm$7.296	$7.490 \pm$0.412
RP+BayesSplit	856.503  $\pm$  5.041	41,964.334  $\pm$  5267.328	60.349  $\pm$  6.065	7.391  $\pm$  0.422
ExtraTrees	$829.345 \pm$8.158	$28,827.176 \pm$20.081	$37.568 \pm$0.809	$6.101 \pm$0.155
ExtraTrees+MABSplit	$824.230 \pm$6.631	$28,838.510 \pm$0.143	$35.958 \pm$0.805	$6.068 \pm$0.103
ExtraTrees+BayesSplit	818.699  $\pm$  4.059	27,953.338  $\pm$  405.546	34.857  $\pm$  1.053	6.312  $\pm$  0.096

shows that using BayesSplit yields lower test MSEs than the naïve solver or MABSplit, and the improvement in generalization performance along with reduced training time highlights the effectiveness of Bayesian optimization in efficiently focusing on the most promising splits.

4.2. Fixed Budget Comparisons

Classification: Under a fixed computational budget defined by a set number of histogram insertions, forests trained with BayesSplit split more nodes and require fewer data-point queries than those using the naïve solver. Consequently, these forests can accommodate more trees and achieve improved generalization.

Figure 5 compares the number of classification trees built under a fixed budget with test accuracies in

Table 4. Classification test accuracy (fixed computational budget).

	Dataset
	MNIST	APS Failure	Forest Covertype	Online Retail	Dry Bean
RF	0.575$\pm$0.006	0.983$\pm$0.0	0.374$\pm$0.019	0.893$\pm$0.003	0.858$\pm$0.003
RF+MABSplit	0.825$\pm$0.001	0.987$\pm$0.0	0.670$\pm$0.001	0.917$\pm$0.001	0.867$\pm$0.002
RF+BayesSplit	0.825  $\pm$  0.001	0.987  $\pm$  0.0	0.670  $\pm$  0.001	0.917  $\pm$  0.001	0.867  $\pm$  0.002
RP	0.589$\pm$0.014	0.984$\pm$0.0	0.553$\pm$0.05	0.893$\pm$0.003	0.876$\pm$0.001
RP+MABSplit	0.834$\pm$0.001	0.988$\pm$0.0	0.676$\pm$0.001	0.929$\pm$0.004	0.882$\pm$0.002
RP+BayesSplit	0.834  $\pm$  0.001	0.988  $\pm$  0.0	0.676  $\pm$  0.001	0.929  $\pm$  0.004	0.882  $\pm$  0.002
ExtraTrees	0.563$\pm$0.008	0.983$\pm$0.0	0.389$\pm$0.022	0.896$\pm$0.003	0.877$\pm$0.004
ExtraTrees+MABSplit	0.814$\pm$0.002	0.988$\pm$0.0	0.653$\pm$0.014	0.919$\pm$0.008	0.885$\pm$0.004
ExtraTrees+BayesSplit	0.820  $\pm$  0.002	0.989  $\pm$  0.0	0.672  $\pm$  0.001	0.933  $\pm$  0.005	0.887  $\pm$  0.002

. For RF and RP, BayesSplit and MABSplit perform similarly across all five datasets. However, in ExtraTrees, BayesSplit surpasses MABSplit in both tree count and test accuracy, indicating that BayesSplit effectively captures and leverages the additional randomness in ExtraTrees’ split-selection process.

Regression: Under a fixed computational budget, integrating BayesSplit consistently outperforms both the naïve approach and MABSplit, allowing more trees to be trained and yielding lower test MSEs in all baseline models. Figure 6 shows that BayesSplit supports additional regression trees within the same budget, and

Table 5. Regression test MSEs (fixed computational budget).

	Dataset
	Beijing Multi-Site Air Quality	SGEMM GPU Kernel Performance	Multivariate Gait Data	Clickstream Data
RF	$1150.986 \pm$1.360	$33,133.193 \pm$229.830	$109.821 \pm$0.178	$17.198 \pm$0.013
RF+MABSplit	1122.964$\pm$0.636	$27,539.144 \pm$188.497	$53.366 \pm$2.076	$10.176 \pm$0.096
RF+BayesSplit	1106.436  $\pm$  0.365	27,421.693  $\pm$  98.705	39.960  $\pm$  0.457	9.017  $\pm$  0.070
RP	$1169.343 \pm$1.107	$73,717.816 \pm$4059.742	$85.964 \pm$3.780	$13.117 \pm$0.820
RP+MABSplit	$1169.573 \pm$3.351	$70,140.627 \pm$3425.126	$76.579 \pm$2.672	$6.755 \pm$0.192
RP+BayesSplit	1135.562  $\pm$  1.281	67,814.118  $\pm$  755.378	70.194  $\pm$  4.203	6.678  $\pm$  0.062
ExtraTrees	$951.450 \pm$1.656	$33,104.573 \pm$213.766	$90.886 \pm$1.426	$13.004 \pm$2.393
ExtraTrees+MABSplit	$797.448 \pm$1.581	$28,969.467 \pm$41.929	$43.639 \pm$2.203	$6.087 \pm$0.078
ExtraTrees+BayesSplit	772.231  $\pm$  0.922	27,762.501  $\pm$  136.819	33.776  $\pm$  0.490	5.855  $\pm$  0.048

confirms notable improvements in predictive performance via reduced test MSEs. Notably, compared to MABSplit, BayesSplit can reduce the test MSEs by up to 25%.

BayesSplit proves particularly well-suited to regression tasks due to the continuous nature of the target variable, which enables more precise estimation of impurity reductions. This precision results in tighter confidence intervals within the Bayesian updating framework, allowing the algorithm to allocate more samples to regions of high uncertainty while limiting evaluations in low-variance segments. Consequently, BayesSplit converges more rapidly toward optimal splits, reducing unnecessary computations and enhancing overall efficiency.

4.3. Feature Stability Comparisons

Beyond predictive performance, Random Forests offer valuable insights into feature importance, thereby enhancing model explainability [32]. We evaluate feature importance using two well-established metrics: Out-of-Bag Permutation Importance (OOB-PI) and Mean Decrease in Impurity (MDI). Specifically, OOB-PI quantifies the change in out-of-bag error when the values of a feature are shuffled, reflecting its contribution to predictive accuracy. On the other hand, MDI calculates the average reduction in impurity (e.g., Gini index or entropy for classification, MSE for regression) across all decision nodes where a given feature is used for splitting, indicating its effectiveness in reducing overall impurity within the model. To ensure robustness, the top$k$features identified by these metrics are further assessed for stability using standardized formulas [33].

Table 6. Stability scores (fixed computational budget).

Model	Stability Metric	Dataset	Stability
RF RF+MABSplit RF+BayesSplit	MDI MDI MDI	Random Classification Random Classification Random Classification	$0.536 \pm$0.039 $0.863 \pm$0.016 0.863$\pm$0.016
RF RF+MABSplit RF+BayesSplit	MDI MDI MDI	Random Regression Random Regression Random Regression	$0.497 \pm$0.046 $0.581 \pm$0.050 0.805$\pm$0.012
RF RF+MABSplit RF+BayesSplit	Permutation Permutation Permutation	Random Classification Random Classification Random Classification	$0.599 \pm$0.022 $0.695 \pm$0.025 0.787$\pm$0.024
RF RF+MABSplit RF+BayesSplit	Permutation Permutation Permutation	Random Regression Random Regression Random Regression	$0.403 \pm$0.047 $0.456 \pm$0.041 0.634$\pm$0.014

shows that under a fixed computational budget, forests trained with BayesSplit achieve a 10–40% improvement in feature stability compared to MABSplit, corresponding to an average increase of approximately 30.28%. This improvement is particularly significant in resource-constrained environments, such as IoT and edge computing applications, where model interpretability and robustness are critical. By consistently identifying the most relevant features across multiple iterations, BayesSplit not only enhances predictive performance but also mitigates the risk of overfitting to irrelevant or noisy data.

5. Conclusions and Future Work

In this work, the BayesSplit algorithm was proposed as a Bayesian enhancement of MABSplit for decision-tree node splitting. While MABSplit introduces multi-armed bandit techniques with frequentist confidence intervals, BayesSplit advances this approach by developing a Bayesian-based impurity estimation framework where impurity reduction events are treated as Bernoulli trials. On the benchmark datasets, BayesSplit reduced wall-clock training time by 20–70% relative to MABSplit and by up to 95% relative to the naïve approach. Compared to MABSplit, it also lowered regression MSEs by as much as 25%.

Beyond computational efficiency, BayesSplit exhibits enhanced feature stability. Our experiments primarily focused on standard computing environments, making it an important future research direction to benchmark and optimize BayesSplit for resource-constrained devices such as smartphones and IoT platforms. Given the growing relevance of edge computing, which involves limited memory, power constraints, and specialized hardware architectures, dedicated optimizations and platform-specific considerations are critical. Future work could also extend BayesSplit to parallel frameworks such as Apache Spark, further broadening its applicability in large-scale real-time processing scenarios.

Finally, we note that gradient boosting decision tree (GBDT) frameworks such as XGBoost, LightGBM, and CatBoost remain strong baselines in predictive performance. These boosting algorithms often achieve higher accuracy than bagging-based methods on many tasks due to their sequential error-correcting training process. In contrast, our work focuses on the Random Forest framework, where trees are constructed in parallel, aiming to improve training efficiency without sacrificing accuracy. Nevertheless, integrating BayesSplit’s adaptive splitting strategy into boosting frameworks offers potential advantages, such as selectively updating the necessary data points’ residual targets rather than recomputing residuals for the entire dataset at each iteration. Although such integration presents practical and algorithmic challenges, it represents a promising direction for future research, as indicated by recent advances such as FastForest [9]. Thus, our proposed approach complements rather than directly competes with optimized gradient boosting and hybrid methods. We intend to explore this topic in greater detail in future work.