LARF: Two-Level Attention-Based Random Forests with a Mixture of Contamination Models

Abstract

This paper provides new models of the attention-based random forests called LARF (leaf attention-based random forest). The first idea behind the models is to introduce a two-level attention, where one of the levels is the “leaf” attention, and the attention mechanism is applied to every leaf of trees. The second level is the tree attention depending on the “leaf” attention. The second idea is to replace the softmax operation in the attention with the weighted sum of the softmax operations with different parameters. It is implemented by applying a mixture of Huber’s contamination models and can be regarded as an analog of the multi-head attention, with “heads” defined by selecting a value of the softmax parameter. Attention parameters are simply trained by solving the quadratic optimization problem. To simplify the tuning process of the models, it is proposed to convert the tuning contamination parameters into trainable parameters and to compute them by solving the quadratic optimization problem. Many numerical experiments with real datasets are performed for studying LARFs. The code of the proposed algorithms is available.

1. Introduction

Several crucial improvements in neural networks have been made in recent years. One of them is the attention mechanism, which has significantly improved classification and regression models in many machine learning areas, including the natural language processing models, computer vision, etc. [1,2,3,4,5]. The idea behind the attention mechanism is to assign weights to features or examples in accordance with their importance and their impact on the model predictions. The attention weights are learned by incorporating an additional feedforward neural network within a neural network architecture. Additionally, the success of the attention models as components of the neural network motivates one to extend this approach to other machine learning models different from neural networks, for example, to random forests (RFs) [6]. Following this idea, a new model called the attention-based random forest (ABRF) has been developed [7,8]. This model incorporates the attention mechanism into ensemble-based models such as RFs and the gradient boosting machine [9,10]. The ABRF models stem from the interesting interpretation [1,11] of the attention mechanism through the Nadaraya–Watson kernel regression model [12,13]. The Nadaraya–Watson regression model learns a non-linear function by using a weighted average of data using a specific normalized kernel as a weighting function. A detailed description of the model can be found in Section 3. According to [7,8], attention weights in the Nadaraya–Watson regression are assigned to decision trees in an RF depending on examples which fall into leaves of trees. Weights in ABRF have trainable parameters and use Huber’s $ϵ$-contamination model [14] for defining the attention weights. Huber’s $ϵ$-contamination model can be regarded as a set of convex combinations of probability distributions, where one of the distributions is considered as a set of trainable attention parameters. A detailed description of the model is provided in Section 5.1. In accordance with the $ϵ$-contamination model, each attention weight consists of two parts: the softmax operation with the tuning coefficient $1-ϵ$ and the trainable bias of the softmax weight with coefficient $ϵ$. One of the improvements of ABRF, which has been proposed in [15], is based on joint incorporating self-attention and attention mechanisms into the RF. The proposed models outperform ABRF, but this outperformance is not sufficient, because this model provided inferior results for several datasets. Therefore, we propose a set of models which can be regarded as extensions of ABRF and which are based on two main ideas.

The first idea is to introduce a two-level attention, where one of the levels is the “leaf” attention, i.e., the attention mechanism is applied to every leaf of a tree. As a result, we obtain the attention weights assigned to leaves and the attention weights assigned to trees. The attention weights of trees depend on the corresponding weights of leaves which belong to these trees. In other words, the attention at the second level depends on the attention at the first level, i.e., we obtain the attention of the attention. Due to the “leaf” attention, the proposed model will be abbreviated as LARF (leaf attention-based random forest).

One of the peculiarities of LARFs is using a mixture of Huber’s $ϵ$-contamination models instead of the single contamination model, as has been conducted in ABRF. This peculiarity stems from the second idea behind the model, which takes into account the softmax operation with different parameters. In fact, we replace the standard softmax operation by the weighted sum of the softmax operations with different parameters. With this idea, we achieve two goals. First of all, we partially solve the problem of the tuning parameters of the softmax operations which are a part of the attention operations. Each value of the tuning parameter from the predefined set (from the predefined grid) is used in a separate softmax operation. Then, weights of the softmax operations in the sum are trained jointly while training other parameters. This approach can also be interpreted as the linear approximation of the softmax operations with trainable weights and with different values of tuning parameters. However, a more interesting goal is that some analogs of the multi-head attention [16] are implemented by using the mixture of contamination models, where “heads” are defined by selecting a value of the corresponding softmax operation parameter.

Additionally, in contrast to ABRF [8], where the contamination parameter $ϵ$ of Huber’s model was a tuning parameter, the LARF model considers this parameter as the training one. This allows us to significantly reduce the model tuning time and avoid the enumeration of the parameter values in accordance with the grid. The same is implemented for the mixture of the Huber’s models.

Different configurations of LARF produce a set of models, which depend on trainable and tuning parameters of the two-level attention and on algorithms for their calculation.

We investigate two types of RFs in the experiments: original RFs and Extremely Randomized Trees (ERT) [17]. According to [17], the ERT algorithm chooses a split point randomly for each feature at each node and then selects the best split among these. In contrast to ERTs, original RFs choose the most optimal (not random) split of a set of features at each node in accordance with a criterion, for example, with the Gini impurity [6].

Our contributions can be summarized as follows:

• We propose new two-level attention-based RF models, where the attention mechanism at the first level is applied to every leaf of trees, the attention at the second level incorporates the “leaf” attention, and it is applied to trees. The training of the two-level attention is reduced to solving the standard quadratic optimization problem.
• A mixture of Huber’s $ϵ$-contamination models is used to implement the attention mechanism at the second level. The mixture allows us to replace a set of tuning attention parameters (the temperature parameters of the softmax operations) with trainable parameters, whose optimal values are computed by solving the quadratic optimization problem. Moreover, this approach can be regarded as an analog of the multi-head attention.
• We propose an approach to convert the tuning contamination parameters ($ϵ$ parameters) in the mixture of the $ϵ$-contamination models into trainable parameters. Their optimal values are also computed by solving the quadratic optimization problem.

Many numerical experiments with real datasets are performed for studying LARFs. They demonstrate outperforming results of some LARF modifications. The code of the proposed algorithms can be found at https://github.com/andruekonst/leaf-attention-forest (accessed on 20 April 2023).

This paper is organized as follows. The related work can be found in Section 2. A brief introduction to the attention mechanism as the Nadaraya–Watson kernel regression is given in Section 3. A general approach to incorporating the two-level attention mechanism into the RF is provided in Section 4. Ways for the implementation of the two-level attention mechanism and constructing several attention-based models by using the mixture of Huber’s e-contamination models are considered in Section 5. Numerical experiments with real data, which illustrate properties of the proposed models, are provided in Section 6. The concluding remarks can be found in Section 7.

2. Related Work

Attention mechanism. Due to the great efficiency of machine learning models with the attention mechanisms, different attention-based models have become of great interest in recent years. Consequently, numerous attention models have been proposed to improve the performance of machine learning algorithms. The most comprehensive analysis and description of various attention-based models can be found in in-depth surveys [1,2,3,4,5,18].

It is important to note that parametric attention models as parts of neural networks are mainly trained by applying the gradient-based algorithms which lead to computational problems, when training is carried out through the softmax function. Many approaches have been proposed to cope with this problem. A large part of the approaches is based on the linear approximation of the softmax attention [19,20,21,22]. Another part of the approaches is based on random feature methods to approximate the softmax function [18,23].

Another improvement of the attention-based models is to use the self-attention which was proposed in [16] as a crucial component of neural networks called Transformers. The self-attention models have also been studied in surveys [4,24,25,26,27,28]. This is only a small part of all the works devoted to attention and self-attention mechanisms.

It should be noted that the aforementioned models are implemented as neural networks, and they have not been studied for applications to other machine learning models, for example, to RFs. Attempts to incorporate the attention and self-attention mechanisms into the RF and the gradient boosting machine were made in [7,8,15]. Following these research works, we extend the proposed models to improve the attention-based models. Apart from this, we propose the attention models, which do not use the gradient-based algorithms for computing optimal attention parameters. The training process of the models is based on solving standard quadratic optimization problems.

Weighted RFs. A lot of approaches have been proposed in recent years to improve RFs. One of the important approaches is based on the assignment of weights to decision trees in the RF. This approach is implemented in various algorithms [29,30,31,32,33,34]. However, most of these algorithms have a disadvantage. The weights are assigned to trees independently of training or testing examples, i.e., each weight characterizes trees on average, over all training examples, and it does not take into account any feature vector. Moreover, the weights do not have training parameters which usually make the model more flexible and accurate.

Contamination model in attention mechanisms. There are several models, which use imprecise probabilities in order to model the lack of sufficient training data. One of the first models is the so-called Credal Decision Tree, which is based on applying the imprecise probability theory to classification and proposed in [35]. Following this work, a number of models, based on imprecise probabilities, were presented in [36,37,38,39], where the imprecise Dirichlet model is used. This model can be regarded as a reparametrization of the imprecise $ϵ$-contamination model, which is applied to LARF. The imprecise $ϵ$-contamination model has been also applied to machine learning methods, for example, to the support vector machine [40] or to the RF [41]. The attention-based RF applying the imprecise $ϵ$-contamination model to the parametric attention mechanism was proposed in [8,15]. However, there are no other works which use the imprecise models in order to implement the attention mechanism.

3. Nadaraya–Watson Regression and the Attention Mechanism

The basis of the attention mechanism can be considered in the framework of the Nadaraya–Watson kernel regression model [12,13], which estimates a function f as a locally weighted average using a kernel as a weighting function. Suppose the dataset is represented by n examples $({\mathbf{x}}_1,y_1),\dots ,({\mathbf{x}}_n,y_n)$, where ${\mathbf{x}}_i=(x_{i1},\dots ,x_{im})\in {\mathbb{R}}^m$ is a feature vector consisting of m features; $y_i\in \mathbb{R}$ is a regression output. The regression task is to construct a regressor $f:{\mathbb{R}}^m\to \mathbb{R}$, which can predict the output value $\tilde{y}$ of a new observation $\mathbf{x}$, using the dataset.

The Nadaraya–Watson kernel regression estimates the regression output $\tilde{y}$ corresponding to a new input feature vector $\mathbf{x}$, as follows [12,13]:

$$\tilde{y}=\sum _{i=1}^n\alpha (\mathbf{x},{\mathbf{x}}_i)y_i,$$

(1)

where weight $\alpha (\mathbf{x},{\mathbf{x}}_i)$ conforms with a relevance of the feature vector ${\mathbf{x}}_i$ to the vector $\mathbf{x}$.

It can be observed from the above that the Nadaraya–Watson regression model estimates $\tilde{y}$ as a weighted sum of training outputs $y_i$ from the dataset so that their weights depend on the location of ${\mathbf{x}}_i$ relative to $\mathbf{x}$. This means that the closer ${\mathbf{x}}_i$ is to $\mathbf{x}$, the greater weight is assigned to $y_i$.

According to the Nadaraya–Watson kernel regression [12,13], weights can be defined by means of the kernel K as a function of the distance between the vectors ${\mathbf{x}}_i$ and $\mathbf{x}$. The kernel estimates how ${\mathbf{x}}_i$ is close to $\mathbf{x}$. Then, the weight is written as follows:

$$\alpha (\mathbf{x},{\mathbf{x}}_i)=\frac{K(\mathbf{x},{\mathbf{x}}_i)}{{\sum }_{j=1}^{n}K(\mathbf{x},{\mathbf{x}}_j)}.$$

(2)

One of the popular kernels is the Gaussian kernel. It produces weights of the form:

$$\alpha (\mathbf{x},{\mathbf{x}}_i)=\sigma \left(-\frac{{∥\mathbf{x}-{\mathbf{x}}_i∥}^2}{\tau }\right),$$

(3)

where $\tau$ is a tuning (temperature) parameter; $\sigma \left(·\right)$ is a notation of the softmax operation.

In terms of the attention mechanism [42], the vector $\mathbf{x}$, vectors ${\mathbf{x}}_i$, outputs $y_i$, and weight $\alpha (\mathbf{x},{\mathbf{x}}_i)$ are called the query, keys, values, and the attention weight, respectively. Weights $\alpha (\mathbf{x},{\mathbf{x}}_i)$ can be extended by incorporating trainable parameters. In particular, parameter $\tau$ can also be regarded as the trainable parameter.

Many forms of parametric attention weights, which also define the attention mechanisms, have been proposed, e.g., the additive attention [42], the multiplicative or dot-product attention [16,43]. We also consider the attention weights based on the Gaussian kernels, i.e., producing the softmax operation. However, the parametric forms of the attention weights will be quite different from many popular attention operations.

4. Two-Level Attention-Based Random Forest

One of the powerful machine learning models handling tabular data is the RF, which can be regarded as an ensemble of T decision trees so that each tree is trained on a subset of examples randomly selected from the training set. In the original RF, the final RF prediction $\tilde{y}$ for the testing example $\mathbf{x}$ is determined by averaging predictions ${\tilde{y}}_1,\dots ,{\tilde{y}}_T$ obtained for all trees.

Let ${\mathcal{J}}_k\left(\mathbf{x}\right)$ be the index set of examples which fall into the same leaf in the k-th tree as $\mathbf{x}$. One of the ways to construct the attention-based RF is to introduce the mean vector ${\mathbf{A}}_k\left(\mathbf{x}\right)$ defined as the mean of the training vectors ${\mathbf{x}}_j$ which fall into the same leaf as $\mathbf{x}$. However, this simple definition can be extended by incorporating the Nadaraya–Watson regression into the leaf. In this case, we can write

$${\mathbf{A}}_k\left(\mathbf{x}\right)=\sum _{j\in {\mathcal{J}}_k\left(\mathbf{x}\right)}\mu \left(\mathbf{x},{\mathbf{x}}_j\right){\mathbf{x}}_j,$$

(4)

where $\mu \left(\mathbf{x},{\mathbf{x}}_j\right)$ is the attention weight in accordance with the Nadaraya–Watson kernel regression.

In fact, (4) can be regarded as the self-attention. The idea behind (4) is that we find the mean value of $\mathbf{x}$ by assigning weights to training examples which fall into the corresponding leaf in accordance with their vicinity to the vector $\mathbf{x}$.

In the same way, we can define the mean value of regression outputs corresponding to examples falling into the same leaf as $\mathbf{x}$:

$$B_k\left(\mathbf{x}\right)=\sum _{j\in {\mathcal{J}}_k\left(\mathbf{x}\right)}\mu \left(\mathbf{x},{\mathbf{x}}_j\right)y_j.$$

(5)

Expression (5) can be regarded as the attention. The idea behind (5) is to obtain the prediction provided by the corresponding leaf by using the standard attention mechanism or Nadaraya–Watson regression. In other words, we weigh predictions provided by the k-th leaf of a tree in accordance with the distance between the feature vector $\mathbf{x}$, which falls into the k-th leaf, and all the feature vectors ${\mathbf{x}}_j$ which fall into the same leaf. It should be noted that the original regression tree provides the averaged prediction; i.e., it corresponds to the case when all $\mu \left(\mathbf{x},{\mathbf{x}}_j\right)$ are identical for all $j\in {\mathcal{J}}_k\left(\mathbf{x}\right)$ and equal to $1/\#{\mathcal{J}}_k\left(\mathbf{x}\right)$.

We suppose that the attention mechanisms used above are non-parametric. This implies that weights do not have trainable parameters. It is assumed that

$$\sum _{j\in {\mathcal{J}}_k\left(\mathbf{x}\right)}\mu \left(\mathbf{x},{\mathbf{x}}_j\right)=1.$$

(6)

The “leaf” attention introduced above can be regarded as the first-level attention in a hierarchy of the attention mechanisms. It characterizes how the feature vector $\mathbf{x}$ fits the corresponding tree.

If we suppose that the whole RF consists of T decision trees, then, the set of ${\mathbf{A}}_k\left(\mathbf{x}\right)$, $k=1,\dots ,T$, in the framework of the attention mechanism, can be regarded as a set of keys for every $\mathbf{x}$, the set of $B_k\left(\mathbf{x}\right)$, $k=1,\dots ,T$, can be regarded as a set of values. This implies that the final prediction $\tilde{y}$ of the RF can be computed by using the Nadaraya–Watson regression, namely,

$$\tilde{y}=f(\mathbf{x},\mathbf{w})=\sum _{k=1}^T\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),\mathbf{w}\right)B_k\left(\mathbf{x}\right).$$

(7)

Here, $\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),\mathbf{w}\right)$ is the attention weight with the vector $\mathbf{w}=(w_1,\dots ,w_T)$ of trainable parameters which belong to a set $\mathcal{W}$ so that they are assigned to each tree. The attention weight $\alpha$ is defined by the distance between $\mathbf{x}$ and ${\mathbf{A}}_k\left(\mathbf{x}\right)$. It is assumed due to properties of the attention weights in the Nadaraya–Watson regression that the following condition is valid:

$$\sum _{k=1}^T\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),\mathbf{w}\right)=1.$$

(8)

The above “random forest” attention can be regarded as the second-level attention which assigns weights to trees in accordance with their impact on the RF prediction corresponding to $\mathbf{x}$.

The main idea behind the approach is to use the above attention mechanisms jointly. After substituting (4) and (5) into (7), we obtain

$$\tilde{y}\left(\mathbf{x}\right)=f(\mathbf{x},\mathbf{w})=\sum _{k=1}^T\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),\mathbf{w}\right)\sum _{j\in {\mathcal{J}}_k\left(\mathbf{x}\right)}{y}_j\mu \left(\mathbf{x},{\mathbf{x}}_j\right),$$

(9)

or

$$\tilde{y}\left(\mathbf{x}\right)=\sum _{k=1}^T\alpha \left(\mathbf{x},\sum _{i\in {\mathcal{J}}_k\left(\mathbf{x}\right)}\mu \left(\mathbf{x},{\mathbf{x}}_i\right){\mathbf{x}}_i,\mathbf{w}\right)\sum _{j\in {\mathcal{J}}_k\left(\mathbf{x}\right)}{y}_j\mu \left(\mathbf{x},{\mathbf{x}}_j\right).$$

(10)

A scheme of the two-level attention is shown in Figure 1. It is observed from Figure 1 how the attention at the second level depends on the “leaf” attention at the first level.

In total, we obtain the trainable attention-based RF with parameters $\mathbf{w}$, which are defined by minimizing the expected loss function over the set $\mathcal{W}$ of parameters, respectively, as follows:

$${\mathbf{w}}_{opt}=arg\underset{\mathbf{w}\in \mathcal{W}}{min}\sum _{s=1}^{n}L\left(\tilde{y}\left({\mathbf{x}}_s\right),y_s,\mathbf{w}\right).$$

(11)

The loss function can be rewritten as the following:

$$\begin{array}{cc}\hfill & {\displaystyle \sum _{s=1}^n}L\left(\tilde{y}\left({\mathbf{x}}_s\right),y_s,\mathbf{w}\right)={\displaystyle \sum _{s=1}^n}{\left(y_s-\tilde{y}\left({\mathbf{x}}_s\right)\right)}^2\hfill \\ \hfill & ={\displaystyle \sum _{s=1}^n}{\left(y_s-{\displaystyle \sum _{k=1}^T\sum _{j\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}}{y}_j\mu \left({\mathbf{x}}_s,{\mathbf{x}}_j\right)·\alpha \left({\mathbf{x}}_s,{\displaystyle \sum _{i\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}}\mu \left({\mathbf{x}}_s,{\mathbf{x}}_i\right){\mathbf{x}}_i,\mathbf{w}\right)\right)}^2.\hfill \end{array}$$

(12)

The optimal trainable parameters $\mathbf{w}$ are computed depending on forms of the attention weights $\alpha$ in the optimization problem (12). It should be noted that the problem (12) may be complex from the computation point of view. Therefore, one of our results is to propose such a form of the attention weights $\alpha$ that reduces the problem (12) to a convex quadratic optimization problem, whose solution does not meet any difficulties.

It is important to point out that the additional sets of trainable parameters can be introduced into the definition of the attention weights $\mu \left(\mathbf{x},{\mathbf{x}}_j\right)$. On the one hand, we obtain more flexible attention mechanisms in this case due to the parametrization of the training weights $\mu \left(\mathbf{x},{\mathbf{x}}_j\right)$. On the other hand, many trainable parameters lead to the increasing complexity of the optimization problem (11) and the possible overfitting of the whole RF.

5. Modifications of the Two-Level Attention-Based Random Forest

Different configurations of LARF produce a set of models which depend on trainable parameters of the two-level attention and its implementation. A classification of models and their notations are shown in

Table 1. Classification of the attention-based RF models proposed and studied in this paper.

	Tuning $\mathit{ϵ}$		Trainable ${\mathit{ϵ}}_1,\dots ,{\mathit{ϵ}}_{\mathit{M}}$
	Fixed $\mathbf{w}$	Trainable $\mathbf{w}$	Fixed $\mathbf{w}$	Trainable $\mathbf{w}$
Without the “leaf” attention	-	$\mathbf{w}$-ARF	$ϵM$-ARF	$ϵM$-$\mathbf{w}$-ARF
With the “leaf” attention	-	$\mathbf{w}$-LARF	$ϵM$-LARF	$ϵM$-$\mathbf{w}$-LARF

. In order to explain the classification, two subsets of the attention parameters should be considered:

• Parameters produced by the contamination probability distributions of Huber’s $ϵ$-contamination model in the form of the vector $\mathbf{w}$, whose length coincides with the number of trees.
• Parameters ${ϵ}_1,\dots ,{ϵ}_M$ of contamination in the mixture of M in the Huber’s contamination models, which define the imprecision of the mixture model.

The following models can be constructed depending on trainable parameters and on using the “leaf” attention, i.e., the two-level attention mechanism:

• $ϵ$-ARF: The attention-based forest with learning $ϵ$ as an attention parameter, but without the training vector $\mathbf{w}$, i.e., $w_k=1/T$, $k=1,\dots ,T$, and without the “leaf” attention;
• w-ARF: The attention-based forest with the learning vector $\mathbf{w}$ as attention parameters and without the “leaf” attention;
• $ϵ$-LARF: The attention-based forest with learning $ϵ$ as an attention parameter, but without the training vector $\mathbf{w}$ and with the “leaf” attention, i.e., by using the two-level attention mechanism;
• w-LARF: The attention-based forest with the learning vector $\mathbf{w}$ as attention parameters and with the “leaf” attention, i.e., by using the two-level attention mechanism;
• $ϵ$-w-ARF: The attention-based forest with the learning vector $\mathbf{w}$ and the parameter $ϵ$ as attention parameters and without the “leaf” attention;
• $ϵ$-w-LARF: The attention-based forest with the learning vector $\mathbf{w}$ and the parameter $ϵ$ as attention parameters and with the “leaf” attention;
• $ϵM$-ARF: The attention-based forest with the learning parameters ${ϵ}_1,\dots ,{ϵ}_M$ as attention parameters with $w_k=1/T$, $k=1,\dots ,T$, and without the “leaf” attention;
• $ϵM$-LARF: The attention-based forest with the learning parameters ${ϵ}_1,\dots ,{ϵ}_M$ as attention parameters with $w_k=1/T$, $k=1,\dots ,T$, and with the “leaf” attention;
• $ϵM$-w-ARF: The attention-based forest with the learning vector $\mathbf{w}$ and the parameters ${ϵ}_1,\dots ,{ϵ}_M$ as attention parameters and without the “leaf” attention;
• $ϵM$-w-LARF: The attention-based forest with the learning vector $\mathbf{w}$ and the parameters ${ϵ}_1,\dots ,{ϵ}_M$ as attention parameters and with the “leaf” attention.

Models $ϵ$-ARF, $ϵ$-LARF, $ϵ$-$\mathbf{w}$-ARF, and $ϵ$-$\mathbf{w}$-LARF are not presented in Table 1 because they are special cases of models $ϵM$-ARF, $ϵM$-LARF, $ϵM$-$\mathbf{w}$-ARF, and $ϵM$-$\mathbf{w}$-LARF, respectively, by $M=1$.

5.1. Huber’s Contamination Model and the Basic Two-Level Attention

In order to simplify the optimization problem (12) and to effectively solve it, we offer to represent the attention weights $\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),\mathbf{w}\right)$ by using Huber’s $ϵ$-contamination model [14]. The idea to represent the attention weight by means of the $ϵ$-contamination model has been proposed in [8]. We use this idea to incorporate the $ϵ$-contamination model into the optimization problem (12) and to construct the first modifications of LARF.

Let us provide a brief introduction to Huber’s $ϵ$-contamination model. The model considers a set of probability distributions of the form $F\left(\mathbf{x}\right)=(1-ϵ)·P\left(\mathbf{x}\right)+ϵ·R$. Here, $P\left(\mathbf{x}\right)=(p_1\left(\mathbf{x}\right),\dots ,p_T\left(\mathbf{x}\right))$ is a discrete probability distribution contaminated by another probability distribution, denoted as $R=(r_1,\dots ,r_T)$, which can be arbitrary in the unit simplex having the dimension T. It is important to note that the distribution P depends on the feature vector $\mathbf{x}$, i.e., it is different for every vector $\mathbf{x}$, whereas the distribution R does not depend on $\mathbf{x}$. The contamination parameter $ϵ\in [0,1]$ controls the impact of the contamination probability distribution R on the distribution $P\left(\mathbf{x}\right)$. Since the distribution R can be arbitrary, the set of the distributions F forms a subset of the unit simplex so that its size depends on the parameter $ϵ$. If $ϵ=0$, then the subset of distributions F is reduced to the single distribution $P\left(\mathbf{x}\right)$. In case $ϵ=1$, the set of $F\left(\mathbf{x}\right)$ is the whole unit simplex.

Following the common definition of the attention weights through the softmax operation with the parameter $\tau$, we propose to define each probability in $P\left(\mathbf{x}\right)$ as

$$p_k\left(\mathbf{x}\right)=\sigma \left(-{∥\mathbf{x}-{\mathbf{A}}_k\left(\mathbf{x}\right)∥}^2/\tau \right).$$

This implies that the distribution $P\left(\mathbf{x}\right)$ characterizes how the feature vector $\mathbf{x}$ is far from the vector ${\mathbf{A}}_k\left(\mathbf{x}\right)$ in all trees of the RF. Let us suppose that the probability distribution R is the vector of the trainable parameters $\mathbf{w}$. The idea is to train the parameters $\mathbf{w}$ to achieve the best accuracy of the RF. After substituting the softmax operation into the attention weight $\alpha$, we obtain:

$$\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),\mathbf{w}\right)=(1-ϵ)·\sigma \left(-{∥\mathbf{x}-{\mathbf{A}}_k\left(\mathbf{x}\right)∥}^2/\tau \right)+ϵ·w_k,k=1,\dots ,T.$$

(13)

One can observe from (13) that the attention weight is linearly dependent on the trainable parameters $\mathbf{w}=(w_1,\dots ,w_T)$. It is important to note that the attention weight assigned to the k-th tree depends only on the k-th parameter $w_k$, but not on other elements of the vector $\mathbf{w}$. The parameter $ϵ$ is a tuning parameter determined by means of the standard validation procedure. It should be noted that elements of the vector $\mathbf{w}$ are probabilities. Hence, we can write

$$\sum _{k=1}^T{w}_k=1,w_k\ge 0,k=1,\dots ,T.$$

(14)

This implies that the set $\mathcal{W}$ is the unit simplex of the dimension T.

Let us return to the attention weight $\mu \left(\mathbf{x},{\mathbf{x}}_j\right)$ of the first level. The attention is non-parametric at the first level; therefore, the attention weight can be defined in the standard way by using the Gaussian kernel or the softmax operation with the parameter ${\tau }_0$; i.e., it can be expressed in this form:

$$\mu \left(\mathbf{x},{\mathbf{x}}_j\right)=\sigma \left(-{∥\mathbf{x}-{\mathbf{x}}_j∥}^2/{\tau }_0\right).$$

(15)

Finally, we can rewrite the loss function (12) by taking into account the above definitions of the attention weights, as follows:

$$\begin{array}{cc}\hfill & {\displaystyle \underset{\mathbf{w}\in \mathcal{W}}{min}\sum _{s=1}^n}L\left(\tilde{y}\left({\mathbf{x}}_s\right),y_s,\mathbf{w}\right)\hfill \\ \hfill & ={\displaystyle \underset{\mathbf{w}\in \mathcal{W}}{min}\sum _{s=1}^n}{\left(y_s-{\displaystyle \sum _{k=1}^T}\left((1-ϵ)C_k\left({\mathbf{x}}_s\right)-ϵD_k\left({\mathbf{x}}_s\right)w_k\right)\right)}^2,\hfill \end{array}$$

(16)

where

$$\begin{array}{cc}\hfill C_k\left({\mathbf{x}}_s\right)& ={\sum }_{l\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}{y}_l·\sigma \left(-\frac{{∥{\mathbf{x}}_s-{\mathbf{x}}_l∥}^2}{{\tau }_0}\right)\hfill \\ \hfill & \times \sigma \left(-\frac{{∥\mathbf{x}-{\sum }_{i\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}{\mathbf{x}}_i·\sigma \left(-{∥{\mathbf{x}}_s-{\mathbf{x}}_i∥}^2/{\tau }_0\right)∥}^2}{\tau }\right),\hfill \end{array}$$

(17)

$$D_k\left({\mathbf{x}}_s\right)={\sum }_{j\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}{y}_j·\sigma \left(-\frac{{∥{\mathbf{x}}_s-{\mathbf{x}}_j∥}^2}{{\tau }_0}\right).$$

(18)

One can observe from the above that $C_k\left({\mathbf{x}}_s\right)$ and $D_k\left({\mathbf{x}}_s\right)$ do not depend on the parameters $\mathbf{w}$. Therefore, the objective function (16) jointly with the simple constraints $\mathbf{w}\in \mathcal{W}$ or (14) is the standard quadratic optimization problem, which can be solved by means of many available efficient algorithms. The corresponding model is called $\mathbf{w}$-LARF. The notation means that trainable parameters are $\mathbf{w}$. The same model without “leaf” attention is denoted as $\mathbf{w}$-ARF. It coincides with the model $ϵ$-ABRF proposed in [8].

It should be worth knowing that the problem (16) is similar to the optimization problem stated in [7,8]. However, it turns out that the addition of the “leaf” attention significantly improves the RF, as it will be demonstrated by many numerical experiments with real data.

5.2. Models with the Trainable Contamination Parameter $ϵ$

One of the important contributions to the work, which makes the proposed model different from the model presented in [7,8], is the idea of learning the contamination parameter $ϵ$ jointly with the parameters $\mathbf{w}$. However, this idea leads to a complex optimization problem, where gradient-based algorithms have to be used. In order to avoid using these algorithms and to tackle a simple optimization problem, we consider two ways. The first way is just to assign the same value $1/T$ to all parameters $w_k$. Then, the optimization problem (16) can be rewritten as the following:

$$\underset{0\le ϵ\le 1}{min}\sum _{s=1}^n{\left(y_s-\sum _{k=1}^T\left((1-ϵ)C_k\left({\mathbf{x}}_s\right)-ϵD_k\left({\mathbf{x}}_s\right)\frac{1}{T}\right)\right)}^2.$$

(19)

We have a simple quadratic optimization problem with one variable $ϵ$. Let us call the corresponding model as $\mathit{ϵ}$-LARF. The notation means that the trainable parameter is $ϵ$. The same model without the “leaf” attention is denoted as $ϵ$-ARF.

Another way is to introduce new variables ${\gamma }_k=ϵw_k$, $k=1,\dots ,T$. Then, the optimization problem (16) can be rewritten in this form:

$$\underset{{\gamma }_1,\dots ,{\gamma }_T,ϵ}{min}\sum _{s=1}^n{\left(y_s-(1-ϵ)\sum _{k=1}^T{C}_k\left({\mathbf{x}}_s\right)-\sum _{k=1}^T{\gamma }_k{D}_k\left({\mathbf{x}}_s\right)\right)}^2,$$

(20)

subject to

$$\sum _{k=1}^T{\gamma }_k=ϵ,{\gamma }_k\ge 0,k=1,\dots ,T,$$

(21)

$$0+\varsigma \le ϵ\le 1.$$

(22)

We again deal with the quadratic optimization problem with new optimization variables ${\gamma }_1,\dots ,{\gamma }_T,ϵ$ and linear constraints. The parameter $\varsigma$ takes a small value to avoid the case $ϵ=0$. The corresponding model is denoted as $ϵ$-$\mathbf{w}$-LARF. The notation means that the trainable parameters are $\mathbf{w}$ and $ϵ$. The same model without the “leaf” attention is denoted as $ϵ$-$\mathbf{w}$-ARF.

5.3. Mixture of Contamination Models

Another important contribution is an attempt to search for an optimal value of the temperature parameter $\tau$ in (13) or in (17). We propose an approximate approach, which can significantly improve the model. Let us introduce a finite set $\{{\tau }_1,\dots ,{\tau }_M\}$ of M values of the parameter $\tau$. Here, M can be regarded as a tuning integer parameter which impacts the number of all training parameters. Large values of M may lead to a large number of training parameters and the corresponding overfitting. Small values of M may lead to an inexact approximation of $\tau$.

Before considering how the optimization problem can be rewritten with allowance for the above information, we again return to the attention weight $\alpha$ in (13) and represent it as follows:

$$\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)=\frac{1}{M}\sum _{j=1}^M{\alpha }_j\left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right),$$

(23)

where

$${\alpha }_j\left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)=(1-{ϵ}_j)\sigma \left(-\frac{{∥\mathbf{x}-{\mathbf{A}}_k\left(\mathbf{x}\right)∥}^2}{{\tau }_j}\right)+{ϵ}_j{w}_k.$$

(24)

We have a mixture of M contamination models with the different contamination parameters ${ϵ}_j$. It is obvious that the sum of new weights $\alpha$ over $k=1,\dots ,T$ is 1 because the sum of each ${\alpha }_j\left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)$ over $k=1,\dots ,T$ is also 1. Each ${\alpha }_j\left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)$ forms a small simplex so that its center is defined by ${\tau }_j$ and its size is defined by ${\tau }_j$. The corresponding sets of possible attention weights are depicted in Figure 2, where the unit simplex by $T=3$ includes small simplices corresponding to three ($M=3$) contamination models ${\alpha }_j\left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)$, $j=1,2,3$, with different centers and different contamination parameters ${ϵ}_1,{ϵ}_2,{ϵ}_3$. The “mean” simplex of weights $\alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)$ is depicted by using dashed sides. Parameters $\mathbf{w}$ are optimized so that the attention weights will be located in the “mean” simplex.

Let us prove that the resulting “mean” model represents the $ϵ$-contamination model with the contamination parameter $ϵ$. We use $P_j$ to denote the probability distibution $P_j=(p_j^{\left(1\right)},\dots ,p_j^{\left(T\right)})$ as follows:

$$P_j=\sigma \left(-\frac{{∥\mathbf{x}-{\mathbf{A}}_k\left(\mathbf{x}\right)∥}^2}{{\tau }_j}\right).$$

(25)

Then, we can write

$$\begin{array}{cc}\hfill \alpha \left(\mathbf{x},{\mathbf{A}}_k\left(\mathbf{x}\right),w_k\right)& =\frac{1}{M}{\displaystyle \sum _{j=1}^M}(1-{ϵ}_j)P_j+\frac{1}{M}{\displaystyle \sum _{j=1}^M}{ϵ}_j{w}_k\hfill \\ \hfill & =\frac{1}{M}{\displaystyle \sum _{j=1}^M}(1-{ϵ}_j)P_j+ϵw_k,\hfill \end{array}$$

(26)

where

$$ϵ=\frac{1}{M}\sum _{j=1}^M{ϵ}_j.$$

(27)

Suppose there is a probability distribution $Q=(q^{\left(1\right)},\dots ,q^{\left(T\right)})$; therefore, it holds

$$\frac{1}{M}\sum _{j=1}^M(1-{ϵ}_j)P_j=(1-ϵ)Q.$$

(28)

If we prove that the probability distribution Q exists, the resulting “mean” model is the $ϵ$-contamination model. Let us find sums of the left and the right sides of (28) over $i=1,\dots ,T$. Hence, we obtain

$$\frac{1}{M}\sum _{j=1}^M(1-{ϵ}_j)\sum _{i=1}^T{p}_j^{\left(i\right)}=(1-ϵ)\sum _{i=1}^T{q}^{\left(i\right)}.$$

(29)

Substituting (27) into (29), we obtain

$$\sum _{i=1}^T{q}^{\left(i\right)}=1,$$

as it was to be proved.

The introduced mixture of the contamination models can be regarded as a multi-head attention to some extent, where every “head” is produced by using a certain parameter ${\tau }_j$.

Let us represent the softmax operation in (13) jointly with the factor $(1-ϵ)$ as follows:

$$(1-ϵ)·\sigma \left(\frac{-{∥\mathbf{x}-{\mathbf{A}}_k\left(\mathbf{x}\right)∥}^2}{\tau }\right)=\frac{1}{M}\sum _{j=1}^M(1-{ϵ}_j)·\sigma \left(\frac{-{∥\mathbf{x}-{\mathbf{A}}_k\left(\mathbf{x}\right)∥}^2}{{\tau }_j}\right).$$

(30)

It can be observed from (30) that new parameters ${ϵ}_1,\dots ,{ϵ}_M$ along with ${\tau }_1,\dots ,{\tau }_M$ are introduced in the place of $ϵ$ and $\tau$, respectively. Term $(1-ϵ)·$ $C_k\left({\mathbf{x}}_s\right)$ in (16) and (17) is replaced with the following terms:

$$\frac{1}{M}\sum _{j=1}^M(1-{ϵ}_j)C_k^{\left(j\right)}\left({\mathbf{x}}_s\right),$$

(31)

where

$$\begin{array}{cc}\hfill C_k^{\left(j\right)}\left({\mathbf{x}}_s\right)& ={\sum }_{l\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}{y}_l·\sigma \left(-\frac{{∥{\mathbf{x}}_s-{\mathbf{x}}_l∥}^2}{{\tau }_0}\right)\hfill \\ \hfill & \times \sigma \left(-\frac{{∥{\mathbf{x}}_s-{\sum }_{i\in {\mathcal{J}}_k\left({\mathbf{x}}_s\right)}{\mathbf{x}}_i\mu \left({\mathbf{x}}_s,{\mathbf{x}}_i\right)∥}^2}{{\tau }_j}\right).\hfill \end{array}$$

(32)

Finally, we observed the following optimization problem:

$$\begin{array}{cc}\hfill & {\displaystyle \underset{{\gamma }_1,\dots ,{\gamma }_T,ϵ}{min}\sum _{s=1}^n}\left(y_s-\frac{1}{M}{\displaystyle \sum _{j=1}^M}(1-{ϵ}_j){\displaystyle \sum _{k=1}^T}{C}_k^{\left(j\right)}\left({\mathbf{x}}_s\right)\right.\hfill \\ \hfill & {\left.-\frac{1}{M}{\displaystyle \sum _{j=1}^M}{ϵ}_j{\displaystyle \sum _{k=1}^T}{w}_k{D}_k\left({\mathbf{x}}_s\right)\right)}^2,\hfill \end{array}$$

(33)

subject to

$$\sum _{k=1}^T{w}_k=1,w_k\ge 0,k=1,\dots ,T.$$

Let us introduce new variables ${\gamma }_k^{\left(j\right)}=w_k{ϵ}_j$, $k=1,\dots ,T$, $j=1,\dots ,M$. Hence, we can write the optimization problem with the new $M·T+M$ variables as

$$\begin{array}{cc}\hfill & {\displaystyle \underset{{\gamma }_1,\dots ,{\gamma }_T,ϵ}{min}\sum _{s=1}^n}\left(y_s-\frac{1}{M}{\displaystyle \sum _{j=1}^M}(1-{ϵ}_j){\displaystyle \sum _{k=1}^T}{C}_k^{\left(j\right)}\left({\mathbf{x}}_s\right)\right.\hfill \\ \hfill & {\left.-\frac{1}{M}{\displaystyle \sum _{j=1}^M\sum _{k=1}^T}{\gamma }_k^{\left(j\right)}{D}_k\left({\mathbf{x}}_s\right)\right)}^2,\hfill \end{array}$$

(34)

subject to

$$\sum _{k=1}^T{\gamma }_k^{\left(j\right)}={ϵ}_j,{\gamma }_k^{\left(j\right)}\ge 0,k=1,\dots ,T,,j=1,\dots ,M,$$

(35)

$$0+\varsigma \le {ϵ}_j\le 1.$$

(36)

We again face the quadratic optimization problem with linear constraints. The corresponding model will be denoted as $ϵM$-$\mathbf{w}$-ARF or $ϵM$-$\mathbf{w}$-LARF, depending on applying the “leaf” attention. The notation means that the trainable parameters are $\mathbf{w}$ and ${ϵ}_1,\dots ,{ϵ}_M$. Additionally, we will use the same model, but with condition $w_k=1/T$ for all $k=1,\dots ,T$. The corresponding models are denoted as $ϵM$-ARF or $ϵM$-LARF.

6. Numerical Experiments

Let us introduce notations for different models of the attention-based RFs.

• RF (ERT): the original RF (the ERT) without applying attention mechanisms;
• ARF (LARF): the attention-based forest which has the following modifications: $ϵM$-ARF, $ϵM$-LARF, $ϵM$-w-ARF, and $ϵM$-w-LARF.

In all experiments, RFs as well as ERTs consist of 100 trees. To select the best tuning parameters in numerical experiments, a 3-fold cross-validation on the training set consisting of $n_{\mathrm{tr}}=4n/5$ examples with 100 repetitions is performed. The search for the best parameter ${\tau }_0$ is carried out by considering all its values in a predefined grid. A cross-validation procedure is subsequently used to select their appropriate values. The testing set for computing the accuracy measures is comprised of $n_{\mathrm{test}}=n/5$ examples. In order to obtain desirable estimates of the vectors ${\mathbf{A}}_k\left(\mathbf{x}\right)$ and $B_k\left(\mathbf{x}\right)$, all trees in the experiments are trained so that at least 10 examples fall into every leaf of a tree. Value 10 for the number of examples is taken for two reasons. On the one hand, we have to compute the mean vectors ${\mathbf{A}}_k\left(\mathbf{x}\right)$ and $B_k\left(\mathbf{x}\right)$ and to obtain unbiased estimators. On the other hand, it is difficult to expect that a large number of examples will fall into every leaf by a small number of training examples. Therefore, our prior experiments have demonstrated that this parameter should be 10. We also use all features at each split of decision trees.

We do not consider models $ϵ$-ARF, $\mathbf{w}$-ARF, $ϵ$-LARF, and $\mathbf{w}$-LARF, because they can be regarded as special cases of the corresponding models $ϵ$M-ARF, $ϵM$-$\mathbf{w}$-ARF, $ϵ$M-LARF, and $ϵM$-$\mathbf{w}$-LARF when $M=1$. The value of M is an integer tuning parameter, and it is tuned in the interval from 1 to 20. Set $\{{\tau }_1,\dots ,{\tau }_M\}$ of the softmax operation parameters is defined as $\left\{{10}^{-⌊M/2⌋},{10}^{-⌊M/2⌋+1},\dots ,{10}^0,\dots ,{10}^{⌊M/2⌋-1},{10}^{⌊M/2⌋}\right\}$. In particular, if $M=1$, then the set of $\tau$ consists of one element $\tau =1$. The parameter of the first-level attention in the “leaf” ${\tau }_0$ is taken equal to 1.

Numerical results are presented in tables where the best results are shown in bold. The coefficient of determination denoted $R^2$ and the mean absolute error (MAE) are used for the regression evaluation. The greater value of the coefficient of determination and the smaller MAE we have, the better results we achieve.

The proposed approach is studied by applying datasets which are taken from open sources. The dataset Diabetes is downloaded from the R Packages; datasets Friedman 1, 2 and 3 are retrieved from the site: https://www.stat.berkeley.edu/~breiman/bagging.pdf (accessed on 20 April 2023); datasets Regression and Sparse are taken from package “Scikit-Learn”; datasets Wine Red, Boston Housing, Concrete, Yacht Hydrodynamics, Airfoil can be found in the UCI Machine Learning Repository [44]. These datasets with their numbers of features m and numbers of examples n are given in

Table 2. A brief introduction about the regression data sets.

Data Set	Abbreviation	m	n
Diabetes	Diabetes	10	442
Friedman 1	Friedman 1	10	100
Friedman 2	Friedman 2	4	100
Friedman 3	Friedman 3	4	100
Scikit-Learn Regression	Regression	100	100
Scikit-Learn Sparse Uncorrelated	Sparse	10	100
UCI Wine red	Wine	11	1599
UCI Boston Housing	Boston	13	506
UCI Concrete	Concrete	8	1030
UCI Yacht Hydrodynamics	Yacht	6	308
UCI Airfoil	Airfoil	5	1503

. A more detailed information can be found from the aforementioned data resources.

Values of the measure $R^2$ for several models, including RF, $ϵM$-$\mathbf{w}$-ARF, $ϵM$-$\mathbf{w}$-LARF, $ϵM$-ARF, and $ϵM$-LARF, are shown in

Table 3. Values of $R^2$ for comparison of models based on the RF.

Data Set	${\mathit{\tau }}_0$	RF	$\mathit{ϵ}\mathit{M}$-w-ARF	$\mathit{ϵ}\mathit{M}$-w-LARF	$\mathit{ϵ}\mathit{M}$-ARF	$\mathit{ϵ}\mathit{M}$-LARF
Diabetes	$0.01$	$0.416$	$0.419$	$\mathbf{0}$.$\mathbf{434}$	$0.425$	$0.426$
Friedman 1	1	$0.459$	$0.470$	$\mathbf{0}$.$\mathbf{524}$	$0.438$	$0.472$
Friedman 2	1	$0.841$	$0.887$	$\mathbf{0}$.$\mathbf{933}$	$0.886$	$0.916$
Friedman 3	1	$0.625$	$0.708$	$\mathbf{0}$.$\mathbf{749}$	$0.675$	$0.704$
Airfoil	100	$0.823$	$0.844$	$0.914$	$0.822$	$\mathbf{0}$.$\mathbf{917}$
Boston	1	$0.814$	$0.820$	$\mathbf{0}$.$\mathbf{870}$	$0.819$	$0.856$
Concrete	10	$0.845$	$0.857$	$\mathbf{0}$.$\mathbf{896}$	$0.844$	$\mathbf{0}$.$\mathbf{896}$
Wine	1	$0.433$	$0.421$	$\mathbf{0}$.$\mathbf{481}$	$0.421$	$0.477$
Yacht	$0.1$	$0.981$	$0.989$	$\mathbf{0}$.$\mathbf{993}$	$0.981$	$0.982$
Regression	$0.1$	$0.380$	$0.434$	$\mathbf{0}$.$\mathbf{455}$	$0.361$	$0.409$
Sparse	1	$0.470$	$0.489$	$\mathbf{0}$.$\mathbf{641}$	$0.535$	$0.630$

The best obtained results on each dataset are shown in bold.

. The results are obtained by training the RF. The optimal values of ${\tau }_0$ are also given in the table. It can be observed from Table 3 that $ϵM$-$\mathbf{w}$-LARF outperforms all models for most datasets. Moreover, Table 3 shows that the two-level attention models ($ϵM$-$\mathbf{w}$-LARF and $ϵM$-LARF) provide better results than models which do not use the “leaf” attention ($ϵM$-$\mathbf{w}$-ARF and $ϵM$-ARF). It should be also noted that all attention-based models outperform the original RF. The same relationship between the models occurs for another accuracy measure (MAE). It is clearly shown in

Table 4. Values of MAE for comparison of models based on the RF.

Data Set	RF	$\mathit{ϵ}\mathit{M}$-w-ARF	$\mathit{ϵ}\mathit{M}$-w-LARF	$\mathit{ϵ}\mathit{M}$-ARF	$\mathit{ϵ}\mathit{M}$-LARF
Diabetes	$44.92$	$44.95$	$\mathbf{44}$.$\mathbf{61}$	$44.79$	$44.81$
Friedman 1	$2.540$	$2.545$	$\mathbf{2}$.$\mathbf{411}$	$2.595$	$2.473$
Friedman 2	$111.7$	$95.29$	$\mathbf{72}$.$\mathbf{71}$	$92.44$	$74.24$
Friedman 3	$0.154$	$0.130$	$0.135$	$0.144$	$\mathbf{0}$.$\mathbf{129}$
Airfoil	$2.203$	$2.065$	$1.451$	$2.217$	$\mathbf{1}$.$\mathbf{416}$
Boston	$2.539$	$2.538$	$\mathbf{2}$.$\mathbf{148}$	$2.489$	$2.217$
Concrete	$4.834$	$4.676$	$\mathbf{3}$.$\mathbf{496}$	$4.883$	$3.615$
Wine	$0.451$	$0.459$	$\mathbf{0}$.$\mathbf{411}$	$0.461$	$0.417$
Yacht	$1.004$	$0.787$	$\mathbf{0}$.$\mathbf{611}$	$1.004$	$0.971$
Regression	$109.1$	$103.6$	$\mathbf{101}$.$\mathbf{3}$	$111.2$	$105.8$
Sparse	$1.908$	$1.871$	$\mathbf{1}$.$\mathbf{528}$	$1.772$	$1.543$

The best obtained results on each dataset are shown in bold.

.

Another important question is how the attention-based models perform when the ERT is used. The corresponding values of $R^2$ and MAE are shown in

Table 5. Values of $R^2$ for comparison of models based on the ERT.

Data Set	${\mathit{\tau }}_0$	ERT	$\mathit{ϵ}\mathit{M}$-w-ARF	$\mathit{ϵ}\mathit{M}$-w-LARF	$\mathit{ϵ}\mathit{M}$-ARF	$\mathit{ϵ}\mathit{M}$-LARF
Diabetes	$0.01$	$0.438$	$0.441$	$0.434$	$\mathbf{0}$.$\mathbf{471}$	$0.444$
Friedman 1	1	$0.471$	$0.471$	$\mathbf{0}$.$\mathbf{524}$	$0.441$	$0.495$
Friedman 2	10	$0.813$	$0.840$	$\mathbf{0}$.$\mathbf{933}$	$0.840$	$0.919$
Friedman 3	1	$0.570$	$0.569$	$\mathbf{0}$.$\mathbf{749}$	$0.569$	$0.637$
Airfoil	100	$0.802$	$0.804$	$\mathbf{0}$.$\mathbf{914}$	$0.804$	$0.909$
Boston	10	$0.831$	$0.834$	$0.870$	$0.834$	$\mathbf{0}$.$\mathbf{882}$
Concrete	10	$0.839$	$0.838$	$\mathbf{0}$.$\mathbf{896}$	$0.838$	$0.895$
Wine	1	$0.418$	$0.418$	$0.481$	$0.418$	$\mathbf{0}$.$\mathbf{486}$
Yacht	1	$0.988$	$0.988$	$\mathbf{0}$.$\mathbf{993}$	$0.988$	$\mathbf{0}$.$\mathbf{993}$
Regression	$0.1$	$0.402$	$0.429$	$0.455$	$0.429$	$\mathbf{0}$.$\mathbf{464}$
Sparse	1	$0.452$	$0.522$	$0.641$	$0.522$	$\mathbf{0}$.$\mathbf{663}$

The best obtained results on each dataset are shown in bold.

and

Table 6. Values of MAE for comparison of models based on the ERT.

Data Set	ERT	w-ARF	w-LARF	$\mathit{ϵ}\mathit{M}$-ARF	$\mathit{ϵ}\mathit{M}$-LARF
Diabetes	$44.549$	$44.271$	$44.614$	$44.271$	$\mathbf{44}$.$\mathbf{21}$
Friedman 1	$2.502$	$2.502$	$2.411$	$2.502$	$\mathbf{2}$.$\mathbf{388}$
Friedman 2	$123.0$	$113.7$	$72.71$	$113.7$	$\mathbf{70}$.$\mathbf{48}$
Friedman 3	$0.179$	$0.179$	$\mathbf{0}$.$\mathbf{135}$	$0.179$	$0.148$
Airfoil	$2.370$	$2.360$	$\mathbf{1}$.$\mathbf{451}$	$2.360$	$1.471$
Boston	$2.481$	$2.451$	$2.148$	$2.451$	$\mathbf{2}$.$\mathbf{023}$
Concrete	$5.119$	$5.124$	$\mathbf{3}$.$\mathbf{496}$	$5.124$	$3.659$
Wine	$0.464$	$0.464$	$\mathbf{0}$.$\mathbf{411}$	$0.464$	$0.412$
Yacht	$0.824$	$0.822$	$\mathbf{0}$.$\mathbf{611}$	$0.822$	$0.612$
Regression	$106.3$	$103.1$	$101.3$	$103.1$	$\mathbf{100}$.$\mathbf{0}$
Sparse	$1.994$	$1.820$	$1.528$	$1.820$	$\mathbf{1}$.$\mathbf{519}$

The best obtained results on each dataset are shown in bold.

, respectively. Table 5 also contains the optimal values ${\tau }_0$. In contrast to the case of using the RF, it can be observed from the tables that $ϵM$-LARF outperforms $ϵM$-$\mathbf{w}$-LARF for several models. It can be explained by reducing the accuracy due to a larger number of training parameters (parameters $\mathbf{w}$) and overfitting for small datasets. It is also worth noting that models based on ERTs provide better results than models based on RFs. However, this improvement is not significant. This is observed in

Table 7. Comparison of the best results provided by models based on RFs and ERTs.

Data Set	RF	ERT
Diabetes	$0.434$	$\mathbf{0}$.$\mathbf{471}$
Friedman 1	$\mathbf{0}$.$\mathbf{524}$	$\mathbf{0}$.$\mathbf{524}$
Friedman 2	$\mathbf{0}$.$\mathbf{933}$	$\mathbf{0}$.$\mathbf{933}$
Friedman 3	$\mathbf{0}$.$\mathbf{749}$	$\mathbf{0}$.$\mathbf{749}$
Airfoil	$\mathbf{0}$.$\mathbf{917}$	$0.914$
Boston	$0.870$	$\mathbf{0}$.$\mathbf{882}$
Concrete	$\mathbf{0}$.$\mathbf{896}$	$\mathbf{0}$.$\mathbf{896}$
Wine	$0.481$	$\mathbf{0}$.$\mathbf{486}$
Yacht	$\mathbf{0}$.$\mathbf{993}$	$\mathbf{0}$.$\mathbf{993}$
Regression	$0.455$	$\mathbf{0}$.$\mathbf{464}$
Sparse	$0.641$	$\mathbf{0}$.$\mathbf{663}$

The best obtained results on each dataset are shown in bold.

, where the best results are collected for models based on ERTs and RFs. Table 7 shows that the results are identical for several datasets, namely, for datasets Friedman 1, 2, 3, Concrete, and Yacht. If one is to apply the t-test to compare the values of $R^2$ obtained for two models, then, according to [45], the t-statistics is distributed in accordance with the Student distribution with the $11-1$ degrees of freedom (11 datasets). The obtained p-value is $p=0.071$. We can conclude that the outperformance of the ERT is not statistically significant because $p>0.05$.

We take the number of trees in RFs equal to 100, because our goal is to compare RFs and the proposed modifications of LARF with the same parameters of numerical experiments. We also study how values of $R^2$ depend on different numbers of decision trees for the considered datasets. The corresponding numerical results for the RF and the ERT are shown in

Table 8. Values of $R^2$ for comparison of RFs by different numbers of decision trees.

Data Set	Numbers of Trees
	100	400	700	1000
Diabetes	$0.416$	$0.418$	$0.418$	$0.419$
Friedman 1	$0.459$	$0.465$	$0.465$	$0.465$
Friedman 2	$0.841$	$0.836$	$0.838$	$0.839$
Friedman 3	$0.625$	$0.625$	$0.626$	$0.627$
Airfoil	$0.823$	$0.824$	$0.824$	$0.824$
Boston	$0.814$	$0.815$	$0.816$	$0.816$
Concrete	$0.845$	$0.847$	$0.847$	$0.847$
Wine	$0.433$	$0.434$	$0.434$	$0.434$
Yacht	$0.981$	$0.982$	$0.982$	$0.982$
Regression	$0.380$	$0.397$	$0.398$	$0.399$
Sparse	$0.470$	$0.470$	$0.471$	$0.473$

and

Table 9. Values of $R^2$ for comparison of ERTs by different numbers of decision trees.

Data Set	Numbers of Trees
	100	400	700	1000
Diabetes	$0.438$	$0.439$	$0.439$	$0.439$
Friedman 1	$0.471$	$0.475$	$0.474$	$0.475$
Friedman 2	$0.813$	$0.813$	$0.814$	$0.815$
Friedman 3	$0.570$	$0.565$	$0.567$	$0.568$
Airfoil	$0.802$	$0.803$	$0.803$	$0.803$
Boston	$0.831$	$0.831$	$0.832$	$0.833$
Concrete	$0.839$	$0.840$	$0.840$	$0.840$
Wine	$0.418$	$0.418$	$0.418$	$0.418$
Yacht	$0.988$	$0.988$	$0.988$	$0.988$
Regression	$0.402$	$0.390$	$0.387$	$0.390$
Sparse	$0.452$	$0.455$	$0.455$	$0.456$

, respectively, by 100, 400, 700, and 1000 trees. It can be observed in Table 8 and Table 9 that $R^2$ insignificantly increases with the number of trees. However, it is important to point out that the largest values of $R^2$ for RFs and ERTs obtained by $T=1000$ do not exceed the values of $R^2$ for LARF modifications presented in Table 3, Table 4, Table 5 and Table 6.

It should be pointed out that the proposed models can be regarded as extensions of the attention-based RF ($ϵ$-ABRF) presented in [8]. Therefore, it is also worth comparing the two-level attention models with $ϵ$-ABRF.

Table 10. Comparison of $ϵ$-ABRF and the proposed models by using the measure $R^2$ when RFs and ERTs are the basis.

Data Set	RF		ERT
	$\mathit{ϵ}$-ABRF	LARF	$\mathit{ϵ}$-ABRF	LARF
Diabetes	$0.424$	$\mathbf{0.434}$	$0.441$	$\mathbf{0.471}$
Friedman 1	$0.470$	$\mathbf{0.524}$	$0.513$	$\mathbf{0.524}$
Friedman 2	$0.877$	$\mathbf{0.933}$	$0.930$	$\mathbf{0.933}$
Friedman 3	$0.686$	$\mathbf{0.749}$	$0.739$	$\mathbf{0.749}$
Airfoil	$0.843$	$\mathbf{0.917}$	$0.837$	$\mathbf{0.914}$
Boston	$0.823$	$\mathbf{0.870}$	$0.838$	$\mathbf{0.882}$
Concrete	$0.857$	$\mathbf{0.896}$	$0.863$	$\mathbf{0.896}$
Wine	$0.423$	$\mathbf{0.481}$	$0.416$	$\mathbf{0.486}$
Yacht	$0.989$	$\mathbf{0.993}$	$0.988$	$\mathbf{0.993}$
Regression	$0.450$	$\mathbf{0.455}$	$0.447$	$\mathbf{0.464}$
Sparse	$0.529$	$\mathbf{0.641}$	$0.536$	$\mathbf{0.663}$

The best obtained results on each dataset are shown in bold.

shows the values of $R^2$ obtained by using $ϵ$-ABRF and the best values of the proposed models when the RF and the ERT are used.

If we compare the results presented in Table 10 by applying the t-tests in accordance with [45], then tests for the proposed models and $ϵ$-ABRF based on the RF and the ERT provide p-values equal to $p=0.00067$ and $p=0.00029$, respectively. The tests demonstrate the clear outperformance of the proposed models in comparison with $ϵ$-ABRF.

It is obvious that the tuning parameters of the proposed modifications, for example, ${\tau }_0$, are not optimal due to the validation procedure and due to the grid of values used in experiments. However, we can observe from the numerical results that the proposed modifications outperform RFs or $\epsilon$-ABRF even with suboptimal values of the tuning parameters.

7. Conclusions

The new attention-based RF models proposed in the paper have supplemented the class attention models incorporated into machine learning models, differently from neural networks [7,8]. Moreover, the proposed models do not use gradient-based algorithms to learn attention parameters, and their training is based on solving the quadratic optimization problem with linear constraints. This peculiarity significantly simplifies the training process.

It is notable to point out that computing the attention weights in leaves of trees is a very simple task from the computational point of view. At the same time, this simple modification leads to the crucial improvement of the RF models. Numerical results with real data have demonstrated this improvement. This fact motivates us to continue developing attention-based modifications of machine learning models in different directions. First of all, the same approach can be applied to the gradient boosting machine [10]. The first successful attempt to use the attention mechanism in the gradient boosting machine with decision trees as base learners has been carried out in [7]. This attempt has confirmed that the boosting model can be improved by adding the attention component. The idea of the “leaf” attention and the optimization over parameters of kernels can be directly transferred to the gradient boosting machine. This is a direction for further research.

One of the important results presented in this paper is using a specific mixture of contamination models, which can be regarded as a variant of the well-known multi-head attention [16], where each “head” is defined by the kernel parameter. However, values of the parameter are selected in accordance with a predefined set. Therefore, the next direction is to consider randomized procedures to select the values of the parameter.

The proposed models consider only a single leaf of a tree for every example and implement the “leaf” attention in this leaf. However, they do not take into account neighbor leaves, which may also provide useful information for improving the models and should be studied.

It has been demonstrated in [8] that attention-based RFs allow us to interpret predictions by using the attention weights. The introduced two-level attention mechanisms may also improve the interpretability of RFs by taking into account additional factors.

Finally, we have developed the proposed modifications by using Huber’s $ϵ$-contamination model and the mixture of the models. Another notable problem is to consider different available statistical models [46] and their mixtures. A proper choice of the mixture components may significantly improve the whole attention-based RF.