Meta-Tree Random Forest: Probabilistic Data-Generative Model and Bayes Optimal Prediction

Abstract

This paper deals with a prediction problem of a new targeting variable corresponding to a new explanatory variable given a training dataset. To predict the targeting variable, we consider a model tree, which is used to represent a conditional probabilistic structure of a targeting variable given an explanatory variable, and discuss statistical optimality for prediction based on the Bayes decision theory. The optimal prediction based on the Bayes decision theory is given by weighting all the model trees in the model tree candidate set, where the model tree candidate set is a set of model trees in which the true model tree is assumed to be included. Because the number of all the model trees in the model tree candidate set increases exponentially according to the maximum depth of model trees, the computational complexity of weighting them increases exponentially according to the maximum depth of model trees. To solve this issue, we introduce a notion of meta-tree and propose an algorithm called MTRF (Meta-Tree Random Forest) by using multiple meta-trees. Theoretical and experimental analyses of the MTRF show the superiority of the MTRF to previous decision tree-based algorithms.

1. Introduction

Various studies in pattern recognition deal with a prediction problem of a targeting variable $y_{n+1}$ corresponding to an explanatory variable ${\mathit{x}}_{n+1}$ given pairs of explanatory and targeting variable ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$. In many of them, the targeting variable $y_{n+1}$ is predicted with a tree T. One way to use a tree T is to represent a function $y_{n+1}=f({\mathit{x}}_{n+1};T)$ and predict $y_{n+1}$. A tree T used to represent the function $y_{n+1}=f({\mathit{x}}_{n+1};T)$ is called a decision tree in the literature. In this paper, however, this tree is called a function tree. This is because we distinguish a tree used to represent a function from a tree used to represent a data-generative model (A tree used to represent a data-generative model will be explained in the next paragraph.). In the previous studies, algorithms in which a single function tree is used are discussed in, e.g., CART [1]; algorithms in which multiple function trees are used are discussed, e.g., Random Forest [2] is an algorithm that constructs multiple function trees from ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$ and aggregates them to predict $y_{n+1}$ from ${\mathit{x}}_{n+1}$. There are various extensions of Random Forest, e.g., Generalized Random Forest [3] generalizes the splitting rule of a function tree. Boosting is an algorithm that constructs a function tree sequentially from ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$ and combines the constructed function trees to predict $y_{n+1}$ from ${\mathit{x}}_{n+1}$. There are various Boosting methods, e.g., gradient boosting method in Gradient Boost [4] and XGBoost [5]. Further, previous studies, such as Alternating Decision Forest [6] and Boosted Random Forest [7], combine the ideas of Random Forest and Boosting method. Moreover, combinations of the function trees and neural networks have also been discussed. In Neural Decision Forest [8], inner nodes are replaced by randomized multi-layer perceptrons. In Deep Neural Decision Forest [9], they are replaced by deep convolutional neural networks (CNN). In Adaptive Neural Trees [10], not only their nodes are replaced by CNNs but also their edges are replaced by nonlinear functions. In Deep Forest [11], Random Forests are combined in a cascade structure like deep CNNs. The function tree is also used to understand the deep neural network as an interpretable input-output function in Reference [12]. In these algorithms, a function tree is used to represent the function from ${\mathit{x}}_{n+1}$ to $y_{n+1}$. Although this function is trained for the data ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$ under a certain criterion, statistical optimality for prediction of $y_{n+1}$ is not necessarily discussed theoretically because these algorithms usually do not assume a probabilistic data-generative structure of x and y. As we will describe later in detail, on the other hand, we assume a probabilistic data-generative structure of x and y, and consider a statistical optimal prediction of $y_{n+1}$. This is the crucial difference between our study and the related works.

To predict the targeting variable $y_{n+1}$, another way to use a tree T is to represent a data-generative model $p\left(y\right|\mathit{x},T)$ that represents a conditional probabilistic structure of y given $\mathit{x}$. A tree T used to represent the data-generative model $p\left(y\right|\mathit{x},T)$ is called a model tree throughout this paper. Although not so many studies have assumed the model tree, Reference [13] has assumed it. Because Reference [13] assumed that a targeting variable $y_{n+1}$ is generated according to $p\left(y\right|{\mathit{x}}_{n+1},T)$, statistical optimality for prediction of $y_{n+1}$ can be discussed theoretically. Specifically, based on the Bayes decision theory [14], Reference [13] proposed the optimal prediction of $y_{n+1}$.

In order to explain the optimal prediction based on the Bayes decision theory in more detail, we introduce the terminology model tree candidate set. The model tree candidate set is a set of model trees in which the true model tree (When the model tree T is used to represent the true data-generative model $p\left(y\right|\mathit{x},T)$, we call it true model tree.) is assumed to be included. One example of a model tree candidate set is a set of model trees whose depth is up to $d\in \mathbb{N}$. The optimal prediction based on the Bayes decision theory is given by weighting all the model trees in the model tree candidate set and the optimal weight of each model tree is given by the posterior probability $p\left(T\right|{\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n)$ of the model tree T.

Because the number of all the model trees in the model tree candidate set increases exponentially according to the maximum depth of model trees, the computational complexity of weighting them increases exponentially according to the maximum depth of model trees. One way to reduce the computational complexity is to restrict the model tree candidate set. To represent a restricted model tree candidate set, Reference [13] proposed a notion of meta-tree. The concept of a meta-tree was originally used for data compression in information theory (see, e.g., Reference [15]) (Recently, the concept of a meta-tree was also used for image compression [16].). As shown in Figure 1, a model tree candidate set is composed of the model trees represented by the meta-tree.

Figure 1. An example of a meta-tree and four model trees represented by it.

A meta-tree is also used for the prediction of $y_{n+1}$ in the algorithm proposed by Reference [13]. In summary, a meta-tree has the following two roles: (i) it represents a model tree candidate set; and (ii) it is used for the prediction algorithm. The characteristics of a meta-tree are as follows:

• If the true model tree is in a model tree candidate set represented by a meta-tree, the statistically optimal prediction—optimal prediction based on the Bayes decision theory—can be calculated.
• The number of model trees in a model tree candidate set represented by a meta-tree increases exponentially according to the depth of the meta-tree.
• The computational cost in learning processes of a single meta-tree has the same order as that of a single function tree.

Under the assumption that the true model tree is in the restricted model tree candidate set represented by a meta-tree, Reference [13] proposed the optimal prediction based on the Bayes decision theory.

As we have described above, if the true model tree is actually included in the model tree candidate set, the optimal prediction based on the Bayes decision theory is calculated. Hence, it is desirable that we construct the model tree candidate set that includes as many model trees as possible within the allowed computational cost. Motivated by this fact, this paper extends the model tree candidate set compared with Reference [13]. Instead of considering a single meta-tree as in Reference [13], we consider multiple meta-trees. By using the model tree candidate set represented by multiple meta-trees, we predict $y_{n+1}$ based on the Bayes decision theory. We call this proposed algorithm $\mathit{MTRF}(\mathit{Meta}$-$\mathit{Tree}\mathit{Random}\mathit{Forest})$. The characteristics of MTRF are as follows:

• If the true model tree is in a model tree candidate set represented by any of the meta-trees of MTRF, the statistically optimal prediction—optimal prediction based on the Bayes decision theory— can be calculated.
• The number of model trees in the model tree candidate set represented by multiple meta-trees is constant times larger than those contained in a single meta-tree.
• The computational cost in learning processes of meta-trees of MTRF is the same order as that of multiple function trees of Random Forest.

Regarding a meta-tree in Reference [13] or multiple meta-trees of MTRF, how to construct the meta-tree/multiple meta-trees is a problem. One way to solve this issue is to use the function tree-based algorithms; for example, a meta-tree in Reference [13] can be constructed by regarding a function tree in CART as a meta-tree; multiple meta-trees of MTRF can be constructed by regarding function trees in Random Forest as meta-trees. In this way, by regarding a function tree as a meta-tree, our proposed method can be applied to any practical applications in which a function-tree based algorithm is used—insurance claim task (e.g., Reference [5]), letter classification task (e.g., Reference [6]), semantic segmentation (e.g., Reference [8]), face recognition (e.g., Reference [11]), music classification (e.g., Reference [11]), etc.—and we can improve every previous function tree-based algorithm. This is because we can perform a prediction of $y_{n+1}$ by using more model trees compared with the previous algorithms. More detailed discussion will be given in Section 3 and Section 4.

The rest of this paper is organized as follows. In Section 2, we state the preliminaries of our study. Section 3 explains our proposed method—MTRF. In Section 4, we revisit the previous studies, such as CART [1] and Random Forest [2], and compare them with MTRF. Section 5 shows the results of experiments. Section 6 concludes this paper.

2. Preliminaries

2.1. Model Tree

Let K denote the dimension of explanatory variables. The space of feature values is denoted by $\mathcal{X}:=\{0,1,\dots ,|\mathcal{X}|-1\}$, and the explanatory variable is denoted by $\mathit{x}:={(x_1,\dots ,x_K)}^{\top }\in {\mathcal{X}}^K$. In addition, the space of label values is denoted by $\mathcal{Y}:=\{0,1\dots ,|\mathcal{Y}|-1\}$, and the targeting variable is denoted by $y\in \mathcal{Y}$.

In the field of pattern recognition, the probabilistic structure of x and y is not necessarily assumed. In particular, it is not assumed in most of function tree-based algorithms. In contrast, we consider a triplet of $(T,\mathit{k},\mathbf{\theta })$ defined in Definition 1 below and assume the probabilistic structure $p\left(y\right|\mathit{x},\mathbf{\theta },T,\mathit{k})$ as in Definition 2. We call $(T,\mathit{k})$ a model tree (In Section 1, we called T a model tree. More precisely speaking, however, a model tree is $(T,\mathit{k})$.) and $\mathbf{\theta }$ a tree parameter.

Definition 1.

The notation T denotes the $\left|\mathcal{X}\right|$-ary regular (Here, “regular” means that all inner nodes have $\left|\mathcal{X}\right|$ child nodes.) tree whose depth is up to $D_{\max }$, and $\mathcal{T}$ denotes the set of T. Let s be a node of a tree and $\mathcal{S}$ be a set of s. The set of the inner nodes of T is denoted by $\mathcal{I}\left(T\right)$, and the set of the leaf nodes of T is denoted by $\mathcal{L}\left(T\right)$. Let $k_s\in \{1,2,\dots ,K\}$ be a component of the feature assign vector of node s. The explanatory variable $x_{k_s}\in \mathcal{X}$ is assigned to the inner node $s\in \mathcal{I}\left(T\right)$. Let $\mathit{k}:={\left(k_s\right)}_{s\in \mathcal{S}}$ denote a feature assign vector, and $\mathcal{K}$ denote a set of $\mathit{k}$. Let ${\mathbf{\theta }}_s:=({\theta }_{0|s},{\theta }_{1|s},\dots ,{\theta }_{\left|\mathcal{Y}\right|-1|s})\in {(0,1)}^{\left|\mathcal{Y}\right|}$, where ${\sum }_{y=0}^{\left|\mathcal{Y}\right|-1}{\theta }_{y|s}=1$, be a parameter assigned to $s\in \mathcal{S}$. We define $\mathbf{\theta }:={\left({\mathbf{\theta }}_s\right)}_{s\in \mathcal{S}}$, and  Θ is the set of $\mathbf{\theta }$.

Definition 2.

Because the value of explanatory variable $\mathit{x}$ corresponds to a path from the root node to the leaf node of T whose feature assign vector is $\mathit{k}$, let $s\left(\mathit{x}\right)\in \mathcal{L}\left(T\right)$ denote the corresponding leaf node. Then, for given $\mathit{x}$, $\mathbf{\theta }$, T, and k, the label variable y is generated according to

$$\begin{array}{c}\hfill p\left(y\right|\mathit{x},\mathbf{\theta },T,\mathit{k}):={\theta }_{y\left|s\right(\mathit{x})}.\end{array}$$

(1)

Figure 2a shows two examples of $(T,\mathit{k},\mathbf{\theta })$, and Figure 2b illustrates $p\left(y\right|\mathit{x},\mathbf{\theta },T,\mathit{k})$.

Figure 2. (a) Two examples of $(T_1,{\mathit{k}}_1,{\mathbf{\theta }}_1)$ and $(T_2,{\mathit{k}}_2,{\mathbf{\theta }}_2)$, where $T_1$ and $T_2$ are 2-ary regular trees, ${\mathit{k}}_1=(k_{s_{\lambda }},k_{s_0})=(1,2)$, and ${\mathit{k}}_2=(k_{s_{\lambda }},k_{s_0},k_{s_1})=(3,4,2)$. (b) $p\left(y\right|\mathit{x}={(0,1)}^{\top },{\mathbf{\theta }}_1,T_1,{\mathit{k}}_1)={\theta }_{y|s_{01}}$.

2.2. Problem Setup

In this subsection, we introduce our problem setup. Let ${\mathit{x}}_i:={(x_{i1},\dots ,x_{iK})}^{\top }\in {\mathcal{X}}^K(i=1,\dots ,n)$ denote the value of explanatory variable of the i-th data and ${\mathit{x}}^n:=({\mathit{x}}_1,\dots ,{\mathit{x}}_n)$. We assume that ${\mathit{x}}_1,\dots ,{\mathit{x}}_n$ are i.i.d. random variables drawn from a probability distribution. Let $y_i\in \mathcal{Y}(i=1,\dots ,n)$ denote the targeting variable of the i-th data and $y^n:=(y_1,\dots ,y_n)$. We assume that $y_1,\dots ,y_n$ are generated according to $p\left(y\right|\mathit{x},\mathbf{\theta },T,\mathit{k})$ defined in Section 2.1. In regard to this, we assume that the true model tree $(T^{★},{\mathit{k}}^{★})$ and true tree parameter ${\mathbf{\theta }}^{★}$ are unknown, but a model tree candidate set $\mathcal{M}\subset \mathcal{T}\times \mathcal{K}$ is known. Note that, as we have explained in Section 1, the model tree candidate set $\mathcal{M}$ is a set of model trees $(T,\mathit{k})$ in which the true model tree is assumed to be included. Further, we assume that a class of parametrized distribution $\left\{p\right(y|\mathit{x},\mathbf{\theta },T,\mathit{k}):\mathbf{\theta }\in \Theta ,(T,\mathit{k})\in \mathcal{M}\}$ is known.

The purpose of the prediction problem is to predict unknown targeting variable $y_{n+1}$ corresponding to an explanatory variable ${\mathit{x}}_{n+1}$ by using the given data ${\mathit{x}}^n$, $y^n$, ${\mathit{x}}_{n+1}$.

2.3. Optimal Prediction Based on the Bayes Decision Theory

We introduce a statistically optimal prediction under the problem setup in Section 2.2. From the viewpoint of the Bayes decision theory [14], we shall derive the optimal prediction based on the Bayes decision theory. To this end, we assume priors $p\left(\mathbf{\theta }\right)$, $p\left(T\right)$, and $p\left(\mathit{k}\right)$.

First, we define a decision function as

$$\begin{array}{c}\hfill \delta :{\mathcal{X}}^{Kn}\times {\mathcal{Y}}^n\times {\mathcal{X}}^K\to \mathcal{Y};({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})\mapsto \delta ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1}).\end{array}$$

(2)

Second, we define a 0–1 loss as follows:

$$\begin{array}{cc}\hfill & l(y_{n+1},\delta ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})):=\left\{\begin{array}{c}0(y_{n+1}=\delta ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})),\hfill \\ 1(y_{n+1}\ne \delta ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})).\hfill \end{array}\right.\hfill \end{array}$$

(3)

Third, using the 0–1 loss, we define the loss function, which is the expectation of the 0–1 loss taken by new targeting variable $y_{n+1}$:

$$\begin{array}{cc}\hfill & L(\delta ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1}),\mathbf{\theta },T,\mathit{k}):=\sum _{y_{n+1}\in \mathcal{Y}}p\left(y_{n+1}\right|{\mathit{x}}_{n+1},\mathbf{\theta },T,\mathit{k})l(y_{n+1},\delta ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})).\hfill \end{array}$$

(4)

Next, given the loss function, we define the risk function, which is the expectation of the loss function taken by training data ${\mathit{x}}^n$, $y^n$, and ${\mathit{x}}_{n+1}$:

$$\begin{array}{cc}\hfill R& \left(\delta \right(·,·,·),\mathbf{\theta },T,\mathit{k})\hfill \end{array}$$

$$\begin{array}{c}\hfill :=\sum _{{\mathit{x}}^n\in {\mathcal{X}}^{Kn}}\sum _{y^n\in {\mathcal{Y}}^n}\sum _{{\mathit{x}}_{n+1}\in {\mathcal{X}}^K}p\left({\mathit{x}}_{n+1}\right)p\left({\mathit{x}}^n\right)p\left(y^n\right|{\mathit{x}}^n,\mathbf{\theta },T,\mathit{k})L(\delta ({\mathit{x}}^n,y^n{\mathit{x}}_{n+1}),\mathbf{\theta },T,\mathit{k}).\end{array}$$

(5)

Finally, the Bayes risk function, which is the expectation of the risk function taken by $\mathbf{\theta }$, T, and $\mathit{k}$, is defined as

$$\begin{array}{cc}\hfill \mathrm{BR}\left(\delta \right(·,·,·\left)\right):=& \sum _{(T,\mathit{k})\in \mathcal{M}}p\left(\mathit{k}\right)p\left(T\right){\int }_{\Theta }p\left(\mathbf{\theta }\right)R(\delta (·,·,·),\mathbf{\theta },T,\mathit{k})d\mathbf{\theta }.\hfill \end{array}$$

(6)

Then, we have the following theorem:

Theorem 1.

Under the setup in Section 2.2, the decision function ${\delta }^{*}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$ that minimizes the Bayes risk function is given as

$$\begin{array}{cc}\hfill {\delta }^{*}& ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})=\hfill \\ \hfill & \underset{y_{n+1}}{\arg \max }\sum _{(T,\mathit{k})\in \mathcal{M}}p\left(\mathit{k}\right|{\mathit{x}}^n,y^n)p\left(T\right|{\mathit{x}}^n,y^n,\mathit{k}){\int }_{\Theta }p\left(\mathbf{\theta }\right|{\mathit{x}}^n,y^n,T,\mathit{k})p\left(y_{n+1}\right|{\mathit{x}}_{n+1},\mathbf{\theta },T,\mathit{k})\mathrm{d}\mathbf{\theta }.\hfill \end{array}$$

(7)

The proof of Theorem 1 is in Appendix A. We call ${\delta }^{*}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$ the Bayes decision.

2.4. Previous Study

In this subsection, we introduce how to calculate the Bayes decision in Reference [13]. Because the proof of Reference [13] lacks some explanation, we give more rigorous discussion in comparison. This is one of the contributions in this paper.

As we have explained in Section 1, Reference [13] introduced the concept of a “meta-tree” to represent a restricted model tree candidate set. For T and $\mathit{k}$, a meta-tree—denoted by ${\mathrm{M}}_{T,\mathit{k}}$—represents a model tree candidate set that is composed of model trees $(T^{\prime },{\mathit{k}}^{\prime })$ where $T^{\prime }$ is a sub-tree of T and ${\mathit{k}}^{\prime }=\mathit{k}$. An example of ${\mathrm{M}}_{T,\mathit{k}}$ is shown in Figure 3, where T is a complete tree with $D_{\max }=2$ and $\mathit{k}=(3,4,2)$. In this example, a meta-tree ${\mathrm{M}}_{T,\mathit{k}}$ represents a model tree candidate set which is composed of four model trees. A meta-tree is also used for the prediction of $y_{n+1}$ in the algorithm proposed by Reference [13]. In short, a meta-tree has the two roles: first, it represents a model tree candidate set, and, second, it is used for the prediction algorithm.

Figure 3. An example of a meta-tree ${\mathrm{M}}_{T,\mathit{k}}$, where T is a complete tree with $D_{\max }=2$ and $\mathit{k}=(3,4,2)$.

Let ${\mathcal{T}}_{{\mathrm{M}}_{T,\mathit{k}}}$ denote a set of T represented by a meta-tree ${\mathrm{M}}_{T,\mathit{k}}$ and let $\mathcal{M}={\mathcal{T}}_{{\mathrm{M}}_{T,\mathit{k}}}\times \left\{\mathit{k}\right\}$. Under the assumption that the true model tree is included in $\mathcal{M}$, the Bayes decision in Theorem 1 is expressed as

$$\begin{array}{cc}\hfill {\delta }^{*}& ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})\hfill \\ \hfill & =\underset{y_{n+1}}{\arg \max }\sum _{T^{\prime }\in {\mathcal{T}}_{{\mathrm{M}}_{T,\mathit{k}}}}p\left(T^{\prime }\right|{\mathit{x}}^n,y^n,\mathit{k}){\int }_{\Theta }p\left(\mathbf{\theta }\right|{\mathit{x}}^n,y^n,T^{\prime },\mathit{k})p\left(y_{n+1}\right|{\mathit{x}}_{n+1},\mathbf{\theta },T^{\prime },\mathit{k})\mathrm{d}\mathbf{\theta }.\hfill \end{array}$$

(8)

Now, (8) can be decomposed into two parts:

$$\begin{array}{cc}1.\hfill & \hfill q\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,T,\mathit{k}):={\int }_{\Theta }p\left(\mathbf{\theta }\right|{\mathit{x}}^n,y^n,T,\mathit{k})p\left(y_{n+1}\right|{\mathit{x}}_{n+1},\mathbf{\theta },T,\mathit{k})\mathrm{d}\mathbf{\theta }.\end{array}$$

(9)

$$\begin{array}{cc}2.\hfill & \hfill \tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k}):=\sum _{T^{\prime }\in {\mathcal{T}}_{{\mathrm{M}}_{T,\mathit{k}}}}p\left(T^{\prime }\right|{\mathit{x}}^n,y^n,\mathit{k})q\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,T^{\prime },\mathit{k}).\end{array}$$

(10)

Reference [13] exactly calculates (9) and (10) under some assumptions. We describe them in Section 2.4.1 and Section 2.4.2, respectively.

2.4.1. Expectation over the Parameters

In this subsection, we explain how to calculate (9). To calculate (9) analytically, a conjugate prior of ${\mathbf{\theta }}_s$ given T and $\mathit{k}$, is assumed.

Assumption 1.

We assume $p\left({\mathbf{\theta }}_s\right|T,\mathit{k})=\mathrm{Dirichlet}\left({\mathbf{\theta }}_s\right|{\mathbf{\alpha }}_s)$ for $s\in \mathcal{L}\left(T\right)$, where ${\mathbf{\alpha }}_s$ denotes the hyper-parameter of the Dirichlet distribution.

Under this assumption, $q\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,T,\mathit{k})$ has a closed form expression and (9) can be calculated exactly. In addition, $q\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,T,\mathit{k})$ has an important property. We describe this in the next lemma as a preparation of Section 2.4.2.

Lemma 1.

For any $T,T^{\prime }\in \mathcal{T}$, ${\mathit{x}}^n\in {\mathcal{X}}^{Kn}$, $y^n\in {\mathcal{Y}}^n$, and ${\mathit{x}}_{n+1}\in {\mathcal{X}}^K$, if $s\left({\mathit{x}}_{n+1}\right)\in \mathcal{L}\left(T\right)\cap \mathcal{L}\left(T^{\prime }\right)$, then

$$\begin{array}{cc}\hfill & q\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,T,\mathit{k})=q\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,T^{\prime },\mathit{k}).\hfill \end{array}$$

(11)

The proof of Lemma 1 is in Appendix B. From this lemma, the right- and left-hand sides of (11) are denoted by $q_{s\left({\mathit{x}}_{n+1}\right)}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})$ because they depend on not T and $T^{\prime }$ but $s\left({\mathit{x}}_{n+1}\right)$.

2.4.2. Summation over All Model Trees Represented by a Meta-Tree

In this subsection, we explain how to calculate (10). The exact calculation of (10) needs to take summation over all model trees represented by a single meta-tree. However, the computational complexity of this calculation is huge because the number of model trees increases exponentially according to the depth of meta-tree. Reference [13] solved this problem. The advantages of using the method proposed in Reference [13] are as follows:

• This method calculates (10) exactly.
• The computational complexity of calculating (10) in learning parameters is $O\left(n{D}_{\max }\right)$, which is the same as that of building a single function tree.

To execute this method, an assumption about the prior probability distribution of T—Assumption A2—is required. It should be noted that $p\left(T\right)$ is different from the construction method of function trees, given ${\mathit{x}}^n$ and $y^n$.

Assumption A2.

Let $g_s\in [0,1]$ denote a hyper-parameter assigned to any node $s\in \mathcal{S}$. Then, we assume the prior of $T^{\prime }\in {\mathcal{T}}_{{\mathrm{M}}_{T,\mathit{k}}}$ as

$$\begin{array}{c}\hfill p\left(T^{\prime }\right)=\prod _{s\in \mathcal{I}\left(T^{\prime }\right)}{g}_s\prod _{s^{\prime }\in \mathcal{L}\left(T^{\prime }\right)}(1-g_{s^{\prime }}),\end{array}$$

(12)

where $g_s=0$ for a leaf node s of a meta-tree ${\mathrm{M}}_{T,\mathit{k}}$.

Remark 1.

It should be noted that (12) is a probability distribution, i.e., ${\sum }_{T^{\prime }\in {\mathcal{T}}_{{\mathrm{M}}_{T,\mathit{k}}}}p\left(T^{\prime }\right)=1$. See Lemma A1 in Appendix C.

As we have described in Section 1, a meta-tree is used for the prediction of $y_{n+1}$ in the algorithm proposed by Reference [13]. By using a meta-tree ${\mathrm{M}}_{T,\mathit{k}}$, the recursive function to calculate the Bayes decision (10) is defined as follows.

Definition 3.

We define the following recursive function ${\tilde{q}}_s\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,{\mathrm{M}}_{T,\mathit{k}})$ for any node s on the path in ${\mathrm{M}}_{T,\mathit{k}}$ corresponding to ${\mathit{x}}_{n+1}$.

$$\begin{array}{cc}\hfill & {\tilde{q}}_s\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,{\mathrm{M}}_{T,\mathit{k}}):=\hfill \\ \hfill & \left\{\begin{array}{cc}{q}_s\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k}),\hfill & \left(\mathrm{if}s\mathrm{is}\mathrm{the}\mathrm{leaf}\mathrm{node}\mathrm{of}{\mathrm{M}}_{T,\mathit{k}}\right),\hfill \\ (1-g_{s|{\mathit{x}}^n,y^n})q_s\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})\hfill \\ +g_{s|{\mathit{x}}^n,y^n}{\tilde{q}}_{s_{\mathrm{child}}}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,{\mathrm{M}}_{T,\mathit{k}}),\hfill & \left(\mathrm{otherwise}\right),\hfill \end{array}\right.\hfill \end{array}$$

(13)

where $s_{\mathrm{child}}$ is the child node of s on the path corresponding to ${\mathit{x}}_{n+1}$ in ${\mathrm{M}}_{T,\mathit{k}}$, and $g_{s|{\mathit{x}}^n,y^n}$ is also recursively updated as follows:

$$\begin{array}{c}\hfill g_{s|{\mathit{x}}^i,y^i}:=\left\{\begin{array}{cc}{g}_s,\hfill & (\mathrm{if}i=0),\hfill \\ \frac{g_{s|{\mathit{x}}^{i-1},y^{i-1}}{\tilde{q}}_{s_{\mathrm{child}}}\left(y_i\right|{\mathit{x}}_i,{\mathit{x}}^{i-1},y^{i-1},{\mathrm{M}}_{T,\mathit{k}})}{{\tilde{q}}_s\left(y_i\right|{\mathit{x}}_i,{\mathit{x}}^{i-1},y^{i-1},{\mathrm{M}}_{T,\mathit{k}})},\hfill & \left(\mathrm{otherwise}\right).\hfill \end{array}\right.\end{array}$$

(14)

Now, (10) can be calculated as shown in the following theorem.

Theorem 2.

$\tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})$ can be calculated by

$$\begin{array}{cc}\hfill & \tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})={\tilde{q}}_{s_{\lambda }}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,{\mathrm{M}}_{T,\mathit{k}}),\hfill \end{array}$$

(15)

where $s_{\lambda }$ is the root node of ${\mathrm{M}}_{T,\mathit{k}}$. In addition, $p\left(T\right|{\mathit{x}}^{n+1},y^{n+1},\mathit{k})$ can be calculated as

$$\begin{array}{c}\hfill p\left(T\right|{\mathit{x}}^{n+1},y^{n+1},\mathit{k})=\prod _{s\in \mathcal{I}\left(T\right)}{g}_{s|{\mathit{x}}^{n+1},y^{n+1}}\prod _{s^{\prime }\in \mathcal{L}\left(T\right)}(1-g_{s^{\prime }|{\mathit{x}}^{n+1},y^{n+1}}).\end{array}$$

(16)

The proof of Theorem 2 is in Appendix C. Surprisingly, the computational complexity of calculating (10) is $O\left(n{D}_{\max }\right)$ by using this theorem.

3. Proposed Method

In this section, we introduce our proposed method. Here, we reconsider the general setup where $\mathcal{M}=\mathcal{T}\times \mathcal{K}$. Recall that the Bayes decision (7) is

$$\begin{array}{c}\hfill {\delta }^{*}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})=\underset{y_{n+1}}{\arg \max }\sum _{\mathit{k}\in \mathcal{K}}p\left(\mathit{k}\right|{\mathit{x}}^n,y^n)\tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k}),\end{array}$$

(17)

where $\tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})$ is defined as in (10). In (17), $\tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})$ can be calculated in the same way as in Section 2.4. However, regarding the calculation of (17), we need to calculate $\tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})$ for all $\mathit{k}$ that satisfies $p\left(\mathit{k}\right)>0$. Because this computational cost increases exponentially depending to $D_{\max }$, it is hard to calculate all of them in general. To reduce the computational complexity of (17), we restrict the model tree candidate set to the set represented by multiple meta-trees. The advantages of building multiple meta-trees are as follows:

• As we have explained in Section 2.4.2, the number of model trees represented by each of the meta-trees increases exponentially according to the depth of meta-trees. In addition, the number of model trees in multiple meta-trees is constant times larger than those contained in a single meta-tree.
• If the true model tree is a sub-tree of any of the meta-trees, the statistically optimal prediction—optimal prediction based on the Bayes decision theory—can be calculated.
• If we build B meta-trees, the computational cost of it in learning parameters is $O\left(nB{D}_{\max }\right)$, which is the same as that of building B function trees.

Now, we introduce the next assumption.

Assumption A3.

For B meta-trees ${\mathrm{M}}_{T_1,{\mathit{k}}_1},\dots ,{\mathrm{M}}_{T_B,{\mathit{k}}_B}$, we define ${\mathcal{K}}^{\prime }\{{\mathit{k}}_1,\dots ,{\mathit{k}}_B\}$. Then, we assume a uniform distribution on ${\mathcal{K}}^{\prime }$ as a prior probability distribution of $\mathit{k}\in \mathcal{K}$.

An example of ${\mathcal{K}}^{\prime }$ is shown in Figure 4.

Figure 4. An example of $\mathcal{K}=\{{\mathit{k}}_1,{\mathit{k}}_2,{\mathit{k}}_3\}$ and ${\mathcal{K}}^{\prime }=\{{\mathit{k}}_1,{\mathit{k}}_2\}$. In this example, the prior probability $p\left({\mathit{k}}_1\right)=p\left({\mathit{k}}_2\right)=1/2$ and $p\left({\mathit{k}}_3\right)=0$.

Next, we prove the following lemma about computing a posterior probability distribution of $\mathit{k}$. In fact, the recursive function we have defined in Definition 3 is also useful to calculate a posterior probability distribution on ${\mathcal{K}}^{\prime }$ sequentially. The proof of Lemma 2 is in Appendix D.

Lemma 2.

By decomposing ${\mathit{x}}^n$ and $y^n$ into the set of i-th data $({\mathit{x}}_i,y_i)(i=1,\dots ,n)$ and performing the recursive calculation of ${\tilde{q}}_s\left(y_i\right|{\mathit{x}}_i,{\mathit{x}}^{i-1},y^{i-1},{\mathrm{M}}_{T,\mathit{k}})$ as in (13), a posterior probability distribution of $\mathit{k}\in {\mathcal{K}}^{\prime }$ is expressed as follows:

$$\begin{array}{c}\hfill p\left(\mathit{k}\right|{\mathit{x}}^n,y^n)\propto \prod _{i=1}^n{\tilde{q}}_{s_{\lambda }}\left(y_i\right|{\mathit{x}}_i,{\mathit{x}}^{i-1},y^{i-1},{\mathrm{M}}_{T,\mathit{k}}).\end{array}$$

(18)

From Theorem 2 and Lemma 2, we immediately obtain the next theorem.

Theorem 3.

Let us consider the model tree candidate set represented by B meta-trees, i.e.,

$$\begin{array}{c}\hfill \bigcup _{b=1}^B\left({\mathcal{T}}_{{\mathrm{M}}_{T_b,{\mathit{k}}_b}}\times \left\{{\mathit{k}}_b\right\}\right).\end{array}$$

(19)

Then, if the true model tree is included in (19), the Bayes decision ${\delta }^{*}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$ can be calculated as

$$\begin{array}{cc}\hfill {\delta }^{*}& ({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})=\underset{y_{n+1}}{\arg \max }\sum _{\mathit{k}\in {\mathcal{K}}^{\prime }}p\left(\mathit{k}\right|{\mathit{x}}^n,y^n)\tilde{q}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,\mathit{k})\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & =\underset{y_{n+1}}{\arg \max }\sum _{\mathit{k}\in {\mathcal{K}}^{\prime }}{\tilde{q}}_{s_{\lambda }}\left(y_{n+1}\right|{\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,{\mathrm{M}}_{T,\mathit{k}})\prod _{i=1}^n{\tilde{q}}_{s_{\lambda }}\left(y_i\right|{\mathit{x}}_i,{\mathit{x}}^{i-1},y^{i-1},{\mathrm{M}}_{T,\mathit{k}}).\hfill \end{array}$$

(21)

If the true model tree is not included in (19), the right-hand side of (20) is a sub-optimal prediction of the Bayes decision.

By using Theorem 3, we can calculate the optimal (possibly sub-optimal) prediction of the Bayes decision effectively. We name this algorithm Meta-Tree Random Forest (MTRF). We summarize MTRF in Algorithm 1.

Algorithm 1 MTRF

Input:${\mathit{x}}_{n+1},{\mathit{x}}^n,y^n,B,g_s,{\mathbf{\alpha }}_s$ Output:${\delta }^{*}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$ Initialize parameters $g_s$. Construct B meta-trees ${\mathrm{M}}_{T_1,{\mathit{k}}_1},\dots ,{\mathrm{M}}_{T_B,{\mathit{k}}_B}$ and ${\mathcal{K}}^{\prime }:=\{{\mathit{k}}_1,\dots ,{\mathit{k}}_B\}$. for all$\mathit{k}\in {\mathcal{K}}^{\prime }$do  for $i=1,\dots ,n,n+1$do   Calculate ${\tilde{q}}_{s_{\lambda }}\left(y_i\right|{\mathit{x}}_i,{\mathit{x}}^{i-1},y^{i-1},{\mathrm{M}}_{T,\mathit{k}})$ with (13).   if $i\ne n+1$then    Renew parameters $g_{s|{\mathit{x}}^i,y^i}$ with (14).   end if  end for end for Calculate ${\delta }^{*}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$ with (20).

Remark 2.

One way to construct multiple meta-trees is to use the ideas of Random Forest. Random Forest builds function trees $\mathcal{FT}$ with the impurity (e.g., the entropy and the Gini coefficient) from the training data ${\mathit{x}}^n$ and $y^n$. By regarding $(T_1,{\mathit{k}}_1)\dots (T_B,{\mathit{k}}_B)$ that are built by Random Forest as meta-trees ${\mathrm{M}}_{T_1,{\mathit{k}}_1}\dots {\mathrm{M}}_{T_B,{\mathit{k}}_B}$, we can construct B meta-trees. In our experiments in Section 5, we adopt this method. The computational complexity of MTRF is $O\left(nB{D}_{\max }\right)$ in learning processes, which is the same as that of Random Forest. This is computable unless n, B, or $D_{\max }$ are not extremely huge. In addition, we can parallelize the procedure on each meta-tree.

4. CART and Random Forest Revisited and Comparison with MTRF

4.1. CART and Random Forest Revisited

Although CART [1] and Random Forest [2] do not assume the model tree and they use trees to express a function $y=f(\mathit{x};T)$ (i.e., function tree), they can be regarded as methods of constructing a model tree candidate set.

First, we consider algorithms that use a single function tree, such as CART [1]. Such algorithms can be regarded as selecting a single model tree and predicting $y_{n+1}$ by using the single function tree that corresponds to the single model tree. For example, the prediction of CART—denoted by ${\delta }_{\mathrm{CART}}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$—is given by

$$\begin{array}{c}\hfill {\delta }_{\mathrm{CART}}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})=\underset{y_{n+1}}{\arg \max }p\left(y_{n+1}\right|{\mathit{x}}_{n+1},\widehat{\mathbf{\theta }}({\mathit{x}}^n,y^n),\widehat{T},\widehat{\mathit{k}}),\end{array}$$

(22)

where $(\widehat{T},\widehat{\mathit{k}})$ and $\widehat{\mathbf{\theta }}({\mathit{x}}^n,y^n)$ denote an estimated model tree and tree parameter, respectively.

Second, as another example, let us consider Random Forest [2]. Random Forest can be regarded as selecting multiple model trees and predicting $y_{n+1}$ by weighting the multiple function trees that correspond to the multiple model trees. It should be noted that the number of function trees is equal to the number of model trees in Random Forest. The prediction of Random Forest—denoted by ${\delta }_{\mathrm{RF}}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})$—is given by

$$\begin{array}{c}\hfill {\delta }_{\mathrm{RF}}({\mathit{x}}^n,y^n,{\mathit{x}}_{n+1})=\underset{y_{n+1}}{\arg \max }\sum _{(T,\mathit{k})\in \widehat{\mathcal{M}}}\frac{1}{|\widehat{\mathcal{M}}|}p\left(y_{n+1}\right|{\mathit{x}}_{n+1},\widehat{\mathbf{\theta }}({\mathit{x}}^n,y^n),T,\mathit{k}),\end{array}$$

(23)

where $\widehat{\mathcal{M}}=\{({\widehat{T}}_1,{\widehat{\mathit{k}}}_1),\dots ,({\widehat{T}}_B,{\widehat{\mathit{k}}}_B)\}\subset \mathcal{M}$ denotes the set of model trees that are constructed by Random Forest.

4.2. Comparison of Random Forest with MTRF

Let us consider the situation where the trees that represent the function trees constructed by Random Forest are meta-trees of MTRF. Then, the size of model tree candidate set is far larger in MTRF than in Random Forest.

Random Forest can be regarded as approximating (9), (10) and (17) as in (23). Further, the approximation of the weight $p(T,\mathit{k}|{\mathit{x}}^n,y^n)\approx 1/\left|\widehat{\mathcal{M}}\right|$ is not accurate and the optimality of each function tree’s weight for predicting $y_{n+1}$ has not usually been discussed. In contrast, MTRF calculates the Bayes decision in (9) and (10) exactly and approximates (17) as in (20). Moreover, the optimal weights of T and $\mathit{k}$ are given by (16) and (18), respectively.

5. Experiments

Most machine learning algorithms require rich numerical experiments to confirm the improvement independent of a specific dataset. However, in our paper, such an improvement is theoretically explained in Section 3 and Section 4. If we redefine all the function trees constructed by any existing method as meta-trees, we can necessarily improve it in a sense of the Bayes risk function since our model tree candidate set contains all the model trees that correspond to the original function trees. This improvement is independent of both datasets and the original methods. Therefore, we perform our experiment only on three types of datasets and compare with the Random Forest. Similar results should be given in any other situation.

5.1. Experiment 1

Although the theoretical superiority of MTRF to Random Forest has been explained in Section 3 and Section 4, we performed Experiment 1 to make sure their performance on synthetic data. The procedure of Experiment 1 is the following:

• We initialize the hyper-parameter ${\mathbf{\alpha }}_s=(1/|\mathcal{X}|,\dots ,1/|\mathcal{X}\left|\right)$, which is assigned on each node s.
• To make true model trees diverse, we determinate an index of a true feature assign vector $\mathit{k}$ and the true model trees as below:
  • In the true model tree (A), we assign $x_j$ to the node whose depth is j.

    –

    true model tree (A-1): the complete 2-ary model tree with depth 4.

    –

    true model tree (A-2): the regular 2-ary model tree with depth 4. This model tree holds the structure that only left nodes have child nodes and right nodes does not.

  • In the true model tree (B), we assign all different variables to the node.

    –

    true model tree (B-1): the complete 2-ary model tree with depth 4.

• We determine parameters ${\mathbf{\theta }}_s$ according to the Dirichlet distribution.
• We generate K-dimensional n training data from the true model tree and generate K-dimensional 100 test data from the true model tree. From (7), Bayes decision takes the same value for any distribution $p\left({\mathit{x}}^n\right)$. Therefore, in our experiments, we set a probability distribution of $\mathit{x}$ as follows: Let $U\left(\right\{0,1\left\}\right)$ denote the uniform distribution on $\{0,1\}$. Then, a probability distribution of $\mathit{x}$ is expressed as

  $$\begin{array}{c}\hfill p\left(x_{ij}\right)\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }U\left(\{0,1\}\right)(i=1,\dots ,n,j=1,\dots ,K).\end{array}$$

  (24)
• We perform Random Forest that utilizes B function trees $(T_1,{\mathit{k}}_1),\dots ,(T_B,{\mathit{k}}_B)$. We set its impurity as the entropy and max-depth as $D_{\max }$. Afterward, we make B meta-trees ${\mathrm{M}}_{T_1,{\mathit{k}}_1},\dots ,{\mathrm{M}}_{T_B,{\mathit{k}}_B}$. (Of course, their max-depth are $D_{\max }$, too.) Then, we calculate the average prediction error rate of Random Forest and MTRF, where we set the hyper-parameter $g_s$ as $g_s=1/2$ in MTRF.
• We repeat Steps 3∼5Q times.

Figure 5 shows an example of these model trees (note that, for simplicity, we illustrate the model trees with depth 2).

Figure 5. An example of true model tree (A-1) (a), model tree (A-2) (b), and model tree (B-1) (c).

We set $\left|\mathcal{X}\right|=\left|\mathcal{Y}\right|=2$, $n=100,200,\dots ,1000,K=500,B=10,D_{\max }=2,4,6$, and $Q=5000$. The result is shown in Figure 6.

Figure 6. Relationships between the number of learning data n and the average prediction error rate where the true model tree is model (A-1) (a), model (A-2) (b), and model (B-1) (c).

From Figure 6, when we compare the performance of Random Forest with that of MTRF, we can confirm that MTRF outperforms Random Forest in all conditions. These results are theoretically reasonable because the trees that represent the function trees constructed by Random Forest are meta-trees of MTRF.

5.2. Experiment 2

In Experiment 2, we used real data (nursery school data and mushroom data) in the UCI repository (University of California, Irvine Machine Learning Repository). The procedure of Experiment 2 is mostly the same as that of Experiment 1, but the way of sampling data is different; instead of Steps 2, 3, and 4 in Experiment 1, we randomly sampled $n+t$ data from the whole dataset and divided them to n training data and t test data. We repeated the random sampling of the data for 2500 times.

The detail on nursery school data is as follows. The explanatory variables are discrete 8 attributes of a family whose child wants to enter a nursery school (e.g., finance, health, and family structure). The dimension of each attributes is up to 5. The targeting variable is whether the nursery school should welcome him/her or not. We want to consider that a new child should be welcomed with his attributes and learning data. We set $\left|\mathcal{X}\right|=5$, $\left|\mathcal{Y}\right|=2$, $n=100,200,\dots ,1000$, $t=1000$, $K=8$, $B=5$, and $D_{\max }=2,3,4$. The result is shown in Figure 7a.

Figure 7. Relationships between the number of the learning data n and the average prediction error rate on nursery school data (a)/mushroom data (b) in UCI repository.

The detail on mushroom data is as follows. The explanatory variables are discrete 22 attributes of a mushroom (e.g., smell, shape, surface, and color). The dimension of each attributes is up to 10. The targeting variable is whether the mushroom can be eaten, not be eaten, or unknown. We want to predict that a new mushroom can be eaten, with its attributes and learning data. We set $\left|\mathcal{X}\right|=22$, $\left|\mathcal{Y}\right|=3$,$n=100,200,\dots ,1000$, $t=1000$, $K=22$, $B=5$, and $D_{\max }=2,3,4$. The result is shown in Figure 7b.

From Figure 7, we can confirm MTRF outperforms Random Forest in all conditions.

6. Conclusions and Future Work

In this paper, we have considered a model tree and derived the Bayes decision. We have addressed the computational complexity problem of the Bayes decision by introducing a notion of meta-tree, and proposed MTRF (Meta-Tree Random Forest), which uses multiple meta-trees. As we have shown in Theorem 3, if the true model tree is included in the model tree candidate set represented by multiple meta-trees, MTRF calculates the Bayes decision with efficient computational cost. Even if the true model tree is not included in the model tree candidate set represented by multiple meta-trees, MTRF gives the approximation of the Bayes decision. The advantages of MTRF have been theoretically analyzed. In Section 4, we have explained that, if we redefine all the function trees constructed by Random Forest as meta-trees, we can necessarily improve it in a sense of the Bayes risk function since our model tree candidate set contains all the model trees that correspond to the original function trees. We have performed experiments in Section 5 to check the theoretical analysis.

As we have described in Section 1, it is desirable that we construct the model tree candidate set that includes as many model trees as possible within the allowed computational cost. This is because if the true model tree is included in the model tree candidate set, the Bayes decision is calculated. Hence, the main problem of MTRF is how to construct multiple meta-trees. In Section 5, we have constructed multiple meta-trees by using the idea of Random Forest. However, we can consider other ways to construct multiple meta-trees. One of the possible future directions is to use other ideas, such as Boosting. This is one of the future works in our research.