Forgetful Forests: Data Structures for Machine Learning on Streaming Data under Concept Drift

Abstract

Database and data structure research can improve machine learning performance in many ways. One way is to design better algorithms on data structures. This paper combines the use of incremental computation as well as sequential and probabilistic filtering to enable “forgetful” tree-based learning algorithms to cope with streaming data that suffers from concept drift. (Concept drift occurs when the functional mapping from input to classification changes over time). The forgetful algorithms described in this paper achieve high performance while maintaining high quality predictions on streaming data. Specifically, the algorithms are up to 24 times faster than state-of-the-art incremental algorithms with, at most, a 2% loss of accuracy, or are at least twice faster without any loss of accuracy. This makes such structures suitable for high volume streaming applications.

1. Introduction

Supervised machine learning [1] tasks start with a set of labeled data. Researchers partition that data into training data and test data. They train their favorite models on the training data and then derive accuracy results on the test data. The hope is that these results will hold on to yet-to-be-seen data because the mapping between input data and output label (for classification tasks) does not change, i.e., is independent and identically distributed (i.i.d.).

This i.i.d. paradigm works well for applications such as medical research. In such settings, if a given set of lab results L indicates a certain diagnosis d at time t, then that same set of input measurements L will suggest diagnosis d at a new time t’.

However, there are many applications where the mapping between the input and output label changes: movie recommendations, variants of epidemics, market forecasting, or many real-time applications [2]. Predicting well in these non-i.i.d. settings is a challenge, but it also presents an opportunity to increase speed because a learning system can judiciously “forget” (i.e., discard) old data and learn a new input–output mapping on only the relevant data and thus do so quickly. In addition to discarding data cleverly, such a system can take advantage of the properties of the data structures to speed up their maintenance.

These intuitions underlie the basic strategy of the forgetful data structures we describe in this paper. As an overview, our methodology is to apply the intuitions of the state-of-the-art algorithms for streaming data with concept drift in a way that achieves high quality and high speed. This entails changing incremental decision tree building techniques as well as random forest maintenance techniques.

We will first introduce the design of our proposed algorithms in Section 3, then we will describe how we tuned the hyperparameters in Section 4, and finally, we will compare our algorithms with four state-of-the-art algorithms in Section 5. The comparison will be based on accuracy, F1 score (where appropriate), and time.

2. Related Work

The training process of many machine learning models is to take a set of training samples of the form $\left(x_1,y_1\right),\left(x_2,y_2\right),\dots ,\left(x_N,y_N\right)$, where in each training sample $\left(x_i,y_i\right)\in \left(X_{train},Y_{train}\right)$, $x_i$ is a vector of feature-values, and $y_i$ is a class label [3,4]. The goal is to learn a functional mapping from the $X$ values to the $y$ values. In the case when the mapping between $X$ and $y$ can change, an incremental algorithm will update the mapping as data arrives. Specifically, after receiving the k-th batch of training data, the parameters of the model $f$ may change to reflect that batch. At the end of the n-th training batch, the model $f_n$ can give a prediction of the following data point such that ${\widehat{y}}_{n+1}=f_n\left(x_{n+1}\right)$. This method of continuously updating the model on top of the previous model is called incremental learning [5,6].

Conventional decision tree methods, such as CART [7], are not incremental. Instead, they learn a tree from an initial set of training data once and for all. A naive incremental approach (needed when the data is not i.i.d.) would be to rebuild the tree from scratch periodically. However, rebuilding the decision tree can be expensive and if one waits too long, accuracy will suffer. State-of-the-art methods, such as VFDT [8] or iSOUP-Tree [9], incrementally update the decision tree with the primary goal of reducing memory consumption. We review various methods here below and outline what we learn from them.

2.1. Hoeffding Tree

In the Hoeffding Tree (or VFDT) [10], each node considers only a fixed subset of the training set, designated by a parameter $n$, and uses that data to choose the splitting attribute and value of that node. In this way, once a node has been fitted on $n$ data points, it will not be updated anymore.

The number of data points $n$ considered by each node is calculated using the Hoeffding bound [11], $n=\frac{R^{2}ln\left(1/\delta \right)}{2{ϵ}^2}$, where $R$ is the range of the variable, $\delta$ is the confidence fraction which is set by user, and $ϵ=\widehat{G}\left(x_1\right)-\widehat{G}\left(x_2\right)$ is the distance between the best splitter $x_1$ and the second best splitter $x_2$ based on the $\widehat{G}$ function. $\widehat{G}\left(\cdot \right)$ (e.g., information gain) is the measure used to choose spitting attributes. Thus, if $ϵ$ is large, $n$ can be small, because a big difference in, say, information gain gives us the confidence to stop considering other training points. Similarly, if $\delta$ is large, then intuitively we are allowed to be wrong with a higher probability, so $n$ can be small.

2.2. Adaptive Hoeffding Tree

The Adaptive Hoeffding Tree [12] will hold a variable-length window $W$ of recently seen data. We will have $1-\delta$ confidence that the splitting attribute has changed if any two sub-windows (say the older sub-window $w$ and the newer sub-window $w\prime$) of $W$ are “large enough”, and their heuristic measurements are “distinct enough”. To define “large enough” and “distinct enough”, the Adaptive Hoeffding tree uses the Hoeffding bound [11]: when $\left|\widehat{G}\left(w\right)-\widehat{G}\left(w\prime \right)\right|$ is larger than $2\ast ϵ$, where $\widehat{G}\left(\cdot \right)$ (e.g., information gain) is the measure used to choose spitting attributes. If the two sub-windows are “distinct enough”, the older data and the newer data have different best splitters. This means a concept drift has happened, and the algorithm will drop all data in the older sub-window in order to remove the data before the concept drift. While we adopt the intuition of dropping old data, we believe a gradual approach can work better in which more or less data can be dropped but in a continuous way depending on the amount of change in accuracy.

2.3. iSOUP-Tree

In contrast to the Hoeffding Tree, the iSOUP-Tree [9] uses the FIMT-MT method [13], which works as follows. There are two learners at each leaf to make predictions. One learner is a linear function $y=wx+b$ used to predict the result, where w and b are variables trained with the data and the results that have already arrived at this leaf, y is the prediction result, and X is the input data. The other learner computes the average value of the y from the training data seen so far. The learner with the lower absolute error will be used to make predictions. Different leaves in the same tree may choose different learners. To handle concept drift, the more recent data will get more weight in these predictions than the older data. We adopt the intuition of giving greater importance to newer data than to older data. Our method is to delete older data with greater probability than newer data. This achieves the same result as weighting and offers greater speed.

2.4. Adaptive Random Forest

In the adaptive random forest [14], each underlying decision tree is a Hoeffding Tree without early pruning. Without early pruning, different trees tend to be more diverse.

To detect concept drift, the adaptive random forest uses the Hoeffding bound described above. Further, each tree has two threshold interval levels to assess its performance in the face of concept drift. When the lower threshold level of a tree $T$ is reached (meaning $T$ has not been performing well) and $T$ has no background tree, the random forest will create a new background tree $T$′ that is trained like any other tree in the forest, but $T$′ will not influence the prediction. If tree $T$ already has a background tree $T″$, $T$ will be replaced by $T$″. When the higher threshold level of a tree $T$ is reached (meaning the tree has been performing very badly), even if no background tree $T^{\prime }$ is present, the random forest will delete $T$ and replace it with a new tree. We adopt the intuition of deleting trees that do not perform well. Our discarding rule is slightly more statistical in nature (with a t-test), but the intuition is the same as in this algorithm.

2.5. Ensemble Extreme Learning Machine

Ensemble Extreme Learning Machine [15] is a single hidden layer feedforward neural network whose goal is to classify time series. To detect concept drift, the method calculates and records the accuracy and the standard deviation of each data block, where a data block represents the data updated from the data stream each time. All data blocks are required to be the same size. Suppose that $p_i$ is the accuracy and $s_i$ is the standard deviation of the newest block i, $p_{best}$ and $s_{best}$ are the highest accuracy and the corresponding standard deviation recorded up to some point i-1 in the stream. Given that $ϵ$ is a hyperparameter representing the accuracy threshold, when $p_i+s_i<p_{best}+2\ast s_{best}$ and $ϵ<p_i$, the system will not change the model at all (for lack of evidence of concept drift). When $p_i+s_i\ge p_{best}+2\ast s_{best}$ but $ϵ<p_i$, the system will update the model using the new data (for evidence of mild concept drift). When $ϵ\ge p_i$, the system will forget all retained data and all previous accuracy and standard deviations recorded and the system will retrain the model with new data (for evidence of abrupt concept drift). Our problem and data structure are different because we are concerned with prediction on tree structures, but we appreciate the intuition of the authors’ detection and treatment of concept drift.

In summary, our forgetful data structures take much from the related work with two primary innovations: (i) We discard data in a continuous fashion with respect to the gain or loss in accuracy. (ii) We have designed our tree structures to support fast incremental updates for streaming data.

3. Forgetful Data Structures

This paper introduces both a forgetful decision tree and a forgetful random forest (having forgetful decision trees as components). These methods probabilistically forget old data and combine the retained old data with new data to track datasets that may undergo concept drift. In the process, the values of several hyperparameters are adjusted depending on the relative accuracy of the current model.

Given an incoming data stream, our method (i) saves time by maintaining the sorted order on each input attribute, (ii) efficiently rebuilds subtrees to process only incoming data whenever possible (i.e., whenever a split condition does not change). This works for both trees and forests, but because forests subsample the attributes, efficient rebuilding is more likely for forests.

To handle concept drift, we use three main ideas: (i) when the accuracy decreases, discard more of the older data and discard trees that have poor accuracy; (ii) when the accuracy increases, retain more old data; and (iii) vary the depth of the decision trees based on the size of the retained data.

The accuracy value in this paper is measured based on the confusion matrix: the proportion of true positive plus true negative instances relative to all test samples:

$$\frac{\left|TruePositive~in~testset\right|+\left|TrueNegative~in~testset\right|}{Size\left(testset\right)}$$

3.1. Forgetful Decision Trees

When a new incoming data batch is acquired from the data stream, the entire decision tree will be incrementally updated from the root node to the leaf nodes. The basic idea of this routine is to retain an amount of old data determined by an accuracy-dependent parameter called $rSize$ (set in the routine Adapt Parameters). Next, the forgetful decision tree is rebuilt recursively but avoids rebuilding the subtree of any node n when the splitting criterion on n does not change. In that case, the subtree is updated rather than rebuilt. Updating entails only sorting the incoming data batch and placing it into the tree, while rebuilding entails sorting the new data with retained data and rebuilding the subtree. When the incoming batch is small, updating is much faster than rebuilding. Please see the detailed pseudocode and its explanation in Appendix A.

3.2. Adaptive Calculation of Retain Size and Max Tree Height

Retaining a substantial amount of historical data will result in higher accuracy when there is no concept drift, because the old information is useful. When concept drift occurs, $rSize$ (the retained data) should be small, because old information will not reflect the new concept (which is some new mapping from input to label). Smaller $rSize$ will result in increased speed. Thus, changing $rSize$ can improve accuracy and reduce time. We use the following rules:

• When accuracy increases (i.e., the more recent predictions have been more accurate than previous ones) a lot, the model can make good use of more data. We want $rSize$ to increase with the effect so that we discard little or no data. When the accuracy increase is mild, the model has perhaps achieved an accuracy plateau, so we increase $rSize$, but only slightly.
• When accuracy changes little or not at all, we allow $rSize$ to slowly increase.
• When accuracy decreases, we want to decrease $rSize$ to forget old data, because this indicates that concept drift has happened. When concept drift is mild and accuracy decreases only a little, we want to retain more old data, so $rSize$ should decrease only a little. When accuracy decreases a lot, the new data may follow a completely different functional mapping from the old data, so we want to forget most of the old data, suggesting $rSize$ should be very small.

To achieve the above requirements, we adaptively change $rSize$. Besides that, we also adaptively change the maximum height of the tree ($maxHeight$). To avoid overfitting or underfitting, we will set $maxHeight$ to be monotonic with $rSize$. In addition, to handle the cold start at the very beginning, we will not forget any data until more than a certain size of data (call it $warmSize$) has arrived. We explain the details of adaptive parameters in Appendix B.

3.3. Forgetful Random Forest

The forgetful random forest is based on the forgetful decision tree described above in Section 3.1. Each random forest contains $nTree$ forgetful decision trees. The update and rebuild algorithms for each decision tree in the random forest are the same as those described in Section 3.1 except:

• Only a limited number of features are considered at each split, increasing the chance of updating (rather than rebuilding) subtrees during recursion, thus saving time by avoiding the need to rebuild subtrees from scratch. The number of features considered by each decision tree is uniformly and randomly chosen within the range $(\lfloor \sqrt{nFeatures}\rfloor +1,nFeatures]$, where $nFeatures$ is the number of features in the dataset. Further, every node inside the same decision tree considers the same features.
• The update function for each tree will randomly and uniformly discard old data without replacement, instead of discarding data based on time of insertion. Because this strategy does not give priority to newer data, this strategy increases the diversity of the data given to the trees.
• To decrease the correlation between trees and increase the diversity in the forest, we give the user the option to choose the leveraging bagging [11] strategy to the data arriving at each random forest tree. The size of the data after bagging is $W$ times the size of original data, where $W$ is a random number with an expected value of $6$, generated by a $Poisson\left(\lambda =6\right)$ distribution. To avoid the performance loss resulting from too many copies of the data, we never allow $W$ to be larger than $10$. Each data item in the expanded data is randomly and uniformly selected from the original data with replacement. We apply bagging to each decision tree inside the random forest.

When the overall random forest accuracy decreases, our method discards trees that suffer from particularly poor performance. If the decrease is large, then forgetful random forests discard many trees. If the decrease is small, then our method discards fewer. The discarded trees will be replaced with the same data but different features. We will not discard any tree unless we have enough confidence (which is $1-tThresh$) that the accuracy has decreased based on a two-sample t-test. The detailed pseudocode and accompanying explanation of the forgetful random forests are in Appendix C.

4. Tuning the Values of the Hyperparameters

Forgetful decision trees and random forests have six hyperparameters to set, which are (i) the evaluation function $G\left(\cdot \right)$ of the decision tree, (ii) the maximum height of trees $maxHeight$, (iii) the minimum size of retained data following initial cold start $warmSize$, (iv) for forgetful random forests, the empirical probability that concept drift has occurred 1-$tThresh$, (v) the number of trees in the forgetful random forest $nTree$, and (vi) the change in the retained data size $iRate$. To find the best values for these hyperparameters, we generated 18 datasets with different intensity of concept drifts, number of concept drifts, and Gaussian noise, using a generator inspired by Harvard Dataverse [16].

Our tests on the 18 new datasets indicated that some hyperparameters have optimal values (with respect to accuracy) that apply to all datasets. Others have optimal values that vary slightly depending on the dataset but can be learned. While the details are in Appendix D, here are the results of that study:

• The evaluation function can be a Gini Impurity [17] score or an entropy reduction coefficient. Which one is chosen does not make a material difference, so we set $G\left(\cdot \right)$ to entropy reduction coefficient for all datasets.
• The maximum tree height ($maxHeight$) is adaptively set based on the methods in Section 3.2 to log base 2 of $rSize$. Applying other stopping criteria does not materially affect the accuracy. For that reason, we ignore other stopping criteria.
• To mitigate the inaccuracies of the initial cold start, the model will not discard any data in cold startup mode. To leave cold startup mode, the accuracy should be better than random guessing on the last $50\%$ of data, when $rSize$ is at least $warmSize$. $warmSize$ adapts if it is too small, so we will set its initial value to $64$ data items.
• To avoid discarding trees too often, we will discard a tree only when we have $1-tThresh$ confidence that the accuracy has changed. We observe that all datasets enjoy a good accuracy when $tThresh=0.05$.
• There are $nTree$ forgetful decision trees in each forgetful random forest. We observe that the accuracy of all datasets stops growing after $nTree>20$ both with bagging and without bagging, so we will set $nTree=20$.
• The adaptation strategy in Section 3.2 needs an initial value for parameter $iRate$, which influences the increase rate of the retained data $rSize$. Too much data will be retained if the initial $iRate$ is large, but the accuracy will be low for lack of training data if the initial value of $iRate$ is small. We observe that the forgetful decision tree does well when $iRate=0.3$ or $iRate=0.4$ initially. We will use $iRate=0.3$ as our initial setting and use it in the experiments of Section 5 for all our algorithms, because most simulated datasets have higher accuracy at $iRate=0.3$ than at $iRate=0.4$.

5. Results

This section compares the following algorithms: forgetful decision tree, forgetful random forest with bagging, forgetful random forest without bagging, Hoeffding Tree [8,18], Hoeffding adaptive tree [12], iSOUP tree [9], train once, and adaptive random forest [14]. The forgetful algorithms use the hyperparameter settings from Section 4 on both the real datasets and the synthetic datasets produced by others.

We measure time consumption, the accuracy and, where appropriate, the F1 score.

The following settings yield the best accuracy for the state-of-the-art algorithms with which we compare:

• Previous papers [8,18] provide two different configurations for the Hoeffding tree. The configuration from [8] usually has the highest accuracy, so we will use it in the following experiment: $split\_confidence={10}^{-7}$, $grace\_period=200$, and $tie\_threshold=0.05$. Because the traditional Hoeffding tree cannot deal with concept drift, we set $leaf\_prediction=Naive Bayes Adaptive$ to allow the model to adapt when concept drift happens.
• The designers of the Hoeffding adaptive tree suggest six possible configurations of the Hoeffding adaptive tree, which are HAT-INC, HATEWMA, HAT-ADWIN, HAT-INC NB, HATEWMA NB, and HAT-ADWIN NB. HAT-ADWIN NB has the best accuracy, and we will use it in the following experiments. The configuration is $leaf\_prediction=Naive Bayes$, $split\_confidence=0.0001$, $grace\_period=200$, and $tie\_threshold=0.05$.
• The designers provide only one configuration for the iSOUP tree [9], so we will use it in the following experiment. The configuration is $leaf\_prediction=adaptive$, $split\_confidence=0.0001$, $grace\_period=200$, and $tie\_threshold=0.05$.
• For the train-once model, we will train the model only once with all of the data before starting to measure accuracy and other metrics. The train-once model is never updated again. In this case, we will use a non-incremental decision tree, which is the CART algorithm [7], to fit the model. We use the setting with the best accuracy, which is $criterion=gini$, and no other restrictions.

The designers of adaptive random forest provided six variant configurations of adaptive random forest [14]: the variants $AR{F}_{moderate}$, $AR{F}_{fast}$, $AR{F}_{PHT}$, $AR{F}_{noBkg}$, $AR{F}_{stdRF}$, and $AR{F}_{maj}$. $AR{F}_{fast}$ has the highest accuracy in most cases that we tested, so we will use that configuration: ${\delta }_w=0.01$, ${\delta }_d=0.001$, and $learners=100$.

5.1. Categorical Variables

Because the real datasets all contain categorical variables and the forgetful methods do not handle those directly, we modified the categorical variables into their one-hot encodings using the OneHotEncoder of scikit-learn [19]. For example, a categorical variable $color=\{R,G,B\}$ will be transferred to three binary variables $isR=\{True,False\}$, $isG=\{True,False\}$, and $isB=\{True,False\}$. In addition, all of the forgetful methods in the following tests use only binary splits at each node.

5.2. Metrics

In addition to accuracy, we use precision, $recall$, and F1 score to evaluate our methods. Precision and recall are appropriate to problems where there is a class of interest, and the question is which percentage of predictions of that class are correct (precision) and how many instances of that class are predicted (recall). This is appropriate for the detection of phishing websites. Accuracy is more appropriate in all other applications. For example, in the electricity datasets, price up and price down are both classes of interest. Therefore, we present precision, recall, and the F1 score for phishing only. We use the following formula based on the confusion matrix:

• $accuracy=\frac{\left|TruePositive\right|+\left|TrueNegative\right|}{Size\left(test-set\right)}$
• $precision=\frac{\left|TruePositive\right|}{\left|TruePositive\right|+\left|FalsePositive\right|}$
• $recall=\frac{\left|TruePositive\right|}{\left|TruePositive\right|+\left|FalseNegative\right|}$
• $F1score=\frac{\left|2\ast TruePositive\right|}{2\ast \left|TruePositive\right|+\left|FalseNegative\right|+\left|FalsePositive\right|}$

In contrast to i.i.d. machine learning tasks, we do not partition the data into a training set and a test set. Instead, when each batch of data arrives, we measure the accuracy and F1 score of the predictions on that batch, before we use the batch to update the models.

We start measuring accuracy and F1 score after the accuracy of the forgetful decision tree flattens out, in order to avoid evaluating the predictions during initial start-up when too little data has arrived to create a good model. For each dataset, all algorithms (state-of-the-art and forgetful) will start measuring the accuracy and F1 score after the same amount of data has arrived.

5.3. Datasets

We use four real datasets and two synthetic datasets to test the performance of our forgetful methods against the state-of-the-art incremental algorithms. The forest cover type, phishing, and power supply datasets suffer from frequent and mild concept drifts, the electricity dataset suffers from frequent and drastic concept drift, and the two synthetic datasets suffer from gradual and abrupt concept drifts, respectively. Thus, our test datasets cover a variety of concept drift scenarios.

• The forest cover type (ForestCover) [20] dataset captures images of forests for each $30\ast 30$ m cell determined from the US Forest Service (USFS) Region 2 Resource Information System (RIS) data. Each increment consists of 400 image observations. The task is to infer the forest type. This dataset suffers from concept drift because later increments have different mappings from input image to forest type than earlier ones. For this dataset, the accuracy first increases and then flattens out after the first $\mathrm{24,000}$ data items have been observed, out of $\mathrm{581,102}$ data items.
• The electricity [21] dataset describes the price and demand of electricity. The task is to forecast the price trend in the next $30$ min. Each increment consists of data from one day. This data suffers from concept drift because of market and other external influences. For this dataset, the accuracy never stabilizes, so we start measuring accuracy after the first increment, which is after the first $49$ data items have arrived, out of $\mathrm{36,407}$ data items.
• Phishing [22] contains $\mathrm{11,055}$ web pages accessed over time, some of which are malicious. The task is to predict which pages are malicious. Each increment consists of $100$ pages. The tactics of phishing purveyors get more sophisticated over time, so this dataset suffers from concept drift. For this dataset, the accuracy flattens out after the first $500$ data items have arrived.
• Power supply [23] contains three years of power supply records of an Italian electrical utility, comprising $\mathrm{29,928}$ data items. Each data item contains two features, which are the amount of power supplied from the main grid and the amount of power transformed from other grids. Each data item is labelled with the hour of the day when it was collected (from $0$ to $23$). The task is to predict the label from the power measurements. Concept drift arises because of season, weather, and the differences between working days and weekends. Each increment consists of $100$ data items, and the accuracy flattens out after the first $1000$ data items have arrived.
• The two synthetic datasets are from [16]. Both are time-ordered and are designed to suffer from concept drift over time. One, called Gradual, has $\mathrm{41,000}$ data points. Gradual is characterized by complete label changes that happen gradually over $1000$ data points at three single points, and $\mathrm{10,000}$ data items between each concept drift. Another dataset, called Abrupt, has $\mathrm{40,000}$ data points. It undergoes complete label changes at three single points, with $\mathrm{10,000}$ data items between each concept drift. Each increment consists of $100$ data points. Unlike the datasets that were used in Section 4, these datasets contain only four features, two of which are binary classes without noise, and the other two are sparse values generated by $sin\left(x\right)$ and $si{n}^{-1}\left(y\right)$, where $x$ and $y$ are the uniformly generated random numbers. For both datasets, the accuracy flattens out after $1000$ data items have arrived.

To measure the statistical stability in the face of the noise caused by the randomized setting of the initial seeds, we test all decision trees and random forests six times with different seeds and record the mean values with a $95\%$ confidence interval for time consumption, accuracy, and F1 score.

The following experiments are performed on an Intel Xeon Platinum 8268 24C 205W 2.9 GHz Processor with $200$ gigabytes of memory, Intel, Santa Clara, CA, USA.

5.4. Quality and Time Performance of the Forgetful Decision Tree

Figure 1 compares the time consumption of different incremental decision trees. For all datasets, the forgetful decision tree is at least three times faster than the other incremental methods.

Figure 2 compares the accuracy of different incremental decision trees and a train-once model. For all datasets, the forgetful decision tree is as accurate or more accurate than other incremental methods.

Figure 3 compares the precision, recall, and F1 score of different incremental decision trees. Because these metrics are not appropriate for other datasets, we use them only on the phishing dataset. The precision and recalls vary. For example, the Hoeffding adaptive tree has a better precision but a worse recall than the forgetful decision tree, while the iSOUP tree has a better recall but a worse precision. Overall, the forgetful decision tree has a similar F1 score to the other incremental methods.

5.5. Quality and Time Performance of the Forgetful Random Forest

Figure 4 compares the time performance of maintaining different random forests. From this figure, we observe that the forgetful random forest without bagging is the fastest algorithm. In particular, it is at least $24$ times faster than the adaptive random forest. The forgetful random forest with bagging is about $10$ times slower than without bagging, but it is still $2.5$ times faster than the adaptive random forest.

Figure 5 compares the accuracy of different random forests. From these figures, we observe that the forgetful random forest without bagging is slightly less accurate than the adaptive random forest (by at most $2\%$). By contrast, the forgetful random forest with bagging has a similar accuracy compared to the adaptive random forest. For some applications, the loss of accuracy might be acceptable in order to handle a high streaming data rate.

Figure 6 compares the precision, recall, and F1 score of training different random forests when these evaluations are appropriate. From these figures, we observe that the forgetful random forest without bagging has a lower precision, recall, and F1 score than the adaptive random forest (by at most $0.02$). By contrast, the forgetful random forest with bagging has a similar precision but a lower recall and F1 score (by at most $0.01$) compared to the adaptive random forest.

6. Discussion and Conclusions

Forgetful decision trees and forgetful random forests constitute simple, fast and accurate incremental data structure algorithms. We have found that:

• The forgetful decision tree is at least three times faster and at least as accurate as state-of-the-art incremental decision tree algorithms for a variety of concept drift datasets. When the precision, recall, and F1 score are appropriate, the forgetful decision tree has a similar F1 score to state-of-the-art incremental decision tree algorithms.
• The forgetful random forest without bagging is at least $24$ times faster than state-of-the-art incremental random forest algorithms, but is less accurate by at most $2\%$.
• By contrast, the forgetful random forest with bagging has a similar accuracy to the most accurate state-of-the-art forest algorithm (adaptive random forest) and is $2.5$ faster.
• At a conceptual level, our experiments show that it was possible to set hyperparameter values based on changes in accuracy on synthetic data and then apply those values to real data. The main such hyperparameters are $iRate$ (increase rate of size of data retained), $tThresh$ (the confidence interval that accuracy has changed), and $nTree$ (the number of decision trees in the forgetful random forests).
• Further our experiments show the robustness of our approach across a variety of applications where concept drift is frequent or infrequent, mild or drastic, and gradual or abrupt.

In summary, forgetful data structures speed up traditional decision trees and random forests for streaming data and help them adapt to concept drift. Further, bagging increases accuracy but at some cost in speed. The most pressing question for future work is whether some alternative method to bagging can be combined with forgetfulness to increase accuracy at less cost in a streaming concept drift setting.