Structure Extension of Tree-Augmented Naive Bayes

Due to the simplicity and competitive classification performance of the naive Bayes (NB), researchers have proposed many approaches to improve NB by weakening its attribute independence assumption. Through the theoretical analysis of Kullback–Leibler divergence, the difference between NB and its variations lies in different orders of conditional mutual information represented by these augmenting edges in the tree-shaped network structure. In this paper, we propose to relax the independence assumption by further generalizing tree-augmented naive Bayes (TAN) from 1-dependence Bayesian network classifiers (BNC) to arbitrary k-dependence. Sub-models of TAN that are built to respectively represent specific conditional dependence relationships may “best match” the conditional probability distribution over the training data. Extensive experimental results reveal that the proposed algorithm achieves bias-variance trade-off and substantially better generalization performance than state-of-the-art classifiers such as logistic regression.


Introduction
Supervised classification is an important task in data-mining and pattern recognition [1]. It requires building a classifier that can map an unlabeled instance into a class label. Traditional approaches to classification problems include decision trees, logistic regression etc. More recently, Bayesian network classifiers (BNCs) have attracted more attention from researchers in terms of explicit, graphical, interpretable representation and competitive performance against state-of-the-art classifiers.
Among numerous BNCs, naive Bayes (NB) is an extremely simple and remarkably effective approach to classification [2]. It infers the conditional probability by assuming that the attributes are independent given the class label [3]. It follows logically that relaxing NB's independence assumption is a feasible and effective approach to build more powerful BNCs [4,5]. Researchers proposed to extend NB from 0-dependence BNC to 1-dependence BNCs [6,7] (e.g., tree-augmented naive Bayes or TAN), and then to arbitrary k-dependence BNCs [8,9] (e.g., k-dependence Bayesian classifier or KDB). These BNCs learn from training data and allow additional edges between attributes that capture the dependence relationships among them. These restricted BNCs also capture another assumption behind NB, i.e., every attribute is dependent on the class variable and thus the class is the root in the network.
Given a random instance x = (x 1 , · · · , x n ), where x i ∈ Ω X i , classification is done by applying Bayes rule to predict the class label y * that corresponds to the highest posterior probability of the class variable, i.e., y * = arg max P(y|x), where y ∈ Ω y . By using Bayes theorem, for restricted BNC we have y * = arg max P(y|x) = arg max P(y, x) P(x) = arg max P(y, x) = arg max P(x|y)P(y) The objective of restricted BNC learning is to induce a network (or a set of networks) that may "best match" the conditional probability distribution P(x|y) given different class labels over the training data and explicitly represent statements about conditional independence. Information theory, which is proposed by Shannon, has established mathematical basis for the rapid development of BN. Mutual information (MI) I(X i ; Y) is the most commonly used criterion to rank attributes for attribute sorting or filtering [10,11], and conditional mutual information (CMI) I(X i ; X j |Y) is used to measure conditional dependence between attribute pair X i and X j for identifying possible dependencies.
Among numerous proposals to improve the accuracy of NB by weakening its attribute independence assumption, TAN demonstrates remarkable classification performance, yet at the same time maintains the computational simplicity and robustness that characterize NB. However, it can only model 1-dependence relationships among attributes. The optimization process of BNCs is implemented in practice by using heuristic search techniques to find the best candidate over the space of possible networks. The search process relies on a scoring function that evaluates each network with respect to the training data, and then to search for the best network according to this function. The likelihood function, e.g., Kullback-Leibler divergence, plays a fundamental role in Bayesian statistics [12,13]. The likelihood principle states that all relevant information for inference is contained in the likelihood function for the observed data given the assumed statistical model. We prove from the viewpoint of Kullback-Leibler divergence that the difference between NB and its variations lies in different orders of CMIs represented by these augmenting edges in the tree-shaped network structure. The CMIs may vary greatly for different class labels. Thus, in this paper we propose to generalize TAN from 1-dependence BNC to arbitrary k-dependence one. Different sub-models of TAN are introduced to respectively represent specific conditional dependence relationships depending on y. The Bayes rule is applied to select the maximum of the joint probability distribution P(y, x) for classification. Extensive experimental results reveal that the proposed algorithm, called Extensive TAN (ETAN), achieves competitive generalization performance and outperforms several state-of-the-art BNCs such as KDB while retaining excellent computational complexity.

Prior Work
A BNC is a graphical representation of the joint probability distribution P(y, x). It comprises two components. Firstly, a directed acyclic graph G = (U , V ), where U = X ∪ Y. X = {X 1 , · · · , X n } and Y respectively represent the attributes and class variable. V represents the set of arcs or direct dependencies. Secondly, a set of parameters, which are usually conditional probability distributions for each attribute in U . Given a training data set D, the goal of learning a BNC is to find the Bayesian network B that best represents P(u) or P(y, x) and predicts the class label for an unlabeled instance by selecting arg max y P(y, x). According to the chain rule of joint probability, P(y, x) is calculated by P(u) = P(y, x) = P(y)P(x 1 |y)P(x 2 |x 1 , y) · · · P(x n |x 1 , · · · , x n−1 , y), For discrete probability distributions P(u) and Q(u), the Kullback-Leibler divergence (also called relative entropy) is a measure of distance between these two probability distributions and is defined to be [14] It is the expectation of the logarithmic difference between P(u) and Q(u), where the expectation is taken using P(u). In other words, it is also the difference between H Q (U ) and H P (U ). Suppose that B is a Bayesian network over U , P B (u) is the joint probability encoded in B, the Kullback-Leibler divergence between the expected P(u) in Equation (2) and P B (u) is where H B (U ) = − ∑ u P(u) log P B (u). The entropy function H P (U ) is the optimal number of bits needed to store all possible combinations of attribute values of U . Thus, KL(P||B) can measure the difference between the information quantity carried by D and that encoded in B. NB, which is the simplest BNC, involves no dependence in its network structure according to conditional independence assumption [15]. Figure 1a shows an example of its network structure. Hence, for NB, P NB (u) is calculated by P NB (u) = P(y)P(x 1 |y)P(x 2 |y) · · · P(x n |y), Thus, H NB (U ) can be calculated by y,x 1 ,··· ,x n P(y, x 1 , · · · , x n )logP(y)P(x 1 |y)P(x 2 |y) · · · P(x n |y) The remarkable classification performance of NB has stimulated the exploration of improving its classification performance [16]. However, the dependency relationships between attributes always violate this assumption in many learning tasks. Many methods [17][18][19][20] attempt to improve the classification performance of NB by relaxing its independence assumption, such as TAN.
TAN constructs the tree structure by finding a maximum weighted spanning tree [21] (see Figure 1b). The structure is determined by extending Chow-Liu tree [22], which uses CMI to measure the weight of arcs. The CMI between X i and X j given the class Y, I(X i ; X j |Y), is defined as follows [23], For each attribute X i ∈ X , its parent set is π i = {X j ∈ X | X j → X i ∈ V }. The learning procedure of TAN is shown in Algorithm 1.

Input:
Training data set D with attribute set X = {X 1 , · · · , X n } and the class Y. Output: TAN 1 Let G be a directed graph G = (U , V ), in which U is a set of attributes and V is a set of arcs. 2 U = {Y}, X * = Ø, V =Ø, where X * represents a set of attributes in G. 3 Calculate I(X i ; X j |Y) from D for each pair of attributes. 10 Calculate conditional probability distribution for a Bayesian network parameter Θ with G. 11 Let TAN be a BNC with Θ and G. 12 return TAN To illustrate the learning process of TAN, we take dataset Balance-Scale as an example. Dataset Balance-Scale is from the University of California Irvine (UCI) machine learning repository and has 625 instances, 4 attributes, and 3 class labels. As a 1-dependence BNC, TAN requires that each attribute can have at most 1 parent. In the first step we need to find the most significant dependence between attributes. As shown in Figure 2a, I(X 2 ; X 4 |Y) corresponds to the maximum of I(X i ; X j |Y) for any attribute pairs. Then arc X 2 − X 4 is added to the topology of TAN. In the second step, we need to find the next significant dependence relationship between attributes. As shown in Figure 2b, I(X 1 ; X 4 |Y) corresponds to the maximum of I(X i ; Π i |Y) where Π i ∈ {X 2 , X 4 } and X i ∈ {X 2 , X 4 }. Then arc X 1 − X 4 is added to the topology. The next iteration begins. As shown in Figure 2c, I(X 2 ; X 3 |Y) corresponds to the maximum of I(X i ; Π i |Y) where Π i ∈ {X 1 , X 2 , X 4 } and X i ∈ {X 1 , X 2 , X 4 }. Then arc X 2 − X 3 is added to the topology. Finally, there exist at least 1 dependence relationship between any attribute X i and other attributes and the learning procedure of TAN stops.
According to the structure of TAN, P TAN (u) is calculated by P TAN (u) = P(y)P(x 1 |y)P(x 2 |x 1 , y) · · · P(x n |π n , y) = P(y)P(x 1 |y) where π i ( i > 1) represents the parent attribute of X i ( i > 1). Correspondingly, H TAN (U ) can be calculated by y,x 1 ,··· ,x n P(y, x 1 , · · · , x n )logP(y)P(x 1 |y)P(x 2 |x 1 , y) · · · P(x n |π n , y) According to Equation (6) and Equation (9), the difference between H NB (U ) and H TAN (U ) can be calculated by Add an arc from Add an arc from Thus, the difference between H NB (U ) and H TAN (U ) is the summation of 1-order CMIs that correspond to the conditional dependence relationships among attributes. Equation (10) can clarify why TAN applies CMI to fully describe the 1-dependence relationships in the maximum weighted spanning tree. As TAN is a successful structure augmentation of NB, many researchers suggest that identifying significant dependencies can help to achieve more precise classification accuracy [24,25]. Ziebart et al. [26] model the selective forest-augmented naive Bayes by allowing attributes to be optionally dependent on the class. Jing and Pavlovi [27] presented the boosted BNC which greedily builds the structure with the arcs with the highest value of CMI.
KDB [28] extends the network structure further by using variable k to control the attribute dependence spectrum (see Figure 1c). KDB first sorts attributes by comparing MI I(X i ; Y). Suppose that the attribute order is {X 1 , X 2 , · · · , X n }, P KDB (u) is calculated by where Π i is the set of k parent attributes of X i when k < i ≤ n. Whereas when k ≥ i, X i takes the first i − 1 attributes in the order as its parent attributes. Correspondingly, H KDB (U ) can be calculated by According to Equation (6) and Equation (12), the difference between H NB (U ) and H KDB (U ) can be calculated by (13) Thus, the difference between H NB (U ) and H KDB (U ) is actually the summation of these CMIs of different orders.
Extending the network structure with high attribute dependence spectrum has become popular to improve the classification performance of BNCs [29]. Pernkopf and Bilmes [30] establish k-graphs via ranking attributes by means of a greedy algorithm and selecting the k best parents by scoring each possibility with the classification accuracy. Lu and Mineichi [31] propose k-dependence classifier chains with label-specific features and demonstrate the effectiveness of the method.

Extensive Tree-Augmented Naive Bayes
KDB allows us to construct classifiers at arbitrary points (values of k) along the attribute dependence spectrum. To build an ideal KDB, we need to learn how to maximize the Kullback-Leibler divergence shown in Equation (13). The original KDB sort attributes by comparing MI and uses a set of 1-order CMIs (e.g., I(X i ; X 1 |Y), I(X i ; X 2 |Y) · · · , I(X i ; X i−1 |Y)) rather than one higher-order CMI (e.g., I(X i ; X 1 , · · · , X i−1 |Y)) to measure the conditional dependencies between X i and its parent attributes. To illustrate the difference between these two measures, we take data set Census-income, which has 299,285 instances, 41 attributes and 2 class labels, for example to learn specific KDB, corresponding distributions of ∑ X j ∈Π i I(X i ; X j |Y) and I(X i ; Π i |Y) are shown in Figure 3, where the X-axis denotes the index of attributes sorted in the decreasing order of I(X i ; Π i |Y). From Figure 3, the distribution of ∑ X j ∈Π i I(X i ; X j |Y) differs greatly to that of I(X i ; Π i |Y), thus the former is not appropriate to approximate the latter.
CMI or I(X i ; X j |Y) is a popular measure to evaluate the dependency relationship between attributes, and the maximum weighted spanning tree learned by TAN describes the most significant dependencies in its 1-dependence structure. While I(X i ; X j |Y) may fail to discriminate the dependency relationships given different class labels. The definition of CMI shown in Equation (7) can be represented as follows, where I(X i ; X j |y) measures the informational correlation between X i and X j given specific class label y and is defined as follows, From Equation (1), for restricted BNCs the most important issue is how to deeply mine the significant conditional dependencies among attributes given class label y. Whereas from Equation (14), CMI or I(X i ; X j |Y) is just a simple uniform averaging of I(X i ; X j |y), and the latter assumes that the data set has been divided into |Y| subsets and each subset corresponds a specific class label. To illustrate the variety of I(X i ; X j |y) with different class labels, we present their comparison on data set Census-income in Figure 4. As Figure 4 shows, the distribution of I(X i ; X j |y) differs greatly as y changes. Correspondingly the network structures, in which the conditional dependencies are measured by I(X i ; X j |y), should be different.
The value of CMI Index of attributes 7 2 8 19 21 24 23 1 33 22 6 15 25 40 18 41 32 28 27 34 16 29 17 10 14 20 9 12 5 11 30 35 13 38 26 31   According to Figures 3 and 4, ∑ X j ∈Π i I(X i ; X j |Y) may not be able to approximate I(X i ; Π i |Y), and to discriminate the conditional dependence between attributes, I(X i , X j |y) rather than I(X i , X j |Y) is appropriate to measure the conditional dependencies implicated in different subspaces of training data. Thus, motivated by the learning schemes of TAN and KDB, the structure of the proposed BNC is just like an extension of TAN from 1-dependence BNC to arbitrary k-dependence BNC, and we respectively learn |Y| sub-classifiers from |Y| subspaces of training data. The attributes are sorted in such a way that the Kullback-Leibler divergence will be maximized and each attribute can have at most k parent attributes. Supposed that the attribute order is {X 1 , X 2 , · · · , X n }, from the chain rule of joint probability of BNC (see Equation (2)) we know that any possible parents of attribute X i must be selected from {X 1 , X 2 , · · · , X i−1 }. As a k-dependence BNC, ETAN uses a heuristic search strategy and the learning procedure of ETAN is divided into two parts: when i ≤ k + 1, all attributes in {X 1 , X 2 , · · · , X i−1 } will be selected as the parents of X i ; when i > k + 1, only k attributes in {X 1 , X 2 , · · · , X i−1 } will be selected as the parents of X i . The learning procedure of one sub-model of ETAN is shown in Algorithm 2.

Algorithm 2: Sub-ETAN (y,k).
Input: Training data set D with attribute set X = {X 1 , · · · , X n } and the specific class label y. Output: A sub-ETAN classifier 1 Let G y be a directed graph G y = (U y , V y ), in which U y is a set of attributes and V y is a set of arcs. 2 U y = {Y},X * y = Ø; V y =Ø; where X * y represents a set of attributes in G y . 3 Calculate I(X i ; X j |y) for each pair of attributes.
13 end 14 end 15 Calculate conditional probability distribution for a Bayesian network parameter Θ y with G y . 16 Let sub-ETAN be a BNC with Θ y and G y .

return sub-ETAN
To illustrate the learning process of ETAN, we also take dataset Balance-Scale as an example and set Y = y 1 , k = 2. In the first step we need to find the most significant dependence between attributes. As shown in Figure 5a, I(X 3 ; X 4 |y 1 ) corresponds to the maximum of I(X i ; X j |y 1 ) for any attribute pairs. Then arc X 3 − X 4 is added to the topology of ETAN. In the second step, we need to find the next significant dependence relationship between attributes. As shown in Figure 5b, I(X 2 ; X 3 , X 4 |y 1 ) corresponds to the maximum of I(X i ; Π i |y 1 ) where Π i ∈ {X 3 , X 4 } and X i ∈ {X 3 , X 4 }. Then arcs X 2 − X 4 and X 2 − X 3 are added to the topology. The next iteration begins. As shown in Figure 5c, I(X 1 ; X 3 , X 4 |y 1 ) corresponds to the maximum of I(X i ; Π i |y 1 ) where Π i ∈ {X 2 , X 3 , X 4 } and X i ∈ {X 2 , X 3 , X 4 }. Then arcs X 1 − X 3 and X 1 − X 4 are added to the topology. Finally, there exist at least k dependence relationship between any attribute X i and other attributes and the learning procedure of ETAN stops.
For testing instance x, its class label y will correspond to one of the |Y| candidate classifiers, whose structure may lead to the maximum of joint probability P(y, x), i.e., y * = arg max P(y, x|BNC). The prediction procedure of our proposed method, the extensive tree-augmented naive Bayes, is shown in Algorithm 3.
Add an arc from X to X . L=L / {X , X }, M =M U{X ,X }. L={X , X }, M ={X , X }. X = arg max I(X ; X X | y ), where X ∈ L , X , X ∈ M. 1 2 Add two arcs from X to X and X to X . L=L / X , M =M U X .
Add two arcs from X to X and X to X . L=L / X , M =M U X .  Figure 5. The learning procedure of ETAN with k = 2 on Balance-Scale.

Algorithm 3: The Extensive tree-augmented naive Bayes (ETAN).
Input: Training data set D with attribute set X = {X 1 , · · · , X n } and the class Y and a test instance x = (x 1 , · · · , x n ), and the parameter k. Output: The predicted class label y * 1 Let B be the model space of BNCs. Each sub-model is represented by B i , 1 ≤ i ≤ |Y|, where |Y| represents the number of class labels in Ω Y .

Experimental Results
We compare the performance of our proposed methods with other algorithms. All experiments are carried out on 40 data sets from UCI machine learning repository. Table 1 shows the details of each data set used, including the number of instances, attributes, and the class. These data sets are arranged in the order of the number of instances. For each data set, numeric attributes are discretized using Minimum Description Length discretization [32]. To allow the proposed algorithm to be compared with Weka's algorithms, missing values for qualitative attributes are replaced with modes and those for quantitative attributes are replaced with means from the training data.  [34].
In machine learning, zero-one loss is the most common function to measure the classification performance. Kohavi and Wolpert [35] presented a bias-variance decomposition of zero-one loss for analyzing supervised learning scenarios. Bias represents the systematic component of error, which measures how closely the classifier can describe surfaces for a data set. Variance represents the component of error that stems from sampling, which represents the sensitivity of the classifier to changes in training data. The estimation of these measures is using 10-fold cross validation to provide an accurate evaluation of the performance of algorithms. The experimental results of zero-one loss, bias and variance are shown in Tables A1-A3 respectively in Appendix A. Statistically, we employ win/draw/loss when two algorithms are compared with respect to a performance measure. A win and a draw respectively indicate that the algorithm has significantly and not significantly lower error than the comparator. We assess a difference as significant if the outcome of a one-tailed binomial sign test is less than 0.05. Base probability estimates with M-estimation which leads to more accurate probabilities are applied in our paper, where the value of M is 1 [36].

ETAN vs. BNCs
In this section, we present experimental results of our proposed algorithms, ETAN. Table 2 displays the win/draw/loss records summarizing the relative zero-one loss, bias, and variance of different algorithms. Cell [i, j] in Table 2 contains win/draw/loss records for the BNC on row i against the BNC on column j. As shown in Table 2, ETAN respectively performs significantly better than NB on 28 datasets in terms of zero-one loss. In particular, ETAN shows obvious advantages when compared with TAN (19 wins). ETAN has a clear edge over KDB with win/draw/loss records of 17/14/9. Ensemble of classifiers, e.g., AODE and WAODE, brings improvement in accuracy in the sense that small variations in the training set would lead them to produce very different models, which help achieve higher classification accuracy compared to these single-structure classifiers. ETAN is also an ensemble but has higher attribute dependence spectrum than AODE or WAODE. It retains the advantage over AODE and WAODE in terms of zero-one loss, although less significantly. As structure complexity increases, ETAN uses higher-order CMI to measure the high-dependence relationships that may help to improve the classification performance.
The win/draw/loss records of bias and variance are also shown in Table 2. Bias-variance analysis of BNCs is given in the following discussion. By modeling BNCs with respect to the class labels, ETAN makes full use of the training data. For bias, ETAN performs better than NB (32/4/4). When ETAN is compared with 1-dependence classifier, ETAN beats TAN on 17 datasets and loses on 8 datasets. As each sub-model in ETAN is k-dependence classifier, ETAN beats AODE on 20 datasets. It indicates that BNCs with more interdependencies can perform better in terms of bias. Variance-wise, NB performs the best, because the structure of NB is definite and insensitive to variations in training data. For single-structure BNCs, higher-dependence BNCs (e.g., KDB) performs worse than lower-dependence BNCs (e.g., TAN). This also holds for ensemble classifiers, and ETAN performs worse than AODE and WAODE. The reason may be that further dependence discovery will result in overfitting.
ETAN is a structural augmentation of TAN where every attribute takes the class variable and at most k other attributes as its parents. k-order CMI is introduced to measure the conditional dependencies among attributes and the final structure is an extended maximum spanning tree. This alleviates some of NB's independence assumption and therefore reduces its bias at the expense of increasing its variance. As can be seen from Table 2, ETAN performs better in terms of variance than NB on 4 datasets, i.e., Anneal, Vowel, Dis and Mushrooms. Each dataset has smaller number of instances and larger number of attributes. That may lead to sparsely distributed data and imprecise estimate of probability distribution. For lower quantities of data, the lower variance results in lower error for ETAN, while for larger quantities of data the lower bias results in lower error. ETAN may underfit the sparsely distributed training data that will lead to lower variance and then higher classification accuracy. The ideal datasets on which ETAN has better variance prediction accuracy are those with small data quantity and sparse data. For example, dataset Anneal has only 898 instances but 38 attributes and 6 class labels.

ETAN vs. Logistic Regression (LR)
In this section, our proposed algorithms are compared with the state-of-the-art algorithm, Logistic Regression (LR). LR can be viewed as a partially parametric approach [37], hence, a BNC can be mapped to a LR model [38]. We use LR's implementation in Weka, an open source provided by the University of Waikato for machine learning. Weka offers an improved implementation of LR, which use a quasi-Newton to search for the optimized values of attributes and considers the instance weights. The experimental results in terms of zero-one one loss, bias and variance have been shown in the fifth column of Tables A1-A3 in Appendix A. Table 3 shows the win/draw/loss results. Due to the computational constraints of LR, the size of data sets has an obvious effect on its training time; hence, we have not been able to learn the classification models for the two largest data sets. That is why the sum of the number of all cells is exactly 38. As we can see from Table 3, ETAN beats LR on 26 data sets, which means ETAN achieves better classification performance than LR. ETAN results in not only better bias performance on 22 data sets but also better variance performance on 21 data sets. In the other words, ETAN is difficult to be beaten by LR.
To further illustrate the advantages of our algorithms, we present the comparison results with respect to zero-one loss in Figure 6, where the X-axis represents the zero-one loss results of ETAN and the Y-axis represents those of LR. Most of the points in Figure 6 are above the diagonal lines, which means that our algorithms have shown better classification performance in general. LR is a popular binary classifier and attempts to predict outcomes based on a set of independent attributes. Among those data sets which correspond to the points under the diagonal lines, most of them have 2 class labels, less than 1000 instances and at least 8 attributes. Obviously, the sparsely distributed data and binary classification may be the main reasons why LR performs better. However, for data sets containing non-binary attributes, ETAN allows us to build (more expressive) non-linear classifiers, which is impossible for LR, unless one "binarizes" all attributes and this may artificially introduce noise.

Comparison of All Algorithms
To compare multiple algorithms over multiple data sets, Friedman test is used in the following discussion, which ranks the algorithms for each data set [39]. We calculate the rank of each algorithm for each data set separately (assign average ranks in case of tie). The null hypothesis is that there is no significant difference in the average ranks. The Friedman statistic is distributed according to χ 2 F with t − 1 degrees of freedom. For any level of significance α, the null hypothesis will be rejected if χ 2 F > χ 2 α . The critical value of χ 2 α for α = 0.05 with t − 1 = 6 degrees of freedom is 14.07. The Friedman statistic for zero-one loss is 37.90. Therefore, the null hypothesis is rejected.
As the null hypothesis is rejected, we perform Nemenyi test which is used to analyze which pairs of algorithms are significantly different [40]. If the difference between the average ranks of two algorithms is less than the critical difference (CD), their performance is significantly different. For these 7 algorithms and 38 data sets, the value of CD is 1.462.
The comparison of all algorithms against each other with Nemenyi test in terms of zero-one loss is shown in Figure 7. We plot the algorithms on the left line according to average ranks, the higher the position of algorithms, the lower average ranks will be, hence the better performance. As we can see, the rank of ETAN is significantly better than that of other algorithms. WAODE and AODE also achieve lower average rank than KDB, TAN, and NB. It indicates that ensemble classifiers may help to improve performance of the single-structure classifiers. The advantage of ETAN over ADOE and that of KDB over TAN may be attributed to the increasing attribute dependence spectrum.

Conclusions
Our work was primarily motivated by the consideration that the structure difference between NB and its variations can be measured by different orders of CMIs in terms of Kullback-Leibler divergence, and conditional dependencies between attributes may vary greatly for different class labels. In this paper, we provide a novel learning algorithm, ETAN, which extends TAN to arbitrary k-dependence BNC. The final network structure is similar to an extended version of maximum weighted spanning tree and corresponds to the maximum of sum of CMIs. ETAN substantially achieves better performance with respect to different evaluation functions and is highly competitive with the state-of-the-art higher-dependence BNC, e.g., KDB.

Conflicts of Interest:
The authors declare no conflict of interest.