Augmenting Naïve Bayes Classifiers with k-Tree Topology

Abstract

The Bayesian network is a directed, acyclic graphical model that can offer a structured description for probabilistic dependencies among random variables. As powerful tools for classification tasks, Bayesian classifiers often require computing joint probability distributions, which can be computationally intractable due to potential full dependencies among feature variables. On the other hand, Naïve Bayes, which presumes zero dependencies among features, trades accuracy for efficiency and often comes with underperformance. As a result, non-zero dependency structures, such as trees, are often used as more feasible probabilistic graph approximations; in particular, Tree Augmented Naïve Bayes (TAN) has been demonstrated to outperform Naïve Bayes and has become a popular choice. For applications where a variable is strongly influenced by multiple other features, TAN has been further extended to the k-dependency Bayesian classifier (KDB), where one feature can depend on up to k other features (for a given $k\ge 2$). In such cases, however, the selection of the k parent features for each variable is often made through heuristic search methods (such as sorting), which do not guarantee an optimal approximation of network topology. In this paper, the novel notion of k-tree Augmented Naïve Bayes (k-TAN) is introduced to augment Naïve Bayesian classifiers with k-tree topology as an approximation of Bayesian networks. It is proved that, under the Kullback–Leibler divergence measurement, k-tree topology approximation of Bayesian classifiers loses the minimum information with the topology of a maximum spanning k-tree, where the edge weights of the graph are mutual information between random variables conditional upon the class label. In addition, while in general finding a maximum spanning k-tree is NP-hard for fixed $k\ge 2$, this work shows that the approximation problem can be solved in time $O\left(n^{k+1}\right)$ if the spanning k-tree also desires to retain a given Hamiltonian path in the graph. Therefore, this algorithm can be employed to ensure efficient approximation of Bayesian networks with k-tree augmented Naïve Bayesian classifiers of the guaranteed minimum loss of information.

1. Introduction

Bayesian networks are acyclic graphical models that can effectively model complex random systems with probabilistic dependencies among random variables. The structured framework plays a crucial role in statistical classification tasks, where accurately assigning class labels to unclassified samples is highly desirable [1]. Bayesian classifiers assume $(n-1)$-dependency among n feature variables [2]; for instance, when computing the probabilities of various diseases for class labels, given n symptoms as features, the network assumes each feature may possibly be influenced by all other $n-1$ features. The utility of Bayesian classifiers extends across various application domains, including health care [3], object detection [4], document classification [5], fraud detection [6], and spam filtering [7], showcasing their versatility. While Bayesian classifiers allow for modeling arbitrary complex dependencies between features across different domains, their computational cost is notable [2]; in particular, to capture complex dependences and higher-order interactions among feature variables, many applications of Bayesian classifiers rely on computing joint probability distributions [8]. Therefore, as the number of features increases, the involved computational complexity escalates due to the potentially full dependencies among feature variables [9,10]. It is widely acknowledged that optimal learning of Bayesian networks is computationally intractable [11,12].

To cope with the intractability in computing with Bayesian networks, one viable approach is to approximate the joint probability distribution. Technically, this strategy is to relax constraints on dependencies between feature variables when their joint probability distribution is considered. The simplest form of the approximation for Bayes classifiers assumes zero dependencies among features, referred to as Naïve Bayes classifier [13]. Such classifiers operate under the assumption that each feature is conditionally independent of other features given the class label. The simplicity of the model makes it an appropriate choice for various applications, such as image classification [14], spam detection [15,16,17], and sentiment analysis [18,19,20]. However, while the independence assumption significantly simplifies the computation for joint probability, it may not be realistic, particularly in applications involving complex attribute dependencies [21] where performance can adversely be limited [12]. Therefore, in many practices, probabilistic graph approximations often prioritize structures where feature variables exhibit dependencies, such as those found in tree structures. Specifically, within a tree topology, each feature variable is influenced by at most one other feature variable [22].

In particular, tree augmented Naïve Bayes (TAN) is a preferred approximation of Bayesian networks. It is based on a tree-dependency structure assumption for feature variables [23]. TAN classifiers mitigate the independence assumption among features in Naïve Bayes while still maintaining lower time complexity compared to full Bayesian networks [12]. TAN classifiers demonstrate much better performance over Naïve Bayes in various applications and are a popular choice for many classification tasks. For example, they have found widespread use across diverse domains, including cryptocurrency trend classification [24], anomaly-based intrusion detection [25], facial biotype classification [26], and other prediction tasks [27]. However, in applications heavily characterized by complex attribute dependencies, TAN classifiers may struggle to represent the probabilistic graphical models accurately. This is especially true when one feature is strongly influenced by more than one other feature. For such applications, the tree topology of TAN has been further improved by considering more dependencies for each feature variable. In particular, the k-dependency Bayes classifier (KDB) [2] and its variant [28] have been introduced, which can represent feature dependencies with up to k feature parents, where $k\ge 2$. In other words, it allows each feature to be influenced by a maximum of k other variables. Unfortunately, in those studies, the construction of k-dependence tree classifiers relied on heuristic ranking methods to determine the k parents of each variable, which do not ensure the optimality of the resulting dependence topology [29].

In this paper, we introduce a novel approach to ensure optimal construction of topology for KDB. We coin k-tree augmented Naïve Bayes (k-TAN) for the Bayesian networks whose structure is of the k-tree topology. A k-tree is a tree-like graph where vertices (representing feature variables) can be defined in some desired order such that each vertex is introduced with respect to other k (parent) vertices that already exist. In other words, a k-TAN is a KDB where the k-dependency of feature variables follows the k-tree definition. In this work, we prove that the proposed k-TAN enables an optimal approximation of the Bayesian network of k-tree topology. In particular, we show that, under the KL-divergence measurement [30], the k-tree topology approximation has the minimum information loss if the structure topology is taken from a maximum spanning k-tree. In this context, the edge weights of the graph are calculated as the mutual information between feature variables conditional upon the class label. Though finding a maximum spanning k-tree (MSkT) is intractable [31] for fixed $k\ge 2$ (and thus neither fixed-parameter tractable [32]), we demonstrate in this paper that solving the MSkTproblem can be done in time $O\left(n^{k+1}\right)$ for every fixed $k\ge 1$ so long as the maximum spanning k-tree retains a Hamiltonian path presented in the input graph. Consequently, a k-TAN as the approximation of Bayesian networks with minimum loss of information can be found efficiently for small to moderate values of k.

The remainder of this paper is organized as follows: Section 2 gives an introduction to Bayesian networks and reviews previous approaches to its approximation, including Naïve Bayes, tree augmented Naïve Bayes (TAN), and k-dependency Bayesian networks (KDBs). Section 3 introduces the k-tree augmented Naïve Bayes (k-TAN) and Section 4 presents a detailed proof that optimal k-tree topology is ensured by the maximum spanning k-tree. We elaborate on our proposed polynomial time algorithm to solve MSkT in Section 5.

2. Backgrounds

Consider dataset D that consists of m data samples for random variable set $\mathbf{X}\cup \left\{Y\right\}$, where $\mathbf{X}=\{X_1,\dots ,X_n\}$ is the set n feature variables, and Y is the class variable. Thus D can be represented as an $m\times (n+1)$ matrix of numerical values on m data samples. For simplicity, we assume all variables have the binary domain $\{0,1\}$ of values. The classification problem is, given a dataset D of samples, to construct a predictive model that can accurately classify any input sample of unknown label. Technically, A classifier $\mathcal{F}$ assigns a class label, either 0 or 1, to the input unclassified sample $\mathbf{z}=(z_1,\dots ,z_n)$, where $z_j$ is a value of $j^{\mathrm{th}}$ feature variable $X_j$ in sample $\mathbf{z}$, $1\le j\le n$, such that $y=\mathcal{F}\left(z\right)\in \{0,1\}$.

Definition 1.

A Bayesian network over random variables $\mathbf{X}\cup \left\{Y\right\}$ is a probabilistic, directed acyclic graph $G=(\mathbf{X}\cup \{Y\},E)$, where vertices denote random variables in $\mathbf{X}$ and directed edges in E between vertices denote causal dependencies between the corresponding random variables. Consequently, the absence of an edge between two particular variables signifies their causal independence.

In the Bayesian network, if $(X_j,X_i)\in E$, variable $X_j$ is called a parent of variable $X_i$. We denote with $\pi \left(i\right)$ the set of indexes of parents for $X_i$. That is, $\pi \left(i\right)=\{X_j:(X_j,X_i)\in E\}$. Due to the causal independence assumption between variables that do not share directed edges, the likelihood of variables in $\mathbf{X}$ given class variable Y can be decomposed and written as

$$P\left(\mathbf{X}\right|Y)=\prod _{i=1}^{n}P\left(X_i|Y,X_j:j\in \pi \left(i\right)\right)$$

(1)

Based on the well-known Bayes formula $P\left(A\right|B)=P(B\left|A\right)P\left(A\right)/P\left(B\right)$ [8] and (1), the predicted class label $y^{*}$ on any given feature values $\mathbf{x}$ can be computed as the one with the highest posterior probability. That is,

$$y^{*}=arg\underset{y}{max}P\left(y\right|\mathbf{x})=arg\underset{y}{max}{\displaystyle \frac{P\left(y\right){\prod }_{i=1}^{n}P\left(x_i\right|y,x_j:j\in \pi \left(i\right))}{P\left(\mathbf{x}\right)}},$$

(2)

where the denominator $P\left(\mathbf{x}\right)$ can be dropped. However, the above formula indicates that $\pi \left(i\right)$ may contain up to $n-1$ indexes of variables and the probability distribution may be of an order of n, making the computation of likelihood challenging with the standard Bayesian network classifiers.

2.1. Naïve Bayes

Naïve Bayes classifier [12] is a Bayesian classification model that makes the Naïve assumption that each feature variable $X_i$ is conditionally independent of other feature variables given the class label variable Y, implying that there are no causal dependencies between feature variables in the graphical model. This assumption greatly simplifies the computation with Bayesian networks because the likelihood of variables in $\mathbf{X}$ given class variable Y can be computed with

$$P_{\mathit{NB}}\left(\mathbf{X}\right|Y)=\prod _{j=1}^n{P}_{\mathit{N}B}\left(X_j\right|Y)$$

Therefore, under the Naïve Bayes modeling, maximizing the posterior probability in Equation (2) to predict the class label on a given collection of feature values $\mathbf{x}=(x_1,\dots ,x_n)$ is computed as

$$y^{*}=\arg \underset{y}{max}\prod _{j=1}^n{P}_{\mathit{NB}}\left(x_j\right|y)·P_{\mathit{NB}}\left(y\right)$$

Note that the above optimization computation can be very efficient since for given feature values $bx=(x_1,\dots ,x_n)$, and the maximization over the choices of y does not need to consider the value of $P_{\mathit{NB}}\left(X\right)$ for the given $bx$. The typical topology structure of the Naïve Bayes classifier is illustrated in Figure 1a.

2.2. Tree Augmented Naïve Bayes

There have been some further significant developments on augmenting the Naïve Bayes classifiers. These efforts aim to relax the independence assumption among features while maintaining lower time complexity compared to full Bayesian networks. Tree structures are preferred as a feasible probabilistic graph approximation, namely tree augmented Naïve Bayes (TAN) [33], which permits causal dependencies between feature variables as well as the dependence between every feature variable and the label variable Y. The overall structure topology of the causal dependencies is yet simple enough to form a tree structure. Therefore, every feature variable, except one, have one other feature variable as its parent [23]. Figure 1b illustrates such a topology. For TAN classifiers, the the likelihood of variables in $\mathbf{X}$ given class variable Y can be written as

$$P_{\mathit{TAN}}\left(\mathbf{X}\right|Y)=\prod _{j=1}^n{P}_{\mathit{TAN}}\left(X_j\right|Y,X_{j^{\prime }}),$$

where for $j=1,\dots ,n$, $X_{j^{\prime }}$ is the parent feature variable of $X_j$, i.e., $(X_{j^{\prime }},X_j)\in E$, for which the parent relation is acyclic, and for exactly one j, $j^{\prime }$ does not exist.

With the TAN model, maximizing the posterior probability in Equation (2) to predict class label on a given collection of feature values $\mathbf{x}=(x_1,\dots ,x_n)$ is expressed as

$$y^{*}=\arg \underset{y}{max}\prod _{j=1}^n{P}_{\mathit{TAN}}\left(x_j\right|y,x_{j^{\prime }})·P_{\mathit{TAN}}\left(y\right)$$

2.3. Optimality of TAN Topology

The structure topology of causal relationships among the feature variables is a tree in a TAN classifier. Unlike the Naïve Bayes where feature variables are assumed independent, TANs of different tree topologies approximate the joint distribution of the underlying (unknown) Bayesian network to different extents. This discrepancy between the unknown joint distribution P and the approximated distribution $P_{\mathit{TAN}}$ is information loss due to structure topology approximation by some TAN, which is measurable with the KL-divergence $D_{\mathit{KL}}(P\parallel P_{\mathit{TAN}})$ [30]. Indeed, the question here is how to represent the $n^{th}$-order relationships of P with binary relationships characterized as a tree structure that has the minimum information loss. In addressing this issue for general graphical models, the seminal work of Chow and Liu [22] showed that the smallest KL-divergence is achieved by the topology of a maximum spanning tree where edge weights take mutual information between feature variables. This idea was adopted to determine the optimal TAN network structure $T^{*}$ through computing a maximum spanning tree over mutual information between feature variables conditional upon the label variable [23], where

$$T^{*}=arg\underset{\mathrm{tree}T}{max}\sum _{(i,i^{\prime })\in T}I(X_i;X_{i^{\prime }}|Y),$$

in which $i^{\prime }$ is the parent of i in tree T.

While optimal approximation of Bayesian networks with tree structure like TAN can be computed efficiently by the maximum spanning tree algorithms, in such a model, every feature variable can only be a causal effect of another (single) feature variable. In many practical applications, classifiers capable of modeling causal dependence upon multiple feature variables become necessary. To cope with this situation, the k-dependency Bayesian classifier (KDB) was introduced in [2] to relax casual independence assumptions by allowing every feature variable to have up to k other feature variable parents, for chosen $k\ge 2$. In addition to the capability to handle multi-dependency among feature variables, the parameter k in a KDB is adjustable and comes in handy for trading between model construction efficiency and classification performance. On the other hand, however, determining an optimal structure topology for KDB poses a great challenge. Technically, representing the $n^{\mathrm{th}}$ order relationships with $k^{\mathrm{th}}$ order relationships in KDB to minimize the divergence $D_{\mathit{KL}}(P\parallel P_{\mathit{KDB}})$ proves a computationally difficult task.

Hence, the construction of a pertinent k-dependence structure of KDB for applications has resorted to heuristics methods. One typical practice is to compute the conditional mutual information $I(X_i;X_j|Y)$ for each pair of feature variables $X_i$ and $X_j$ given class label Y. On an existing variable $X_i$ of interest, k other variables $X_j$ with the highest mutual information with $X_i$ are identified as the feature parent variables for $X_i$. Yet another k-dependence classifier named Extended Tree Augmented Naïve Bayes (ETAN) [28] has also been introduced. Unlike KDB, ETAN employs higher-order conditional mutual information, e.g., $I(X_i;X_1,\dots ,X_{i-1}|Y)$, to capture the conditional dependencies between $X_i$ and its parent attributes. A drawback of these strategies is that they would generate a large number of redundant dependencies between feature variables in the model and result in over-fitting. Other methods based on sorting of feature variables and on filtering are all heuristic and do not guarantee the optimality of the determined structure topology [29,34].

3. $\mathit{k}$-Tree Augmented Naïve Bayes

3.1. Acyclic k-Tree

In this section, we introduce the approximation of Bayesian networks with the k-tree topology. Intuitively, a k-tree is a tree-like graph with positive integer k representing the “thickness of the tree trunk and branches”; the smaller the value of k, the more the graph resembles a tree, and 1-tree is a tree. Formally,

Proposition 1.

Let $k\ge 1$ be a fixed integer. Let Δ be a non-empty, directed, acyclic clique with vertex set $V(\Delta )$ and edge set $E(\Delta )$. Then there must be exactly one vertex $v\in V(\Delta )$ such that $\forall u\in V(\Delta ),u\ne v\Rightarrow (u,v)\in E(\Delta )$.

Definition 2.

Let $k\ge 1$ be an integer. The class of (acyclic) k-trees of $n\ge k$ vertices are defined as follows, recursively,

1. 

A k-tree of k vertices is an acyclic k-clique;

2. 

A k-tree $G=(V,E)$ of n vertices, for $n>k$, consists of a k-tree $H=(U,F)$ of $n-1$ vertices and a vertex $v\notin U$, where $V=U\cup \left\{v\right\}$, and $E=F\cup \left\{(u,v):(u,v) is a directed edge u\in C\right\}$, for some acyclic k-clique C in H.

Based on the definition, any acyclic k-tree G containing vertices $\{x_1,\dots ,x_n\}$ is created recursively by starting from an acyclic k-clique and then introducing one new vertex at a step. Thus the acyclic k-tree T can be expressed as an orthodox sequence that represents the creation process as:

$$G=C_1{v}_{k+1}{C}_2{v}_{k+2}\dots C_{n-k}{v}_n,$$

(3)

where $C_1=\{v_1,\dots ,v_k\}$ is the first acyclic k-clique created, and, for $k+1\le i\le n$, directed edges $(u,v_i)$ are created from all vertices u in some acyclic k-clique $C_{i-k}$ that already exists to newly introduced vertex $v_i$. Despite that expression (3) may not be unique for a k-tree G, it gives a definite structure topology via the following definition. Figure 2a illustrates Definition 2 with creation of a 2-tree example.

Definition 3.

Let $G=(V,E)$ be an acyclic k-tree containing vertices $\{v_1,\dots ,v_n\}$. For any two vertices $u,v\in V$, u is called a parent of v if directed edge $(u,v)\in E$. All parents of a vertex form the parent set for the vertex. Specifically, $\pi \left(i\right)$ denotes the parent set of vertex $v_i$ and is defined as $\pi \left(i\right)=\{j:(v_j,v_i)\in E\}$.

In particular, for the k-tree expressed with (3), the parent set of vertex $v_i\in V$, for every index $i>k$, is $\pi \left(i\right)=\{j:v_j\in C_{i-k}\}$. For those vertices $v_i\in C_1$, the initial k-clique, the parent set is defined as: $\pi \left(i\right)=\{j:(v_j,v_i)\in E \mathrm{and} v_j\in C_1\}$. By Proposition 1, the k vertices in $C_1$ should have $0,1,2,\dots ,k-1$ exactly as their parent set sizes, respectively.

Proposition 2.

Let $G=(V,E)$ be an acyclic k-tree. Then any vertex $v\in V$, together with its parent vertices, forms an acyclic clique, which is denoted with ${\Delta }_v$.

By the proposition and definition of acyclic k-trees, $(k+1)$-cliques in a k-tree of n vertices, with $n\ge k+1$, can be denoted as ${\Delta }_v$, for every vertex $v\in V\setminus C_1$ and the number of acyclic $(k+1)$-cliques in the k-tree is $n-k$. It is also not difficult to see that the creation process of the k-tree given in (3) actually also describes the created k-tree as a tree that connects $(k+1)$-cliques in it (see Figure 2b).

Definition 4.

Let $G=(V,E)$ be an acyclic k-tree of $n\ge k+1$ vertices created by the process in (3). Then the tree-representation of G is a tree in which tree nodes are $(k+1)$-cliques in G and a tree edge $({\Delta }_{v_i},{\Delta }_{v_j})$ exists between $(k+1)$-cliques ${\Delta }_{v_i}$ and ${\Delta }_{v_j}$ if and only if ${\Delta }_{v_i}\cap {\Delta }_{v_j}$ is the parent set of vertex $v_j$ in the k-tree and for i, $i<j$, the largest.

Figure 2a,b illustrate the concept of tree-representation of k-trees. Note that tree-representations are not necessarily unique for a given k-tree. While there may be more than one ${\Delta }_{v_i}$ that satisfies the condition to establish a tree-edge with ${\Delta }_{v_j}$, our definition chooses the most recently created ${\Delta }_{v_i}$. Therefore, a representation is unique with respect to any given creation process. The notion of tree-representations will facilitate discussions in Section 5.

3.2. k-Tree Augmented Naïve Bayes

Definition 5.

Let $k\ge 1$ be an integer. A k-tree augmented Naïve Bayes (k-TAN) is a Bayesian network $G=(\mathbf{X}\cup \{Y\},E)$ where the induced subgraph by feature variables in $\mathbf{X}$ forms exactly an acyclic k-tree of n vertices, where $n=\left|\mathbf{X}\right|$, and of edge set $E\setminus \left\{\right(Y,X):X\in \mathbf{X}\}$.

Figure 2c shows a k-TAN in which the set of feature variables forms an acyclic 2-tree. Note that k-TAN is TAN for $k=1$ and generalizes TAN with the permission for a feature variable to have a causal effect from multiple (k or fewer) other (parent) variables. As later analyses will show, an optimal topology structure does not need to require every variable to have exactly k parent variables, making it flexible to deal with sophisticated causal dependencies in various applications.

Now we consider the problem of classification by k-TAN of topology G over feature variables $\mathbf{X}=\{X_1,\dots ,X_n\}$ and class variable Y. First, the likelihood (i.e., conditional probability) of variables $\mathbf{X}$ with the K-TAN is computed as

$$P_G\left(\mathbf{X}\right|Y)=\prod _{i=1}^{n}P\left(X_i|Y,X_j:j\in \pi \left(i\right)\right),$$

(4)

where parent set $\{X_j:j\in \pi \left(i\right)\}$ of variable $X_i$, $i=1,\dots ,n$, is determined by the acyclic k-tree imposed on the variables $\mathbf{X}$. Let $\mathbf{x}=(x_1,\dots ,x_n)$ be a given sample of values for feature variables $\mathbf{X}=\{X_1,\dots ,X_n\}$ of the the k-TAN. Then predicted label value $y^{*}$ for class variable Y can be computed with the maximization of the posterior probability as

$$y^{*}=\arg \underset{y}{max}P\left(y\right|\mathbf{x})=\arg \underset{y}{max}{\displaystyle \frac{P\left(y\right){\prod }_{i=1}^{n}P\left(x_i|y,\{x_j:j\in \pi \left(i\right)\}\right)}{P\left(\mathbf{x}\right)}},$$

where $P\left(\mathbf{x}\right)$ can be omitted as it remains unchanged for different values of y.

4. Optimal $\mathit{k}$-Tree Topology

In general, different graph topologies for k-TAN may result in different performances of the model. Theoretical performance of approximated graphical models can be effectively measured with information-theoretics. Let P be the true yet unknown probability distribution of random variables defined on some graphical model, and let G be the structure topology to approximate the model. Then the information loss due to the approximation can be measured with the KL-divergence

$$D_{\mathit{KL}}(P \Vert P_G)=\sum P{log}_2{\displaystyle \frac{P}{P_G}},$$

(5)

where $P_G$ is the joint probability distribution of random variables pertaining to the approximated structure topology G. An optimal structure topology for the approximation is the one that minimizes the information loss $D_{\mathit{KL}}(P \Vert P_G)$. Chow and Liu initiated such approximation with tree topologies in their seminal work [22]. They proved that the optimal approximation can be achieved via a tree topology that yields the maximum sum of mutual information. The result has been applied to finding optimal tree topology for the construction of tree augmented Bayesian networks, TANs [23]. Unfortunately, using the KL-divergence measurement to compute information loss due to topology approximation with G, especially to find an optimal topology approximation has not been successful on non-tree structures, in particular, for KDB and its variants.

An optimal structure topology of k-TAN for classification tasks can be identified by minimizing the information loss due to topology approximation with k-TAN. Specifically, the optimal structure $G^{*}$ is computed with

$$G^{*}=\arg \underset{\mathit{k}\text{-}\mathit{TAN}G}{min}{D}_{\mathit{KL}}\left(P(\mathbf{X},Y) \Vert P_G(\mathbf{X},Y)\right),$$

(6)

where $P(\mathbf{X},Y)$ is the unknown probability distribution over feature variables $\mathbf{X}$ and the label variable Y, and $P_G(\mathbf{X},Y)$ is the distribution enabled by the structure topology G over the same set of variables in the k-TAN.

We now give a derivation of an optimal k-tree topology of k-TAN. By (5), the goal of Equation (6) can be achieved by minimizing computation of the relative entropy as:

$$\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_2{\displaystyle \frac{P(\mathbf{x},y)}{P_G(\mathbf{x},y)}},$$

(7)

where pair $(\mathbf{x},y)$ is a data sample for variables $\mathbf{X}$ and Y. By conditional entropy, expression in (7) can be computed exactly as,

$$\begin{array}{cc}\hfill \sum _{\mathbf{x},y}P(\mathbf{x},y){log}_2{\displaystyle \frac{P\left(\mathbf{x}\right|y\left)P\right(y)}{P_G\left(\mathbf{x}\right|y)P_G\left(y\right)}}& =\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_2{\displaystyle \frac{P\left(\mathbf{x}\right|y)}{P_G\left(\mathbf{x}\right|y)}}\hfill \\ & =-H\left(\mathbf{X}\right|Y)-\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_2{P}_G\left(\mathbf{x}\right|y)\hfill \end{array}$$

The second term of the right-hand side of the above equation can be turned into

$$\begin{array}{cc}& -\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_2\prod _{i=1}^{n}P\left(x_i|y,\{x_j:j\in \pi \left(i\right)\}\right)\hfill \\ \hfill =& -\sum _{\mathbf{x},y}P(\mathbf{x},y)\sum _{i=1}^n{log}_{2}P\left(x_i|y,\{x_j:j\in \pi \left(i\right)\}\right)\hfill \\ \hfill =& -\sum _{\mathbf{x},y}P(\mathbf{x},y)\sum _{i=1}^n{log}_2{\displaystyle \frac{P\left(x_i\right|y)·P(x_i,\{x_j:j\in \pi \left(i\right)\}|y)}{P\left(x_i\right|y)·P\left(\{x_j:j\in \pi \left(i\right)\}\right|y)}}\hfill \end{array}$$

The last term can be decomposed into:

$$\begin{array}{cc}& -\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_{2}P\left(x_i\right|y)-\sum _{\mathbf{x},y}P(\mathbf{x},y)\sum _{i=1}^n{log}_2{\displaystyle \frac{P(x_i,\{x_j:j\in \pi \left(i\right)\}|y)}{P\left(x_i\right|y)·P\left(\{x_j:j\in \pi \left(i\right)\}\right|y)}}\hfill \\ \hfill =& -\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_{2}P\left(x_i\right|y)-\sum _{i=1}^n\sum _{\mathbf{x},y}P(\mathbf{x},y){log}_2{\displaystyle \frac{P(x_i,\{x_j:j\in \pi \left(i\right)\}|y)}{P\left(x_i\right|y)·P\left(\{x_j:j\in \pi \left(i\right)\}\right|y)}}\hfill \\ \hfill =& -\sum _{i=1}^n\sum _{x_i,y}P(x_i,y){log}_{2}P\left(x_i\right|y)-\sum _{i=1}^n\sum _{\mathbf{x},y}P(y,x_i,\{x_j:j\in \pi \left(i\right)\}){log}_2{\displaystyle \frac{P(x_i,\{x_j:j\in \pi \left(i\right)\}|y)}{P\left(x_i\right|y)·P\left(\{x_j:j\in \pi \left(i\right)\}\right|y)}}\hfill \\ \hfill =& \sum _{i=1}^{n}H\left(X_i\right|Y)-\sum _{i=1}^{n}I\left(X_i;\{X_j:j\in \pi \left(i\right)\}|Y\right)\hfill \end{array}$$

The second last equality holds because the probability $P(\mathbf{x},y)$ is projected onto the $i\mathrm{th}$ variable, resulting in $P(x_i,y)$ and omitting other variables due to summation of their probabilities. It is true also because likewise $P(\mathbf{x},y)$ is projected onto the $i\mathrm{th}$ variable and the $j\mathrm{th}$ variables for all $j\in \pi \left(i\right)$, resulting in $P(y,x_i,\{x_j:j\in \pi \left(i\right)\})$. The last equality is due to the definition of entropy and mutual information.

The above steps yield

$$\sum _{\mathbf{x},y}P(\underset{-}{\mathbf{x}},y){log}_2{\displaystyle \frac{P(\mathbf{x},y)}{P_G(\mathbf{x},y)}}=-H\left(\mathbf{X}\right|Y)+\sum _{i=1}^{n}H\left(X_i\right|Y)-\sum _{i=1}^{n}I\left(X_i;\{X_j:j\in \pi \left(i\right)\}|Y\right)$$

Since the terms of entropies are independent of choices of $\pi$, the minimization of the left-hand-side is realized by the maximization of the sum ${\sum }_{i=1}^{n}I\left(X_i;\{X_j:j\in \pi \left(i\right)\}|Y\right)$. Note that $I\left(X_i;\{X_j:j\in \pi \left(i\right)\}|Y\right)$ is mutual information between variable $X_i$ and its variable parent set $\{X_j:j\in \pi \left(i\right)\}$ conditional upon class variable Y, where $\pi$ is drawn from the acyclic k-tree topology G that is imposed on the k-TAN. Therefore, we conclude:

Theorem 1.

The optimal structure topology G for feature variables in k-TAN can be determined by maximizing sum ${\sum }_{i=1}^{n}I\left(X_i;\{X_j:j\in \pi \left(i\right)\}|Y\right)$, where π represents the k-tree structure among feature variables in G.

5. Finding Maximum Spanning $\mathit{k}$-Tree

We show in the next section that the maximization problem in information theoretics given in Theorem 1 has an equivalent problem in graph theoretics.

5.1. Maximum Spanning k-Tree

Definition 6.

Let $G=(V,E)$ be an acyclic graph. A real-value function $f:V\to \mathbb{R}$ is a neighborhood function if for any subset $\Delta \subseteq V$, $F(\Delta )\ne -\infty$ if and only if Δ is an acyclic clique.

Definition 7.

Let $k\ge 1$ be a fixed integer and f be a neighborhood function. The Maximum spanning k-tree problem associated with f, denoted as MSkT$\left(f\right)$, is, given a directed acyclic graph $G=(V,E)$, finding a spanning acyclic k-tree H that maximizes the sum:

$$f\left(H\right)=\sum _{v\in V,{\Delta }_v\subseteq H}f\left({\Delta }_v\right),$$

where ${\Delta }_v$ denotes the acyclic clique formed by v along with its parent vertices in H. Note that problem MSkT$\left(f\right)$ is a generic problem that can be specific when the neighborhood function f is specified.

MSkT$\left(f\right)$ generalizes the traditional maximum spanning tree problem where the neighborhood function $f\left({\Delta }_v\right)$ is simply $w(u,v)$, the weight between v and its parent u. Another further generalization is to consider ${\Delta }_v$ to be a hyper edge shared by v and its parent vertices and the neighborhood function $f\left({\Delta }_v\right)$ to be the weight on the hyper edge. In particular, we can cast the maximization problem in Theorem 1 as the problem MSkT$\left(f\right)$ for the choice of the neighborhood function f as follows:

$$f\left({\Delta }_v\right)=I(X_v;\{X_u:u\in \pi \left(v\right)\}|Y),$$

(8)

where vertex v denotes the random feature variable $X_v$ in the topology of the k-tree associated with the k-TAN being constructed and $\pi \left(v\right)$ be the set of v’s parent vertices.

We will show that the MSkT$\left(f\right)$ problem is intimately related to the known decision problem spanning k-tree, which answers the following question: given a graph, does it possess a subgraph that is a spanning k-tree? The spanning k-tree appears to be difficult to solve because it has been proved NP-hard [31] even for fixed $k=2$. We shall connect the intractable problem to MSkT$\left(f\right)$ for some specific functions f, and the connection will carry the intractability over to the latter, unfortunately.

Definition 8.

Let $k\ge 1$ be a fixed integer and f be a neighborhood function. The problem D-MSkT$\left(f\right)$is a decision problem that: given a directed acyclic graph $G=(V,E)$ and real number threshold w, determines if G contains a spanning k-tree H such that the value $f\left(H\right)\ge w$.

It is not difficult to see, if we choose the neighborhood function $f\left({\Delta }_v\right)$ to be the count of directed edges from parent vertices of v to v on k-tree H, then $f\left(H\right)=\left({\sum }_{j=1}^{k-1}j\right)+k(n-k)$, where the first sum is the total count of edges in the intial k-clique and term $k(n-k)$ is the count of new edges due to vertices introduced to existing k-cliques. Clearly the count is ${\displaystyle \frac{k}{2}}(2n-k-1)$. Therefore, the input directed acyclic graph G contains a directed k-tree H if and only if the value $f\left(H\right)\ge {\displaystyle \frac{k}{2}}(2n-k-1)$. The above analysis yields a polynomial-time transformation from problem spanning k-tree to D-MSkT$\left(f\right)$ and thus a justification for the intractability of the latter. That is, there are (simple) neighborhood functions f for which problem D-MSkT$\left(f\right)$ is NP-hard. This leads to the intractability of the MSkT$\left(f\right)$ problem as well.

Theorem 2.

For each fixed value of $k\ge 2$, there are neighborhood weight functions f for which problem MSkT$\left(f\right)$is NP-hard.

The above analyses and those from the previous section suggest that the problem MSkT$\left(f\right)$ (and the problem of finding an optimal structure topology for k-TAN as well) is computationally intractable even for fixed values of $k\ge 2$. Nevertheless, we will show in the next section that, with a meaningful restriction on the topology structure of the k-tree, polynomial-time algorithms can be obtained for MSkT$\left(f\right)$.

5.2. Backbone k-Trees

In this section, we focus on a specific type of k-tree whose topology imposes a linear order on the involved vertices. For such k-trees, we derive efficient algorithms for MSkT$\left(f\right)$.

Definition 9.

Let $G=(V,E)$ be a graph, where $V=\{v_1,\dots ,v_n\}$. If the edge set E contains a Hamiltonian path $\{(v_i,v_{i+1}):1\le i<n\}$, then G is called a backbone graph. We also call the Hamiltonian path the backbone, and edge $(v_i,v_{i+1})$ a backbone edge. If G is a k-tree, it is called a backbone k-tree.

Graphs with a built-in Hamiltonian path can be ideal as the structural topology of graphical models arising from meaningful scientific applications. The linearity underlying the importance aspect of relationships among random variables upon which higher-order, more sophisticated relations are established. For example, backbone graph topology is most suitable for modeling time-series events and processes, known significant pathways in gene networks, and biomolecule 3D structure, where the backbone of the molecule is coupled in a linear fashion.

We now introduce and prove a few properties associated with backbone k-trees. For this purpose, our discussion will be based on the tree-representation for k-trees in Definition 4. Since backbone edges are of special interest, a more elaborate definition of tree-representation for the backbone k-tree is needed.

Proposition 3.

Let G be a backbone k-tree of size $n\ge k+1$ and let interval $[1,2,\dots ,n]$ denote the ordered consecutive positions of all vertices in the backbone. Let Δ be a $(k+1)$-clique in G with $\{i_1,\dots ,i_{k+1}\}$ being the positions of vertices in Δ on the backbone in the assumed order $1\le i_1<i_2<\dots <i_{k+1}\le n$. Then these positions partition the consecutive interval $[1,2,\dots ,n]$ into at most $k+2$ non-empty, consecutive intervals as

$$[1,\dots ,i_1],[i_1,\dots ,i_2],\dots ,[i_k,\dots ,i_{k+1}],[i_{k+1},\dots ,n]$$

Let $v\notin \Delta$. We denote with $\mathcal{B}(\Delta ,v)$ the subset of backbone edges formed by vertices whose positions on the backbone are in the same interval containing the position of v.

We now consider a new characterization for backbone k-trees. Specifically, we define the collection of partial backbone k-trees (abbreviated with pbkts) recursively as follows:

Definition 10.

Let H be a graph of size $n\ge k+1$, Δ be a $(k+1)$-clique in H, and α is a subset of backbone edges. Tuple $(H,\Delta ,\alpha )$ is a pbkt rooted at $(k+1)$-clique Δ, which retains backbone edges in α if

(1) 

either $H=\Delta$ and $\alpha \setminus (\Delta \times \Delta )=\varnothing$;

(2) 

or $H=H_1\cup H_2$, where $(H_1,\Delta ,\beta )$ and $(H_2,{\Delta }^{\prime },\gamma )$ are pbkts rooted at $(k+1)$-cliques Δ and ${\Delta }^{\prime }$ and retain backbone edges in β and γ, respectively, such that ${\Delta }^{\prime }{=\Delta |}_v^u$ for some vertices $u\in \Delta$ and v that occur in α but not in Δ, and $\gamma =\mathcal{B}(\Delta ,v)$ and $\beta =\alpha \setminus \gamma$.

${\Delta |}_v^u$ denotes the $(k+1)$-clique derived from Δ, such that u is removed and replaced by v along with new edges $\left\{\right(x,v):x\in \Delta \setminus \{u\left\}\right\}$ added on.

Note that the term partial k-tree has been used for subgraphs of k-trees [35]. Since a subgraph of a backbone k-tree does not necessarily contain all the backbone edges, pbkt is defined with an associated subset of backbone edges. The use of a $(k+1)$-clique as the root is technical, which makes it feasible to view backbone k-trees from the perspective of a relation among $(k+1)$-cliques.

The atomic case of pbkt is simply a $(k+1)$-clique that only contains backbone edges that are already on the clique. Furthermore, it is not difficult to see that, if G is a (full) backbone k-tree, there is a $(k+1)$-clique $\Delta$ in G to serve as the root such that tuple $(G,\Delta ,\alpha )$ is a pbkt with $\alpha$ being the complete set of backbone edges.

We now examine two properties of backbone k-trees important to developing efficient algorithms that find the maximum backbone k-tree.

Lemma 1.

Let $k\ge 1$ and $G=(V,E)$ be a backbone k-tree. Then, any $(k+1)$-clique Δ in G separates G into at most $k+2$ connected components.

Proof. 

By Proposition 3, the positions of vertices in $\Delta$ partitions the consecutive interval into at most $k+2$ consecutive intervals. We claim that any set of vertices whose positions belong to the same, non-empty interval, say $[i_h,\dots ,i_{h+1}]$, form at most one connected component separated by $\Delta$. Suppose otherwise there are two connected components $C_1$ and $C_2$ separated by $\Delta$ and there exists an integer t, where $i_h\le t<t+1\le i_{h+1}$, such that $v_t\in C_1\setminus C_2$ and $v_{t+1}\in C_2\setminus C_1$. Since backbone edge $(v_t,v_{t+1})$ does not belong to $C_1$ nor $C_2$, there would have to be another component that contains both $v_t$ and $v_{t+1}$, which cannot exist, leading to a contradiction. Based on the above analysis, the number of connected components in the backbone k-tree G separated by $(k+1)$-clique is at most $k+2$. (See Figure 3 for an illustration of the above argument). □

The notation ${\Delta |}_v^u$ used in Definition 10 underlies how one $(k+1)$-clique comes to exist based on another. We give a formal term to this relationship between two $(k+1)$-cliques.

Definition 11.

Let Δ be a $(k+1)$-clique and ${\Delta }^{\prime }{=\Delta |}_v^u$ for some $u\in \Delta$ and $v\notin \Delta$. ${\Delta }^{\prime }$ is said to be a neighbor of Δ.

Lemma 2.

Let Δ be a $(k+1)$-clique in a backbone k-tree, which has two different neighbors ${\Delta }_1$ and ${\Delta }_2$. If $x_1\in {\Delta }_1\setminus \Delta$ and $x_2\in {\Delta }_2\setminus \Delta$, then the positions of $x_1$ and $x_2$ on the backbone belong to two different intervals separated by the positions of vertices in Δ.

Proof. 

According to Definition 4, ${\Delta }_1$ and ${\Delta }_2$ should be the roots of two different pbkts that are two connected components separated by $\Delta$ to which $x_1$ and $x_2$ belong, respectively. By the proof for Lemma 1, the positions of $x_1$ and $x_2$ on the backbone should belong to two different consecutive intervals separated by the positions of the vertices in $\Delta$. □

5.3. A Polynomial-Time Algorithm

We now present an efficient algorithm for solving the MSkT$\left(f\right)$ problem where the desired maximum spanning k-tree should contain a certain backbone (i.e., Hamiltonian path) designated in the input graph. We only consider neighborhood functions $f(\Delta )$ that can be computed in time $\delta \left(k\right)$ on any $(k+1)$-clique $\Delta$ for some computable function $\delta$. The properties demonstrated earlier will facilitate the following discussion of the algorithm.

The problem MSBkT$\left(f\right)$, tailored from MSkT$\left(f\right)$, for backbone k-trees, is defined as: given a directed acyclic graph $G=V,E)$ with a designated backbone B, finds a spanning directed acyclic k-tree $H^{*}$ in graph G such that it retains $B\subseteq H^{*}$, and the following sum is maximized:

$$f\left(H^{*}\right)=\sum _{v\in V,{\Delta }_v\subseteq H^{*}}f\left({\Delta }_v\right)$$

By default, we assume the designated backbone B in the input graph can always be denoted as $B=\left\{\right(i,i+1):1\le i<n\}$ by a simple relabeling on the vertices of the graph.

To facilitate discussion, we denote with $\mathcal{E}(\Delta )$ the set of edges formed by vertices in clique $\Delta$. Also recall notation $\mathcal{B}(\Delta ,v)$ introduced earlier, which represents the subset of backbone edges formed by vertices whose positions on the backbone are in the same interval containing the position of v.

Theorem 3.

Problem MSBkT $\left(f\right)$ can be solved in time $O\left(k^2\delta \left(k\right)n^{k+2}\right)$ on input directed, acyclic graphs of n vertices.

Proof. 

The algorithm finds an optimal spanning backbone k-tree based on the tree-representation of k-trees in Definition 4, which offers a recursive construction of pbkts in the input graph G. We define function $S(\Delta ,\alpha )$ to be the maximum sum of neighborhood function f values over all $(k+1)$-cliques in a pbkt that is rooted at $\Delta$ and retains all backbone edges in $\alpha$. Then

$$S(\Delta ,\alpha )=\left\{\begin{array}{cc}f(\Delta )\hfill & \alpha \setminus \mathcal{E}(\Delta )=\varnothing \hfill \\ \underset{u\in \Delta ,v\notin \Delta \mathrm{nor} \mathrm{in} \alpha }{max}\left\{S(\Delta ,\beta )+S(\Delta {|}_v^u,\mathcal{B}(\Delta ,v))\right\}\hfill & \beta =\alpha \setminus \mathcal{B}(\Delta ,v),\hfill \end{array}\right.$$

which strictly follows the rules for pbkt given in Definition 4.

Function $S(\Delta ,\alpha )$ maximizes its value over all pbkts rooted at $\Delta$ and retains backbone edges in set $\alpha$. Then a desired spanning backbone k-tree is one of the pbkts rooted at some $(k+1)$-clique $\Delta$, which maximizes function $S(\Delta ,B)$.

The recurrence relation for function S makes it computable with a dynamic programming process. For every pbkt that is simply $(k+1)$-clique $\Delta$, the value is the neighborhood function value $f(\Delta )$. For every more general case of pbkt, the computation maximizes over all possible vertex pair u and v that are used to split the pbkt into two subcases of pbkts, one rooted at $\Delta$ and another at ${\Delta |}_v^u$. The corresponding subset $\alpha$ of backbone edges that needs to be retained is also split into two. One subset is $\mathcal{B}(\Delta ,v)$ containing those backbone edges whose positions are in the same interval as the position of vertex v. The other one is the rest of the backbone edges in subset $\alpha$. This is because, by the proof of Lemma 1, the backbone edges whose positions are in the same interval as the position of vertex v belong to the same connected component as v.

For the maximization operation, there are $k+1$ choices of vertices in $\Delta$ for u and n options for v. Then a table of $n^{k+1}\times m$ entries can be used for storing values of function $S(\Delta ,\alpha )$, where m is the number of backbone subsets. By Lemma 2, the positions $[1,2\dots ,n]$ of all backbone edges are partitioned into $k+1$ consecutive intervals, and those with positions belonging to the same interval should belong to the same connected pbkt. Consequently, $m\le k+2$ and $\alpha$ can be encoded with bitmap of length $k+2$. The overall time used by the algorithm is thus $O\left(k{2}^k\delta \left(k\right)n^{k+2}\right)$. □

Corollary 1.

For every fixed $k\ge 1$, problem MSBkT $\left(f\right)$ can be solved in time $O\left(n^{k+2}\right)$ on input directed, acyclic graphs of n vertices.

To connect the problem MSBkT$\left(f\right)$ to the problem of finding optimal structure topology for k-TAN that retains an designated Hamiltonian path, the neighborhood function $f(\Delta )=I(X_i;X_j:j\in \pi \left(i\right))$ is the mutual information between variable $X_i$ and its parent set variables in the acyclic directed $(k+1)$-clique, where $x_i$ is the vertex such that $\left|\right\{(x_i,x_j):j\in \pi \left(i\right)\left\}\right|=k$. The function can be computed in time a function in k.

In addition, an earlier investigation [36] showed that the dynamic programming to computing functions around backbone k-tree can be approached in a slightly different way with the argument $\Delta$ being a k-clique instead of a $(k+1)$-clique, yielding a time complexity $O\left(n^{k+1}\right)$ instead of $O\left(n^{k+2}\right)$ for every fixed $k\ge 1$.

Theorem 4.

For every fixed $k\ge 1$, the optimal structure topology for feature variables in k-TAN can be determined in time $O\left(n^{k+1}\right)$ provided that the structure retains a predefined Hamiltonian path in the input graph of n vertices.

Corollary 2.

For every fixed $k\ge 1$, there are polynomial time algorithms for the minimum loss of information approximation of Bayesian networks with a conditional k-tree topology that retains a designated Hamiltonian path in the input network.

6. Conclusions

Bayesian networks offer a structured framework for modeling dependencies among features and class labels, making them invaluable in classification tasks. Generally, there are $n-1$ dependencies among features, where n is the total number of features, which need to be considered. Thus, the complexity of acquiring a complete Bayesian network may be intractable with increasing feature numbers. Naïve Bayes assumes no dependencies among features given the class label, which is often unrealistic in real-world scenarios. To address this issue, Tree Augmented Naïve Bayes (TAN) is introduced to capture the most important dependencies between features by constructing a simple topology, a tree. Therefore, TAN is notably preferred for probabilistic graph approximations. However, more complex topologies such as KDB are required for situations where features strongly influence each other. Previous research on k-Dependency Bayesian classifiers has largely depended on heuristic algorithms that do not guarantee an optimal topology for the approximation.

We introduced an augmented Naïve Bayes classifier, k-TAN, of a k-dependency topology where structure among feature variables can be represented as k-tree graph. We demonstrated that when the k-tree is a maximum spanning k-tree, it provides the optimal approximation of the Bayesian network and ensures minimum information loss under the KL-divergence metric. The edge weights represent the mutual information between feature variables conditioned on the class label. Although finding the maximum spanning k-tree is an intractable task for fixed $k\ge 2$, this paper presents a polynomial-time approach to solve a constrained case of the problem. That is, if the maximum spanning k-tree requires retaining a designated Hamiltonian path in the graph topology, the maximum spanning k-tree problem can be solved in time $O\left(n^{k+1}\right)$. Therefore, employing this algorithm ensures the efficient construction of k-TAN classifiers with minimal information loss. The topology constraint of retaining a Hamiltonian path has a wide spectrum of applications with Bayesian networks.

Our final note is that the proposed k-TAN method may be incorporated with some recently developed extensions to Naïve Bayesian classifiers, which are orthogonal to structural topology extension to Naïve Bayes. This includes, for example, a method in which the integration of a latent component into the Naïve Bayes classifier can account for hidden dependencies among attributes in domains such as health care [9], where data complexity and inter-feature correlations are common. Another work investigates relationships between computing conditional log likelihood and marginal likelihood under the exact and approximation learning setting [37]. Moreover, rather than assuming all features are conditionally independent, Comonotonic-Inference-Based classifier (CIBer) [38] identifies an optimal partition of features, grouping those that exhibit strong dependencies.