Consistency of Learning Bayesian Network Structures with Continuous Variables: An Information Theoretic Approach

Abstract

We consider the problem of learning a Bayesian network structure given n examples and the prior probability based on maximizing the posterior probability. We propose an algorithm that runs in $O\left(n\text{log}n\right)$ time and that addresses continuous variables and discrete variables without assuming any class of distribution. We prove that the decision is strongly consistent, i.e., correct with probability one as $n\to \infty$. To date, consistency has only been obtained for discrete variables for this class of problem, and many authors have attempted to prove consistency when continuous variables are present. Furthermore, we prove that the “$logn$” term that appears in the penalty term of the description length can be replaced by $2(1+ϵ)loglogn$ to obtain strong consistency, where $ϵ>0$ is arbitrary, which implies that the Hannan–Quinn proposition holds.

1. Introduction

In this paper, we address the problem of learning a Bayesian network structure from examples.

For sets $A,B,C$ of random variables, we say that A and B are conditionally independent given C if the conditional probability of A and B given C is the product of the conditional probabilities of A given C and B given C. A Bayesian network (BN) is a graphical model that expresses conditional independence (CI) relations among the prepared variables using a directed acyclic graph (DAG). We define a BN by the DAG with vertexes $V=\{1,\cdots ,N\}$ and directed edges $E=\left\{\right(j,i\left)\right|i\in V,j\in \pi \left(i\right)\}$, where edge $(j,k)\in V^2$ directs from j to k, via minimal parent sets $\pi \left(i\right)\subseteq V$, $i\in V$, such that the distribution is factorized by:

$$P(X^{\left(1\right)},\cdots ,X^{\left(N\right)})=\prod _{i=1}^{N}P\left(X^{\left(i\right)}\right|{\left\{X^{\left(j\right)}\right\}}_{j\in \pi \left(i\right)}).$$

First, suppose that we wish to know whether two random binary variables X and Y are independent (hereafter, we write $X\perp \perp Y$). If we have n pairs of actually emitted examples $(X=x_1,Y=y_1),\cdots ,(X=x_n,Y=y_n)$ and know the prior probability p of $X\perp \perp Y$, then it would be reasonable to maximize the posterior probability of $X\perp \perp Y$ given $x^n=(x_1,\cdots ,x_n)$ and $y^n=(y_1,\cdots ,y_n)$. If we assume that the probabilities $P(X=x),P(Y=y)$ and $P(X=x,Y=y)$ are parameterized by $p\left(x\right|{\theta }_X),p\left(y\right|{\theta }_Y)$, and $p(x,y|{\theta }_{XY})$ and that the prior probabilities $W_X,W_Y,$ and $W_{XY}$ over the probabilities ${\theta }_X,{\theta }_Y,$ and ${\theta }_{XY}$ of $X\in \{0,1\}$, $Y\in \{0,1\}$ and $(X,Y)\in {\{0,1\}}^2$ are available, respectively, then we can construct the quantities:

$$\begin{array}{ccc}\hfill Q_X^n\left(x^n\right)& :=& \int \prod _{i=1}^{n}p\left(x_i\right|{\theta }_X)W_X\left(d{\theta }_X\right),\hfill \\ \hfill Q_Y^n\left(y^n\right)& :=& \int \prod _{i=1}^{n}p\left(y_i\right|{\theta }_Y)W_Y\left(d{\theta }_Y\right),\hfill \\ \hfill Q_{XY}^n(x^n,y^n)& :=& \int \prod _{i=1}^{n}p(x_i,y_i|{\theta }_{XY})W_{XY}\left(d{\theta }_{XY}\right).\hfill \end{array}$$

In this setting, maximizing the posterior probability of $X\perp \perp Y$ given examples $x^n,y^n$ w.r.t. the prior probability p is equivalent to deciding $X\perp \perp Y$ if and only if:

$$p{Q}_X^n\left(x^n\right)Q_Y^n\left(y^n\right)\ge (1-p)Q_{XY}^n(x^n,y^n).$$

(1)

The decision based on (1) is strongly consistent, i.e., it is correct with probability one as $n\to \infty$ [1] (see Section 3.1 for the proof). We say that a model selection procedure satisfies weak consistency if the probability of choosing the correct model goes to unity as n grows (probability convergence) and that it satisfies strong consistency if the probability one is assigned to the set of infinite example sequences that choose the correct model, except for at most finite times (almost sure convergence). In general, strong consistency implies weak consistency, but the converse is not true [2]. In any model selection, in particular for large n, the correct answer is required. If continuous variables are present, the BN structure learning is not easy, and strong consistency is hard to obtain.

The same scenario is applied to the case in which X and Y take values from finite sets A and B rather than $\{0,1\}$.

Next, suppose that we wish to know the factorization of three random binary variables $X,Y,Z$: $P\left(X\right)P\left(Y\right)P\left(Z\right)$, $P\left(X\right)P(Y,Z)$, $P\left(Y\right)P(Z,X)$, ${\displaystyle P\left(Z\right)P(X,Y),\frac{P(X,Y)P(X,Z)}{P\left(X\right)}}$, ${\displaystyle \frac{P(X,Y)P(Y,Z)}{P\left(Y\right)}}$, ${\displaystyle \frac{P(X,Z)P(Y,Z)}{P\left(Z\right)}}$, ${\displaystyle \frac{P\left(Y\right)P\left(Z\right)P(X,Y,Z)}{P(Y,Z)}}$, ${\displaystyle \frac{P\left(Z\right)P\left(X\right)P(X,Y,Z)}{P(Z,X)}}$, ${\displaystyle \frac{P\left(X\right)P\left(Y\right)P(X,Y,Z)}{P(X,Y)}}$ and $P(X,Y,Z)$. If we have n triples of actually emitted examples $(X=x_1,Y=y_1,Z=z_1),\cdots ,(X=x_n,Y=y_n,Z=z_n)$ and know the prior probabilities $p_1,\cdots ,p_{11}$ over the eleven factorizations, then it would be reasonable to choose the one that maximizes:

$$\begin{array}{ccc}{p}_1{Q}_X^n\left(x^n\right)Q_Y^n\left(y^n\right)Q_Z\left(z^n\right),\hfill & p_2{Q}_X^n\left(x^n\right)Q_{YZ}^n(y^n,z^n),\hfill & p_3{Q}_Y^n\left(y^n\right)Q_{XZ}^n(x^n,z^n),\hfill \\ p_4{Q}_Z^n\left(z^n\right)Q_{XY}^n(x^n,y^n),\hfill & p_5\frac{Q_{XY}^n(x^n,y^n)Q_{XZ}^n(x^n,z^n)}{Q_X^n\left(x^n\right)},\hfill & p_6\frac{Q_{XY}^n(x^n,y^n)Q_{YZ}^n(y^n,z^n)}{Q_Y^n\left(y^n\right)},\hfill \\ p_7\frac{Q_{XZ}^n(x^n,z^n)Q_{YZ}^n(y^n,z^n)}{Q_Z^n\left(z^n\right)},\hfill & p_8\frac{Q_Y^n\left(y^n\right)Q_Z^n\left(z^n\right)Q_{XYZ}^n(x^n,y^n,z^n)}{Q_{YZ}^n(y^n,z^n)},\hfill & p_9\frac{Q_Z^n\left(z^n\right)Q_X^n\left(x^n\right)Q_{XYZ}^n(x^n,y^n,z^n)}{Q_{XZ}^n(x^n,z^n)},\hfill \\ p_{10}\frac{Q_X^n\left(x^n\right)Q_Y^n\left(y^n\right)Q_{XYZ}^n(x^n,y^n,z^n)}{Q_{XY}^n(x^n,y^n)},\hfill & p_{11}{Q}_{XYZ}^n(x^n,y^n,z^n),\hfill \end{array}$$

to maximize the posterior probability of the factorization given $x^n=(x_1,\cdots ,x_n)$, $y^n=(y_1,\cdots ,y_n)$ and $z^n=(z_1,\cdots ,z_n)$. For example, between the last two distributions, we choose the last if and only if:

$$p_{10}{Q}_X^n\left(x^n\right)Q_Y^n\left(y^n\right)\le p_{11}{Q}_{XY}^n(x^n,y^n).$$

In fact, for example, we can check that the factorizations:

$$P\left(Y\right)P\left(X\right|Y\left)P\right(Z\left|X\right),P\left(X\right)P\left(Y\right|X\left)P\right(Z\left|X\right),P\left(Z\right)P\left(X\right|Z\left)P\right(Y\left|Z\right)$$

in Figure 1a–c share the same form ${\displaystyle \frac{P\left(XY\right)P\left(XZ\right)}{P\left(X\right)}}$, and we say that they share the same Markov-equivalent class. On the other hand, the factorization

$$P\left(Y\right)P\left(Z\right)P\left(X\right|YZ)=\frac{P\left(XYZ\right)P\left(Y\right)P\left(Z\right)}{P\left(YZ\right)}$$

in Figure 1d has nothing to share with the same Markov equivalent class, except itself. In the case of three variables, there are 25 DAGs, but they reduce to the eleven Markov equivalent classes.

Figure 1. Markov-equivalent classes (a–d).

Figure 1. Markov-equivalent classes (a–d).

The method that maximizes the posterior probability is strongly consistent [1] (see Section 3.1 for the proof), and a scenario with two and three variables as above can be extended to cases with N variables in a straightforward manner, if the variables are discrete.

In this paper, we consider the case when continuous variables are present. The idea is to construct measures $g_X^n\left(x^n\right)$, $g_Y^n\left(y^n\right)$ and $g_{XY}^n(x^n,y^n)$ over $X^n$, $Y^n$ and $X^n\times Y^n$ for continuous ranges X and Y to make the decision whether $X\perp \perp Y$ based on:

$$p{g}_X^n\left(x^n\right)g_Y^n\left(y^n\right)\ge (1-p)g_{XY}^n(x^n,y^n).$$

(2)

The main problem is whether the decision is strongly consistent. Many authors have attempted to address continuous variables. For example, Nir Friedman [3] experimentally demonstrated the construction of a genetic network based on expression data using the E-Malgorithm. However, the variables were assumed to be linearly related and included Gaussian noise, and the dataset was not sufficiently fit to the model. Imoto et al. [4] improved the model such that the relation is expressed by B-spline curves rather than lines. However, all of the authors, including Friedman and Imoto, failed to maximize the posterior probability, and thus, the decision is not consistent. This paper proves that the decision based on (2) and its extension for general $N\ge 2$ is strongly consistent.

In any Bayesian approach of BN structure learning, whether continuous variables are present or not, the procedure consists of two stages:

(1)

Compute the local scores for the nonempty subsets of $\{X^{\left(1\right)},\cdots ,X^{\left(N\right)}\}$; for example, if $N=3$, the seven quantities $Q_X^n\left(x^n\right),\cdots ,Q_{XYZ}^n(x^n,y^n,z^n)$ are obtained; and

(2)

Find a BN structure that maximizes the global scores among the $M\left(N\right)(\le 3^N)$ candidate BN structures; there are at most $3^N$ DAGs in the case of N variables; for example, if $N=3$, the eleven quantities are computed and a structure with the largest is chosen.

Note that the second stage does not care about whether each variable is continuous or not. In this paper, we mainly discuss about the performance of the first stage. The number of local scores to be computed can be saved, although it is generally exponential with N. We consider the problem inSection 3.3.

On the other hand, Zhang, Peters, Janzing and Scholkopf [5] proposed a BN structure learning method using conditional independence (CI) tests based on kernel statistics. However, for the CI test that is close to the Hilbert–Schmidt information criterion (HSIC), it is very hard to simulate the null distribution. They only proposed to approximate it by a Gamma distribution, but no consistency, is obtained because the threshold of the statistical test is not correct in practice. Furthermore, for the independence test approach, it often results in conflicting assertions of independence for finite samples. In particular, for small samples, the obtained DAG sometimes contain a directed loop. The Bayesian approach we consider in this paper does not suffer from the inconvenience, because we seek a structure that maximizes the global score [6].

Another contribution of this paper is identifying the border between consistency and non-consistency in learning Bayesian networks. For discrete X, maximizing $Q_X^n\left(x^n\right)$ is equivalent to minimizing the description length [1]:

$$-log{Q}_X^n\left(x^n\right)\approx H^n\left(x^n\right)+\frac{\alpha -1}{2}logn,$$

(3)

where ${\displaystyle H^n\left(x^n\right)}$ is the empirical entropy of $x^n\in X^n$ (we write $A\approx B$ when $|A-B|$ is bounded by a constant) and α is the cardinality of set X. The problem at hand is whether the $logn$ term is the minimum function of n for ensuring strong consistency. If $logn$ is replaced by two (AIC), we cannot obtain consistency. We prove that $2(1+ϵ)loglogn$ with $ϵ>0$ is the minimum for strong consistency based on the law of iterated logarithms. The same property is known as the Hannan–Quinn principle [7], and similar results have been obtained for autoregression, linear regression [8] and classification [9], among others. The derivation in this paper does not depend on these previous results. The Hannan–Quinn principle will also be applied to continuous variables.

This paper is organized as follows. Section 2.1 introduces the general concept of learning Bayesian network structures based on maximizing the posterior probability, and Section 2.2 discusses the concept of density functions developed by Boris Ryabko [10] and extended by Suzuki [11]. Section 3 presents our contributions: Section 3.1 proves the Hannan–Quinn property in the current problem, andSection 3.2 proves consistency when continuous variables are present. Section 4 concludes the paper by summarizing the results and states the paper's significance in the field of model selection.

2. Preliminaries

2.1. Learning the Bayesian Structure for Discrete Variables and Its Consistency

We choose $w_X$, such that $\int w_X\left(\theta \right)d\theta =1$ and $0\le \theta \left(x\right)\le 1$ by $w_X\left(\theta \right)\propto {\prod }_{x\in X}\theta {\left(x\right)}^{-1/2}$, where X is the set from which X takes its values. Let $\alpha =\left|X\right|$, and let $c_i\left(x\right)$ be the frequency of $x\in X$ in $x^i=(x_1,\cdots ,x_i)\in X^i$, $i=1,\cdots ,n$. It is known that the following quantities satisfies (3) [12]:

$$Q_X^n\left(x^n\right):=\prod _{i=1}^n\frac{c_{i-1}\left(x_i\right)+1/2}{i-1+\left|X\right|/2}=\frac{\Gamma (\alpha /2){\prod }_{x\in X}\Gamma (c_n\left(x\right)+1/2)}{\Gamma {(1/2)}^{\alpha }\Gamma (n+\alpha /2)},$$

where Γ is the Gamma function, and Stirling's formula ${\displaystyle \Gamma \left(z\right)=\sqrt{2\pi z}{\left(\frac{z}{e}\right)}^z\{1+O\left(z^{-1/3}\right)\}}$ has been applied. Thus, for $x\in X$, from the law of large numbers, $c_n\left(x\right)/n$ converges to $P(X=x)$ with probability one as $n\to \infty$, such that:

$$-\frac{1}{n}log{Q}^n\left(x^n\right)\to H\left(X\right):=\sum _{x\in X}-P(X=x)logP(X=x)$$

with probability one as $n\to \infty$.

Moreover, from the law of large numbers, with probability one as $n\to \infty$,

$$-\frac{1}{n}logP(X^n=x^n)=\frac{1}{n}\sum _{i=1}^n\{-logP(X=x_i)\}\to E[-logP\left(X\right)]=H\left(X\right)$$

(Shannon–McMillan–Breiman [13]). This proves that there exists a $Q_X^n$ (universal measure), such that for any probability P over the finite set X,

$$\frac{1}{n}log\frac{P^n\left(x^n\right)}{Q^n\left(x^n\right)}\to 0$$

(4)

with probability one as $n\to \infty$, where we write $P^n\left(x^n\right):=P(X^n=x^n)$. The same property holds for:

$$-log{Q}_Y^n\left(y^n\right)\approx H^n\left(y^n\right)+\frac{\beta -1}{2}logn,$$

(5)

and:

$$-log{Q}_{XY}^n(x^n,y^n)\approx H^n(x^n,y^n)+\frac{\alpha \beta -1}{2}logn,$$

(6)

where $\beta =\left|Y\right|$, ${\displaystyle H^n\left(y^n\right)=\sum _{y\in Y}-c_n\left(y\right)log\frac{c_n\left(y\right)}{n}}$ and ${\displaystyle H^n(x^n,y^n)=\sum _{x\in X}\sum _{y\in Y}-c_n(x,y)log\frac{c_n(x,y)}{n}}$ are the empirical entropies of $y^n\in Y^n$ and $(x^n,y^n)\in X^n\times Y^n$, and $c_n\left(y\right)$ and $c_n(x,y)$ are the numbers of occurrences of $y\in Y$ and $(x,y)\in X\times Y$ in $y^n=(y_1,\cdots ,y_n)\in Y^n$ and $(x^n,y^n)\in X^n\times Y^n$, respectively.

Thus, we have:

$$J^n(x^n,y^n):=\frac{1}{n}log\frac{Q_{XY}(x^n,y^x)}{Q_X\left(x^n\right)Q_Y\left(y^n\right)}\to I(X,Y):=E\left\{\frac{P(X,Y)}{P\left(X\right)P\left(Y\right)}\right\}.$$

with probability one as $n\to \infty$. However, $X\perp \perp Y$ if and only if $I(X,Y)=0$. Hence, if $X\neg \perp \perp Y$, the value of $J^n(x^n,y^n)$ is positive with probability one as $n\to \infty$. However, how can we detect $X\perp \perp Y$ when $X\perp \perp Y$? $J^n(x^n,y^n)$ cannot be exactly zero with probability one as $n\to \infty$.

However, when X and Y are discrete, the estimation based on $J^n(x^n,y^n)$ is consistent: if $X\perp \perp Y$, the value of $J^n(x^n,y^n)$ is not greater than zero with probability one as $n\to \infty$. For example, the decision based on (1) is strongly consistent because the values of $\frac{1}{n}logp$ and $\frac{1}{n}log(1-p)$ are negligible for large n, and asymptotically, (1) is equivalent to $J^n(x^n,y^n,z^n)\le 0$.

In Section 3.1, we provide a stronger result of consistency and a more intuitive and elegant proof.

In general, if N variables exist ($N\ge 2$), we must consider two cases: $D\left(P^{*}\right|\left|P\right)>0$ and$D\left(P^{*}\right|\left|P\right)=0$, where $P^{*}$ and P are the probabilities based on the correct and estimated factorizations and $D\left(P^{*}\right|\left|P\right)$ denotes the Kullback–Leibler divergence between $P^{*}$ and P. If $N=2$, then:

$$D(P^{*}\left|\right|P):=\sum _x\sum _y{P}^{*}(x,y)log\frac{P^{*}(x,y)}{P(x,y)}>0$$

if and only if $X\neg \perp \perp Y$ in $P^{*}$ and $X\perp \perp Y$ in P.

The same property holds for three variables $X,Y,Z$ ($N=3$):

$$J^n(x^n,y^n,z^n):=\frac{1}{n}log\frac{Q_{XYZ}(x^n,y^n,z^n)Q_Z^n\left(z^n\right)}{Q_{XZ}^n(x^n,y^n)Q_{YZ}^n(y^n,z^n)}\to I(X,Y,Z):=E\left\{\frac{P\left(XYZ\right)P\left(Z\right)}{P\left(XZ\right)P\left(YZ\right)}\right\}$$

with probability one as $n\to \infty$, and $X\perp \perp Y|Z$ if and only if $I(X,Y,Z)=0$. Then, we can show $J^n(x^n,y^n,z^n)\le 0$ if and only if $I(X,Y,Z)=0$, with probability one as $n\to \infty$ (see Section 3.1). For example, between the seventh and eleventh factorizations, if $J^n(x^n,y^n,z^n)\le 0$ and $J^n(x^n,y^n,z^n)>0$, then we choose the seventh and eleventh, respectively. In fact,

$$p_7\frac{Q_{XZ}^n(x^n,z^n)Q_{YZ}^n(y^n,z^n)}{Q_Z^n\left(z^n\right)}\ge p_{11}{Q}_{XYZ}^n(x^n,y^n,z^n)⟺J^n(x^n,y^n,z^n)\le 0$$

for large n, because $\frac{1}{n}log\frac{p_7}{p_{11}}$ diminishes.

Then, the decision is correct with probability one as $n\to \infty$. Similarly, we calculate:

$$-log{Q}_Z\left(z^n\right)\approx H^n\left(z^n\right)+\frac{\gamma -1}{2}logn,$$

$$-log{Q}_{YZ}(y^n,z^n)\approx H^n(y^n,z^n)+\frac{\beta \gamma -1}{2}logn,$$

$$-log{Q}_{ZX}(z^n,x^n)\approx H^n(z^n,x^n)+\frac{\gamma \alpha -1}{2}logn,$$

and:

$$-log{Q}_{XYZ}(x^n,y^n,z^n)\approx H^n(x^n,y^n,z^n)+\frac{\alpha \beta \gamma -1}{2}logn,$$

where $\gamma =\left|Z\right|$. In general, for N variables, given P and $P^{*}$, we have all of the CI statements for each of them, and $D\left(P^{*}\right|\left|P\right)=0$ if and only if the CI statements in P imply those in $P^{*}$; in other words, P induces an I-map, which is not necessarily minimal.

Note that for any subsets $a,b,c$ of $\{1,\cdots ,N\}$, we can construct the estimation $J^n(x^n,y^n,z^n)$, with $X={\left\{X^{\left(i\right)}\right\}}_{i\in a},Y={\left\{Y^{\left(j\right)}\right\}}_{j\in b},Z={\left\{X^{\left(k\right)}\right\}}_{k\in c}$, and obtain consistency, i.e., we will have the correct CI statements, where c may be empty.

Table 1 depicts whether $D\left(P^{*}\right|\left|P\right)>0$ or $D\left(P^{*}\right|\left|P\right)=0$ for each $P^{*}$ and P. For example, if the factorizations of $P^{*}$ and P are the fourth and sixth, then $D\left(P^{*}\right|\left|P\right)=0$ from the table. In general, $D\left(P^{*}\right|\left|P\right)=0$ if and only if $P^{*}$ is realized using the factorization and an appropriate parameterset for P.

Table 1. Three-variable case: $D\left(P^{*}\right|\left|P\right)>0$ or $D\left(P^{*}\right|\left|P\right)=0$: “+” and “0” denote $D\left(P^{*}\right|\left|P\right)>0$ and $D\left(P^{*}\right|\left|P\right)=0$, respectively.

	Estimated P
		1	2	3	4	5	6	7	8	9	10	11
True $P^{*}$	1	*	0	0	0	0	0	0	0	0	0	0
	2	+	*	+	+	+	0	0	+	+	+	0
	3	+	+	*	+	0	+	0	+	+	+	0
	4	+	+	+	*	0	0	+	+	+	+	0
	5	+	+	+	+	*	+	+	+	+	+	0
	6	+	+	+	+	+	*	+	+	+	+	0
	7	+	+	+	+	+	+	*	+	+	+	0
	8	+	+	+	+	+	+	+	*	+	+	0
	9	+	+	+	+	+	+	+	+	*	+	0
	10	+	+	+	+	+	+	+	+	+	*	0
	11	+	+	+	+	+	+	+	+	+	+	*

Table 1. Three-variable case: $D\left(P^{*}\right|\left|P\right)>0$ or $D\left(P^{*}\right|\left|P\right)=0$: “+” and “0” denote $D\left(P^{*}\right|\left|P\right)>0$ and $D\left(P^{*}\right|\left|P\right)=0$, respectively.

	Estimated P
		1	2	3	4	5	6	7	8	9	10	11
True $P^{*}$	1	*	0	0	0	0	0	0	0	0	0	0
	2	+	*	+	+	+	0	0	+	+	+	0
	3	+	+	*	+	0	+	0	+	+	+	0
	4	+	+	+	*	0	0	+	+	+	+	0
	5	+	+	+	+	*	+	+	+	+	+	0
	6	+	+	+	+	+	*	+	+	+	+	0
	7	+	+	+	+	+	+	*	+	+	+	0
	8	+	+	+	+	+	+	+	*	+	+	0
	9	+	+	+	+	+	+	+	+	*	+	0
	10	+	+	+	+	+	+	+	+	+	*	0
	11	+	+	+	+	+	+	+	+	+	+	*

2.2. Universal Measures for Continuous Variables

In this section, we primarily address continuous variables.

Let $\left\{A_j\right\}$ be such that $A_0=\left\{X\right\}$, and let $A_{j+1}$ be a refinement of $A_j$. For example, suppose that the random variable X takes values in $X=[0,1]$, and we generate a sequence as follows:

$${\displaystyle \begin{array}{cc}{A}_1& =\left\{\right[0,\frac{1}{2}),[\frac{1}{2},1\left)\right\}\hfill \\ A_2& =\left\{\right[0,\frac{1}{4}),[\frac{1}{4},\frac{1}{2}),[\frac{1}{2},\frac{3}{4},[\frac{3}{4},1)\}\hfill \\ ⋮& \\ A_j& =\left\{\right[0,2^{-(j-1)}),[2^{-(j-1)},2·2^{-(j-1)}),\cdots ,2^{j-1}-1)2^{-(j-1)},1\left)\right\}\hfill \\ ⋮& \end{array}}$$

For each j, we quantize each $x\in [0,1]$ into the $a\in A_j$, such that $x\in a$. For example, for $j=2$, $x=0.4$ is quantized into $a=[\frac{1}{4},\frac{1}{2})\in A_2$. Let λ be the Lebesgue measure (width of the interval). For example, $\lambda \left([\frac{1}{4},\frac{1}{2})\right)=\frac{1}{4}$ and $\lambda \left(\left\{\frac{1}{2}\right\}\right)=0$.

Note that each $A_j$ is a finite set. Therefore, we can construct a universal measure $Q_j^n$ w.r.t. a finite set $A_j$ for each j. Given $x^n=(x_1,\cdots ,x_n)\in {[0,1]}^n$, we obtain a quantized sequence $(a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)})\in A_j^n$ for each j and use it to compute the quantity:

$${\displaystyle g_j^n\left(x^n\right):=\frac{Q_j^n(a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)})}{\lambda \left(a_1^{\left(j\right)}\right)\cdots \lambda \left(a_n^{\left(j\right)}\right)}}$$

for each j. If we prepare a sequence of positive reals $w_1,w_2,\cdots$, such that ${\sum }_j{w}_j=1$ and $w_j>0$, we can compute the quantity:

$${\displaystyle g_X^n\left(x^n\right):=\sum _{j=1}^{\infty }{w}_j{g}_j^n\left(x^n\right).}$$

Moreover, let $f_X$ be the true density function and $f_j\left(x\right):=P(X\in a)/\lambda \left(a\right)$ for $a\in A_j$ and $j=1,2,\cdots$ if $x\in a$. We may consider $f_j$ to be an approximated density function assuming the quantization sequence $\left\{A_j\right\}$ (Figure 2). For the given $x^n$, we define $f_X^n\left(x^n\right)=f_X\left(x_1\right)\cdots f_X\left(x_n\right)$ and $f_j^n\left(x^n\right):=f_j\left(x_1\right)\cdots f_j\left(x_n\right)$.

Figure 2. Quantization at level k: $x^n=(x_1,\cdots ,x_n)\mapsto (a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)})$

Figure 2. Quantization at level k: $x^n=(x_1,\cdots ,x_n)\mapsto (a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)})$

Thus, we have the following proposition, which is a continuous version of the universality (4) that was proven in Section 2.1.

Proposition 1 ([10]). For any density function f, such that $D\left(f_X\right|\left|f_j\right)\to 0$ as $j\to \infty$,

$$\frac{1}{n}log\frac{f_X^n\left(x^n\right)}{g_X^n\left(x^n\right)}\to 0$$

as $n\to \infty$ with probability one, where $D\left(f_X\right|\left|f_j\right)$ is the Kullback–Leibler divergence between $f_X$ and $f_j$.

The same concept is applied to the case where no density function exists [11] in the usual sense (w.r.t. the Lebesgue measure λ). For example, suppose that we wish to estimate a distribution over the positive integers N. Apparently, N is not a finite set and has no density function. We consider the quantization sequence $\left\{B_k\right\}$: $B_0=\left\{N\right\}$, $B_1:=\{\left\{1\right\},\{2,3,\cdots \}\}$, $B_2:=\{\left\{1\right\},\left\{2\right\},\{3,4,\cdots \}\}$, ..., $B_k:=\{\left\{1\right\},\left\{2\right\},\cdots ,\left\{k\right\},\{k+1,k+2,\cdots \}\}$, ....

For each k, we quantize each $y\in N$ into a $b\in B_k$, such that $y\in b$. For example, for $k=2$, $y=4$ is quantized into $b=\{3,4,\cdots \}\in B_2$. Let η be a measure, such that:

$$\eta \left(\left\{k\right\}\right)=\frac{1}{k}-\frac{1}{k+1},k\in N.$$

The measure $\eta \left(a\right)$ for closed interval a gives:

$$\eta \left(a\right)=\sum _{k\in a}\eta \left(\left\{k\right\}\right)=\sum _{k\in a}(\frac{1}{k}-\frac{1}{k+1})=\frac{1}{k_{min}}-\frac{1}{k_{max}}$$

if $k_{min}$ and $k_{max}$ are the minimum and maximum integers in a, and evaluates each bin width in a nonstandard way. For example, $\eta \left(\left\{2\right\}\right)=\frac{1}{6}$ and $\eta \left(\{3,4\}\right)=\frac{2}{15}$. For multiple variables, we compute the measure by:

$$\eta (\left\{j\right\},\left\{k\right\})=(\frac{1}{j}-\frac{1}{j+1})(\frac{1}{k}-\frac{1}{k+1}).$$

Note that each $B_k$ is a finite set, and we construct a universal measure $Q_k^n$ w.r.t. a finite set $B_k$ for each k. Given $y^n=(y_1,\cdots ,y_n)\in N^n$, we obtain a quantized sequence $(b_1^{\left(k\right)},\cdots ,b_n^{\left(k\right)})\in B_k^n$ for each k, such that we can compute the quantity:

$${\displaystyle g_k^n\left(y^n\right):=\frac{Q_k^n(b_1^{\left(k\right)},\cdots ,b_n^{\left(k\right)})}{\eta \left(b_1^{\left(k\right)}\right)\cdots \eta \left(b_n^{\left(k\right)}\right)}}$$

for each k. If we prepare a sequence of positive reals $w_1,w_2,\cdots$, such that ${\sum }_k{w}_k=1$ and $w_k>0$, we can compute the quantity ${\displaystyle g_Y^n\left(y^n\right):=\sum _{k=1}^{\infty }{w}_k{g}_k^n\left(y^n\right)}$. In this case, ${\displaystyle f_Y\left(y\right)=\frac{P(Y=y)}{\eta \left(\right\{y\left\}\right)}}$ for $y\in N$ ($f\left(y\right)$ with $y\notin N$ may take any arbitrary value) is considered to be a generalized density function (w.r.t. the measure η).

In general, if $\eta \left(D\right)=0$ implies $P(Y\in D)=0$ for the Borel sets (the Borel sets w.r.t. R being the set consisting of the sets generated via a countable number of union, intersection and set difference from the closed intervals of R [2]), we state that P is absolutely continuous w.r.t. η and that there exists a density function w.r.t. η (Radon–Nikodym [2]).

The following proposition addresses generalized densities and eliminates the condition$D\left(f_Y\right|\left|f_j\right)\to 0$ as $j\to \infty$ in Proposition 1.

Proposition 2 ([11]). For any generalized density function $f_Y$,

$$\frac{1}{n}log\frac{f_Y^n\left(y^n\right)}{g_Y^n\left(y^n\right)}\to 0$$

as $n\to \infty$ with probability one.

Proposition 1 assumes a specific quantization sequence, such as $\left\{A_n\right\}$. The universality holds for the densities that satisfy $D\left(f_X\right|\left|f_k\right)\to \infty$ as $k\to \infty$ [10]. However, in the proof of Proposition 2, a universal quantization, such that $D\left(f_X\right|\left|f_k\right)\to 0$ as $k\to \infty$ for any density $f_X$, was constructed [11].

3. Contributions

3.1. The Hannan and Quinn Principle

We know that $H^n\left(x^n\right)+H^n\left(y^n\right)-H^n(x^n,y^n)$ is at most $\frac{(\alpha -1)(\beta -1)}{2}logn$ with probability one as $n\to \infty$ when $X\perp \perp Y$ because the decision based on (1) is strongly consistent.

In this section, we prove a stronger result: let:

$$I^n(x^n,y^n,z^n):=H^n(x^n,z^n)+H^n(y^n,z^n)-H^n(x^n,y^n,z^n)-H^n\left(z^n\right).$$

We show that the quantity $I^n(x^n,y^n,z^n)$ is at most $(\alpha -1)(\beta -1)\gamma loglogn$ rather than${\displaystyle \frac{1}{2}(\alpha -1)(\beta -1)\gamma logn}$, when $X\perp \perp Y|Z$:

Theorem 1. If $X\perp \perp Y|Z$:

$$I^n(x^n,y^n,z^n)\le (1+ϵ)(\alpha -1)(\beta -1)\gamma loglogn$$

(7)

with probability one as $n\to \infty$ for any $ϵ>0$.

In order to show the claim, we approximate $I^n(x^n,y^n,z^n)$ by ${\sum }_{z\in Z}I\left(z\right)$ with ${\displaystyle I\left(z\right)=\frac{1}{2}\sum _{i=1}^{\alpha -1}\sum _{j=1}^{\beta -1}{r}_{i,j}^2}$, where $r_{i,j}$, $i=1,\cdots ,\alpha -1,j=1,\cdots ,\beta -1$, are mutually independent random variables with mean zero and variance ${\sigma }_{i,j}^2$, such that:

$$\sum _{i=1}^{\alpha -1}\sum _{j=1}^{\beta -1}{\sigma }_{i,j}^2=(\alpha -1)(\beta -1).$$

Then, from the law of iterated logarithms below (Lemma 1) [2], it will be proven that $r_{i,j}^2$ is almost surely upper-bounded by $2(1+ϵ){\sigma }_{i,j}^{2}loglogn$ for any $ϵ>0$ and each $z\in Z$, which implies Theorem 1 because:

$$\begin{array}{ccc}\hfill I^n(x^n,y^n,z^n)& \approx & \sum _{z}I\left(z\right)=\gamma ·\frac{1}{2}\sum _i\sum _j{r}_{i,j}^2\hfill \\ & \le & \gamma ·\frac{1}{2}\sum _i\sum _{j}2(1+ϵ){\sigma }_{i,j}^{2}loglogn\hfill \\ & =& (1+ϵ)(\alpha -1)(\beta -1)\gamma loglogn\hfill \end{array}$$

(see the Appendix for the details of the derivation).

Lemma 1 ([2]). Let ${\left\{U_k\right\}}_{k=1}^n$ be random variables that obey an identical distribution with zero mean and unit variance, and $S_n:={\sum }_{k=1}^n{U}_k$. Then, with probability one,

$$\underset{n\to \infty }{lim\; sup}\frac{S_n}{\sqrt{2nlognlogn}}=1.$$

Theorem 1 implies the strong consistency of the decision based on (1). However, a stronger statement can be obtained:

Theorem 2. We define $R_Z^n\left(z^n\right)$, $R_{XZ}^n(x^n,z^n)$, $R_{YZ}^n(y^n,z^n)$ and $R_{XYZ}^n(x^n,y^n,z^n)$ by:

$$-log{R}_Z^n\left(z^n\right)=H^n\left(z^n\right)+(1+ϵ)(\gamma -1)loglogn,$$

$$-log{R}_{XZ}^n(x^n,z^n)=H^n(x^n,z^n)+(1+ϵ)(\beta \gamma -1)loglogn,$$

$$-log{R}_{YZ}^n(y^n,z^n)=H^n(y^n,z^n)+(1+ϵ)(\beta \gamma -1)loglogn,$$

and:

$$-log{R}_{XYZ}^n(x^n,y^n,z^n)=H^n(x^n,y^n,z^n)+(1+ϵ)(\alpha \beta \gamma -1)loglogn.$$

Then, the decision based on:

$$R_{XZ}^n(x^n,z^n)R_{YZ}^n(y^n,z^n)\ge R_{XYZ}^n(x^n,y^n,z^n)R_Z^n\left(z^n\right)⟺X\perp \perp Y|Z$$

is strongly consistent.

Proof. We note two properties:

• $R_{XZ}^n(x^n,z^n)R_{YZ}^n(y^n,z^n)\ge R_{XYZ}^n(x^n,y^n,z^n)R_Z^n\left(z^n\right)$ is equivalent to (7); and
• ${\displaystyle \underset{n\to \infty }{lim}\frac{1}{n}log\frac{R_{XYZ}^n(x^n,y^n,z^n)R_Z^n\left(z^n\right)}{R_{XZ}^n(x^n,z^n)R_{YZ}^n(x^n,z^n)}=\underset{n\to \infty }{lim}\frac{1}{n}log\frac{Q_{XYZ}^n(x^n,y^n,z^n)Q_Z^n\left(z^n\right)}{Q_{XZ}^n(x^n,z^n)Q_{YZ}^n(x^n,z^n)}\to I(X,Y,Z)}$

If $X\perp \perp Y|Z$, then from Theorem 1 and the first property, we have $R_{XZ}^n(x^n,z^n)R_{YZ}^n(y^n,z^n)\ge R_{XYZ}^n(x^n,y^n,z^n)R_Z^n\left(z^n\right)$ almost surely. If $R_{XZ}^n(x^n,z^n)R_{YZ}^n(y^n,z^n)\ge R_{XYZ}^n(x^n,y^n,z^n)R_Z^n\left(z^n\right)$ almost surely holds, then the value in the second property should be no greater than zero, which means that $X\perp \perp Y|Z$. This completes the proof. ☐

Theorem 2 is related to the Hannan and Quinn theorem [7] for model selection. To obtain strong consistency, they proved that $loglogn$ rather than $\frac{1}{2}logn$ is sufficient for the penalty terms of autoregressive model selection. Recently, several authors have proven this in other settings, such as classification [9] and linear regression [8].

3.2. Consistency for Continuous Variables

Suppose that we wish to estimate the distribution over $[0,1]\times N$ in Section 2.2. The set $[0,1]\times N$ is not a finite set and has no density function.

Because $A_j\times B_k$ is a finite set, we can construct a universal measure $Q_{j,k}^n$ for $A_j\times B_k$:

$${\displaystyle g_{jk}^n(x^n,y^n):=\frac{Q_{j,k}^n(a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)},b_1^{\left(k\right)},\cdots ,b_n^{\left(k\right)})}{\lambda \left(a_1^{\left(j\right)}\right)\cdots \lambda \left(a_n^{\left(j\right)}\right)\eta \left(b_1^{\left(k\right)}\right)\cdots \eta \left(b_n^{\left(k\right)}\right)}.}$$

If we prepare the sequence such that ${\sum }_{jk}{\omega }_{jk}=1$, ${\omega }_{jk}>0$, we obtain the quantity:

$${\displaystyle g_{XY}^n(x^n,y^n):=\sum _{j=1}^{\infty }\sum _{k=1}^{\infty }{w}_{j,k}{g}_{jk}^n(x^n,y^n).}$$

In this case, the (generalized) density function is obtained via:

$$f_{XY}(x,y)=\frac{F_X\left(x\right|y)}{dx}·\frac{P(Y=y)}{\eta \left(\right\{y\left\}\right)}$$

where $y\in N$ ($f_{XY}$ takes arbitrary values for $x\notin [0,1]$ and $y\notin N$), where $F_X(·|y)$ is the conditional distribution function of X given $Y=y$.

In general, we have the following result:

Proposition 3 For any generalized density function f:

$$\frac{1}{n}log\frac{f_{XY}^n(x^n,y^n)}{g_{XY}^n(x^n,y^n)}\to 0$$

as $n\to \infty$ with probability one.

The measures $g_X^n\left(x^n\right)$ and $g_{XY}^n(x^n,y^n)$ are computed using (A) and (B) of Algorithm 1, where the value of K is the number of quantizations, and ${\widehat{g}}_X^n\left(x^n\right)$ and ${\widehat{g}}_{XY}^n(x^n,y^n)$ denote the approximated scores using finite quantization of level K.

Algorithm 1 Calculating gn.
(A) Input xn ∈ An, Output ${\widehat{g}}_X^n\left(x^n\right)$
1.	For each $k=1,\cdots ,K$, $g_k^n\left(x^n\right):=0$
2.	For each $k=1,\cdots ,K$ and each $a\in A_k$, $c_k\left(a\right):=0$
3.	For each $i=1,\cdots ,n$,
	(a) $A_0=X$, $a_i^{\left(0\right)}=x_i$
	(b) for each $k=1,\cdots ,K$
	i. Find $a_i^{\left(k\right)}\in A_k$ from $a_i^{(k-1)}\in A_{k-1}$
	ii. $log{g}_k^n\left(x^n\right):=log{g}_k^n\left(x^n\right)+log\frac{c_{i,k}\left(a_i^{\left(k\right)}\right)+1/2}{i-1+|A_k|/2}-log\left({\eta }_X\left(a_i^{\left(k\right)}\right)\right)$
	iii. $c_{i,k}\left(a_i^{\left(k\right)}\right):=c_{i,k}\left(a_i^{\left(k\right)}\right)+1$
4.	${\widehat{g}}_X^n\left(x^n\right):={\sum }_{k=1}^K\frac{1}{K}{g}_k^n\left(x^n\right)$
(B) Input $x^n\in A^n$ and $y^n\in B^n$, Output ${\widehat{g}}_{XY}^n(x^n,y^n)$
1.	For each $j,k=1,\cdots ,K$, $g_{j,k}^n(x^n,y^n):=0$
2.	For each $j,k=1,\cdots ,K$ and each $a\in A_j$ and $b\in B_k$, $c_{j,k}(a,b):=0$
3.	For each $i=1,\cdots ,n$
	(a) $A_0=X$, $B_0=Y$, $a_i^{\left(0\right)}=x_i$, $b_i^{\left(0\right)}=y_i$
	(b) for each $j,k=1,\cdots ,K$
	i. Find $a_i^{\left(j\right)}\in A_j$ and $b_i^{\left(k\right)}\in B_k$ from $a_i^{(j-1)}\in A_{j-1}$ and $b_i^{(k-1)}\in B_{k-1}$
	ii. $log{g}_{j,k}^n(x^n,y^n):=log{g}_{j,k}^n(x^n,y^n)+log\frac{c_{i,j,k}(a_i^{\left(j\right)},b_i^{\left(k\right)})+1/2}{i-1+|A_j\left|\right|B_k|/2}-log\left({\eta }_X\left(a_i^{\left(j\right)}\right){\eta }_Y\left(b_i^{\left(k\right)}\right)\right)$
	iii. $c_{i,j,k}(a_i^{\left(j\right)},b_i^{\left(k\right)}):=c_{i,j,k}(a_i^{\left(j\right)},b_i^{\left(k\right)})+1$
4.	${\widehat{g}}_{XY}^n(x^n,y^n):={\sum }_{j=1}^K{\sum }_{k=1}^K\frac{1}{K^2}{g}_{j,k}^n(x^n,y^n)$

Propositions 1–3 are obtained for large K. However, we can prepare only a finite number of quantizations. Furthermore, if n is small, then the number of examples that each bin contains is small, and we cannot estimate the histogram well. Therefore, given n, K must be moderately sized, and we recommend to set $K=\frac{1}{m}logn$ because the number of examples contained in a bin decreases exponentially with increasing depth, where m is the number of variables in the local score. For example, $m=1$ and $m=2$ for (A) and (B), respectively. Algorithm 1 (A)(B) of do not guarantee anything for the theoretical property assured in Proposition 3 and Theorems 3–5 for finite K, however, as K grows, consistency holds.

In Step 3(a) of Algorithm 1(A)(B), we calculate $a_i^{\left(k\right)}$ from $a_i^{(k-1)}$ and not from $x_i$, which means that the computational time required to obtain $(a_i^{\left(1\right)},\cdots ,a_i^{\left(K\right)})$ from $x_i$ is $O\left(K\right)$. Thus, the total computation times of Algorithm 1 (A)(B) are at most $O\left(nK\right)$.

In Step 3(b) of Algorithm 1(A), we compute for $i=1,\cdots ,n$ and $k=1,\cdots ,K$:

$$log\frac{g_k^i\left(x^i\right)}{g_k^{i-1}\left(x^{i-1}\right)}=log\frac{Q_k^i(a_1^{\left(k\right)},\cdots ,a_i^{\left(k\right)})}{Q_k^{i-1}(a_1^{\left(k\right)},\cdots ,a_{i-1}^{\left(k\right)})}-log{\eta }_X\left(a_i^{\left(k\right)}\right)$$

if $x_i$ is quantized into $a_i^{\left(k\right)}\in A_k$, $i=1,\cdots ,n$.

For the memory requirements, we require exponential orders of K. However, because we set $K=\frac{1}{m}logn$, the computational time and memory requirements are at most $O(nlogn)$ and $O\left(n\right)$ for Algorithm 1(A)(B).

Based on the same notion, we can construct $g_Z^n\left(z^n\right)$, $g_{XZ}^n(x^n,z^n)$, $g_{YZ}(y^n.z^n)$, $g_{XYZ}^n(x^n,y^n,z^n)$ from examples $x^n\in X^n$, $y^n\in Y^n$ and $z^n\in Z^n$, and Propositions 2 and 3 hold for three variables.

Theorem 3. With probability one as $n\to \infty$:

$$\frac{1}{n}log\frac{g_{XYZ}^n(x^n,y^n,z^n)g_Z\left(z^n\right)}{g_{XY}^n(x^n,z^n)g_{YZ}^n(y^n,z^n)}\to I(X,Y,Z)$$

(8)

Proof. From Propositions 2 and 3 for two and three variables and the law of large numbers, we have:

$$\begin{array}{ccc}& & \underset{n\to \infty }{lim}\frac{1}{n}log\frac{g_{XYZ}^n(x^n,y^n,z^n)g_Z^n\left(z^n\right)}{g_{XZ}^n(x^n,z^n)g_{YZ}^n(y^n,z^n)}=\underset{n\to \infty }{lim}\frac{1}{n}log\frac{f_{XYZ}^n(x^n,y^n,z^n)f_Z^n\left(z^n\right)}{f_{XZ}^n(x^n,z^n)f_{YZ}^n(y^n,z^n)}\hfill \\ & =& \underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^n\{log\frac{f_{XYZ}(x_i,y_i,z_i)f_Z\left(z_i\right)}{f_{XZ}(x_i,z_i)f_{YZ}(y_i,z_i)}\}=Elog\frac{f_{XYZ}(X,Y,Z)}{f_{XZ}(X,Z)f_{YZ}(Y,Z)}=I(X,Y,Z)\hfill \end{array}$$

with probability one, which completes the proof. ☐

From the discussion in Section 2.1, even when more than two variables are present, if $D\left(P^{*}\right|\left|P\right)>0$, we can choose $P^{*}$ rather than P with probability one as $n\to \infty$.

Now, we prove that the continuous counterpart of the decision based on (1) is strongly consistent:

Theorem 4. With probability one as $n\to \infty$:

$$X\perp \perp Y|Z⟺p{g}_{XZ}^n(x^n,z^n)g_{YZ}^n(y^n,z^n)\ge (1-p)g_{XYZ}^n(x^n,y^n,z^n)g_Z^n\left(z^n\right),$$

(9)

where p is the prior probability of $X\perp \perp Y|Z$.

Proof: Suppose that $X\neg \perp \perp Y|Z$. Then, the conditional mutual information between X and Y given Z is positive, and from Theorem 3, the estimator converges to a positive value with probability one as $n\to \infty$; thus, $p{g}_{XZ}^n(x^n,z^n)g_{YZ}^n(y^n,z^n)\ge (1-p)g_{XYZ}^n(x^n,y^n,z^n)g_Z^n\left(z^n\right)$ holds almost surely. Suppose that $X\perp \perp Y|Z$. The discrete variables X and Y are conditionally independent given Z if and only if:

$$c{Q}_{XZ}^n(x^n,z^n)Q_{YZ}^n(y^n,z^n)\ge (1-c)Q_{XYZ}^n(x^n,y^n,z^n)Q_Z^n\left(z^n\right)$$

with probability one as $n\to \infty$ for any constant $0<c<1$, even if c does not coincide with the prior probability p. If $X,Y$ and Z are continuous, we quantize $x^n$, $y^n$ and $z^n$ into $(a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)})$, $(b_1^{\left(k\right)},\cdots ,b_n^{\left(k\right)})$ and $(c_1^{\left(l\right)},\cdots ,c_n^{\left(l\right)})$. Thus, for each $j,k$ and l, we have:

$$\begin{array}{ccc}& & p{w}_{jl}{w}_{kl}{Q}_{jl}^n(a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)},c_1^{\left(l\right)},\cdots ,c_n^{\left(l\right)})Q_{kl}^n(b_1^{\left(k\right)},\cdots ,b_n^{\left(k\right)},c_1^{\left(l\right)},\cdots ,c_n^{\left(l\right)})\hfill \\ & \ge & (1-p)w_{jkl}{w}_l{Q}_{jkl}^n(a_1^{\left(j\right)},\cdots ,a_n^{\left(j\right)},b_1^{\left(k\right)},\cdots ,b_n^{\left(k\right)},c_1^{\left(l\right)},\cdots ,c_n^{\left(l\right)})Q_n^{\left(l\right)}(c_1^{\left(l\right)},\cdots ,c_n^{\left(l\right)})\hfill \end{array}$$

with probability one as $n\to \infty$. Thus, if we divide both sides by:

$${\eta }_X\left(a_1^{\left(j\right)}\right)\cdots {\eta }_X\left(a_n^{\left(j\right)}\right){\eta }_Y\left(b_1^{\left(k\right)}\right)\cdots {\eta }_Y\left(b_n^{\left(k\right)}\right){\eta }_Z\left(c_1^{\left(l\right)}\right)\cdots {\eta }_Z\left(c_n^{\left(l\right)}\right)$$

and take summations of both sides over $j,k.l=1,2,\cdots$, we have:

$$p{g}_{XZ}^n(x^n,z^n)g_{YZ}^n(y^n,z^n)\ge (1-p)g_{XYZ}^n(x^n,y^n,z^n)g_Z^n\left(z^n\right)$$

with probability one, where we have assumed $w_{j,k,l}>0⟹w_{jl},w_{kl}>0$ because of $K=\frac{1}{m}logn$, which completes the proof.

Note that even if either X or Y is discrete, the same conclusion will be obtained. The generalized density functions cover the discrete distributions as a special case.

From the discussion in Section 2.1, even when more than two variables are present, if $D\left(P^{*}\right|\left|P\right)=0$, we can choose $P^{*}$ rather than P with probability one as $n\to \infty$.

Let $h_Z^n\left(z^n\right)$,$h_{XZ}^n(x^n,z^n)$, $h_{YZ}^n(y^n,z^n)$ and $h_{XYZ}^n(x^n,y^n,z^n)$ take the same values of $g_Z^n\left(z^n\right)$, $g_{XZ}^n(x^n,z^n)$, $g_{YZ}^n(y^n,z^n)$ and $g_{XYZ}^n(x^n,y^n,z^n)$, except that the $logn$ terms in $-log{Q}_Z^n\left(z^n\right)$, $-log{Q}_{XZ}^n(x^n,z^n)$,$-log{Q}_{YZ}^n(y^n,z^n)$ and $-log{Q}_{XYZ}^n(x^n,y^n,z^n)$ are replaced by $2(1+ϵ)loglogn$, respectively, where $ϵ>0$ is arbitrary. Then, we obtain the final result:

Theorem 5. With probability one as $n\to \infty$:

$$p{h}_{XZ}^n(x^n,z^n)h_{YZ}^n(y^n,z^n)\ge (1-p)h_{XYZ}^n(x^n,y^n,z^n)h_Z^n\left(z^n\right)⟺X\perp \perp Y|Z.$$

(10)

This paper focuses on the theoretical aspects of the BN structure learning, in particular for consistency when continuous variables are present. For the details of the practical matters we deal with in this section, see the conference paper [14].

3.3. The Number of Local Scores to be Computed

We refer the conditional independence (CF) score w.r.t. X and Y given Z to the left of (8). Suppose we follow the fastest Bayesian network structure learning due to [6]: let $Pa(X,V)$ be the optimal parent set of $X\in V$ contained in $V-\left\{X\right\}$ for $V\subseteq U:=\{1,\cdots ,N\}$ and $S(X,V)$ its local score. Then, we can obtain:

$$T\left(V\right):=\underset{x\in V}{max}\{S(X,V)+T(V-\left\{X\right\})\}$$

For each $V\subseteq U$, the sinks:

$$X_N={\mathrm{argmax}}_{X\in U}T\left(U\right),X_{N-1}={\mathrm{argmax}}_{X\in U-\left\{X_N\right\}}T(U-\left\{X_N\right\}),\cdots ,$$

and the parent sets:

$$Pa(X_N,U),P(X_{N-1},U-\left\{X_N\right\}),\cdots ,\left\{\right\}.$$

For each fixed pair ($X,V)$, maximizing the local score ${\displaystyle \frac{1}{n}log\frac{g_{W+\left\{X\right\}}}{g_W}}$ and maximizing the CF score ${\displaystyle \frac{1}{n}log\frac{g_{V-\left\{X\right\}}{g}_{W+\left\{X\right\}}}{g_V{g}_W}}$ w.r.t. $V-\left\{X\right\}$ and $W+\left\{X\right\}$, given W are equivalent. In other words,

$$\frac{1}{n}log\frac{g_{W+\left\{X\right\}}}{g_W}\le \frac{1}{n}log\frac{g_{W^{\text{'}}+\left\{X\right\}}}{g_{W^{\text{'}}}}⟺\frac{1}{n}log\frac{g_{V-\left\{X\right\}}{g}_{W^{\text{'}}+\left\{X\right\}}}{g_V{g}_W}\le \frac{1}{n}log\frac{g_{V-\left\{X\right\}}{g}_{W^{\text{'}}+\left\{X\right\}}}{g_V{g}_{W^{\text{'}}}}$$

for $W,W^{\text{'}}\subseteq V-\left\{X\right\}$.

On the other hand, from [15,16], we know that the relationship between the complexity term and the likelihood term gives tight bounds on the maximum number of parents in the optimal BN for any given dataset. In particular, the number of elements in each parent set $Pa(X,V)$ is at most $O(logn)$ for $X\in V$ and $V\subseteq U$. Hence, the number for computing the CF scores is much less than exponentialwith N.

4. Concluding Remarks

In this paper, we considered the problem of learning a Bayesian network structure from examples and provided two contributions.

First, we found that the $logn$ terms in the penalty terms of the description length can be replaced by $2(1+ϵ)loglogn$ to obtain strong consistency, where the derivation is based on the law of iterated logarithms. We claim that the Hannan and Quinn principle [7] is applicable to this problem.

Second, we constructed an extended version of the score function for finding a Bayesian network structure with the maximum posterior probability and proved that the decision is strongly consistent even when continuous variables are present. Thus far, consistency has been obtained only for discrete variables, and many authors have been seeking consistency when continuous variables are present.

Consistency has been proven in many model selection methods that maximize the posterior probability or, equivalently, minimize the description length [1]. However, almost all such methods assume that the variables are either discrete or that the variables obey Gaussian distributions. This paper proposed an extended version of the MDL/Bayesian principle without assuming such constraints and proved its strong consistency in a precise manner, which we believe provides a substantial contribution to the statistics and machine learning communities.