Entropy 2013, 15(7), 2716-2735; doi:10.3390/e15072716
Article

Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers

1,2,* email, 3,4email and 4,5email
Received: 8 June 2013; in revised form: 3 July 2013 / Accepted: 3 July 2013 / Published: 12 July 2013
(This article belongs to the Special Issue Estimating Information-Theoretic Quantities from Data)
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract: We propose a minimum variance unbiased approximation to the conditional relative entropy of the distribution induced by the observed frequency estimates, for multi-classification tasks. Such approximation is an extension of a decomposable scoring criterion, named approximate conditional log-likelihood (aCLL), primarily used for discriminative learning of augmented Bayesian network classifiers. Our contribution is twofold: (i) it addresses multi-classification tasks and not only binary-classification ones; and (ii) it covers broader stochastic assumptions than uniform distribution over the parameters. Specifically, we considered a Dirichlet distribution over the parameters, which was experimentally shown to be a very good approximation to CLL. In addition, for Bayesian network classifiers, a closed-form equation is found for the parameters that maximize the scoring criterion.
Keywords: conditional relative entropy; approximation; discriminative learning; Bayesian network classifiers
PDF Full-text Download PDF Full-Text [2244 KB, uploaded 12 July 2013 12:26 CEST]

Export to BibTeX |
EndNote


MDPI and ACS Style

Carvalho, A.M.; Adão, P.; Mateus, P. Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers. Entropy 2013, 15, 2716-2735.

AMA Style

Carvalho AM, Adão P, Mateus P. Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers. Entropy. 2013; 15(7):2716-2735.

Chicago/Turabian Style

Carvalho, Alexandra M.; Adão, Pedro; Mateus, Paulo. 2013. "Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers." Entropy 15, no. 7: 2716-2735.


Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert