Entropy 2013, 15(7), 2716-2735; doi:10.3390/e15072716
Article

Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers

1 Department of Electrical Engineering, IST, University of Lisbon, Lisbon 1049-001, Portugal 2 PIA, Instituto de Telecomunicações, Lisbon 1049-001, Portugal 3 Department of Computer Science, IST, University of Lisbon, Lisbon 1049-001, Portugal 4 SQIG, Instituto de Telecomunicações, Lisbon 1049-001, Portugal 5 Department of Mathematics, IST, University of Lisbon, Lisbon 1049-001, Portugal
* Author to whom correspondence should be addressed.
Received: 8 June 2013; in revised form: 3 July 2013 / Accepted: 3 July 2013 / Published: 12 July 2013
(This article belongs to the Special Issue Estimating Information-Theoretic Quantities from Data)
PDF Full-text Download PDF Full-Text [2244 KB, uploaded 12 July 2013 12:26 CEST]
Abstract: We propose a minimum variance unbiased approximation to the conditional relative entropy of the distribution induced by the observed frequency estimates, for multi-classification tasks. Such approximation is an extension of a decomposable scoring criterion, named approximate conditional log-likelihood (aCLL), primarily used for discriminative learning of augmented Bayesian network classifiers. Our contribution is twofold: (i) it addresses multi-classification tasks and not only binary-classification ones; and (ii) it covers broader stochastic assumptions than uniform distribution over the parameters. Specifically, we considered a Dirichlet distribution over the parameters, which was experimentally shown to be a very good approximation to CLL. In addition, for Bayesian network classifiers, a closed-form equation is found for the parameters that maximize the scoring criterion.
Keywords: conditional relative entropy; approximation; discriminative learning; Bayesian network classifiers

Article Statistics

Load and display the download statistics.

Citations to this Article

Cite This Article

MDPI and ACS Style

Carvalho, A.M.; Adão, P.; Mateus, P. Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers. Entropy 2013, 15, 2716-2735.

AMA Style

Carvalho AM, Adão P, Mateus P. Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers. Entropy. 2013; 15(7):2716-2735.

Chicago/Turabian Style

Carvalho, Alexandra M.; Adão, Pedro; Mateus, Paulo. 2013. "Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers." Entropy 15, no. 7: 2716-2735.

Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert