• Submit to Entropy Login Register
• MDPI
• Journals A-Z
• For Authors
• For Editors
• About
• Open Access Policy

• Abstract
• Full-Text PDF [356 KB]

• Article Statistics
• Google Scholar
• Order Reprints

• Cite this article
• Export to BibTeX
• Export to EndNote

• [+] on DOAJ
• Gupta, M
• Srivastava, S
• [+] on Google Scholar
• Gupta, M
• Srivastava, S
• [+] on PubMed
• Gupta, M
• Srivastava, S

•  CiteULike
•  Facebook
•  Mendeley
•  Twitter

This article is

• freely available
• re-usable

Entropy 2010, 12(4), 818-843; doi:10.3390/e12040818

Article

Parametric Bayesian Estimation of Differential Entropy and Relative Entropy

Maya Gupta ^1,*  and Santosh Srivastava ²

¹ Department of Electrical Engineering, University of Washington, Seattle WA 98195-2500, USA ² Computational Biology, Fred Hutchinson Cancer Research Center, Seattle WA 98109, USA

* Author to whom correspondence should be addressed.

Received: 16 November 2009; in revised form: 28 March 2010 / Accepted: 2 April 2010 / Published: 9 April 2010

(This article belongs to the Special Issue Distance in Information and Statistical Physics Volume 2)

Download PDF Full-Text [356 KB, uploaded 9 April 2010 11:05 CEST]

Abstract: Given iid samples drawn from a distribution with known parametric form, we propose the minimization of expected Bregman divergence to form Bayesian estimates of differential entropy and relative entropy, and derive such estimators for the uniform, Gaussian, Wishart, and inverse Wishart distributions. Additionally, formulas are given for a log gamma Bregman divergence and the differential entropy and relative entropy for the Wishart and inverse Wishart. The results, as always with Bayesian estimates, depend on the accuracy of the prior parameters, but example simulations show that the performance can be substantially improved compared to maximum likelihood or state-of-the-art nonparametric estimators.

Keywords: Kullback-Leibler; relative entropy; differential entropy; Pareto; Wishart

Article Statistics

Cite This Article