Next Article in Journal
Sound Symbolism in Basic Vocabulary
Next Article in Special Issue
Distances in Probability Space and the Statistical Complexity Setup
Previous Article in Journal
Article Omission in Dutch Children with SLI: A Processing Approach
Previous Article in Special Issue
Entropy and Divergence Associated with Power Function and the Statistical Application
Article Menu

Export Article

Open AccessArticle

Parametric Bayesian Estimation of Differential Entropy and Relative Entropy

Department of Electrical Engineering, University of Washington, Seattle WA 98195-2500, USA
Computational Biology, Fred Hutchinson Cancer Research Center, Seattle WA 98109, USA
Author to whom correspondence should be addressed.
Entropy 2010, 12(4), 818-843;
Received: 16 November 2009 / Revised: 28 March 2010 / Accepted: 2 April 2010 / Published: 9 April 2010
(This article belongs to the Special Issue Distance in Information and Statistical Physics Volume 2)
PDF [356 KB, uploaded 24 February 2015]


Given iid samples drawn from a distribution with known parametric form, we propose the minimization of expected Bregman divergence to form Bayesian estimates of differential entropy and relative entropy, and derive such estimators for the uniform, Gaussian, Wishart, and inverse Wishart distributions. Additionally, formulas are given for a log gamma Bregman divergence and the differential entropy and relative entropy for the Wishart and inverse Wishart. The results, as always with Bayesian estimates, depend on the accuracy of the prior parameters, but example simulations show that the performance can be substantially improved compared to maximum likelihood or state-of-the-art nonparametric estimators. View Full-Text
Keywords: Kullback-Leibler; relative entropy; differential entropy; Pareto; Wishart Kullback-Leibler; relative entropy; differential entropy; Pareto; Wishart

Figure 1

This is an open access article distributed under the Creative Commons Attribution License (CC BY 3.0).

Share & Cite This Article

MDPI and ACS Style

Gupta, M.; Srivastava, S. Parametric Bayesian Estimation of Differential Entropy and Relative Entropy. Entropy 2010, 12, 818-843.

Show more citation formats Show less citations formats

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top