Next Article in Journal
The Particle as a Statistical Ensemble of Events in Stueckelberg–Horwitz–Piron Electrodynamics
Previous Article in Journal
A Novel Faults Diagnosis Method for Rolling Element Bearings Based on EWT and Ambiguity Correlation Classifiers
Article Menu
Issue 5 (May) cover image

Export Article

Open AccessArticle
Entropy 2017, 19(5), 232; doi:10.3390/e19050232

A Kullback–Leibler View of Maximum Entropy and Maximum Log-Probability Methods

Industrial & Systems Engineering and Public Policy, University of Southern California, Los Angeles, CA 90089, USA
Supply Chain & Analytics, University of Missouri-St. Louis, St. Louis, MO 63121, USA
Industrial & Systems Engineering, University of Southern California, Los Angeles, CA 90007, USA
Author to whom correspondence should be addressed.
Academic Editor: Raúl Alcaraz Martínez
Received: 2 March 2017 / Revised: 30 April 2017 / Accepted: 15 May 2017 / Published: 19 May 2017
(This article belongs to the Section Information Theory)
View Full-Text   |   Download PDF [940 KB, uploaded 22 May 2017]   |  


Entropy methods enable a convenient general approach to providing a probability distribution with partial information. The minimum cross-entropy principle selects the distribution that minimizes the Kullback–Leibler divergence subject to the given constraints. This general principle encompasses a wide variety of distributions, and generalizes other methods that have been proposed independently. There remains, however, some confusion about the breadth of entropy methods in the literature. In particular, the asymmetry of the Kullback–Leibler divergence provides two important special cases when the target distribution is uniform: the maximum entropy method and the maximum log-probability method. This paper compares the performance of both methods under a variety of conditions. We also examine a generalized maximum log-probability method as a further demonstration of the generality of the entropy approach. View Full-Text
Keywords: entropy; minimum cross entropy; joint probability distribution entropy; minimum cross entropy; joint probability distribution

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. (CC BY 4.0).

Scifeed alert for new publications

Never miss any articles matching your research from any publisher
  • Get alerts for new papers matching your research
  • Find out the new papers from selected authors
  • Updated daily for 49'000+ journals and 6000+ publishers
  • Define your Scifeed now

SciFeed Share & Cite This Article

MDPI and ACS Style

Abbas, A.E.; H. Cadenbach, A.; Salimi, E. A Kullback–Leibler View of Maximum Entropy and Maximum Log-Probability Methods. Entropy 2017, 19, 232.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top