Entropy 2014, 16(7), 3552-3572; doi:10.3390/e16073552

Duality of Maximum Entropy and Minimum Divergence

1,* email, 2email and 3email
Received: 28 April 2014; in revised form: 19 June 2014 / Accepted: 24 June 2014 / Published: 26 June 2014
(This article belongs to the Special Issue Maximum Entropy and Its Application)
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract: We discuss a special class of generalized divergence measures by the use of generator functions. Any divergence measure in the class is separated into the difference between cross and diagonal entropy. The diagonal entropy measure in the class associates with a model of maximum entropy distributions; the divergence measure leads to statistical estimation via minimization, for arbitrarily giving a statistical model. The dualistic relationship between the maximum entropy model and the minimum divergence estimation is explored in the framework of information geometry. The model of maximum entropy distributions is characterized to be totally geodesic with respect to the linear connection associated with the divergence. A natural extension for the classical theory for the maximum likelihood method under the maximum entropy model in terms of the Boltzmann-Gibbs-Shannon entropy is given. We discuss the duality in detail for Tsallis entropy as a typical example.
Keywords: β-divergence; dual connections; information geometry; MaxEnt; multivariate t-distribution; power exponential family; sufficiency
PDF Full-text Download PDF Full-Text [286 KB, uploaded 26 June 2014 09:58 CEST]

Export to BibTeX |

MDPI and ACS Style

Eguchi, S.; Komori, O.; Ohara, A. Duality of Maximum Entropy and Minimum Divergence. Entropy 2014, 16, 3552-3572.

AMA Style

Eguchi S, Komori O, Ohara A. Duality of Maximum Entropy and Minimum Divergence. Entropy. 2014; 16(7):3552-3572.

Chicago/Turabian Style

Eguchi, Shinto; Komori, Osamu; Ohara, Atsumi. 2014. "Duality of Maximum Entropy and Minimum Divergence." Entropy 16, no. 7: 3552-3572.

Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert