Next Article in Journal
Communicating through Probabilities: Does Quantum Theory Optimize the Transfer of Information?
Next Article in Special Issue
The Measurement of Information Transmitted by a Neural Population: Promises and Challenges
Previous Article in Journal
On the Topological Entropy of Some Skew-Product Maps
Previous Article in Special Issue
Efficient Approximation of the Conditional Relative Entropy with Applications to Discriminative Learning of Bayesian Network Classifiers

Estimation Bias in Maximum Entropy Models

Gatsby Computational Neuroscience Unit, UCL, London WC1N 3AR, UK
Max Planck Institute for Biological Cybernetics, Bernstein Center for Computational Neuroscience and Werner Reichardt Centre for Integrative Neuroscience, 72076 Tübingen, Germany
School for Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK
Author to whom correspondence should be addressed.
Entropy 2013, 15(8), 3109-3129;
Received: 25 June 2013 / Revised: 25 July 2013 / Accepted: 29 July 2013 / Published: 2 August 2013
(This article belongs to the Special Issue Estimating Information-Theoretic Quantities from Data)
Maximum entropy models have become popular statistical models in neuroscience and other areas in biology and can be useful tools for obtaining estimates of mutual information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e., the true entropy of the data can be severely underestimated. Here, we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We focus on pairwise binary models, which are used extensively to model neural population activity. We show that if the data is well described by a pairwise model, the bias is equal to the number of parameters divided by twice the number of observations. If, however, the higher order correlations in the data deviate from those predicted by the model, the bias can be larger. Using a phenomenological model of neural population recordings, we find that this additional bias is highest for small firing probabilities, strong correlations and large population sizes—for the parameters we tested, a factor of about four higher. We derive guidelines for how long a neurophysiological experiment needs to be in order to ensure that the bias is less than a specified criterion. Finally, we show how a modified plug-in estimate of the entropy can be used for bias correction. View Full-Text
Keywords: maximum entropy; sampling bias; asymptotic bias; model-misspecification; neurophysiology; neural population coding; Ising model; Dichotomized Gaussian maximum entropy; sampling bias; asymptotic bias; model-misspecification; neurophysiology; neural population coding; Ising model; Dichotomized Gaussian
Show Figures

Figure 1

MDPI and ACS Style

Macke, J.H.; Murray, I.; Latham, P.E. Estimation Bias in Maximum Entropy Models. Entropy 2013, 15, 3109-3129.

AMA Style

Macke JH, Murray I, Latham PE. Estimation Bias in Maximum Entropy Models. Entropy. 2013; 15(8):3109-3129.

Chicago/Turabian Style

Macke, Jakob H., Iain Murray, and Peter E. Latham. 2013. "Estimation Bias in Maximum Entropy Models" Entropy 15, no. 8: 3109-3129.

Find Other Styles

Article Access Map by Country/Region

Only visits after 24 November 2015 are recorded.
Back to TopTop