Next Article in Journal
ULF Pre-Seismic Geomagnetic Anomalous Signal Related to Mw8.1 Offshore Chiapas Earthquake, Mexico on 8 September 2017
Next Article in Special Issue
Poincaré and Log–Sobolev Inequalities for Mixtures
Previous Article in Journal
Parameter Identification of Fractional-Order Discrete Chaotic Systems
Previous Article in Special Issue
Entropy Inequalities for Lattices
Article Menu
Issue 1 (January) cover image

Export Article

Open AccessFeature PaperArticle
Entropy 2019, 21(1), 28;

On the Impossibility of Learning the Missing Mass

Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA 02142, USA
Toyota Technological Institute at Chicago, Chicago, IL 60637, USA
This work was conducted when both authors were visiting the Information Theory Program, 13 January–15 May 2015, at the Simons Institute for the Theory of Computing, University of California, Berkeley.
Author to whom correspondence should be addressed.
Received: 18 June 2018 / Revised: 25 November 2018 / Accepted: 6 December 2018 / Published: 2 January 2019
(This article belongs to the Special Issue Entropy and Information Inequalities)
Full-Text   |   PDF [861 KB, uploaded 4 January 2019]


This paper shows that one cannot learn the probability of rare events without imposing further structural assumptions. The event of interest is that of obtaining an outcome outside the coverage of an i.i.d. sample from a discrete distribution. The probability of this event is referred to as the “missing mass”. The impossibility result can then be stated as: the missing mass is not distribution-free learnable in relative error. The proof is semi-constructive and relies on a coupling argument using a dithered geometric distribution. Via a reduction, this impossibility also extends to both discrete and continuous tail estimation. These results formalize the folklore that in order to predict rare events without restrictive modeling, one necessarily needs distributions with “heavy tails”. View Full-Text
Keywords: missing mass; rare events; Good–Turing; light tails; heavy tails; no free lunch missing mass; rare events; Good–Turing; light tails; heavy tails; no free lunch
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).

Share & Cite This Article

MDPI and ACS Style

Mossel, E.; Ohannessian, M.I. On the Impossibility of Learning the Missing Mass. Entropy 2019, 21, 28.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics



[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top