On the Estimation of Mutual Information †
Abstract
:1. Introduction
2. Mutual Information
3. Non-Parametric Estimation
3.1. The Vanilla KSG Estimator
3.2. The LNC Correction to KSG
4. Robustness Tests of NP Estimators
4.1. Coordinate Transformations
4.2. Redundancy vs. Noise
5. Discussion
Acknowledgments
References
- Carrara, N.; Ernst, J.A. On the Upper Limit of Separability. arXiv 2017, arXiv:1708.09449. [Google Scholar]
- Tishby, N.; Pereira, F.C.; Bialek, W. The information bottleneck method. arXiv 2000, arXiv:physics/0004057. [Google Scholar]
- Tishby, N.; Zaslavsky, N. Deep Learning and the Information Bottleneck Principle. arXiv 2015, arXiv:1503.02406. [Google Scholar]
- Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; Abbeel, P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. arXiv 2016, arXiv:1606.03657. [Google Scholar]
- Ver Steeg, G.; Galstyan, A. The Information Sieve. arXiv 2015, arXiv:1507.02284. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley-Interscience: Hoboken, NJ, USA, 2006. [Google Scholar]
- Carrara, N.; Vanslette, K. The Inferential Foundations of Mutual Information. arXiv 2019, arXiv:1907.06992. [Google Scholar]
- Kraskov, A.; Stögbauer, H.; Grassberger, P. Estimating mutual information. Phys. Rev. E 2004, 69, 066138. [Google Scholar]
- Gao, S.; Ver Steeg, G.; Galstyan, A. Efficient Estimation of Mutual Information for Strongly Dependent Variables. arXiv 2014, arXiv:1411.2003. [Google Scholar]
- Kozachenko, L.F.; Leonenko, N.N. A statistical estimate for the entropy of a random vector. Probl. PeredachiInformats 1987, 2, 9–16. [Google Scholar]
- Ver Steeg, G. Non-parametric Entropy Estimation Toolbox. Available online: https://github.com/gregversteeg/NPEET (accessed on 14 January 2020).
- Baldi, P.; Sadowski, P.; Whiteson, D. Searching for exotic particles in high-energy physics with deep learning. Nat. Commun. 2014, 5, 4308. [Google Scholar] [CrossRef] [PubMed]
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Carrara, N.; Ernst, J. On the Estimation of Mutual Information. Proceedings 2019, 33, 31. https://doi.org/10.3390/proceedings2019033031
Carrara N, Ernst J. On the Estimation of Mutual Information. Proceedings. 2019; 33(1):31. https://doi.org/10.3390/proceedings2019033031
Chicago/Turabian StyleCarrara, Nicholas, and Jesse Ernst. 2019. "On the Estimation of Mutual Information" Proceedings 33, no. 1: 31. https://doi.org/10.3390/proceedings2019033031
APA StyleCarrara, N., & Ernst, J. (2019). On the Estimation of Mutual Information. Proceedings, 33(1), 31. https://doi.org/10.3390/proceedings2019033031