Next Article in Journal
Identifying the Probability Distribution of Fatigue Life Using the Maximum Entropy Principle
Previous Article in Journal
Introduction to Supersymmetric Theory of Stochastics
Open AccessArticle

An Estimator of Mutual Information and its Application to Independence Testing

Department of Mathematics, Graduate School of Science, Osaka University, Toyonaka, Osaka 560-0043, Japan
Academic Editor: Raúl Alcaraz Martínez
Entropy 2016, 18(4), 109; https://doi.org/10.3390/e18040109
Received: 22 February 2016 / Revised: 14 March 2016 / Accepted: 23 March 2016 / Published: 29 March 2016
(This article belongs to the Section Information Theory, Probability and Statistics)
This paper proposes a novel estimator of mutual information for discrete and continuous variables. The main feature of this estimator is that it is zero for a large sample size n if and only if the two variables are independent. The estimator can be used to construct several histograms, compute estimations of mutual information, and choose the maximum value. We prove that the number of histograms constructed has an upper bound of O(log n) and apply this fact to the search. We compare the performance of the proposed estimator with an estimator of the Hilbert-Schmidt independence criterion (HSIC), though the proposed method is based on the minimum description length (MDL) principle and the HSIC provides a statistical test. The proposed method completes the estimation in O(n log n) time, whereas the HSIC kernel computation requires O(n3) time. We also present examples in which the HSIC fails to detect independence but the proposed method successfully detects it. View Full-Text
Keywords: mutual information; kernel; independence testing; Hilbert-Schmidt independence criterion (HSIC); minimum description length (MDL) principle; histogram mutual information; kernel; independence testing; Hilbert-Schmidt independence criterion (HSIC); minimum description length (MDL) principle; histogram
Show Figures

Figure 1

MDPI and ACS Style

Suzuki, J. An Estimator of Mutual Information and its Application to Independence Testing. Entropy 2016, 18, 109.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop