Entropy 2013, 15(5), 1690-1704; doi:10.3390/e15051690

An Estimate of Mutual Information that Permits Closed-Form Optimisation

* email and email
Received: 1 February 2013; in revised form: 18 April 2013 / Accepted: 28 April 2013 / Published: 8 May 2013
(This article belongs to the Special Issue Estimating Information-Theoretic Quantities from Data)
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract: We introduce a new estimate of mutual information between a dataset and a target variable that can be maximised analytically and has broad applicability in the field of machine learning and statistical pattern recognition. This estimate has previously been employed implicitly as an approximation to quadratic mutual information. In this paper we will study the properties of these estimates of mutual information in more detail, and provide a derivation from a perspective of pairwise interactions. From this perspective, we will show a connection between our proposed estimate and Laplacian eigenmaps, which so far has not been shown to be related to mutual information. Compared with other popular measures of mutual information, which can only be maximised through an iterative process, ours can be maximised much more efficiently and reliably via closed-form eigendecomposition.
Keywords: mutual information; dimensionality reduction; feature extraction; pattern recognition; machine learning
PDF Full-text Download PDF Full-Text [740 KB, uploaded 8 May 2013 11:04 CEST]

Export to BibTeX |

MDPI and ACS Style

Liu, R.; Gillies, D.F. An Estimate of Mutual Information that Permits Closed-Form Optimisation. Entropy 2013, 15, 1690-1704.

AMA Style

Liu R, Gillies DF. An Estimate of Mutual Information that Permits Closed-Form Optimisation. Entropy. 2013; 15(5):1690-1704.

Chicago/Turabian Style

Liu, Raymond; Gillies, Duncan F. 2013. "An Estimate of Mutual Information that Permits Closed-Form Optimisation." Entropy 15, no. 5: 1690-1704.

Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert