Next Article in Journal / Special Issue
Incorporating Spatial Structures in Ecological Inference: An Information Theory Approach
Previous Article in Journal
On the Microscopic Perspective of Black Branes Thermodynamic Geometry
Article Menu

Export Article

Open AccessArticle
Entropy 2010, 12(10), 2144-2170; doi:10.3390/e12102144

Increasing and Decreasing Returns and Losses in Mutual Information Feature Subset Selection

Laboratorium voor Neuro- en Psychofysiologie, Computational Neuroscience Research Group, Katholieke Universiteit Leuven, O&N II Herestraat 49 - bus 1021, 3000 Leuven, Belgium
*
Author to whom correspondence should be addressed.
Received: 28 August 2010 / Accepted: 19 September 2010 / Published: 11 October 2010
(This article belongs to the Special Issue Advances in Information Theory)
View Full-Text   |   Download PDF [853 KB, uploaded 24 February 2015]   |  

Abstract

Mutual information between a target variable and a feature subset is extensively used as a feature subset selection criterion. This work contributes to a more thorough understanding of the evolution of the mutual information as a function of the number of features selected. We describe decreasing returns and increasing returns behavior in sequential forward search and increasing losses and decreasing losses behavior in sequential backward search. We derive conditions under which the decreasing returns and the increasing losses behavior hold and prove the occurrence of this behavior in some Bayesian networks. The decreasing returns behavior implies that the mutual information is concave as a function of the number of features selected, whereas the increasing returns behavior implies this function is convex. The increasing returns and decreasing losses behavior are proven to occur in an XOR hypercube. View Full-Text
Keywords: Bayesian networks; bit parity; conditional entropy; conditional mutual information; decreasing losses; decreasing returns; feature subset selection; increasing losses; increasing returns Bayesian networks; bit parity; conditional entropy; conditional mutual information; decreasing losses; decreasing returns; feature subset selection; increasing losses; increasing returns
This is an open access article distributed under the Creative Commons Attribution License (CC BY 3.0).

Scifeed alert for new publications

Never miss any articles matching your research from any publisher
  • Get alerts for new papers matching your research
  • Find out the new papers from selected authors
  • Updated daily for 49'000+ journals and 6000+ publishers
  • Define your Scifeed now

SciFeed Share & Cite This Article

MDPI and ACS Style

Van Dijck, G.; Van Hulle, M.M. Increasing and Decreasing Returns and Losses in Mutual Information Feature Subset Selection. Entropy 2010, 12, 2144-2170.

Show more citation formats Show less citations formats

Related Articles

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top