Open AccessThis article is
- freely available
The Mathematical Structure of Information Bottleneck Methods
Department of Mathematical Sciences, Montana State University, Bozeman, MT 59717, USA
Center for Biofilm Engineering, Montana State University, Bozeman, MT 59717, USA
Department of Mathematics and Science Programs, Washington State University Vancouver, Vancouver, WA 98686, USA
* Author to whom correspondence should be addressed.
Received: 2 December 2011; in revised form: 7 February 2012 / Accepted: 24 February 2012 / Published: 1 March 2012
Abstract: Information Bottleneck-based methods use mutual information as a distortion function in order to extract relevant details about the structure of a complex system by compression. One of the approaches used to generate optimal compressed representations is by annealing a parameter. In this manuscript we present a common framework for the study of annealing in information distortion problems. We identify features that should be common to any annealing optimization problem. The main mathematical tools that we use come from the analysis of dynamical systems in the presence of symmetry (equivariant bifurcation theory). Through the compression problem, we make connections to the world of combinatorial optimization and pattern recognition. The two approaches use very different vocabularies and consider different problems to be “interesting”. We provide an initial link, through the Normalized Cut Problem, where the two disciplines can exchange tools and ideas.
Keywords: information distortion; spontaneous symmetry breaking; bifurcations; phase transition
Citations to this Article
Cite This Article
MDPI and ACS Style
Gedeon, T.; Parker, A.E.; Dimitrov, A.G. The Mathematical Structure of Information Bottleneck Methods. Entropy 2012, 14, 456-479.
Gedeon T, Parker AE, Dimitrov AG. The Mathematical Structure of Information Bottleneck Methods. Entropy. 2012; 14(3):456-479.
Gedeon, Tomáš; Parker, Albert E.; Dimitrov, Alexander G. 2012. "The Mathematical Structure of Information Bottleneck Methods." Entropy 14, no. 3: 456-479.