Next Article in Journal
Nonlocal Means Two Dimensional Histogram-Based Image Segmentation via Minimizing Relative Entropy
Next Article in Special Issue
Information Dynamics in Urban Crime
Previous Article in Journal
Special Issue on Rate Distortion Theory and Information Theory
Previous Article in Special Issue
Intrinsic Computation of a Monod-Wyman-Changeux Molecule
Open AccessArticle

Probability Mass Exclusions and the Directed Components of Mutual Information

by Conor Finn 1,2,* and Joseph T. Lizier 1
Complex Systems Research Group and Centre for Complex Systems, Faculty of Engineering & IT, The University of Sydney, NSW 2006, Australia
CSIRO Data61, Marsfield, NSW 2122, Australia
Author to whom correspondence should be addressed.
Entropy 2018, 20(11), 826;
Received: 26 September 2018 / Revised: 22 October 2018 / Accepted: 23 October 2018 / Published: 28 October 2018
(This article belongs to the Special Issue Information Theory in Complex Systems)
Information is often described as a reduction of uncertainty associated with a restriction of possible choices. Despite appearing in Hartley’s foundational work on information theory, there is a surprising lack of a formal treatment of this interpretation in terms of exclusions. This paper addresses the gap by providing an explicit characterisation of information in terms of probability mass exclusions. It then demonstrates that different exclusions can yield the same amount of information and discusses the insight this provides about how information is shared amongst random variables—lack of progress in this area is a key barrier preventing us from understanding how information is distributed in complex systems. The paper closes by deriving a decomposition of the mutual information which can distinguish between differing exclusions; this provides surprising insight into the nature of directed information. View Full-Text
Keywords: entropy; mutual information; pointwise; directed; information decomposition entropy; mutual information; pointwise; directed; information decomposition
Show Figures

Figure 1

MDPI and ACS Style

Finn, C.; Lizier, J.T. Probability Mass Exclusions and the Directed Components of Mutual Information. Entropy 2018, 20, 826.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Search more from Scilit
Back to TopTop