Next Article in Journal
Nonlocal Means Two Dimensional Histogram-Based Image Segmentation via Minimizing Relative Entropy
Next Article in Special Issue
Information Dynamics in Urban Crime
Previous Article in Journal
Special Issue on Rate Distortion Theory and Information Theory
Previous Article in Special Issue
Intrinsic Computation of a Monod-Wyman-Changeux Molecule
Article Menu

Export Article

Open AccessArticle
Entropy 2018, 20(11), 826; https://doi.org/10.3390/e20110826

Probability Mass Exclusions and the Directed Components of Mutual Information

1
Complex Systems Research Group and Centre for Complex Systems, Faculty of Engineering & IT, The University of Sydney, NSW 2006, Australia
2
CSIRO Data61, Marsfield, NSW 2122, Australia
*
Author to whom correspondence should be addressed.
Received: 26 September 2018 / Revised: 22 October 2018 / Accepted: 23 October 2018 / Published: 28 October 2018
(This article belongs to the Special Issue Information Theory in Complex Systems)
Full-Text   |   PDF [269 KB, uploaded 31 October 2018]   |  

Abstract

Information is often described as a reduction of uncertainty associated with a restriction of possible choices. Despite appearing in Hartley’s foundational work on information theory, there is a surprising lack of a formal treatment of this interpretation in terms of exclusions. This paper addresses the gap by providing an explicit characterisation of information in terms of probability mass exclusions. It then demonstrates that different exclusions can yield the same amount of information and discusses the insight this provides about how information is shared amongst random variables—lack of progress in this area is a key barrier preventing us from understanding how information is distributed in complex systems. The paper closes by deriving a decomposition of the mutual information which can distinguish between differing exclusions; this provides surprising insight into the nature of directed information. View Full-Text
Keywords: entropy; mutual information; pointwise; directed; information decomposition entropy; mutual information; pointwise; directed; information decomposition
Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
SciFeed

Share & Cite This Article

MDPI and ACS Style

Finn, C.; Lizier, J.T. Probability Mass Exclusions and the Directed Components of Mutual Information. Entropy 2018, 20, 826.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Entropy EISSN 1099-4300 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top